Monitoring the Orbs Platform
By: Ido Zilberberg
Tags: Ethereum, Blockchain, Identity, Crypto, DevOps

Hi everyone! Ido here. One of the important milestones we’ve hit in April, post-launch, was bringing our monitoring system into production. Having an efficient monitoring system is crucial for the optimization of all aspects of the Orbs blockchain. This update will elaborate on pre-production load testing, and post-production performance monitoring. Both use cases employ the Orbs monitoring dashboard as a key component.

In the weeks leading up to the planned launch of the Orbs network, Orbs core team was running stability tests on various network configurations. The purpose of such testing was twofold: identify bugs that manifest themselves only after a long time the system has been online, and test the load limits of the system.

We wanted to share the progress of stability tests with the whole team, so we decided to create a Dashboard that would be posted on a TV at the Orbs Ltd. office, where everyone can see real-time metrics of the running tests.

The requirement was to let everyone have a casual glance once in a while and immediately spot anomalies. Digging deeper into problems is left to other tools.

As a starting point, we already had a /metrics endpoint on the Orbs node, that prints out the node’s metrics in JSON format. For example, see the Orbs Audit node metrics.

Here you can see sample metrics on a validator node in production — It shows the current block height as 452631:

For the presentation layer, we checked several dashboard solutions: Geckoboard is a great looking dashboard especially on a TV screen and if your use case is covered by what it supports out of the box, it is highly recommended. In our case we needed some features which were not available, such as displaying a time frame of less than 1 day, and the ability to perform calculations over the data directly in the dashboard.

Another solution we tried is DataDog, which showed initial promise as it supports scraping JSON metrics directly from a URL, and performing calculations over it. However, we were not satisfied with the visual quality when displayed on a TV screen (specifically the lack of a Dark Mode), and decided to keep looking.

Eventually, we decided to use the Standard Plan of Grafana Cloud with a Prometheus backend for these reasons:

Hosted solution — no need to maintain a Grafana server Renders beautifully on a TV screen Flexible in performing calculations over incoming data in realtime (together with Prometheus).

Grafana is a presentation layer only and can receive data from multiple data sources, in our case it was either from Prometheus or Graphite — both of which are industry standards for a Time Series Database (TSDB). Since Prometheus has client libraries for both Go and Javascript we went along with it.

As for the actually displaying data, we started with a TV monitor attached to an off the shelf Android TV device, but it turned out that several browsers we tested on the Android TV did not render correctly any of our tested dashboards, and it was not possible to disable the screen saver and have it load the dashboard on startup without hacking the device.

So we decided to go with a Raspberry PI B+ v3 which is a mean little machine, capable of rendering the graphs without a hitch. Since it’s a Linux machine and has a wide community around it, it was easy to find solutions and guides on how to set up Grafana to our liking.

Here is our real monitoring TV (complete with window-glare):