Troubleshooting the Orbs Platform
By: Ido Zilberberg
Tags: Development, Identity, Ethereum, Cryptocurrency, Blockchain

As new software meets the real world in production for the first time, unforeseen issues pop up. Anyone in the business long enough knows this for a fact.

Orbs platform is no different, and while considerable effort was, and still is being expended to retain the highest code quality, post-production incidents do happen, but they provide opportunities to make the software better yet.

Part of improving software and keep learning from experience is creating case studies such as this one.

This post includes technical notes to help you, the reader, make sense of the information you are seeing. It is not an exhaustive description of Lean Helix or any other service of the Orbs platform. For that, visit the Orbs Spec.

Such was the case that a few weeks post-launch, the Orbs core team observed less than optimal behavior of the Lean Helix consensus algorithm in production. The dashboard showed that frequently, a block passes consensus only on the third attempt, instead of the expected first or second attempt. This meant it took longer to reach consensus and consequently transaction requests took longer to return a result to the caller.

Technical note: Before going into more detail, let’s understand a bit how Lean Helix works (with gross oversimplification):

Choose a Leader node at random

node at random The leader collects pending transactions (i.e. transactions sent by clients and still waiting to be included in a block) into a “proposed block”

The leader suggests this block to the other non-leader nodes

The non-leader nodes try to reach an agreement (a.k.a consensus) by passing around messages of various types (more on that later) relating to the content of the proposed block. There is a time limit as to how long this can take to propose the block and reach an agreement. If the time limit is exceeded and consensus has still not been reached, a new leader is chosen among the non-leader nodes (the previous leader itself becomes a non-leader node) and the process repeats itself.

The above flow is called a View. The first iteration (view) is denoted V=0, the following iteration (with a new leader) is V=1 and so on. The time limit per View is exponentially increased for every View. For V=0 the leader has 4 seconds to reach consensus (production configuration as of May 2019) before it is replaced. For V=1 the next leader has 8 seconds, for V=2 the next leader has 16 seconds and so on. When consensus on a block is reached, every node commits the proposed (now accepted) block, the View is reset to V=0 and the process repeats itself. Read more about Lean Helix: Orbs’ Unified Theory of Randomness: Explaining the Helix Consensus Algorithm

Now, given the production configuration, it was expected that the first leader at V=0 will sometimes be able to reach consensus, but mostly the second leader at V=1 will do that (exactly why is out of scope). There was no apparent reason to reach the third leader at V=2 unless there was some considerable network lag.

Having said all that, here you can see the problem:

Instead of having all nodes report they are either on V=0 or V=1, the frequency of V=2 was quite high.

So, we started digging deeper.

It was clear that a specific node (let’s call it Node A, its actual identity is irrelevant to this case study) as exhibiting the majority of V=2 incidents.

At this point, the dashboard could not provide further clues. Suffice to say that one of the Orbs core team’s monitoring goals is to improve on what can be easily identified in the context of the dashboard and postpone the following costly log research to more complex issues.

In this case, logs were the next step.

From a troubleshooting perspective, we were fortunate to have frequent V=2 incidents so it was easy to obtain fresh logs around the timeframe of such incidents. It can be quite frustrating trying to debug a rare issue for which logs are not readily available.

The logs use the node’s Orbs address for filtering. In node A’s case, it is “0123456789abcdef”, sometimes contracted to “012345”.

Since we did see Node A’s block height advancing we knew it was up to date on the blocks.

Now, when a node does not commit a block through the consensus algorithm, it can still receive new blocks via Block Sync.

Technical note: Block Sync is a service within the node that wakes up after some predetermined time during which no new block was committed. The node assumes that it’s being left behind for whatever reason, so it requests newer blocks from other nodes in the network. This is useful in case the node was down for a while and now needs to sync to the rest of the network. It is also useful in this case, where blocks are not being committed as part of the consensus algorithm. Read more about Block Sync in Orbs Spec: Block Sync in Orbs Spec

We suspected that node A was advancing through Block Sync rather than through consensus.

So, the initial filter was to show blocks being committed as part of the consensus algorithm flow.

Analyzing logs is like looking at the data with various lenses, one at a time, each of which captures only a subset. For logs over some size, there is simply too much data for a human to grasp in one go.

Here is the first screenshot from logz.io. In it you can see positive filters (in blue), negative filters (in red) and a search string (logz.io uses the Lucene query format):