Fractal Debugging: Redefining the Blockchain Developer Experience
Tags: Blockchain, Development, Neo Column, Debugging

With Time-Travel Debugging and Fractal Debugging, Neo continues to raise the bar and push new frontiers to become the most developer-friendly blockchain platform.

Brian Kernighan, the man who coined the name Unix and who was instrumental in taking the C programming language mainstream, said in his 1974 masterpiece, The Elements of Programming Style, “Everyone knows that debugging is twice as hard as writing a program in the first place. So, if you are as clever as you can be when you write it, how will you ever debug it?”

Well, we realize how prescient Kernighan was, because today, software developers often spend up to a quarter of their time verifying, validating and debugging their software components, services and systems; and, the cost of quality assessment, testing, debugging and fixing and maintaining bugs is estimated to account for a third or higher of the budget of software development.

Looking Back — Key Recurring Themes with respect to Debugging

From our collective 70+ years of previous experience building software tools and platforms for one of the largest software companies in the world, we see three recurring top-line themes from professional developer in their requirements and desires with respect to debugging software components, services, and systems.

Firstly, there is the challenge of Isolating and Understanding Faults. The main-stream software ecosystem continues to get better at capturing and presenting events that occurred in a production system at or during the time of a fault. Our tools to enable the capture and the isolation of faults will continue to get asymptotically better in their application in production scenarios, caveat security and system performance constraints.

Second, enhanced support for Log stream and file analysis i.e. the on-going need for better and better information captured in logs. The key here is to acquire and provide enhanced contextual information about the program state from the analysis. The better the tools are at aggregating and harmonizing this information, the greater the developer productivity,

And thirdly, the abstraction of what is colloquially referred to as the Backward debugger i.e. one that allows the user to step backwards in execution mode and to understand and exploit the “history” akin to a video recording. Microsoft pioneered Time-Travel Debugging for the mainstream software industry, and Neo was the first blockchain platform to make this available for developers world-wide.

Time-Travel Debugging — How the Neo Smart Contract Debugger Raised the Bar for the Blockchain Industry

Blockchain platforms enable powerful new abstractions and present unique challenges for developers. State Machines can make perfectly accurate mistakes; the code is written by human beings after all and bugs are a fact of human endeavor. However, bugs on a blockchain have enormous repercussions due to the immutable nature of the underlying ledger, as any of us who are familiar with the DAO are aware.

Traditionally, logs tend to be effective at giving us the identity of the actual problem, and as an industry, we have gotten good at capturing exceptions and errors in our logs. However, logs address at best one half of the problem. The logs simulate the effect of leaving bread crumbs for the developer, and the optimistic hope is that they are adequate at giving the developer a compass to predict and forecast the various scenarios that the code is being executed in the Mainnet.

Orienting based on the information about the identity of the actual problem, the developer then sets about in a tight inner loop of setting breakpoints, running to the cursor, stepping in and out of methods. Hours, and often days are spent in this cycle of software archeology; however, as soon as a new relatively complex problem occurs in the Testnet or Mainnet we are forced back to square one.

Now, to make matters worse, there are usually a few of the insidious bugs where the observable effects are often separated in time and/or space from the location of the bug. At this point, the traditional approaches pretty much end up with the developers abdicating and providing the classic response: ‘but, it works on my machine …’.

Time-Travel Debugging greatly simplifies debugging by enabling developers to easily step forwards and backward through the code execution. Further, by going back in time and with a greater understanding of the context, the developer can “replay” the scenario as many times as they want to while they examine every step that leads up to the bug and figures out how to best to fix the issue with the code.

Time-Travel Debugging accelerates the developer’s edit, compile, build and debug loop by empowering the developer to record the execution of the running code on the blockchain, and then subsequently re-run it without regard to the past or the future; it does this efficiently by adding as little overhead as possible by capturing code execution in trace files.

Fractal Debugging — Now, the Neo Smart Contract Debugger will Redefine Debugging for the Blockchain Industry

No system is an island; similarly, no chain is an island. In the public blockchain domain, a variety of projects are pursuing diverse and sometimes intersecting goals spanning, scalability, security, privacy, economic models, governance structures etc. In the private and consortium chain domain, long-standing pillars of heterogeneity and interoperability drive a multiplicity of technologies and chain stacks.

Generally, cross-chain scenarios subsume use cases where external data sources (ala Neo3 built-in Oracle) need to be accessed, as well as other network resources (ala Neo3 built-in NeoFS). In all of these cases, the ability to debug “cross-chain” (what we refer to as the “space between the chains” from our previous experience building service-oriented and cloud platforms) becomes paramount.

In the academic world, Model Checking and associated Theorem-proving approaches have been two areas of research towards debugging distributed systems. Model Checking is a process of exhaustive analysis: there are two sub-categories, Symbolic model checking represents and explores possible executions using abstract math, whereas Explicit-state model checking focuses on program execution.

However, these approaches are constrained with respect to effective real-world modeling as well as limitations with respect to scalability and are incapable of translating from the world of academia to the world of production systems. Theorem proving may be used to prove that a distributed system does not have any faults (or restrict the universe of faults); however, in practice, these techniques do not scale into the real-world.

With our approach to Fractal Debugging, in order to enable cross-chain scenarios, we take a multi-pronged approach spanning Tracing, Log Analysis, and Time-Travel Debugging and Visualization.

Tracing tracks the flow of messages and data through the system(s), across protocols; instrumenting the sub-systems to collect and collate pre-defined metrics at a number of junctions and enable an cross-chain view of the data and the insights spanning multiple execution instances. A pre-requisite for tracing is to instrument protocols, capabilities et al.

In those scenarios where the underlying protocols and/or capabilities pose constraints to instrumentation, the complementary lighter-weight Log analysis is applied. This black-box approach is able to provide detailed data spanning a variety of sources; the key is to make this consumable to the developer at the right level of abstraction using their familiar debugging paradigms.

Lastly, and as described earlier, Time-travel debugging captures execution scenarios so that a particular instance of the scenario can be later replayed; this is extraordinarily useful when debugging nondeterministic behaviors. Complemented with Visualization techniques, Time-travel debugging, in conjunction with Trace and Log Analysis enables developers to better understand the “space between the chains”.

Focus on Cross-Chain Developer Productivity

It has been said that programming allows one to think about thinking, and that while debugging code, one learns about learning. It is in this spirit that we shipped the first versions of the Neo Smart Contract Debugger, and we continue to have a singular focus on ensuring the premier debugging tools in the blockchain industry.

Our goal is to continue to raise the bar for the developer experience and to push the envelope with respect to cross-chain development scenarios, including Oracles, FS and more. With the launch wave of Neo3 and the forthcoming enhancements for the Neo Smart Contract Debugger, we are looking forward to delivering Fractal Debugging for cross-chain scenarios to our developers world-wide.

Author: John deVadoss, NGD Enterprise