Bugs! Bugs! Bugs! Gotta Catch ’Em All
By: Satyam Porwal
Tags: Distributed Systems, Failure Detection, Operating Systems

The following blog is a combined (and equal) effort of authors:

Prashant Ramnani, Rithin Manoj, Satyam Porwal, and Shivam Kumar Jha, Indian Institute of Technology, Kharagpur.

Fault tolerance is in the center of distributed system design that covers various methodologies. In past, there have been cases where critical applications buckled under faults because of an insufficient level of fault tolerance.

“A distributed system is one in which the failure of a computer you didn’t even know existed can render your own computer unusable”

Leslie Lamport, May 1987.

Through this blog, we tried to bring out the ways various researchers have tried to detect failures in operating and distributed systems in recent times.

We’ll discuss 4 different papers, hence 4 different approaches for failure detection, and briefly describe their working and certain limitations that each faced.

Capturing and enhancing In Situ system observability for Failure Detection

Huang et al., OSDI’18

Faults are common in large systems, they can be Hardware faults, Software bugs, or even misconfiguration. Failure detection is a fundamental building block for fault-tolerant systems. An important factor in the availability of data is detecting failure reliably and rapidly.

Extensive work has been done in this field. Studies done over several decades have focussed on the detection of fail-stop failures in asynchronous systems. The problem to reliably determine whether a process has crashed or is just “very slow” has proved to be a daunting task.

Gray Failures

If you’re going to fail, fail properly dammit! Masking problems turn out to be one of the key causes of major availability breakdowns and performance anomalies in cloud-scale systems.

Gray failures are when some component of the system fails but the whole system does not crash. Making this even more nuanced and harder to detect.

… cloud practitioners are frequently challenged by gray failure: component failures whose manifestations are fairly subtle and thus defy quick and definitive detection. Examples of gray failure are severe performance degradation, random packet loss, flaky I/O, memory thrashing, capacity pressure, and non-fatal exceptions.

Talking out gray failure is a blog of its own. If you’re really interested you can refer to our inspiration for choosing this paper, Adrian Colyer.

The paper presents its own version of Gray Failure, ZooKeeper cluster, which can no longer service write requests event even though the leader was still actively exchanging heartbeat messages with its followers.

The key insights of the paper are

1. Critical Gray Failures are Observable

2. Error Handling can be used for Error Reporting

The paper tries to design a failure detector that can correctly report the status of each component. The quality of which is measured by

1) Completeness, which says that if a component fails, a detector suspects it

2) Accuracy, which says that a component is not suspected before it fails.

THE SOLUTION: PANORAMA

A collaborative based approach to detect complex production failures by gathering observations about each component from different sources in real-time.

Panorama is divided into two phases (we will see this is a very common trend across multiple papers):

Offline Phase

The panorama includes an offline tool that statically analyses a program’s source code to find critical points for observation (another common observation among various papers; usually coined different terms but refer to a very similar idea) capture to inject hooks (depicted in green lines) for reporting observations.

In the above example, we can see an observation point to be an exception handler, which is invoked when an exception occurs at the observation boundary.

Hooks also capture first-hand observations, especially the runtime errors which are generated from executions of these code regions in production. So whenever Panorama reports a failure, there is concrete evidence and context to help localize where the failure occurred.

2. Runtime Phase

Each Panorama instance maintains a Local Observation Store(LOS) that stores all observations reports made by colocated components.

Each LOS, using a local decision engine, makes judgments about the status of observed components’ status; the status can be any one of HEALTHY, DEAD, and a few levels of UNHEALTHY. These LOSes are synchronized and, in a decentralized fashion, a central verdict server allows easy querying of judgments regarding the failure of components.

There is also a central verdict server that allows easy querying of judgments and arbitrates amongst the decentralized LOSes.

Overview of Panorama. Each Panorama instance runs at the same endpoint with the monitored component.

With the observations collected about a subject, Panorama uses a decision engine to reach a verdict and stores the result in the LOS. A process is judged as unhealthy if the latest status is unhealthy or the health status does not have a recent majority. The summaries across all observers are then aggregated and status decided using a simple majority.

Panorama was applied to ZooKeeper, HDFS, HBase, and Cassandra, at both process and thread level. The system proved to be a messiah which helped us get rid of all production gray failures. While this might be the case with 1000 lines of code, it is a very tedious task to annotate all production code.

Developers find it hard to write proper documentation, making them do annotations would be even harder. Automating this process with minimum manual work would be the best way to go but that itself requires a lot of effort since it needs to be done for a large set of languages.

Even though Panorama has shown positives it has a lot of challenges to become a practical production solution.

CrashTuner: Detecting Crash Recovery Bugs in Cloud Systems via Meta-info Analysis

Lu et. Al, SOSP’19

Moving towards another theme of failure which would result in much more clearly visible and catastrophic outcomes. Crash recovery codes are often buggy and random fault injections have proven ineffective for detecting these. Model-checking at the code level is infeasible (unlike our previous problem), due to the large possible state space.

Recovery must be a first-class (independent) operation for any distributed system. The image below shows a statistic depicting the high number of failures that occur over an interval of one month.

How common are node crashes?

Node Crash Events can be common in large clusters. Existing detection approaches include:

Random fault injection: Ineffective

Model-checking: Inefficient and requires manual specifications

Existing Crash-recovery bugs can be easily triggered nodes:

Crash before reading variables

Crash after writing variables

THE SOLUTION: CRASHTUNER

The paper observes that crash-recovery bugs involve meta-info variables. Meta-info variables include variables denoting nodes, jobs, tasks, applications, containers, attempts, sessions, etc.

The paper infers that crash-recovery bugs can be easily triggered when nodes crash before reading meta-info variables and/or crash after writing meta-info variables.

Overview of the CrashTuner approach.

The basic working of CrashTuner as depicted in the image can be divided into two parts:

Identifying all crash points

a. Using log-based program analysis on runtime logs to discover the meta-info variables (nodes referencing variables and related variables). Mines runtime logs to discover meta-info variables and record their runtime values.

All 4 distributed systems, which CrashTuner was evaluated had a very distinctive common theme: a common logging library that provides common logging interfaces with the following terms: fatal, error, warn, info, debug and trace. Using these log patterns and instances are generated effectively, which are then used to generate the meta-info variables.

b. Static crash point analysis is done on these to identify all other meta-info variables. This is a good approach as it derives all the meta-info variables in the system without precisely tracking program dependencies, which can be a highly complex task in the field of distributed systems.

These “static crash points” are access points to the fields of meta-info types. These are also referred to as pre-read/post-write points (the name says it all).

If a read reference is only used in the return statements the corresponding static crash point is promoted. Such promotion helps to simplify the call stacks of corresponding dynamic pre-read points.

c. The profiler generates dynamic crash points, i.e., executed static crash points with distinct calling stacks. Those static crash points not exercised are discarded.

2. Performs fault injection testing at each crash point, to carefully inject the right faulty event we need to figure out which node to crash when hitting a dynamic crash point.

a. Online Log Analysis is performed where Runtime logs are collected at each node of a cluster using a LogStash Agent, which is optimized to send only the runtime values of meta-info variables to a custom stash.

These are then processed in FIFO order using a HashSet.

b. Trigger instruments the system to inject crash events at dynamic crash points, where pre-read and post-write are instrumented differently.

For pre-read points, a shutdown RPC is followed by a wait, which works as a timeout period for the shutdown event to be handled. For post-write points, a crash RPC is instrumented. Both are invoked with arguments <P, context> and id. P is a dynamic crash point, context is the call stack and id is the access runtime meta-info value. The bugs are reported in 3 cases:

i. Job failures

ii. System hangs

iii. There exist uncommon exceptions in logs

CrashTurner was evaluated on Yarn, HDFS, HBase, and ZooKeeper and proved highly effective in reporting bugs in each of the systems. Using this Crashtuner inserts crash points at read/ write of meta-info variables. This results in a 99.91% reduction in crash points with previous testing techniques.

Silent errors are not reported which leads to unexpected behaviors. Even though Meta-info is a well-suited abstraction for distributed systems. There are questions about identifying meta-info variables and what would be false-positive and false-negative rates for finding meta-info variables via heuristic definition.

Efficient Scalable Thread-Safety-Violation Detection

Li et. Al, SOSP’19

Similar to the two papers discussed above, here also we present a specific problem which the researchers try to deal with, which in this case is Thread-Safety Violation (TSV).

TSV occurs if two threads concurrently invoke two conflicting methods upon the same object. Similar to the gray failures, these are hard to find via testing as they don’t show up in most of the test runs.

Thread-safety violations are a generalization of data races to objects and data structures and occur when the client fails to meet the thread-safety contract of a class or library. It is possible to use data-race detection tools in a manually intensive process to find some of these bugs on a small scale. To reduce the manual effort, it is possible to adopt dynamic data race analysis while running the program under test inputs, but these require a lot of false-positive pruning

THE SOLUTION: TSVD

TSVD is a scalable dynamic analysis tool for TSVs. It uses lightweight instrumentation and dynamic analysis to identify potential conflicting accesses without paying the overhead while being effective in finding bugs. We provide TSVD only the thread-safety contract, and it finds the results with zero false positives.

TSVD requires a static analysis that identifies all the call sites(called TSVD points) to relevant data structures’ methods whose thread-safety needs to be checked in the code.

The only interface to TSVD is the OnCall with arguments that show the thread making the call thread_id, the object being accessed obj_id, and the operation being performed op_id.

Algorithm for OnCall

The trap mechanism works as follows. When a thread calls this OnCall method and it is chosen by the shoud_delay, set_trap will register the current method call to some global table, and the thread will sleep for some period. During this period, every other thread entering OnCall checks if it conflicts with the currently registered traps. If two traps conflict and at least one operation is a write, TSVD finds a bug. This is a delay-based strategy which guarantees that if a violation is found, it is a true bug (i.e., no false positives).

A potential violation is used in multiple test executions by injecting delays to trigger a real violation. If a real violation is found it is a true bug, else TSVD can guarantee it is a false-positive and is ignored.

TSVD maintains a trap set that contains dangerous-pairs of program locations that can likely contribute to thread-safety violations and injects delays at only these locations. TSVD looks for conflicting calls with close-by physical timestamps. If two calls from different threads to the same object occur within a short physical time window, TSVD will add these two calls into the set. This is a highly efficient and scalable method than trying to do a happened-before analysis and finding calls with concurrent logical timestamps.

TSVD was deployed at Microsoft for several months and found 96% TSVs which were previously unknown to developers and 47% were causing severe customer-facing issues.

Comparison of TSVD against other benchmark models

A seemingly straight-forward drawback of the TSVD approach is that it may cause a false-negative by adding the random delay, but when we try to run the tool multiple times, those missed false negatives are captured due to different random delays tried.

FCatch: Automatically Detecting Time-of-fault Bugs in Cloud Systems

Wang et. Al, ASPLOS’18

Moving with the common theme of trying to achieve high availability, the paper introduces Time-of-Fault (TOF) bugs that only manifest when a node crashes or a message drops at a special moment. TOF bugs are composed of trigger, error, and failure.

Previous works relied on Fault injection to expose TOF bugs. These works have shown great progress, but all of them required a deep understanding of the system and a lot of manual effort. Furthermore, they struggle at searching in a huge state space of distributed systems. [Looks like this is another common theme. I wonder why?] These systems failed to reveal any TOF bugs under certain conditions.

SOLUTION: FCATCH

The paper presents a new model for TOF bugs, which have two key properties:

Triggering Conditions: the bug is triggered by special timing of fault f, if f occurs earlier or later, the system would behave correctly.

Root causes: Once triggered, a TOF bug can cause system failure when faulty node Ncrash leaves a shared resource in a state that can’t be handled by another node.

An example of TOF bugs from MapReduce

The image above depicts these two properties.

Take a page from previous work the paper also presents the idea of predict TOF bugs by analyzing operations that read/write the same resource from different nodes (conflicting operations), helping reduce the bug search space. Based on the triggering conditions, predicting TOF bugs can be done by analyzing the correct runs to see if/how the system behaves when TOF changes, this helps reduce the manual work considerably.

Crash Regular Bugs: conflicting operations are from crash node Ncrash and a non-crash node Nregular Crash-Recovery Bugs: conflicting operations are from crash node Ncrash and recovery node Nrecovery.

Conflicting operations in a crash-regular bug. →: define-use relation; gray items disappear due to crash.

Conflicting operations in a crash-recovery bug. →: define-use relation; gray items disappear due to crash.

The flow of FCatch

How are these bugs detected?

FCatch monitors a fault-free run of the target system, recording resource-access operations, happens-before operations, and time-out operations FCatch identifies pairs of conflicting operations from different nodes that have blocking happens-before relationship with each other FCatch checks time-out mechanisms in the target system and reports conflicting and fault-intolerant crash-regular operation pairs as candidate bugs FCatch tries to trigger a reported {W, R} crash-regular TOF bug by crashing the node of W right before W or dropping the message where W is from

The model evaluations show that FCatch can indeed effectively detect TOF bugs with high accuracy and a decent overhead.

The model does not cover all bugs that are related to faults in distributed systems. It only covers node-crash and some message drop faults. It does not cover faults that are not related to the timing of faults or ones that may involve interaction among more than one fault/ resource. Previous works on detecting these multi-variable TOF bugs could help extend this to FCatch.

What we learned…

In distributed systems, failure detectors are used to monitor the processes and to reduce the risk of failures by detecting them before system crashes. Accuracy and completeness are the key attributes to measure the quality of failure detectors. A common way to handles these involves two steps: detecting failure and recovering using periodic checkpoints.

In the papers discussed, we looked at how different failures were dealt with. The first paper describes a unique way to approach gray failures, which are really hard to detect. The second paper describes a method to detect crash recovery bugs using meta-info and fault injection testing. The fourth paper on the other hand says that there exist injection-based methods to detect TOF bugs and present its unique approach to solving the problem, which involves it to describe a new approach to look at the TOF bugs. The third paper tries to detect TSV, which has been said to be hard like gray failures.

A common theme that all the papers present are to use some sort of “crash point” which they use in their own way to detect bugs and try to achieve it without using too much overhead.

Large scale distributed systems are hardly ever “perfect”. Due to their complexity, it is extremely difficult to produce flawless designed distributed systems. Traditional fault detection solutions, in particular, fail to work properly in the context of large scale distributed systems. These papers present their innovative solution to this problem.

Hope you have fun reading them as much as we did.

:)