An Improved "Flight Data Recorder" of Shared Memory Races

ADDING SIMPLE RECORDER HARDWARE TO A MULTICORE CHIP CAN ENABLE DETERMINISTIC REPLAY FOR CYCLIC DEBUGGING. Hardware vendors are currently transitioning from singlethreaded microprocessors to chips that integrate multiple processor cores and threads in what is variously called multicore, chip multiprocessing, and chip multithreading. At the same time, developing and maintaining reliable software continues to challenge software vendors. Effectively debugging the software is critical. Valuable to any debugger, deterministic replay enables a developer to re-execute the (buggy) program and zero in on bugs that faithfully re-appear. Moreover, deterministic replay can also be useful for fault detection/recovery, intrusion detection, and other applications. To effectively use multicores, however, programmers must write and debug multithreaded applications. Unfortunately, neither the software or hardware environments of multicores provide deterministic replay. If nothing is done, software could be delivered later, with more bugs, or both. If users fail to see continued doubling of effective computer performance, vendor profits could be negatively affected. To mitigate some of the gloom in this situation, we advocate augmenting multicores with modest hardware, called a Flight Data Recorder (FDR), that records sufficient information in a log to enable deterministic replay. Like an aircraft flight data recorder, FDR continually records information at low overhead during normal operation. This paper provides a gentle introduction to how FDR solves the critical problem of recording memory races, while our previous work discusses all aspects, including providing algorithms and related work [9, 10, 8]. A race recorder must log sufficient information to order the outcomes of all conflicting memory accesses. Two accesses (reads or writes) conflict if they are from different threads, access the same memory block, and at least one of them is a write. In particular, FDR’s race recording: • supports multicore designs using sequential consistency (SC) or total store order (TSO), which is x86-like; • augments each core with a dynamic instruction counter and a small local memory for logical timestamps (which are derived from instruction counts); • piggybacks timestamps on some coherence messages; and • exploits transitivity to log about one byte per thousand instructions executed. Memory Race Recording Context Before we develop FDR, we summarize the context in which it operates. We assume a recorder and a replayer. The recorder logs information during multithreaded program execution that is sufficient to enable deterministic replay. Ideally, it should allow program to run at (nearly) full speed so the recorder can be “always on” for postmortem analysis of bugs in the field (like aircraft flight data recorders). The recorder must solve three sub-problems. First, it must record initial state. This is trivially done if recording begins as a program starts execution. Otherwise, the problem reduces to taking a checkpoint. Checkpointing is a wellstudied problem with many old (e.g., copy-on-write pages) and new solutions (logging selected read values [5]). With sufficiently long replay intervals, even high checkpoint overhead will only minimally affect performance. Second, a recorder must log inputs from outside the system being recorded, which is also a well-studied problem. An input may include both a value and a timestamp. Input values include program reads of devices (I/O space), values copied into memory by devices with direct memory access (DMA), and values that affect (e.g., vector) interrupts. Input timestamps record the I/O “timing” (e.g., the dynamic instruction count when a processor services an interrupt). Third, the recorder must log the outcome of all memory races, which is the subject of this paper. After recording, the replayer uses the logged information, together with the program binary, to faithfully replay the original execution. The replay will always exercise the same bug(s) and produce the same output(s). In many cases, replayer execution time is less critical, because it is used only when bugs are discovered, which is (hopefully) a small fraction of all executions. Thus, it may be sufficient to do replay with a software simulator. We have only developed a simple replayer for testing recorders. Memory race recording holds the key for future recorders for multithreaded programs on multicore systems. At VMworld 2006, Mendel Rosenblum demonstrated a recordreplay prototype where a colleague painted a picture with Microsoft Paint and replay re-painted it. They received a rousing ovation. In the future, we would like to do such a demonstration with a multithreaded program. In our judgement (prior to Xu joining VMware), doing so will require hardware memory race recording. This is because existing software race recorders slow down program