Precise and Accurate Processor Simulation

Harold W. Cain, Kevin M. Lepak, Brando Precise and accurate simulation of processors and computer systems is a painstaking, time-consuming, and error-prone task. Abstraction and simplification are powerful tools for reducing the cost and complexity of simulation, but are known to reduce precision. Similarly, limiting and simplifying the workloads that are used to drive simulation can simplify the task of the computer architect, while placing the accuracy of the simulation at risk. Historically, precision has been favored over accuracy, resulting in simulators that are able to analyze minute machine model variations while mispredicting— sometimes dramatically—the actual performance of the processor being simulated. In this paper, we argue that both precise and accurate simulators are needed, and provide evidence that counters conventional wisdom for three factors that affect precision and accuracy in simulation. First, we show that operating system effects are important not just for commercial workloads, but also for SPEC integer benchmarks. Next, we show that simulating incorrect speculative paths is largely unimportant for both commercial workloads and SPEC integer benchmarks. Finally, we argue that correct simulation of I/O behavior, even in uniprocessors, can affect simulator accuracy. 1.0 Introduction and Motivation For many years, simulation at various levels of abstraction has played a key role in the design of computer systems. There are numerous compelling reasons for implementing simulators, most of them obvious. Design teams need simulators throughout all phases of the design cycle. Initially, during high-level design, simulation is used to narrow the design space and establish credible and feasible alternatives that are likely to meet competitive performance objectives. Later, during microarchitectural definition, a simulator helps guide engineering trade-offs by enabling quantitative comparison of various alternatives. During design implementation, simulators are employed for testing, functional validation, and late-cycle design trade-offs. Finally, simulators provide a useful reference for performance validation once real hardware becomes available. Outside of the industrial design cycle, simulators are also heavily used in the computer architecture academic research community. Within this context, simulators are primarily used as a vehicle for demonstrating or comparing the utility of new architectural features, compilation techniques, or microarchitectural techniques, rather than for helping to guide an actual design project. As a result, academic simulators are rarely used for functional or performance validation, but strictly for proof of concept, design space exploration, or quantitative trade-off analysis. Simulation can occur at various levels of abstraction. Some possible approaches and their benefits and drawbacks are summarized in Table 1. For example, flexible and powerful analytical models that exploit queueing theory can be used to examine system-level trade-offs, identify performance bottlenecks, and make coarse performance projections. Alternatively, equations that compute cycles per instruction (CPI) based on cache miss rates and fixed latencies can also be used to estimate performance, though this approach is not effective for systems that are able to mask latency with concurrent activity. These approaches are powerful and widely employed, but will not be considered further in this paper. Instead, we will focus on the last three alternatives: the use of either trace-driven, execution-driven, or full-system simulation to simulate processors and computer systems. Trace-driven simulation utilizes execution traces collected from real systems. Collection schemes range from software-only schemes that instrument program binaries all the way to proprietary hardware devices that connect to processor debug ports. The former pollute the collected trace, since the software overhead slows program execution relative to I/O devices and other external events. The latter can run at full speed, but require expensive investment in proprietary hardware, knowledge of debug port interfaces, and are probably not feasible for future multigigahertz processors. Trace collection is also hampered by the fact that usually only non-speculative or committed instructions are recorded in the trace. Hence, the effects of speculatively executed instructions from incorrect branch paths are lost. Furthermore, once a trace has been collected, the cost associated with the disk space used to store the trace may be a limiting factor. Prior work has argued that trace-driven simulation is no longer adequate for simulating modern, out-of-order processors (e.g. [1]). In fact, the vast majority of research papers published today employ execution-driven simulation and utilize relatively detailed and presumably precise simulation. A recent paper argued that precise simulation is very important and can dramatically affect the conclusions one might draw about the relative benefits of specific microarchitectural techniques [6]. Some of these conclusions were toned down in a subsequent publication [5]. One can reasonably conclude that the majority of recently published computer architecture research papers place a great deal of emphasis and effort on precision of simulation, and researchers invest large amounts of time implementing and exercising detailed simulation models. In this paper, we show that conventional approaches to exercising processor simulators do so poorly vis-a-vis accuracy that, practically speaking, precision is unimportant. We argue that the correct approach is to build a simulator that is both precise and accurate. We accomplish this by building a simulator--PHARMsim--that does not cheat with respect to any aspect of simulation. Without the investment in such a simulator, we assert that it is impossible to determine whether or not the right abstractions and simplifications have been applied to either the simulator or the workload that is driving it. We provide evidence that counters conventional wisdom for three factors that affect precision and accuracy in simulation. First, we show that operating system effects are important not just for commercial workloads (as shown by [8] and numerous others), but also for SPEC integer benchmarks [13]. Surprisingly, omitting the operating system can introduce error that exceeds 100% for benchmarks that, according to conventional wisdom, hardly exercise the operating system. Next, we show that simulating incorrectly predicted speculative paths is largely unimportant for both commercial workloads and SPEC integer benchmarks. In most cases, even with an aggressively speculative processor model that issues twice as many instructions as it retires, the bottom line effect on performance is usually less than 1%, and only 2.4% in the worst case. Finally, we argue that correct simulation of I/O behavior, even in uniprocessors, can affect simulator accuracy. We find that a direct-memory-access (DMA) engine that correctly models the timing of cache line invalidates to lines that are written by I/O devices can affect miss rates by up to 2% and performance by up to 1%, even in a uniprocessor with plenty of available memory bandwidth. We have found that the main drawback of a simulator that does not cheat is the expense and overhead of correctly implementing such a simulator. For example, since our DMA engine implementation relies on the coherence protocol to operate correctly, the coherence protocol must be correctly implemented. Similarly, since the processors actually read values from the caches, rather than cheating by reading from an artificially-maintained flat memory image, DMA and multiprocessor coherence must be correctly maintained in the caches. Furthermore, within the processor core, the register renaming, branch redirection, Modeling Technique Inputs Benefits Drawbacks Analytical models Cache miss rates; I/O rates Flexible, fast, convenient, provide intuition Cannot model concurrency;