Improving cache performance by exploiting redundancy, temporal affinity, and reuse of data

Recently, the speed-gap between processor and DRAM has widened. A cache-miss can cause a stall time of hundreds of processor cycles; so that stall times caused by cache-misses now constitute one of the most serious bottlenecks in modern computer systems. Therefore it has become imperative to reduce cache misses for speeding up computer systems. Cache misses are mainly caused by insufficient capacity, and can be classified into two classes: (1) cache-misses that occur even when the replacement decisions are correct and (2) cache-misses caused by the wrong replacement decisions. The purpose of this thesis is to propose methods to reduce cache-misses of both types (1) and (2). Two methods that aim to reduce each class of cache-misses are proposed. Type (1) cache misses are reduced by exploiting the redundancy and temporal affinity of data. Two data items are said to have temporal affinity when they are referenced within a short period of time. A static compiler analysis can detect and exploit this affinity in numeric programs. On the other hand, it is difficult for compilers to detect temporal affinity in programs using a recursive data structure, which is used widely in non-numeric programs. We propose a software and hardware cooperative scheme, called the Field Array Compression Technique (FACT), to reduce cache misses caused by a recursive data structure. FACT gathers data with temporal affinity in contiguous memory using a transformation of the data layout in memory. It then compresses recursive pointer and integer fields. As a result, one cache-fill can load together a greater number of data items with temporal affinity. In addition, the compression enlarges the effective cache capacity. The effectiveness of FACT is evaluated through an execution-driven simulation. For eight of the Olden pointer-intensive programs, FACT on average reduces memory stall time by 41.6%, and increases speed by 37.4%. Type (2) cache-misses are reduced by improving the performance of cache replacement algorithm. We propose a novel replacement algorithm, called Inter-Reference Gap Distribution Replacement (IGDR), for set-associative secondary caches of processors. IGDR attaches a weight to each memory-block; on a replacement request, it selects a memory-block with the smallest weight for eviction. The time difference between successive references of a memory-block is called the Inter-Reference Gap (IRG). IGDR estimates the ideal weight of a memory-block using the reciprocal of its IRG. To estimate this reciprocal, we propose a reference model where each memory-block has its own probability distribution of IRGs; and it is assumed that each memory-block follows this model. IGDR calculates the expected value of the reciprocal of the IRG from this distribution to use as the weight of the memory-block. This reference model exploits the fact that the IRG distribution of each memory-block has its own peaks. On the implementation side, IGDR does not have probability distribution; instead it records the IRG distribution statistics at run-time and estimates the weight using these statistics. IGDR classifies memory-blocks into several classes and records statistics for each class. It is shown that IRG distributions of memoryblocks correlate their reference counts; this enables memory-blocks to be classified by their reference counts. The effectiveness of IGDR is evaluated by comparing its performance with the Least Recently Used (LRU) algorithm using an execution-driven simulation. For ten of the SPEC CPU2000 programs, IGDR achieves up to 46.1% (on average 19.8%) miss reduction and up to 48.9% (on average 12.9%) speedup. The combined method of FACT and IGDR is then evaluated. For eight of the Olden pointer-intensive programs, the combined method achieves a speedup by up to 19.9% (on average 3.9%) over FACT. It is confirmed that FACT and IGDR reduce cache-misses in two different ways; so that FACT and IGDR are complementary methods, and that their combination reduces more cache-misses than either of them on their own. Stall time caused by cache-misses presents one of the most serious bottlenecks found in modern computer systems. It is shown that FACT and IGDR are effective methods for alleviating this bottleneck.