When the processor makes a memory request, the request first passes to the primary cache. If the data item is found in this cache, we have a cache hit . If the data item is not found in the primary cache, we have a cache miss and the memory request is forwarded to the L2 cache. If the data item is found in this cache, we have an L2 cache hit and the data is passed back to the primary cache. If the data is not found in the L2 cache, the request is finally forwarded to the main memory.
Caches work well because the memory request can usually be serviced by the primary cache.
Thus, the L2 and main memory are rarely accessed. If a memory location is referenced, it is very likely that the memory location will be referenced again in the near future. If a memory location is referenced, it is very likely that a nearby memory location will also be referenced in the near future. Consider a program that spends much of its time executing loops of some type, or the inner portions of loop nests. These loops usually contain only a few hundred bytes of code.