All current computer architectures involve advanced techniques to overcome the memory bottleneck. It is also important to distinguish bewteen the latency and bandwidth . The latency measures the time to complete an operation, like a memory read or a multiplication. It is very difficult to improve the latency within a given technology, e.g. it is very costly to build memory modules much faster than the figures given above. Therefore the typical latency for a floating point operation may be 10 basic machine cycles for a single multiplication, while the maximal throughput may be several multiplications in each cycle.
Similar considerations hold for memory throughput and latency.
In consequence, even for numerical algorithms the storage organization and data structures are gaining in importance. In practice the best algorithm is not necessarily the one with the smallest possible number of arithmetic operations but the one where arithmetic cost is well balanced with memory access requirements. Additionally, all programs must be organized such that the CPU-internal parallelism can be exploited. This motivates to study memory organization principles in the Section 3.2 of this chapter.