Caches and pipelines were a perfect match for each other. Previously, it didn't make much sense to build a pipeline that could run faster than the access latency of off-chip cash memory. Using on-chip cache memory instead, meant that a pipeline could run at the speed of the cache access latency, a much smaller length of time. This allowed the operating frequencies of processors to increase at a much faster rate than that of off-chip memory.
Pipeline stalls and flushes due to branches are the two main things preventing achieving higher performance through instruction level parallelism. The from time that the processor's instruction decoder has found that it has encountered a conditional branch instruction to the time that the deciding jumping register value can be read out, the pipeline might be stalled for several cycles. On average, every fifth instruction executed is a branch, so that's a high amount of stalling. If the branch is taken, its even worse, as then all of the subsequent instructions which were in the pipeline needs to be flushed.
Techniques such as branch prediction and speculative execution are used to reduce these branch penalties. Branch prediction is where the hardware makes educated guesses on whether a particular branch will be taken. The guess allows the hardware to prefetch instructions without waiting for the register read. Speculative execution is a further enhancement in which the code along the predicted path is executed before it is known whether the branch should be taken or not.
The addition of caches reduces the frequency or duration of stalls due to waiting for data to be fetched from the main memory hierarchy, but does not get rid of these stalls entirely. In early designs a "cache miss" would force the cache controller to stall the processor and wait. Of course there may be some other instruction in the program whose data "is" available in the cache at that point. Out-of-order execution allows that ready instruction to be processed while an older instruction waits on the cache, then re-orders the results to make it appear that everything happened in the programmed order.
Even with all of the added complexity and gates needed to support the concepts outlined above, improvements in semiconductor manufacturing soon allowed even more logic gates to be used.
In the outline above the processor processes parts of a single instruction at a time. Computer programs could be executed faster if multiple instructions were processed simultaneously. This is what superscalar processors achieve, by replicating functional units such as ALUs. The replication of functional units was only made possible when the integrated circuit (some times called "die") area of a single-issue processor no longer stretched the limits of what could be reliably manufactured. By the late 1980s, superscalar designs started to enter the market place.
In modern designs it is common to find two load units, one store (many instructions have no results to store), two or more integer math units, two or more floating point units, and often a SIMD unit of some sort. The instruction issue logic grows in complexity by reading in a huge list of instructions from memory and handing them off to the different execution units that are idle at that point. The results are then collected and re-ordered at the end.
Register renaming refers to a technique used to avoid unnecessary serialized execution of program instructions because of the reuse of the same registers by those instructions. Suppose we have to groups of instruction that will use the same register, one set of instruction is executed first to leave the register to the other set, but if the other set is assigned to a different similar register both sets of instructions can be executed in parallel.
Due to the growing gap between CPU operating frequencies and DRAM access times, none of the techniques that enhance instruction-level parallelism (ILP) within one program could overcome the long stalls (delays) that occurred when data had to be fetched from main memory. Additionally, the large transistor counts and high operating frequencies needed for the more advanced ILP techniques required power dissipation levels that could no longer be cheaply cooled. For these reasons, newer generations of computers have started to utilize higher levels of parallelism that exist outside of a single program or program thread.
This trend is sometimes known as "throughput computing". This idea originated in the mainframe market where online transaction processing emphasized not just the execution speed of one transaction, but the capacity to deal with big numbers of transactions at the same time. With transaction-based applications such as network routing and web-site serving greatly increasing in the last decade, the computer industry has re-emphasized capacity and throughput issues.
One technique of how this parallelism is achieved is through multiprocessing systems, computer systems with multiple CPUs. In the past this was reserved for high-end mainframes but now small scale (2-8) multiprocessors servers have become commonplace for the small business market. For large corporations, large scale (16-256) multiprocessors are common. Even personal computers with multiple CPUs have appeared since the 1990s.
Advances in semiconductor technology reduced transistor size; multicore CPUs have appeared where multiple CPUs are implemented on the same silicon chip. Initially used in-chips targeting embedded markets, where simpler and smaller CPUs would allow multiple instantiations to fit on one piece of silicon. By 2005, semiconductor technology allowed dual high-end desktop CPUs "CMP" chips to be manufactured in volume. Some designs, such as UltraSPARC T1 used a simpler (scalar, in-order) designs in order to fit more processors on one piece of silicon.
Recently, another technique that has become more popular is multithreading. In multithreading, when the processor has to fetch data from slow system memory, instead of stalling for the data to arrive, the processor switches to another program or program thread which is ready to execute. Though this does not speed up a particular program/thread, it increases the overall system throughput by reducing the time the CPU is idle.
Conceptually, multithreading is equivalent to a context switch at the operating system level. The difference is that a multithreaded CPU can do a thread switch in one CPU cycle instead of the hundreds or thousands of CPU cycles a context switch normally requires. This is achieved by replicating the state hardware (such as the register file and program counter) for each active thread.
A further enhancement is simultaneous multithreading. This technique allows superscalar CPUs to execute instructions from different programs/threads simultaneously in the same cycle.
Southampton Corporation Tramways
Southampton Corporation Tramways was a transport company that opened in 1879 and closed in 1949. At first the trams were horse-drawn, but later they were powered by electricity.
Southampton, a city of the United Kingdom located in Hampshire, was becoming bigger in the nineteenth century. The population was 27,000 people in 1841. This was because the docks were becoming much more used. Also, a railway openened in 1839 and a horse-drawn omnibus service started at as well. By 1843 a regular service was running from the station to Millbrook, Eling, Bitterne, Shirley, Portswood, Totton and Romsey. Because of this, by 1861 the population had become 42,000.
By 1872 tramways were a lot more popular. The British and Foreign Tramway Company contacted Southampton Corporation with an idea to set up a tramway in the town. Lots of people were against the idea. The Corporation did not allow tram tracks to be put in the High Street and Above Bar. The British and Foreign Tramway Company stopped thinking about the idea.