Thus, before discussing the various alternatives, it is important to become familiar with a few commonly used terms. This model naturally fits the concept of performing the same operation on every element of an array, and is thus often associated with vector or array manipulation. Because all operations are inherently synchronized, interactions among SIMD processors tend to be easily and efficiently implemented. This is a more flexible model than SIMD execution, but it is achieved at the risk of debugging nightmares called race conditions , in which a program may intermittently fail due to timing variations reordering the operations of one processor relative to those of another. Unlike SIMD, each processor executing SPMD code may take a different control flow path through the program.
High bandwidth allows large blocks of data to be transferred efficiently between processors.
The latency of a communication system is the minimum time taken to transmit one object, including any send and receive software overhead. Latency is very important in parallel processing because it determines the minimum useful grain size , the minimum run time for a segment of code to yield speed-up through parallel execution. In general, a message is constructed by software on one processor and is sent through an interconnection network to another processor, which then must accept and act upon the message contents. Thus, message passing can yield high bandwidth making it a very effective way to transmit a large block of data from one processor to another. Systems like the multi-processor Pentium machines running Linux physically share a single memory among their processors, so that a value written to shared memory by one processor can be directly accessed by any processor.