The performance differences observed below are due to the differences in instruction rate and available local-memory data rate. The measurement plotted above was performed between a pair of Supermicro PCs, which have 64-bit, 66MHz, PCI slots, and excellent PCI performance. The test was performed by one host sending and another host receiving messages of each plotted length repeatedly. The plot is qualitatively similar to the data-rate performance of other types of interconnect, e.g., Ethernet. For small message sizes, the data-rate performance is limited by the number of DMA transfers and packets the interface can handle per unit time.
Longer messages convey bytes in larger units, hence, more efficiently.
The reason for the jagged patterns in the central part of this log-scale plot is that GM fragments long messages into packets of at most 4KB at the sender, and reassembles the packets into messages at the receiver. This fragmentation and reassembly is performed in order to limit the packet size in the network, so that a long message will not block a channel for an extended period, but will allow other packets to be interleaved on the channel. Note the comparison given on this web page with Gigabit Ethernet and multiple 100Base Ethernet links.