Our idea is to use the processor on the Myrinet network card to do the Communication while the host processor is busy with the computation. Message matching on the Network processor allows for overlapping of computation and the communication. We achieved one way message latency of about 25 micro seconds. When sending a large message, the Network processor DMA the message body without any intervention from the host processor.
Host simply copies the message envelope and the starting address of the data buffer.
Current implementation is in a very basic stage. It can send and receive data from only pre pinned memory pages and can only handle simple data types. The implementation is not portable either. The myrinet communication layer is implemented as a separate RPI layer. We have published the details in following papers.