open-mpi/ompi — pull request #246 by yosefe (2014-10-21 13:22:26+00:00)
Title: [RFC] Add new PML yalla, which uses mxm directly to reduce overhead.

WHAT: Add support for Mellanox Technologies’ Messaging library as a new PML named “yalla”.

WHERE: Trunk and 1.9

WHEN: Oct. 15

WHY: Reduce MPI overhead by plugging MXM directly to communication calls.

Mellanox Technologies’ Messaging library (MXM) provides support for high-level API of tag-matched send/receive operations which can be directly mapped to MPI point-to-point communications. Currently it’s plugged into PML CM as an MTL implementation. However, the CM layer adds unnecessary overheads, mostly induced by management of large request structures, and an additional software layer. By using MXM as a PML, we get rid of these overheads.

Some performance results:
* latency - mtl_mxm: 1.14 usec, pml_yalla: 1.08 usec
* single core message rate - mtl_mxm: 4.9 Mpps, pml_yalla: 7.3 Mpps

http://starwars.wikia.com/wiki/Ubed_Yalla