ComputationalRadiationPhysics/picongpu — pull request #438 by ax3l (2014-05-22 14:18:01+00:00)
Title: Disable mpi_leave_pinned for non-GPU-aware MPIs

### See bug
http://www.open-mpi.org/community/lists/users/2012/11/20659.php

### First seen in PIConGPU
ComputationalRadiationPhysics#407 (comment)

### Problem description:
cudaMalloc/cudaFree are not correctly intercepted for non-GPU aware
MPI implementations.
The default value of `mpi_leave_pinned` `-1` lets the underlying network stack
decide if it prefers to enable this option.
Since cudaMalloc'ed memory is pinned, MPI does not realize the memory
has changed in the follow case:
- `cudaMalloc` some host buffer
- (work with it)
- `cudaFree` the buffer (MPI does not realize that)
- `cudaMalloc` a new buffer and repeat
For MPI the memory behind the second malloc does still *look* as if it
would be unchanged. This causes undefinded behaviour (e.g. the OpenIB
stack could try to reuse the last cached value of the pointer).

Even if we triggered this problem with our local IB network stack, the
bug is not *limited* to Infiniband. It is a generic flag and can in
principle happen for any network stack that tries to pin allocations.
For that reason, the flag was not added to our file openib.conf

### Incomplete list of GPU aware MPI implementations
https://devblogs.nvidia.com/parallelforall/introduction-cuda-aware-mpi/

- MVAPICH2 1.8/1.9b
- OpenMPI 1.7 (beta)
- CRAY MPI (MPT 5.6.2)
- IBM Platform MPI (8.3)

[Titan's](https://www.olcf.ornl.gov/tutorials/gpudirect-mpich-enabled-cuda/) MPI is GPU-aware. The default value in **taurus** is also `-1` and a support ticked about GPU-awareness of the bullxmpi is pending.