Split ExchangeHalo into BeginExchangeHalo and EndExchangeHalo.

This is to enable overlapping of communication with communication.
Merge branch 'master' into asynch-halo-exchange
Adding scatterKernel.

This can be used to collect data to be communicated to other ranks on
the GPU into a contiguous array.  In the case were we communicate via
the host this saves PCIe bandwidth.  It can also be used as a
preparatory step for GPU aware MPI messaging of arrays.
Break matrix up into diagonal and off-diagonal block.
Add device array for scatter indices to MatrixOptimizationDataTx.
Begin/EndExchangeHalo take GPU data.

Previously these assumed that CPU data was up to date.  Now GPU data is
assumed up to date and the scatter and gather act on the GPU data.  This
is to get consistent behavior with and without GPUDirect.
Add declaration of legacy ExchangeHalo to ExchangeHalo.hpp.
A bunch of bug fixes in halo exchange related functions.
Fix communication sequence in ComputeSYMGS.