Consider an exchange operaion, proc 1 and proc 2 exchange some data. In MPI this can be done a number of ways, I have a benchmark code to measure this. Here is a subset of its results.
Times are in secs.
Now consider a global sum. On a power of 2 number of nodes the most efficient algorithm is a hypercube algorithm. Writing our own version using way3, gdsum, as the exchange algoruthm results in significant performance increase.