Alright I have 1.28 ms for 100K nodes on 1 K20m. Thats 15 GFLOPs. I get 22GFLOPs on 2 k20s both of these are using the overlap testcase.
Overlap benchmarks for the two kepler k20s are done
Identified a bug in condition to choose isend vs alltoall (for procs > 8 it was doing both)
I have a new idea for encoding
Boom! squashed the R_by_rank decoding bottleneck. The R_by_rank is ordered by rank already, and I have the R set of nodes ordered the same way. I was able to directly copy the memory up to the GPU.
Awesome. The direct copy to GPU is working so my comm time is cut in half
Fast comm times from overlap_fast