Note that even for dgemm , the message-passing activity slows down local computation on other processors. This effect is more extreme on 8 nodes than on smaller configurations. We believe the cause is an inefficient broadcast algorithm, which has a running time that is linear in the number of nodes. We are currently investigating enhancements to this algorithm. Our results imply that improvements to the communication layer that reduced memory bandwidth requirements could result in improved application performance, through the indirect effect of speeding up local computation.
Times are in seconds.