This effect is demonstrated by the fact that the load miss time is reduced by almost half for each doubling in cluster size. Smaller problem sizes running on a larger number of processors would have higher communication miss rates. The results, in Figure 3 show that the performance impact of clustering is indeed greater. However, with the smaller problem size there is an increase in synchronization due to load imbalance. This leads us to the following conclusion for regular grid-based iterative computations.
Clustering does reduce the communication to computation ratio substantially.
When the problem is small enough relative to the number of processors for communication to be significant, clustering helps performance significantly although other overheads such as load imbalance and synchronization can become substantial. While we do not expect clustering to have much impact on large regular-grid problems, clustering may push out the number of processors that can be used effectively on a fixed problem size. Barnes-Hut and FMM fall into the first class of applications in which there is almost no performance benefit as the size of the cluster is increased from 1 to 8 processors. These applications do not have much inherent communication and the reduction because of clustering is not nearly as large as for Ocean, so that clustering has almost no impact on overall performance. Larger numbers of processors would still show little gain in communication, and the synchronization wait times and load imbalance would dominate by the time the communication became large enough for clustering to help.