I kept my code fairly modular, and sacrificed memory for speed and caching of data. For example, for each processor, I kept a node representation for each of its neighbors which cached all the grid sections that the neighbor would need for its ghosts, and which grid sections it owned that we needed ghosts from. One nice feature of keeping my code modular was that I was able to easily simulate the message passing between several virtual processors on a single process. In this way, I could just remove the references to MPI, and run my code on a single processor, while simulating a run of several processors. This allowed for much easier debugging and verification of correct results.
Parallel programming debugging is extremely difficult.
I think this ability to simulate the message passing of MPI is a definite advantage. In general, I found the MPI seemed to work well in speeding up the code, providing good results. I will go into the specific results down below. This is a very detailed description of my choices. For the quick, five sentence summary, see this .