Mpi

Let me try to explain the reason for tags again.

Short answer: The rank is simply not enough to handle multiple connections.

Long answer: The MPI channel between any two ranks is like a FIFO queue and MPI guarantees that send-recv pairs are atomic at the message level. By default all messages have tag 0 associated with it, but other tags can be used to use the single channel as multiple logical channels. 

If we use only tag 0, messages need to be multiplexed for different Collage::Connections between two peers some other way, otherwise the data will be mixed up because we senders and receivers in each process can belong to different threads. One way to achieve the multiplexing is to add a connection identifier to the MPI message, but still, you must use a single MPI call for each Connection::write and Connection::send message, because if you split the call in two MPI messages you end up having race conditions (a prerequisite is that we mustn't change connection.cpp at all so adding a lock there is not an option). Adding a connection ID implies an additional data copy to prepare the buffer to be given to MPI or do some other fancy stuff with MPI user defined types (which I don't even know if it's possible). In any case it's a protocol and implementation complexity than can be avoided using tags. 

So tags are the multiplexing mechanism that we need, simple and straightforward. Levaraging the MPI implementation we can be sure that it's made better than we can in a reasonable amount of time by any other means (at least it should).