thorium: add a queue for messages when request count is too high

Any message sent through transport needs a couple of calls to MPI_Test
before the send operation is completed. Sometimes, the number of MPI
requests is very high, which causes some clutter to accumulate in the
MPI request queue and therefore increases the node latency.

This patch limits the amount of MPI requests at any given time.

Link: https://github.com/GeneAssembly/biosal/issues/826
Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
thorium: add compilation option for clutter avoidance

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
examples: remove trailing spaces

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>