thorium: increase actor-level throughput by +75%

The ring that is currently used for recycling messages is a bottleneck
(both for the number of CPU cycles in the main node thread and in the
contention and/or cache misses caused by the spinlock on this ring).

By using a fast delivery path (worker-to-worker) for recycled messages,
the level of contention is diminished significantly. Also, the burden of
recycling is moved from the node thread to worker threads (distributed).

Performance indicator:
(actor-level throughpout increase: +75%)

Configuration:
4 nodes, 28 worker threads (4 * 7), 2800 actors (28 * 100)
Total sent message count: 112000000 (2800 * 40000)

Before:
Run 1: Actor messaging rate: 235.919687 messages / second
Run 2: Actor messaging rate: 235.187357 messages / second
Run 3: Actor messaging rate: 205.375243 messages / second
Run 4: Actor messaging rate: 179.353341 messages / second

After:
Run 1: Actor messaging rate: 347.582644 messages / second
Run 2: Actor messaging rate: 354.333163 messages / second
Run 3: Actor messaging rate: 343.403769 messages / second
Run 4: Actor messaging rate: 340.239907 messages / second

LTTng-UST was very useful in tracking the bottleneck.

Link: https://github.com/GeneAssembly/biosal/issues/751
Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>