thorium: use a different timeout parameter for each multiplexing channel

Link: https://github.com/GeneAssembly/biosal/issues/805
Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>
thorium: predict the message count for future flush operations

The message multiplexer increases the throughput (number of
messages per second) when there is a lot of actor messages.

However, using a static timeout (currently 200*1000 ns) for
every message that goes through the multiplexer code path can
reduce throughput in the actor computation. In particular, reply
actor messages are usually empty, which means that they will go
through the multiplexer path without much benefit because
the volume of messages may be insufficient for actually filling
up multiplexer buffers.
Therefore, waiting over 200 μs to flush a small buffer adds unnecessary
latency.

This commit fixes this problem by predicting the next message count
using data in the current period.

The data gathered in a period (for each destination node managed
by a given exporter worker) looks like this:

MULTIPLEXER DATA #0 age 84425 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #1 age 186645 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #2 age 280202 buffer_size 680 / 1024 message_count 34
MULTIPLEXER DATA #3 age 206250 buffer_size 860 / 1024 message_count 43
MULTIPLEXER DATA #4 age 167994 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #5 age 208418 buffer_size 780 / 1024 message_count 39
MULTIPLEXER DATA #6 age 205482 buffer_size 800 / 1024 message_count 40
MULTIPLEXER DATA #7 age 205143 buffer_size 920 / 1024 message_count 46
MULTIPLEXER DATA #8 age 199640 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #9 age 138147 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #10 age 201786 buffer_size 820 / 1024 message_count 41
MULTIPLEXER DATA #11 age 193457 buffer_size 1020 / 1024 message_count 51
MULTIPLEXER DATA #12 age 202264 buffer_size 480 / 1024 message_count 24
MULTIPLEXER DATA #13 age 201622 buffer_size 660 / 1024 message_count 33
MULTIPLEXER DATA #14 age 205166 buffer_size 440 / 1024 message_count 22
MULTIPLEXER DATA #15 age 203170 buffer_size 520 / 1024 message_count 26

[boisvert@bigmem biosal]$ time ./scripts/performance/latency-probe-4x7.sh | tee log
With PREDICT_FUTURE
0m9.295s

Without PREDICT_FUTURE
>= 2 minutes

Link: https://github.com/GeneAssembly/biosal/issues/805
Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>

Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>