stable to default
stable to default
stable to default
stable to default
stable to default
Explain
stable to default
merge stable into default
stable to default
stable to default
stable to default
merge stable into default
drive remove_from_queue with DeadGMPids
...instead of LiveGMPids

The latter may be out of date s.t. it contains fewer pids than are
actually alive, due to new GMs having sprung into live recently. We'd
then, incorrectly, remove the corresponding entries from the queue's
mnesia record, causing havoc.

DeadGMPids can be out of date too; it may contain fewer pids than have
actually died, due to GMs having died more recently. That is harmless
though since it never leads us to remove an entry that we shouldn't,
and we will learn about any new deaths eventually.
stable to default
gm_pids is the source of truth
We base promotion decisions on the sub-set of [QPid | SPids] that is
referenced in gm_pids. In particular, QPid may be missing from that,
if another slave previously noticed its death.
always return DeadPids, so that all deaths are logged
record workers in a set
...instead of a queue.

That way when an idle worker is restarted (and sends an 'idle' message
to the pool), it won't end up in the pool twice.

Note that we always hand out work to the first worker in the
ordset. That is actually more efficient than the round-robin strategy
we had with the queue since it keeps a smaller number of workers busy
while others can hibernate.
track workers by Pid instead of name
monitor workers
In a way this is a damage limitation exercise...

When a worker dies while working it is not in the pool anyway, so
noticing its death is a no-op.

When a worker dies while idle, it will not return to the pool when we
next submit work to it. But obviously the submitted work won't be
carried out, and the submitter will get an error (unless they
submitted the work asynchronously).

All the monitoring does is reduce the likelihood of the latter
happening. It cannot eliminate it though since the worker may die just
as work was being submitted to it.
Remove rabbit_amqqueue:force_event_refresh/1 synchrony