HPCC-12409 Pathological distribute slowdown fix

Prevent sender queing partial buckets if writers already actively
working on buckets to same target.
Also allow multiple write handler threads to work on the same
target, this allows the serialization/compression (and potentially
dedupping) to occur in parallel even if to same writer.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Yes - I'm not sure that using atomic is really the right answer.

atomics should probably be used when multiple sources might be incrementing a data item at the same time.

In this case you only need to ensure that code that reads the value reads the updated value.  I think adding a compiler_memory_barrier() is probably the more correct way of doing that.

volatile also performs a similar function but doesn't prevent reordering with respect to other non-volatile variables.  (In fact this is all that the atomic_read does - it should possibly have a barrier before it as well.)

We should probably come up with some agreed patterns.  (At the moment there are quite a few places that assume reads from volatile/non-volatile bools are ok, some probably are, but others might not be.)