Community Edition 3.10.0 Release Candidate 3

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-8234 Avoid deadlock on balanced splitter

If some of the outputs on a splitter are conditional and therefore
never started, a balanced splitter will deadlock when it reaches
it's buffering limit

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-8052 nested ChildDatasets + ECL Playground generated results

Regression introduced when rolling back "disable paging if child
datasets exists" changes.

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
HPCC-8235 Ensure end-of-groups stripped for sorted join input

A grouped input to a global join, which was presorted by the join
keys, was not stripping the end of group markers off, as a
consequence it would stop reading the input at the first end of group.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-8229 Fix occasional crash generating code to cast a string to data

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #3653 from jakesmith/joinsortedgrouped-bug

HPCC-8235 Ensure end-of-groups stripped for sorted join input

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3652 from ghalliday/issue8229

HPCC-8229 Fix occasional crash generating code to cast a string to data

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3651 from GordonSmith/HPCC-8052

HPCC-8052 nested ChildDatasets + ECL Playground generated results

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3650 from jakesmith/split-if-stall

HPCC-8234 Avoid deadlock on balanced splitter

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Community Edition Release 3.8.6-9

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Community Edition 3.10.0 Release Candidate 3a

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-8236 Avoid spilling callback deadlock

A race condition in Thor's use of the spilling callback mechanism,
could mean that it deadlocks. A call back occurs whilst it is
preparing to finish and setup a stream of existing spills, at
that point, it sets up another callback, but that will get
blocked whilst in the existing callback = deadlock.

Fix is to clear the existing callback which is finished with,
before entering the lock and setting up the new one.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #3654 from jakesmith/spillcallback-deadlock

HPCC-8236 Avoid spilling callback deadlock

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Community Edition 3.10.0 Release Candidate 3a

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-8151 Group dedup all, was stripping end of group markers.

The non-hashing variety of group dedup all, was not returning the
end of group markers to the output, i.e. it returned the results
effectively as one large group.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-8250 Activities derived from disk read asserting on multiway files

When reading a multiway file, some activities derived from diskread are
failing with assertion errors due to changes related to the delayed
decision on whether remote read is needed.

Now that the queryOutput never returns the remote adaptor, the redirect
to use remote needs to be added to the nextInGroup methods for csv, xml
and normalize activities.

In addition, where OPT is used and a file ends up with no data, some
fields used by the local read code were not set up.

This error was introduced in rc3 and should be addressed before rc4/gold.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3661 from richardkchapman/diskread-8250

HPCC-8250 Activities derived from disk read asserting on multiway files

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-8255 Csv headings deadlock

A csvread with headings wasn't marked global, as a consequence
there was a race, between initalizing on one slave and sending
data from previous instances to next.

Ensure csvread with headings is marked global, since the activity
instances need to coordinate.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #3669 from jakesmith/candidate-3.8.6

HPCC-8255 Csv headings deadlock

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-8189 JobQueue updating dali too frequently

Both eclccserver and roxie were setting short timeouts on the call to
jobq.dequeue, in order to ensure they could close down cleanly and quickly.

However, this resulted in frequent writes to dali to update the queue
status (several per second), meaning dali would never be idle and the
inc file would grow unnecessarily large.

Use the async cancelAcceptConversation method means we can close down even
faster, and avoids the dali noise.

There is still a timeout every 900s (in a defualt configuration) on the
dfuserver monitor queue. That cannot be eliminated without more significant
refactoring.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>