HPCC-3326 Thor Lookup Join improvements

Lookup join gathers the RHS into an expanding row array, then exapands that
row array to fit a proportianally sized hash table in at the end.
Change so that:

1) The hash table is a separate allocate, this a) means it's not requiring
and even bigger contiguous single array and b) the initial row array can be
released after hashing
2) If the input row count is known, pre-allocate the hash table
3) The broadcasting/collection of RHS was very inefficient. It involved
compressing blocks on all nodes, node 1 requesting them serially,
uncompressing, collecting into HT, recompressing and broadcasting back to
others.
Change scheme, so streams and broadcasts from all and keeps blocks
compressed until arrive at final destination

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Minor changes following review
HPCC-7941 Update ConfigMgr Images and Docs

Update docs and images depicting configmgr
to reflect or capture changes

Signed-off-by: G Panagiotatos <greg.panagiotatos@lexisnexis.com>
HPCC-7941 Update ConfigMgr Images and Docs

Update docs and images depicting configmgr
to reflect or capture changes

Signed-off-by: G Panagiotatos <greg.panagiotatos@lexisnexis.com>
HPCC-2885 not all LIMIT options are documented on JOIN

Signed-off-by: Richard Taylor <richard.taylor@lexisnexis.com>
fix whitespace issues
HPCC-7961 Fix problems with eclcc handling of -main

* eclcc -main with no file now compiles the referenced definition

* eclcc -main with a supplied file now replaces the definition with
the query text.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-7930 Rename ecl-package to ecl-packagemap

Signed-off-by: Stuart Ort <stuart.ort@lexisnexis.com>

Conflicts:

	ecl/ecl-package/ecl-package.cpp
HPCC-7884 Detect superfile error on reload subs

This check throws an error on invalid ranges, which will trigger an error
message on both listing a superfile and trying to add new subfiles to it.
Also relevant to HPCC-7938.

Signed-off-by: Renato Golin <rengolin@hpccsystems.com>
HPCC-7941 Update ConfigMgr Images and Docs

Update docs and images depicting configmgr
to reflect or capture changes

Signed-off-by: G Panagiotatos <greg.panagiotatos@lexisnexis.com>
Merge pull request #3451 from rengolin/dfs-badsuper

HPCC-7884 Detect superfile error on reload subs

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge branch 'candidate-3.8.x' into candidate-3.10.x

Conflicts:
	dali/base/dadfs.cpp

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7971 Fix typo in fromxml.xml regression suite key

A typo in the key file is causing spurious regression suite failures on
this test.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7966 Spurious errors reported by remote.ecl in regression test

The remote.ecl test as coded would give different results depending on the
size of the cluster that the test was run on. Code that had been added to
try to compensate for this was not correct.

Hopefully this latest attempt is.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7880 - Additional thor memory tracing

Trace # maxCores setting per job
Trace roxiemem and heap usage at end of each job.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #3463 from richardkchapman/roxie-regress

HPCC-7966 Spurious errors reported by remote.ecl in regression test

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #3462 from richardkchapman/fromxml-7971

HPCC-7971 Fix typo in fromxml.xml regression suite key

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
HPCC-7953 Roxiemem unit tests fail if all run together

Running roxie -selftest to execute all unit tests may cause some to fail, as
the roxiemem heap size ends up being set by whichever test executes first, and
this may not be enough for later tests.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3391 from jakesmith/lookupjoin-opt

HPCC-3326 Thor Lookup Join improvements

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #3456 from richardkchapman/roxie-selftest

HPCC-7953 Roxiemem unit tests fail if all run together

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>