HPCC-8686 - Allow global join merge stream to spill

Global join used to merge partitioned side to disk, change to
merge to memory and conditional spill on demand.
Also refactor global join code to remove duplicated code shared
between rightPartition/leftPartition implementations.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3022 - Allow buildindex not to require maxlength if >4k

For legacy backward compatibility reasons, we have forced the max
key size (key+payload) to default to the max record size, which
in legacy systems used to be 4K. This can been overridden
with ,MAXLENGTH on the record structure.
However, this imposes a unnecessary restriction in the current
system, which has no default max record length and therefore
doesn't need to use a 'max' in the key header at all.

Fix by removing the max length check in the key builder, but for
continued backward compatibility, track the largest key+payload
and store it in the header. If the max is < old default legacy(4k)
set header max to 4K.
This will mean any key built on a new system with this
compatibility option on, will create a compatible key for legacy
system as long as the key+payload <= 4k.
If it is higher, it will issue a warning. Legacy systems can still
use the key, but they will need to specify MAXLENGTH.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-8686 - minor change following review
HPCC-10911 DOCS:start up logging

HPCC-10911 DOCS:start up logging
Document the new hpcc-init log which
logs start up init activity

Signed-off-by: g-pan <greg.panagiotatos@lexisnexis.com>
HPCC-10920 Return ClusterName in WsDFU.DFUInfo response

If a logical file is associated with only one cluster, the cluster
name is returned in WsDFU.DFUInfo response. If a logical file is
associated with multiple clusters, the cluster names are returned
inside a comma separated string in the response.

Based on review comments, I created a new structure to hold both
cluster name and file parts on that cluster. Add an array of the
structure into DFUInfo response. Fill in data into that array. In
DFUInfo response, 'depr' existing cluster and DFUFileParts array.

The 'Copy' variable inside the DFUPart structure indicates whether
the copy is primary copy (Copy=1) or not.

Signed-off-by: Kevin Wang kevin.wang@lexisnexis.com
HPCC-10920 Revise a comment to avoid confusing
Community Edition 4.2.4-rc1 Release Candidate 1

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-4.2.4' into closedown-4.2.x

Conflicts:
	version.cmake

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11000 Update numFilesToProcess when removing a file from todo

Incorrect NumFilesToProcess is found in ESP Roxie metrics. This fix
decrements the numFilesToProcess when removing a file from todo queue.
Merge pull request #5523 from wangkx/h11000a

HPCC-11000 Update numFilesToProcess when removing a file from todo

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11007 Failed wildcard spray, empty directory, assert

Fix empty directory handling error.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-11024 Reorder class to improve 64bit packing

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-11025 - Fix various MP close socket issues

Fix:
1) Spot non-MP clients connecting to the MP port early and ignore.
2) IpAddress::setNetAddress - set 'null' ip if null IPv4 provided.
3) Ignore InterCommunicator close sockets as exceptions unless
specifically listening to socket closed.
4) Ingore Communicator close socket exceptions unless part
internal to group communicator is based on.
5) Ensure close socket exception on Thor watchdog recycles Thor.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11025 - changes after review
HPCC-10854 Roxie does not retry file resolution once query fails to load

If a query fails to load (because of missing files, the query will stay
suspended until either Roxie is restarted or the query is deleted and
republished.

Add an option to always retry loading queries if they failed to load
correctly (defaults true, but can be disabled in case it causes
performance issues on clusters where there are a lot of suspended
queries).

Also add an option to control:reload to foce retrying of failed queries

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-10952 Add AuthenticateFeature 'FileUploadAccess' to env setting

The 'FileUploadAccess' permission is missing from the "Esp Features
for SMC". In this fix, the AuthenticateFeature 'FileUploadAccess' is
added to environment.xml. When ECLwatch is started, CLdapSecManager::
createFeatureMap() reads the 'FileUploadAccess' from esp.xml and adds
the permission into LDAP server.

Signed-off-by: Kevin Wang kevin.wang@lexisnexis.com
HPCC-9892 Remove need for LOADXML for template language

Code to so already implemented however, it was still being reported as an error
and thus finaly aborting compilation. Also removed dummy uses of LOADXML within
ecl/regress.

Signed-off-by: jamienoss <james.noss@lexisnexis.com>
Merge pull request #5537 from jakesmith/hpcc-11025

HPCC-11025 - Fix various MP close socket issues

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11007 Failed wildcard spray, empty directory, assert

Update after review.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-11041 Improve tracing when Roxie can not load file

If traceLevel is 3 or higher, print out all locations attempted when Roxie
fails to load a file.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>