HPCC-8686 - Allow global join merge stream to spill

Global join used to merge partitioned side to disk, change to
merge to memory and conditional spill on demand.
Also refactor global join code to remove duplicated code shared
between rightPartition/leftPartition implementations.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3022 - Allow buildindex not to require maxlength if >4k

For legacy backward compatibility reasons, we have forced the max
key size (key+payload) to default to the max record size, which
in legacy systems used to be 4K. This can been overridden
with ,MAXLENGTH on the record structure.
However, this imposes a unnecessary restriction in the current
system, which has no default max record length and therefore
doesn't need to use a 'max' in the key header at all.

Fix by removing the max length check in the key builder, but for
continued backward compatibility, track the largest key+payload
and store it in the header. If the max is < old default legacy(4k)
set header max to 4K.
This will mean any key built on a new system with this
compatibility option on, will create a compatible key for legacy
system as long as the key+payload <= 4k.
If it is higher, it will issue a warning. Legacy systems can still
use the key, but they will need to specify MAXLENGTH.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-8686 - minor change following review
HPCC-10911 DOCS:start up logging

HPCC-10911 DOCS:start up logging
Document the new hpcc-init log which
logs start up init activity

Signed-off-by: g-pan <greg.panagiotatos@lexisnexis.com>
HPCC-10920 Return ClusterName in WsDFU.DFUInfo response

If a logical file is associated with only one cluster, the cluster
name is returned in WsDFU.DFUInfo response. If a logical file is
associated with multiple clusters, the cluster names are returned
inside a comma separated string in the response.

Based on review comments, I created a new structure to hold both
cluster name and file parts on that cluster. Add an array of the
structure into DFUInfo response. Fill in data into that array. In
DFUInfo response, 'depr' existing cluster and DFUFileParts array.

The 'Copy' variable inside the DFUPart structure indicates whether
the copy is primary copy (Copy=1) or not.

Signed-off-by: Kevin Wang kevin.wang@lexisnexis.com
HPCC-10821 Fix roxie job not shown on Activity page

The existing code goes through each cluster in environment settings and
find out the jobs of the cluster using /status/servers/server/@name=cluster's
'AgentQueue' in the environment settings. That does not work for roxie
cluster because @name='RoxieServer', not ='roxie.roxie'.

In this fix, the existing data collection code for cluster status and jobs is
divided into three steps. The step 1 focuses on collecting data about cluster
and cluster status. This step collects cluster information for each target
cluster list in environment settings and finds out whether cluster queue
is in status/servers or not. In the step 2, the code goes through each
target cluster (and other servers including ECLagent) list in status/servers
and collect job information (both running jobs and queued jobs) of the
cluster (and server). The data for roxie jobs will be included in this step.
The information collected in the step 1 and 2 will be added into
ActivityResponse by the code of step 3.

Signed-off-by: Kevin Wang kevin.wang@lexisnexis.com
HPCC-10821 Simplify readRunningWUsOnStatusServer()
HPCC-10920 Revise a comment to avoid confusing
Community Edition 4.2.4-rc1 Release Candidate 1

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-4.2.4' into closedown-4.2.x

Conflicts:
	version.cmake

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-10460 Streamed dataset support for Java

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-10821 Revise based on review comments

Replace the structure WsSMCStatusServerTypeName with an enum and
a parallel array of strings; Rename the getStatusServerName() to
getStatusServerTypeName(); Rename the foundQueueInStatusServer()
to findQueueInStatusServer(); Rename the getWUClusterInfo() to
findWUClusterInfo(); Fix a connection leak when calling querySDS;
Fix two possible leaks in case of exceptions; Move 7 lines of
code into a new function findTargetCluster() which is called
three times.
HPCC-11000 Update numFilesToProcess when removing a file from todo

Incorrect NumFilesToProcess is found in ESP Roxie metrics. This fix
decrements the numFilesToProcess when removing a file from todo queue.
HPCC-10459 Streamed dataset support for R

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11020 Leaking rows in the streamed embed plugins

Make sure rows are released after use.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-10460 Streamed dataset support for Java

Fix memory leak of incoming Roxie rows

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5523 from wangkx/h11000a

HPCC-11000 Update numFilesToProcess when removing a file from todo

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11007 Failed wildcard spray, empty directory, assert

Fix empty directory handling error.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-11024 Reorder class to improve 64bit packing

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-11025 - Fix various MP close socket issues

Fix:
1) Spot non-MP clients connecting to the MP port early and ignore.
2) IpAddress::setNetAddress - set 'null' ip if null IPv4 provided.
3) Ignore InterCommunicator close sockets as exceptions unless
specifically listening to socket closed.
4) Ingore Communicator close socket exceptions unless part
internal to group communicator is based on.
5) Ensure close socket exception on Thor watchdog recycles Thor.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>