HPCC-9791 XML file spray logging improvement

Add
- splitOffset value log at findSplitPoint() call
- log predicted recordsize, maxRecordSize and blockSize when "End of record
  not found (need to increase maxRecordSize?)" exception occurs
- extension to exception message with same info as logged (offset, predicted
  recordsize, etc.
- relation between blocksize and maxrecordsize parameter

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-9589 Introduce pseudo tables flags for xml/nlp parse

These can be used instead of the current flags in the context.  However because
of issues with hoisting table invariants currently ignoring counters the
switch from the flags can't be done yet.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-9791 XML file spray logging improvement

Add
- splitOffset value log at findSplitPoint() call
- log predicted recordsize, maxRecordSize and blockSize when "End of record
  not found (need to increase maxRecordSize?)" exception occurs
- extension to exception message with same info as logged (offset, predicted
  recordsize, etc.
- relation between blocksize and maxrecordsize parameter

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-9589 Remove colocal/parent extract from some child activities

If the child activity doesn't need to get anything from the parent activity
then it doesn't need a colocal or parent extract.  This can significantly
cut down the size of some queries (and simplify the classes).

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-9779 Add an internal option to generate the ecl for a subgraph

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-9588 - Avoid holding onto lots of open files

XRef's 'Delete Empty Directories' causes all sub directories to
be collated, then themselves recursed, leaving each open and in
use. This commit, opens and recurses the sub directory then
ensures it's released(closed) before moving on.
This should greatly reduce the # of concurent open files this
process uses, which in environments with many sibling directories
and depth, was causing 1000's of open sockets and handles to
dafilesrv.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-9860 - Avoid holding session crit whilst destroying subs

A destructing subscription can involve other locks/crits.
The session manager should ensure the session crit is released
before their final destruction. Otherwise deadlock can ensure
if another thread is involved in e.g. locking and queries the
session manager.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-9869 Make packages easier to manage on multiple targets

Ad the target name as a scope in front of the pmid.. keeping packagemaps
unique per target cluster.  This protects users from accidently
overwriting shared packagemaps.  Also fix bugs that would deactivate or
activate the wrong packagemap, and some bugs that wouldn't support
PackageSets with process masks other than "*".

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
HPCC-9869 Add support for explicitly using packagemaps shared globally

With Packagemaps now defaulting to be unique copies per target,
--global-scope now provides a way of sharing the same packagemap
globally across multiple targets.

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
HPCC-9841 Deleting files from cluster other than original fails

If a file is present on more than one cluster, deleting the file from one
cluster is liable to delete the physical files from the original location of
the file rather than any subsequent copy.

Typically this means that when a file is created on thor then copied to Roxie,
an attempt to delete the roxie copy (via eclwatch) is liable to remove the
logical entry for the roxie copy, but the physical files for the thor entry.

The entire mechanism for multiple copies of data on more then one cluster
relies on the cluster descriptors being properly set up with directory
location at all times (but they were often not). This change should ensure
that they are, at least more often.

It may end up expanding the size of information stored in dali about file
locations though.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9841 Deleting files from cluster other than original fails

Files created by roxie cluster could not be deleted.

Get rid of the legacy 'roxieLabel' setting on groups that seems to serve only
to confuse.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9841 Deleting files from cluster other than original fails

Flags were not properly being set when a file was cloned to Roxie, so that the
'tlk part repeated' flag was not present.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9884 Roxie does not copy files from remote if local orphan exists

If there is a local orphan copy of a file, Roxie will fail to resolve the file
rather than copying the file from remote location (as it does if the file is
not present). This tends to leave to queries suspended on only SOME nodes,
which confuses everyone.

Added an option ignoreOrphans, defaulting to true, to treat orphans as if they
were not present (and thus overwrite them with new good copies).

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9864 Does not pull data from buddy nodes

Logic for looking for local copies was wrong.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-8844 Document LOCAL keyword

Signed-off-by: RichardTaylorHPCC <richard.taylor@lexisnexis.com>
HPCC-9864 Does not pull data from buddy nodes

Remove unused code

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9874 Improve WsEcl JSON formatting for xml types used by ECL

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
INTERNAL - OSX/Clang build break

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9890 Dynamic loading assumes .so extension (so can fail on OSX)

The code should have been using the defines from platform.h.

It was also very sloppy in the assumptions about when and where to add the lib
prefix.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4756 from richardkchapman/mac-build-break

INTERNAL - OSX/Clang build break

Reviewed-By: Gordon Smith <gordon.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>