HPCC-9588 - Avoid holding onto lots of open files

XRef's 'Delete Empty Directories' causes all sub directories to
be collated, then themselves recursed, leaving each open and in
use. This commit, opens and recurses the sub directory then
ensures it's released(closed) before moving on.
This should greatly reduce the # of concurent open files this
process uses, which in environments with many sibling directories
and depth, was causing 1000's of open sockets and handles to
dafilesrv.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-9860 - Avoid holding session crit whilst destroying subs

A destructing subscription can involve other locks/crits.
The session manager should ensure the session crit is released
before their final destruction. Otherwise deadlock can ensure
if another thread is involved in e.g. locking and queries the
session manager.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-9869 Make packages easier to manage on multiple targets

Ad the target name as a scope in front of the pmid.. keeping packagemaps
unique per target cluster.  This protects users from accidently
overwriting shared packagemaps.  Also fix bugs that would deactivate or
activate the wrong packagemap, and some bugs that wouldn't support
PackageSets with process masks other than "*".

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
HPCC-9869 Add support for explicitly using packagemaps shared globally

With Packagemaps now defaulting to be unique copies per target,
--global-scope now provides a way of sharing the same packagemap
globally across multiple targets.

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
HPCC-9841 Deleting files from cluster other than original fails

If a file is present on more than one cluster, deleting the file from one
cluster is liable to delete the physical files from the original location of
the file rather than any subsequent copy.

Typically this means that when a file is created on thor then copied to Roxie,
an attempt to delete the roxie copy (via eclwatch) is liable to remove the
logical entry for the roxie copy, but the physical files for the thor entry.

The entire mechanism for multiple copies of data on more then one cluster
relies on the cluster descriptors being properly set up with directory
location at all times (but they were often not). This change should ensure
that they are, at least more often.

It may end up expanding the size of information stored in dali about file
locations though.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9841 Deleting files from cluster other than original fails

Files created by roxie cluster could not be deleted.

Get rid of the legacy 'roxieLabel' setting on groups that seems to serve only
to confuse.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9841 Deleting files from cluster other than original fails

Flags were not properly being set when a file was cloned to Roxie, so that the
'tlk part repeated' flag was not present.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9884 Roxie does not copy files from remote if local orphan exists

If there is a local orphan copy of a file, Roxie will fail to resolve the file
rather than copying the file from remote location (as it does if the file is
not present). This tends to leave to queries suspended on only SOME nodes,
which confuses everyone.

Added an option ignoreOrphans, defaulting to true, to treat orphans as if they
were not present (and thus overwrite them with new good copies).

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9864 Does not pull data from buddy nodes

Logic for looking for local copies was wrong.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9864 Does not pull data from buddy nodes

Remove unused code

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9874 Improve WsEcl JSON formatting for xml types used by ECL

Signed-off-by: Anthony Fishbeck <anthony.fishbeck@lexisnexis.com>
INTERNAL - OSX/Clang build break

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9890 Dynamic loading assumes .so extension (so can fail on OSX)

The code should have been using the defines from platform.h.

It was also very sloppy in the assumptions about when and where to add the lib
prefix.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4756 from richardkchapman/mac-build-break

INTERNAL - OSX/Clang build break

Reviewed-By: Gordon Smith <gordon.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4752 from afishbeck/jsonUnsignedx

HPCC-9874 Improve WsEcl JSON formatting for xml types used by ECL

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9883 OSX RPATH Fixing for Plugins

Fixes HPCC-9883

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
Merge pull request #4757 from GordonSmith/HPCC-9883-OSX_Plugin_Patching

HPCC-9883 OSX RPATH Fixing for Plugins

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9891 Generate meta info even if syntax errors

Smart editors like to be able to edit code that is not perfect...

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4758 from richardkchapman/meta-on-errors

HPCC-9891 Generate meta info even if syntax errors

Reviewed-By: Gordon Smith <gordon.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9380 Document STD Library fileservices LogicalFileList param

Signed-off-by: RichardTaylorHPCC <richard.taylor@lexisnexis.com>