gh-2704 Setup and copy files to roxie when copying/publishing queries

Add support in ESP for cloning DFU file information when publishing
or copying queries.  When copying a remote query include information
about the remote source so that roxie will know where to find files
located on a remote DALI environment.

Add support to roxie for using the remote DALI information to copy
files from another DALI environment when necessary.

Add ecl-queries command line options for optionally setting file
information overwrite or not setting up the file information at all.

Previously Roxie would copy files if they were already in
the local environment, but DFU/DALI would not be aware of the new
copies.  Roxie was also not able to copy files for queries copied
from a remote environment.

Fixes gh-2704

Signed-off-by: Anthony Fishbeck <Anthony.Fishbeck@lexisnexis.com>
gh-2394 Don't ignore xpath within ECL IFBLOCK when outputing XML

Some xml tags were being output incorrectly because the xpath
attribute used by fileview2 to format xml output were
dropped when FVDataSource::setReturnedInfoFromResult() called
getSimpliedRecord to create the HqlExpression used for processing ouput.

getSimplifiedRecord transformed strings that came after an ifblock,
when the field was transformed field attributes (in this case, the
xpath) were dropped.  Any xpaths used on strings in an ifblock
were ignored.

Fixes gh-2394

Signed-off-by: Anthony Fishbeck <Anthony.Fishbeck@lexisnexis.com>
HPCC-3278 Fix File Spray rate displayed in DFU Workunit Details page

In DFU Workunit Details page, a user found that two different formats
of File Spray speed are displayed . The ProcessMessage shows 253,000KB/s
as 'current rate'. The SummaryMessage shows 253,000Kb/sec as 'average
transfer'. I think both of them should be in 'KB/sec'.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
HPCC-3286 Minisort could spill and lose records

Mini sort is supposed to estimate when all [global] records can fit into
memory, then gather, sort, partition on 1 node. Due to a few reasons and in
particular circumstances, it could spill whilst it was gathering the rows
from nodes 2->n onto node 1, and as a consequence lose records from the
final output.

The main bug was that a spilling flag was being ignored and therefore when
OOM it spilled instead of errored.
The OOM itself was probably mainly as a consequence of the memory manager
issue HPCC-3321, but also in part because the estimate for memory usage for
the minisort was too conservative (wasn't taking into account overheads).

Changes:
+ Fix disabled spilling flag conditions.
+ If minisort is used, the total row count is known, so pre-expand the large
row array.
+ Improve abort handling.
+ Improve size estiamte.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-7815 Remove redundant/wrong default dir

Spurious call to queryBaseDirectory() in Thor backup handler.
Unused and wrong anyway. It gets the paths for primary/replicates from DFS
file descriptor

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge remote-tracking branch 'origin/candidate-3.8.x' into candidate-3.8.4

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
gh-3210 Ensure temp files are under user scope

Redirect all transient files to an internal per user scope.
This will enable security to be enabled on a per user basis and prevent
unauthorised users from seeing each others temp files

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3286 Provide new helper function to estimate row memory consumption

Add a new getExpectedFootprint function to estimate the amount of memory
that an allocation of a given size will actually require.

Also includes a fix to getExpectedCapacity for a packed row (probably
unused at the moment), and remove a VS warning about a double/size32_t
conversion.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-3336 update to reflect review comments

simplify 'spilling enabled' tests
+ couple of comments

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3116 Deschedule WU even if the WU is not found

A user deleted a workunit where the workunit was in the WAIT state. In
the existing code, the workunit cannot be descheduled because the
deschedule() is a member of IWorkunit and the IWorkunit cannot be opened
after the workunit is deleted. In this fix, an extern descheduleWorkunit
is added to deschedule a workunit. Before the descheduleWorkunit is
called, EclWatch verifies that the user has SecAccess_Full right to both
OWN_WU_ACCESS and OTHERS_WU_ACCESS.

This fix also revises the doAction() function which contains the
deschedule code.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
Merge pull request #3339 from ghalliday/issue3286

HPCC-3286 Provide new helper function to estimate row memory consumption

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-3286 Update to use new getExepctedFootprint()

Use new roxiemem method to better estimate memory usage

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
gh-2704 Use const methods in IReferencedFile interface

These methods should be declared const, the interface does not change the
state of the underlying class.

Signed-off-by: Anthony Fishbeck <Anthony.Fishbeck@lexisnexis.com>
HPCC-3297 Show usage when ecl queries copy is called with no parameter

Other CLI commands that have required parameters show usage when no
parameters are provided.

Signed-off-by: Anthony Fishbeck <Anthony.Fishbeck@lexisnexis.com>
Merge pull request #3343 from afishbeck/queries_copy_usage3297

HPCC-3297 Show usage when ecl queries copy is called with no parameter

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3341 from wangkx/h3278

HPCC-3278 Fix File Spray rate displayed in DFU Workunit Details page

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3340 from wangkx/h3116

HPCC-3116 Deschedule WU even if the WU is not found

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3338 from jakesmith/backup-remove-basedir

HPCC-7815 Remove redundant/wrong default dir

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3336 from jakesmith/minisort-fixes

HPCC-3286 Minisort could spill and lose records

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3296 from afishbeck/roxie_clone_files3

HPCC-2704 Setup and copy files to roxie when copying/publishing queries

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>