HPCC-3278 Fix File Spray rate displayed in DFU Workunit Details page

In DFU Workunit Details page, a user found that two different formats
of File Spray speed are displayed . The ProcessMessage shows 253,000KB/s
as 'current rate'. The SummaryMessage shows 253,000Kb/sec as 'average
transfer'. I think both of them should be in 'KB/sec'.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
HPCC-3286 Minisort could spill and lose records

Mini sort is supposed to estimate when all [global] records can fit into
memory, then gather, sort, partition on 1 node. Due to a few reasons and in
particular circumstances, it could spill whilst it was gathering the rows
from nodes 2->n onto node 1, and as a consequence lose records from the
final output.

The main bug was that a spilling flag was being ignored and therefore when
OOM it spilled instead of errored.
The OOM itself was probably mainly as a consequence of the memory manager
issue HPCC-3321, but also in part because the estimate for memory usage for
the minisort was too conservative (wasn't taking into account overheads).

Changes:
+ Fix disabled spilling flag conditions.
+ If minisort is used, the total row count is known, so pre-expand the large
row array.
+ Improve abort handling.
+ Improve size estiamte.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-7815 Remove redundant/wrong default dir

Spurious call to queryBaseDirectory() in Thor backup handler.
Unused and wrong anyway. It gets the paths for primary/replicates from DFS
file descriptor

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3336 update to reflect review comments

simplify 'spilling enabled' tests
+ couple of comments

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-3116 Deschedule WU even if the WU is not found

A user deleted a workunit where the workunit was in the WAIT state. In
the existing code, the workunit cannot be descheduled because the
deschedule() is a member of IWorkunit and the IWorkunit cannot be opened
after the workunit is deleted. In this fix, an extern descheduleWorkunit
is added to deschedule a workunit. Before the descheduleWorkunit is
called, EclWatch verifies that the user has SecAccess_Full right to both
OWN_WU_ACCESS and OTHERS_WU_ACCESS.

This fix also revises the doAction() function which contains the
deschedule code.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
HPCC-3286 Update to use new getExepctedFootprint()

Use new roxiemem method to better estimate memory usage

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #3343 from afishbeck/queries_copy_usage3297

HPCC-3297 Show usage when ecl queries copy is called with no parameter

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3341 from wangkx/h3278

HPCC-3278 Fix File Spray rate displayed in DFU Workunit Details page

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3340 from wangkx/h3116

HPCC-3116 Deschedule WU even if the WU is not found

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3338 from jakesmith/backup-remove-basedir

HPCC-7815 Remove redundant/wrong default dir

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3336 from jakesmith/minisort-fixes

HPCC-3286 Minisort could spill and lose records

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3296 from afishbeck/roxie_clone_files3

HPCC-2704 Setup and copy files to roxie when copying/publishing queries

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.8.x' into candidate-3.10.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7817 Memory leak in Std.Uni.GetNthWord

The RuleBasedBreakIterator object allocated in the unicode plugin function
ulUnicodeLocaleGetNthWord was not released, resulting in a leak every time the
function is called.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3215 from afishbeck/fromxml_regression

gh-2394 Don't ignore xpath within ECL IFBLOCK when outputing XML

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #3344 from richardkchapman/unicode-memleak-7817

HPCC-7817 Memory leak in Std.Uni.GetNthWord

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.8.x' into candidate-3.10.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7813 Add sub-del test in daregress

The bug reported is fixed in 3.8.4, so I'm adding a unit-test to it
so we can make sure we don't regress in the future. Run it alone
with: "daregress localhost subdel".

Signed-off-by: Renato Golin <rengolin@hpccsystems.com>
Merge pull request #3345 from rengolin/dfs-delsub

HPCC-7813 Add sub-del test in daregress

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-3324 ESP can crash if multiple queries received together

If the first time ESP needs to lookup query dlls, it does so on several
threads at the same time, ESP may core due to the global dllServer
object being created multiple times.

Add a CriticalSection to protect the dllServer object. Also copy
strings rather than pointers, in case the lifetime of the dllServer
is ever legitimately shorter than the lifetime of the objects it creates.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>