HPCC-8477 - DFU server is CPU bound for the first phase of CSV files

These fix improves CSV file spray performance when the target is multiple
nodes.

Changes:

- In this particular case a new CCsvMatcher class introduced because the
  original matcher was a bottleneck. If the Special characters of CSV file
  (separator, terminator and quote) are doesn't change or change to only one
  character long new character with a command line parameter, the the process
  use CCsvMatcher. Any other case the old matcher used.

- The used blocksize increased to decrease IO overload.

- When the split size is bigger than the blocksize
  CCsvPartitioner::getSplitRecordSize() function returns with the summarised
  record size in the buffer instead of individual one until it reaches the last
  part ( which is < blocksize). For the last part is switch back to returns
  with individual record size.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
Reworked after review.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-9248 Option to expand hash distribute and match existing distribution

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-8938 - Exception can cause sort partition deadlock

Need to ensure next in asyncfor loop is signalled to continue
to avoid deadlock.
Also add a stack trace when roxiemem runs out of memory.
HPCC-9399 Support smart stepping through a dedup activity

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-9399 Add stepped dedup to hthor and clean up roxie code

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-8316 Clean up the code to remove lots of workflow loops

Instead of applying each stage in turn to all the workflow items,
switch to walking through each workflow item and applying each stage
in turn.  It removes lots of loops.  It could also provide
scope for executing in parallel (although the benefits are likely to
be marginal for most queries).

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-9356 Add depth and distance fields to Graph Page

Added depth to overview and local views.
Added distance to local view.

HPCC-9356

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
HPCC-9347 Relocate Spray forms

Integrate Spray froms into LandingZone browse page.
Add upload/download functionality

Fixes HPCC-9347

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
Move bool processFullBuffer from member of class to parameter list of getSplitRecordSize() method.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
Merge remote-tracking branch 'origin/closedown-4.0.x'

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9391 Refactor DataHandling

Fix HPCC-9391 Refactor DataHandling
doc to faciliate including of sections
in other docs, as needed.

Signed-off-by: g-pan <greg.panagiotatos@lexisnexis.com>
Remove CCsvMatcher.

Signed-off-by: Attila Vamos <attila.vamos@gmail.com>
HPCC-9338 Add --no-files option to ecl-publish

Add --no-files option to prevent ecl-publish from copying DFU file
information used by the query to the target cluster.

Signed-off-by: Anthony Fishbeck <Anthony.Fishbeck@lexisnexis.com>
Merge pull request #4443 from afishbeck/publishNoFiles9338

HPCC-9338 Add --no-files option to ecl-publish

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4441 from g-pan/H9391-DatHan

HPCC-9391 Refactor DataHandling

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #4440 from GordonSmith/HPCC-9347

HPCC-9347 Relocate Spray forms

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9320 Add WUID web links to Browse Queries page

This fix creates a web link for each workunit ID on Browse
Queries page. When the link is clicked, Workunit Details
page will be displayed for the workunit.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
Merge branch 'h9320' into candidate-4.0.0

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-9414 Add XML/JSON responses to FileSpray/UploadFile

On EclWatch Upload File page, when a file is uploaded, the page will
be redirected to Browse Dropzone File page. That does not work when
rawxml_ flag or JSON flag is true. This fix returns upload file results
if the rawxml_ flag or JSON call is detected.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>

Corrected tabs

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>