Prevent FROMXML being forced to execute inside a graph

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Add progress of loop (counter)

See mix/max/skew details of loop progress
Add ability to turn on logging for child queries, which are normally
supressed

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Remove blocking progress crit

a legacy critical block was blocking graph progress updates (and watchdog),
whilst child queries were being processed

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Remove misleading CThorExpandingRowArray constructor

More trouble than it was worth, remove constructor which assumes row
interfaces is same as activities. There were a couple of cases there were
wrong. Probably had no consequence, as array row interface only used if rows are
serialized/allocated within array. In cases where they are not spilling
derivities that doesn't happen.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Allow graph results to have spill priority

This was mainly to address a problem with an attempt to spill the counter
result, which had no serializer and hence crashed. But don't want it to try
to spill the counter result anyway (as always 1 row).

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2172 from g-pan/master

Revised Version XML files (cherry picked from master)

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Fix issue with childif and graph progress

The time stat. collection for a child if activity was causing an exception,
if connected to anything other than the 1st input

Also fix potential race in progress collection in join and diskread

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Added CMakeLists.txt files to allow for system build of HDFS_to_HPCC_Connector doc
Merge pull request #2286 from g-pan/gpan-36x

Added CMakeLists.txt files to allow for system build of HDFS_to_HPCC

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Extend implicit row projection to nested rows

The previous version of the implicit project code only analysed the top
level of fields within a dataset.  If a dataset contained a nested
row field then it was either all included, or all removed.  This
commit extends the functionality so it also removes fields from nested
records if they are unsued.

The main differences are
* Ensuring each set of fields knows the original record structure
* Keeping track of nested fields
* Replacing selectors now also ensures that fields of type datarow are
  mapped if they have been changed.

The advantage is often smaller generated code, and reduced data pulled
from datafiles and indexes.  It also improves the accuracy of an
upcoming feature for describing what data is actually used by a query.

The disadvantage is that this generates more unique row formats - which
increases the size of the metadata classes.  An issue is open to work
on reducing the size of those for the common cases.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Minor clean up of the enumerated type

Add an extra case into the activities enum for rollup and iterate
since it is simpler not to combine them and they have very slightly
different semantics.  (No changes in functionality introduced.)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Final fixes and minor optimization

The fields used in the rollup condition weren't being included in the
fields required by the output if they weren't qualified with LEFT or
RIGHT.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2282 from ghalliday/fromxmlchild

Prevent FROMXML being forced to execute inside a graph

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2272 from jakesmith/loop-progress

Add progress of loop (counter) to graph stats

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2257 from jakesmith/result-spillpriority

Allow graph results to have spill priority

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2285 from jakesmith/graphprogress-crash

Fix issue with childif and graph progress

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2281 from ghalliday/newiproj

Extend implicit row projection to nested rows

Reviewed-By: Renato Golin <rengolin@hpccsystems.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.6.x' into candidate-3.8.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.8.x'

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>