Loop subgraph init optimzation

Initialize global loop subgraphs from master if master is executing the loop
globally, instead of requesting init data per slave if subgraph itself is
executing independently

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Removal and rename of HDFS_Stream. Commit to 3.8.x
Adding Initial Genesis User Guide docs to 3.8x branch
Merge branch 'candidate-3.8.x' of github.com:hpcc-systems/HPCC-Platform into gpan-38x
Catch another possible progress serialization crash

At the time the conditional is serialized, the selected input can have
changed. Protect against invalid input.
Also avoid serializing graph if hasn't yet been initialized (caused a
warning on master)

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Fix #2315 - child query varfilename handling

Child queries with activties based on variable/dynamic filenames, need to
be re-initialized with new meta data from the master each iteration.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge branch 'candidate-3.8.x' of github.com:hpcc-systems/HPCC-Platform into gpan-38x
Changes to remove references to optional packages for 3.8
Updated Doc Changes for 3.8
Refactor watchdog a bit and impl. MP version

Introduce MP version of watchdog mechanism, switchable via thor option.
Default to MP, keeping UDP version for now, for debugging reasons.
UDP version remains limited to UDP packet size.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Minmize graph progress

avoid re-sending collected inactive sub graphs

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Minor changes, reflecting comments
Fix gh-2293.  Modify Enterpise Edition Only
message to a friendlier Comming Soon message.

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
Fix gh-2293.  Modify Enterpise Edition Only
message to a friendlier Comming Soon message.

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
gh-2325 Ecl package copyFiles needs to properly read dali ip param

fixes gh-2325

Signed-off-by: Stuart Ort <stuart.ort@lexisnexis.com>
Use a spill result for a dataset hoisted from a loop

A child query that accesses a dataset that is invarient within the child
query has that dataset executed outside of the child query.  However
because the child query only executes on a single node it uses a workunit
result instead of a spill to cope hold the temporary result.

In thor, a loop that isn't in a child query contains global operations.
So any loop invariant datasets that are extracted should still spill to
a disk spill rather than using a workunit result.

A future better fix is probably to always write to graph results for
any values hoisted within graphs (for hthor, roxie and thor).  However
that will require work in the engines, so not for 3.8.

Fixes #2081.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2334 from garonsky/issue2293-EE-only-msg3

Fix gh-2293.  Modify Enterpise Edition Only Message

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2335 from stuartort/pkg_filecopy_nodaliip

gh-2325 Ecl package copyFiles needs to properly read dali ip param

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Fix #1083, use 64bit row count on 32/64 systems

And make ThorExpandingRowArray max row count 32bit (in mem only)
Remove a couple of 32bit restrictions, in particular, sort's local count was
restricted to 2^32

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2330 from g-pan/gpan-38x

Changes to remove references to 'Optional packages'

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>