Catch another possible progress serialization crash

At the time the conditional is serialized, the selected input can have
changed. Protect against invalid input.
Also avoid serializing graph if hasn't yet been initialized (caused a
warning on master)

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Fix #2315 - child query varfilename handling

Child queries with activties based on variable/dynamic filenames, need to
be re-initialized with new meta data from the master each iteration.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Refactor watchdog a bit and impl. MP version

Introduce MP version of watchdog mechanism, switchable via thor option.
Default to MP, keeping UDP version for now, for debugging reasons.
UDP version remains limited to UDP packet size.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Minmize graph progress

avoid re-sending collected inactive sub graphs

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Fix gh-2293.  Modify Enterpise Edition Only
message to a friendlier Comming Soon message.

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
Use a spill result for a dataset hoisted from a loop

A child query that accesses a dataset that is invarient within the child
query has that dataset executed outside of the child query.  However
because the child query only executes on a single node it uses a workunit
result instead of a spill to cope hold the temporary result.

In thor, a loop that isn't in a child query contains global operations.
So any loop invariant datasets that are extracted should still spill to
a disk spill rather than using a workunit result.

A future better fix is probably to always write to graph results for
any values hoisted within graphs (for hthor, roxie and thor).  However
that will require work in the engines, so not for 3.8.

Fixes #2081.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Fix #1083, use 64bit row count on 32/64 systems

And make ThorExpandingRowArray max row count 32bit (in mem only)
Remove a couple of 32bit restrictions, in particular, sort's local count was
restricted to 2^32

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge remote-tracking branch 'upstream/candidate-3.8.x' into candidate-3.8.x
Adds eclfunction classes to keep track of ECL function properties
Cleans up several sections of code (removes unneeded comments,etc.)
Sends cluster parameter with ecldirect requests.
Extends payload read support.
Adds MAX/MIN support.

Signed-off-by: Rodrigo Pastrana <Rodrigo.Pastrana@lexisnexis.com>
Merge pull request #2348 from rpastrana/ecljdbc_aggreate_fn_work

#2344 and #2327 This change includes fixes/enhancements

Reviewed-By: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2323 from jakesmith/childquery-varfilename

Fix #2315 - child query varfilename handling

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
gh-2216 Add dependency on libarchive

Add dependency on libarchive for all suported distros.

Fixes gh-2216.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2338 from jakesmith/rowcount64

Fix #1083, use 64bit row count on 32/64 systems

Reviewed-By: Renato Golin <rengolin@hpccsystems.com>
Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2332 from jakesmith/graphprogress

Graph progress improvements

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Added missing dojox files.

I am not 100% sure why the issue is intermittent, but there does seem
to be different behavior when certain files are cached locally versus
first load, which _may_ explain it.

Fixes gh-2343

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
Mark link with _dependsOn if different nesting level

A disk write in one graph, followed by a disk read in a loop or
child query wasn't marked as dependent - which causes thor problems.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
fixes gh-2274 Added el6 dependency list.

Signed-off-by: Philip Schwartz <philip.schwartz@lexisnexis.com>
Merge pull request #2353 from pschwartz/centos6Deps

fixes gh-2274 Added el6 dependency list.

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Deleting external temp file dependencies inside CQ

gh-2336

temp/spill files, generated by dependent subgraphs that are read within CQ
need to expect that the temp file may not have been generated within the CQ
set, i.e. they dependency may be external to the CQ set.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2349 from richardkchapman/archive-dependency-2216

Archive dependency 2216

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>