Disk aggregate and index aggregate could corrupt memory

If the result of a disk aggregate or index aggregate was a variable
size row, roxie heap could get overwritten

Removed the unsafe version of the cloneRow function - which is never
 safe to use in new memory paradigms - to ensure that no other issues
remain. A couple of changes in hthor to avoid the unsafe functions
were unlikely to cause corruptions in practice.

Legacy thor does not appear to use these deprecated functions.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #1876 from jakesmith/lookupjoin-stop

Lookup join stop race deadlock

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Fold filenames to ensure they are consistent

- Otherwise very occasiaonly the usage count could be wrong because the
  reads and writes weren't tied up correctly

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Implement LOG(MIN) and LOG(expr) for SOAPCALL

Add new options to SOAPCALL to allow minimal logging to be added to the
log file.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Allow dataset as well as LEFT inside the LOG() expr

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Add soapcall logging tests to regression suite

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
gh-1834 Remove validation before starting ConfigMgr

Remove validation of the default environment before starting Configmgr as it is legacy behaviour when ConfigMgr was working with just one file.

Signed-off-by: Sridhar Meda <sridhar.meda@lexisnexis.com>
Merge pull request #1889 from richardkchapman/roxie-aggregate-var

Disk aggregate and index aggregate could corrupt memory

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #1885 from richardkchapman/prefixjoin-core

Prefixjoin core

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #1896 from smeda/issue1834

gh-1834 Remove validation before starting ConfigMgr

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #1890 from ghalliday/foldfilename

Fold filenames to ensure they are consistent

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #1891 from ghalliday/soaplog

Implement LOG(MIN) and LOG(expr) for SOAPCALL

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #1862 from ghalliday/transformfix

Fix scope of transform on no_select.

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #1841 from ghalliday/splitterfix

Fix problem with extra splitters being created

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Minor reduction in number of aliases created.

Don't create simple aliases of items that are always aliased
- getResults are always saved in member variables

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Fix problems with global optimizer

Previously the optimizer was based on a merging transformer - which
transforms each expression in the tree once for each context it is used
in.  This was because selectors can have ambiguous meanings depending
on the context they were used.

However the optimizer also relies on couning the number of active
references to each dataset, and if an expression is transformed more
than once those counts are no longer valid.  Instead this now
transforms and then calls updateOrphanedSelectors() to ensure the
expressions are only updated once.

That function required fixing to cope with sqnormidx2 and other
examples that implicitly access fields of the parent dataset
(a fairly unusual situation.)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #1897 from hpcc-systems/candidate-3.6.x

Merge latest 3.6 changes back to master

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #1898 from ghalliday/alwaysalias

Minor reduction in number of aliases created.

Reviewed-By: Renato Golin <rengolin@hpccsystems.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
gh-1901 Lookup join leaking rows on failure

Roxie lookup join activity will leak rows if its input throws an
exception while gathering the RHS. Normally any such leak will be
mopped up at the end of the query (though see also gh-1900 for a
situation where they are not.

Fixes gh-1901

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Add test case to regression suite to cover lookupjoin leak

Add a test case that will check the lookupjoin handling for an exception
while gather RHS performs correctly.

Roxie-only as I don't think other platforms support catch.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>