HPCC-3326 Thor Lookup Join improvements

Lookup join gathers the RHS into an expanding row array, then exapands that
row array to fit a proportianally sized hash table in at the end.
Change so that:

1) The hash table is a separate allocate, this a) means it's not requiring
and even bigger contiguous single array and b) the initial row array can be
released after hashing
2) If the input row count is known, pre-allocate the hash table
3) The broadcasting/collection of RHS was very inefficient. It involved
compressing blocks on all nodes, node 1 requesting them serially,
uncompressing, collecting into HT, recompressing and broadcasting back to
others.
Change scheme, so streams and broadcasts from all and keeps blocks
compressed until arrive at final destination

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-2197 Ecl package command line parameters need to be more consistent

The same parameters for different actions should be spelled
the same and should be consistent with ecl command line parameters
when applicable.  Required parameters should not be listed as optional
(i.e. target has replaced queryset and is required)

Signed-off-by: Stuart Ort <stuart.ort@lexisnexis.com>
HPCC-2880 Report files currently in use by cluster

Added a new control message control:getQueryXrefInfo. This can be passed a list of
<Query id='name'/> elements to indicate what queries to return info for, or will return
info for all queries if no query is specified.

Return info is of the form
<QueryInfo>
 <Query id='name'>
  <File name='xxx'/>
  <Library name='yyy'/>
 </Query>
</QueryInfo>

Client tools can use this information to present information to the user as they see fit.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7890 Default to outputting multiple c++ files

Gcc (and other compilers) have problems on very large queries (e.g., >10,000)
activities, because of the total size of a single c++ file.  This changes the
default to split into multiple c++ files.  The regression suite still
generates single files in most cases since that is much easier to compare.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
HPCC-7904 Allow xref max memory to be configured

Previously xref was hard-coded to use a maximum of 1GB
There is now a configuration setting to set, the default is now 4GB
On machines with less, it will trigger a out of memory exception, rather
than the forced internal hard limit

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #3407 from jakesmith/xref-memlimit

HPCC-7904 Allow xref max memory to be configured

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7909 Compile failure on OSX

The fixes to the compile error on Windows in HPCC-3162 have broken
compilations on OSX under Clang.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7892 Roxie should default to predictable order

Indexread in roxie will now default to providing a stable result EXCEPT
where UNSOERTED flag explicitly set, rather than providing a stable
result ONLY when SORTED flag is set.

I have added the TIRunsorted flag, so that the compiler can set it
if we see fit. If it doesn't, we will always default to stable.

This matches behaviour of other Roxie activities like disk read and
keyed join, that are assumed to want stable result orders unless proven
otherwise.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3403 from richardkchapman/roxie-xref

HPCC-2880 Report files currently in use by cluster

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #3405 from ghalliday/issue7890

HPCC-7890 Default to outputting multiple c++ files

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #3409 from richardkchapman/roxie-order-7892

HPCC-7892 Roxie should default to predictable order

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #3408 from richardkchapman/compile-osx-HPCC-7909

HPCC-7909 Compile failure on OSX

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.8.x' into candidate-3.10.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7910 Fix building unittests under windows

The cmake file contained a reference to the linux .so instead of a symbolic
reference.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #3410 from ghalliday/issue7910

HPCC-7910 Fix building unittests under windows

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
HPCC-7913 Regression in control:reload caused by HPCC-7897 fix

ReadWriteLock is not reentrant the way a criticalsecetion is - you must release
a read lock before you can get a write lock, even if it's on the same thread.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-3365 Add params for per-query memory and time limits from CLI

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
HPCC-7916 Add ecl roxie commands

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
HPCC-7918 Configmgr - Save As from context menu locks up browser

- Change menu name to 'Save Environment As' from 'Save Environment
  As...' to properly capture context menu click

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
HPCC-2309 Configmgr - Entering LDAP ou info is error-prone

- When modifying the 'ldapServer' field under the 'LDAP' tab for
  the 'Dali Server' component, pop-up will ask to update the 'filesBasedn'
  field to match that of the selected LDAP server.

- When modifying the 'ldapServer' field under the 'Authentication' tab for
  the 'Esp' component, a pop-up will ask to update the 'filesBasedn' field
  to match that of the selected LDAP server for all bound esp services
  that contain a filesBasedn field.

- When modifying the 'serverice' field under the 'ESP Service Bindings' tab
  for the 'Esp' component, a pop-up will ask to update the 'filesBasedn'
  field for all bound esp services that contain a filesBasedn field.

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>