Various index count and aggregate fixes

Remove projects from count(keyed-limit(project(index)))
  A missing case from a switch statement meant projects weren't
  stripped when there was a keyed limit on the index read.

Reduce the generated code size for an index count/exists
  The numValid(size, ptr) member was never called for an index, so default
  it to throwing an exception in the base class, and clean up numValid(ptr)
  code.

CHOOSEN(LIMIT(index-read))
  The choosen was not being combined within the compound indexread if there
  was a limit.  It is however valid to do so since the choosen is done last.
  (A limit cannot be combined when there is an existing choosen.)
  This was unfortunate because it was frustrating  another optimization.
  count(index-read) > n being converted to count(choosen(index-read,n+1)) > n
  (This is temporarily retained, but will be removed in a subsequent commit).

IF(count(x) > 0, x, y) optimization
  Currently the intention is to perform a count-index on x before reading x.
  However if x contains a limit,skip it is going to need to read the entire
  result, so don't perform the count separately.

Don't generate count(index) if there is a skip limit

Optimize count(x) > 0 to exists(x)
  Instead of (count(x) > 0) becoming count(choosen(x, 1)) > 0 it is now
  transformed to exists(x) - which is either as efficient, or more
  efficient as the count representation.

Remove no_notexists since never created

Terminate hthor exists aggregate activity early

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
ISSUE:#1714 Restrict special chars from comp names

configgen will indicate and error when encountering
special characters in component names.

configmgr will pop-up an error when users attempt
to enter special characters for component names

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
gh-1997 Fix usability issues for User Permission view

Existing user permission page has two usability issues.
(1) multiple permission tables are displayed in one page
which makes the page difficult to read. (2) for each
permission table (permission group), there is an Add
button which allows to add more permission to that group.
The page shows the Add button and the reource list even
though no more permission can be added for some resources.
This fix only shows the first permission table and the
expend links for other tables. This fix also filters out
the resources from the resource list if no more permission
can be added. If the resource list is empty, the Add
button will not be displayed for that permission table.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
ISSUE:1190 envgen error checking for integer params

Add integer validation for command line
parameters.

Add meaningful error messages when integer
validation fails.

Signed-off-by: Gleb Aronsky <gleb.aronsky@lexisnexis.com>
cannotRemove refactoring

Renamed cannotModify to canModify, same with remove,
and moving the logic inside the file, since it didn't
belong to directories anyway. Instead of using flags,
now using what the methods mean and one call the other
when need.

There is a kludge in Dir::cannotRemove() that will be
cleaned in a subsequent patch, as soon as doRemoveEntry
gets a file as argument, not a filename.

If I manage to finish this second step before this pull
request is pulled, I'll merge them two.
Add superfile name to error
This commit only reformats the code.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
gh-1776 EclWatch pauses/clears queues for Roxie/HThor

A drop down menu is added for Roxie/HThor cluster to
pauses/clears queues. I also improved the index.xslt:
better confirmation messages; removed old code lines
which are no longer used; changed some function names
and variable names for better read.

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
Revised based on review

Signed-off-by: Kevin Wang <kevin.wang@lexisnexis.com>
LDAP check for non-expiring passwords

Even when the domain policy specifies that account passwords expire after
a certain time period, individual accounts can still be specified as never
expiring (admin accounts, for instance).  Add an addition test for this
case when a user logs in.

Signed-off-by: William Whitehead <william.whitehead@lexisnexis.com>
Ensure subgraphs of a global loop are global

If the loop is global and executing subgraphs synchronously, then ensure all
subgraphs of the loop are marked global. Otherwise, the locally executed
subgraphs, will call back to the master and can cause race conditions, when
creating the graph on the master.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Guard against potential race condition

For child graphs, that are initiating creation on master, there's a possible
race condition if multiple slaves initialize the graph and/or request graph
initialization data at the same time

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Ensure jobtemp spill files are replicated

There can be a large period of time between thor executing different
graphs in a workunit, e.g., if other high priority jobs take precedence.
This means it is more likely that a machine may have been swapped in
the meantime.  Therefore any file which has lifetime beyond a single
graph should be replicated - otherwise the entire job may need to be
rerun.

Fixes #1944.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Common up null datasets with minor differences

A minor improvement to the generated code for a few queries.
The type of a dataset may contain an annotation indicating the name of
the original record.  If one null dataset expression contains an
annotation and another doesn't they should be treated as identical.

(Had a minor effect on a couple of queries).

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2107 from ghalliday/nullconstant

Common up null datasets with minor differences

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2118 from ghalliday/issue1944

Ensure jobtemp spill files are replicated

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Fix limitations with meta information for FAIL

IF(cond, dataset, FAIL) should inherit the meta information from the
left dataset.  This fix adds an "ANY" sort order/distribution which
allows the meta information to be calculated correctly.

It is possible this could also be used for some other node types (e.g.,
anything that has a single row).

Fixes Bugzilla bug #100253

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Stop LIMIT SKIP|ONFAIL attributes being ignored

On very rare occasions the SKIP/ONFAIL attribute of a limit could
be ignored.  It occurred when a limit was incorrectly evaluated
using inline code, rather than executed in a child query.  This fix
ensures it is executed in the correct place.

Example was SET(LIMIT(ds, ONFAIL), field).

There were no other examples in any archived queries.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2108 from ghalliday/failmeta

Fix limitations with meta information for FAIL

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Add ConfigMgr support for ws_ecl VIPS

Added ability to configure VIPS in ws_ecl esp service via ConfigMgr.
Updated XSD files to read in the new fields. Updated the XSL files to generate the VIPS into the esp configuration.
Also, updated the handling of the xpathType attribute to allow resolution against multiple values.
This was needed to handle the requirement that each Roxie only appear in atmost one VIP.

Fixes gh-2077

Signed-off-by: Sridhar Meda <sridhar.meda@lexisnexis.com>