FEATURE - support for reading directly from zip and other archives

Similar to the way that we support reading from git repositories, and
using the same hooks and filename format.

Uses libarchive.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
gh-2166 Add the ability to properly activate / deactive package sets

Signed-off-by: Stuart Ort <stuart.ort@lexisnexis.com>
Fix internal error on complex nested field selection

In some circumstances it was possible to get the following error:
Attempting to lookup field <name> in a dataset which has no active element.
This is preventing another option being enabled that would solve some
problems with conditional evaluation of expressions.

The root cause is the handling of no_select.  A no_select should have a
newAtom attribute associated with it if it is selecting a field from a
new row.  So a[1].b.c is represented as
  select(select(a[1],b,new),c)

Some places were incorrectly checking if the current select had an
associated new attribute instead of checking if the root select had
one.

no_select(no_selectnth) now always has a newAttribute associated with it.
This simplifies and optimizes the processing of unnormalized trees.

This also contains a fix for hqliproj projecting a datarow, and better
expansion of (a.b.c.d within transforms)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
FEATURE - support for reading directly from zip and other archives

Refactored zip and git file hook functionality to use dyaically loaded plugin
library structure.
Default USE_LIBARCHIVE of fon Windows, to avoid breaking everyone's builds
Update lastPos properly for more accurate detection of unsupported seek
requests.
Removed some unnecessary tracing.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2179 from ghalliday/newselector

Fix internal error on complex nested field selection

Reviewed-By: Renato Golin <rengolin@hpccsystems.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
gh-2182 Pkg file delete should not require a local copy of the pkg file

Signed-off-by: Stuart Ort <stuart.ort@lexisnexis.com>
Protect against potential act. init race

Possibly only an issue for global loop initialization, where slaves
subgraphs can cause requests to master as master graph is initializing

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2183 from stuartort/pkg_delete_nofileneeded

gh-2182 Pkg file delete should not require a local copy of the pkg file

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
SelfJoin object leak, cause error if spilled

SelfJoin was leaking the lhs input stream, as a result, if a local selfjoin
had spilt whilst sorting, the spill files remained opened and an error
ensued.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Restructure child dataset flags so they are more useful

This change only modifies the values for the different flags.
It means that the values can be masked to determine subsets of the
information (e.g., does it refer to LEFT/RIGHT), does it have a single
input dataset.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2185 from jakesmith/selfjoin-leak

SelfJoin object leak, cause error if spilled

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2159 from richardkchapman/zipinclude

FEATURE - support for reading directly from zip and other archives

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
gh-2187 hpcc-init fails to stop thor on Ubuntu 12.04

There is a regression in the killall program in the current Ubuntu 12.04
distro's repositories, which prevents killall from working as advertised
on longer executable names.

See https://bugs.launchpad.net/ubuntu/+source/psmisc/+bug/970638

Testing suggests that providing a full pathname that the executable was launched
with, as opposed to just the tail name, will work around the issue.

Without this workaround, thor cannot be properly stopped on Ubuntu 12.04 systems.

Fixes gh-2187.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2180 from ghalliday/childflags

Restructure child dataset flags so they are more useful

Reviewed-By: Renato Golin <rengolin@hpccsystems.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Report "ESP Password Expired" to user

If a user's password is expired, report that error to the user instead of
misleading "Invalid Credentials" error.

Signed-off-by: William Whitehead <william.whitehead@lexisnexis.com>
Merge pull request #2173 from stuartort/pkg_activate

Package activate

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2189 from richardkchapman/ldapPWExpired

Report password expired error to ESP user

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2188 from richardkchapman/precise-workaround-2187

gh-2187 hpcc-init fails to stop thor on Ubuntu 12.04

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2191 from richardkchapman/precise-workaround-2187

gh-2187 hpcc-init fails to stop thor on Ubuntu 12.04

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Fix regression if memory leak option is enabled

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>