Merge latest changes from candidate-3.6.2 into candidate-3.6.x

Ensure changes that were directly taken to 3.6.2 are in subsequent 3.6.x
versions too.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Fix #2090 Rename ECL macro args and enhance descriptions

Renamed the pipe macro arguments and comments to better reflect their meaning

Signed-off-by: russ <russ@russ-VirtualBox.(none)>
Roxie generated filenames for index parts were incorrect

Roxie generating file names for index parts other than the top-level
key incorrectly. This would lead to some confusion, and possibly additional
copying of files, but no other ill effect as far as I can see.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Roxie file cache was not checking timestamps

The code for checking and comparing filestamps when deciding if Roxie's
cached copy of a file was up to date seems to have been lost in the
big refactoring of package files, meaning that file mismatches would only
be detected when the size changed.

This had a few unfortunate effects:

1. Index changes are likely to be only detected by slaves but not by server
   nodes. This leads to 'partially suspended' queries which were are not
   giving helpful error messages.
2. If data is refreshed on thor, Roxie might continue to read a previously
   cached version of the data, even if Roxie was restarted.

See gh-2096. This is not a complete fix for all the issues there, since Roxie
is still falsely reporting that two versions of a data file are required
at the same time when all queries using the previous version have been
unloaded.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Roxie assert error on partially suspended queries

If a Roxie slave thinks that a query is suspended, but the server does not
(this can happen when a file part is missing, for example), then the error
reported if the query is executed wille be an internal assert failure
rather than something more helpful.

See issue gh-2096.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Roxie caching local files too long

Roxie was nto clearing the local file cache when the last query
that referenced a file was released. This means that it was not
possible to load a new query that referenced an updated version
of a file without restarting roxie

Fixes gh-2096.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2099 from richardkchapman/roxie-filenames

Roxie generated filenames for index parts were incorrect

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #2100 from richardkchapman/roxie-filedates

Roxie filedates

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #2101 from richardkchapman/roxie-partially-suspended

Roxie assert error on partially suspended queries

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Roxie filecache - prevent crash if cache destroyed before files.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2103 from richardkchapman/roxie-filecache-release

Roxie filecache release

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #2098 from RussWhitehead/h2h

Fix #2090 Rename ECL macro args and enhance descriptions

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
First draft of HDFS_to_HPCC_Connector_User_s_Guide

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
edits to HDFS_to_HPCC_Connector_User_s_Guide to fix bluebold

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
edits to HDFS docs. formatting plus PipeOut missing 7th arg)

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
edits to HDFS_to_HPCC_Connector_User_s_Guide minor

Signed-off-by: JamesDeFabia <James.Defabia@LexisNexis.com>
Merge pull request #2110 from richardkchapman/HDFS_Connector

Hdfs connector

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Community Edition release 3.6.2-6rc Release Candidate 6

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>