HPCC-11302 - Fix implicit super file creation transaction retry

Hitting a timeout and retrying caused the automatically creted
superfile to be missing to any transaction steps that relied on
it.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11308 - ensure transaction object files are released on retry

Files transaction looked up in prepartion for manipulation inside
transaction steps were held (linked) and therefore retained a
read lock on those files. If the preparation failed to lock the
files they were release, however if they succeeded, but
subsequent steps failed they were retained whilst the transaction
mechanism sat in it's retry sleep phase.
The net effect would be that other jobs/users might be blocked
from gaining exclusive access to files that had been referenced
inside a transaction.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11304 Field translation options not working correctly

There are several mechanisms for controlling whether field translation should
be allowed.

 1. In the workunit, using #option (layoutTranslationEnabled,1)
 2. In the package file, using control:enableFieldTranslation=”1”
 3. In the RoxieTopology.xml file, using fieldTranslationEnabled=”1”

The global setting can be overridden temporarily (until the next time “service
hpcc-init start” is executed) using testsocket

In theory, the workunit setting should (if present) override the package file,
which should in turn (if present) override the global setting. If none are
present, field translation is disabled.

However, the workunit setting was always used, defaulting to false if not
present.

Also make the control: optional in the package file for back-compatibility

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
HPCC-11324 - Allow DFS to delete external file via logical ref.

Permit a couple of couple of places to delete external files

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11143 List Queries associated with a WU

Fixes HPCC-11143

Signed-off-by: Gordon Smith <gordon.smith@lexisnexis.com>
HPCC-11309 - Ensure IDFUHelper cooperate with transactions

Ensure they release their locks before committing transaction.
Also move remove super file operation into transaction.

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
HPCC-11253 Do not lock superfiles in roxie

Don't include spill files or write activities.

Handle opt flag on indexes properly.

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5788 from GordonSmith/HPCC-11143

HPCC-11143 List Queries associated with a WU

Reviewed-By: Kevin Wang <kevin.wang@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5768 from jakesmith/hpcc-11302

HPCC-11302 - Fix implicit super file creation transaction retry

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5771 from jakesmith/hpcc-11309

HPCC-11309 - Ensure IDFUHelper cooperate with transactions

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5769 from jakesmith/hpcc-11308

HPCC-11308 - ensure transaction object files are released on retry

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5786 from jakesmith/hpcc-11324

HPCC-11324 - Allow DFS to delete external file via logical ref.

Reviewed-By: Gavin Halliday <gavin.halliday@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #5790 from richardkchapman/roxie-file-fixes

HPCC-11253 Do not lock superfiles in roxie

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge pull request #5774 from richardkchapman/enable-field-translation

HPCC-11304 Field translation options not working correctly

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Merge branch 'candidate-4.2.6' into candidate-5.0.0

Conflicts:
	roxie/ccd/ccdquery.cpp

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge branch 'master' into candidate-5.0.0

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-5.0.0' into closedown-5.0.x

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>