Fix gh-2340 - spot invalid files-in-progress file

Detect and rename fip file and refresh

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Split off 3.8.0

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Extend implicit row projection to nested rows

The previous version of the implicit project code only analysed the top
level of fields within a dataset.  If a dataset contained a nested
row field then it was either all included, or all removed.  This
commit extends the functionality so it also removes fields from nested
records if they are unsued.

The main differences are
* Ensuring each set of fields knows the original record structure
* Keeping track of nested fields
* Replacing selectors now also ensures that fields of type datarow are
  mapped if they have been changed.

The advantage is often smaller generated code, and reduced data pulled
from datafiles and indexes.  It also improves the accuracy of an
upcoming feature for describing what data is actually used by a query.

The disadvantage is that this generates more unique row formats - which
increases the size of the metadata classes.  An issue is open to work
on reducing the size of those for the common cases.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Minor clean up of the enumerated type

Add an extra case into the activities enum for rollup and iterate
since it is simpler not to combine them and they have very slightly
different semantics.  (No changes in functionality introduced.)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Final fixes and minor optimization

The fields used in the rollup condition weren't being included in the
fields required by the output if they weren't qualified with LEFT or
RIGHT.

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Merge pull request #2361 from richardkchapman/2281-cherry-pick

2281 cherry pick

Reviewed-by: Gavin Halliday <ghalliday@hpccsystems.com>
Fix compile problem with CWsDeployFileInfo

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Housekeeping, remove redundant code/files

Signed-off-by: Jake Smith <jake.smith@lexisnexis.com>
Merge pull request #2364 from jakesmith/wsdeploy-compilefix

Fix compile problem with CWsDeployFileInfo

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2367 from jakesmith/redundant-code-removal

Housekeeping, remove redundant code/files

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge pull request #2355 from jakesmith/gh-2340

Fix gh-2340 - spot invalid files-in-progress file

Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>
Merge remote-tracking branch 'origin/candidate-3.8.x'

Signed-off-by: Richard Chapman <rchapman@hpccsystems.com>
Fix assert when ebcdic string used in filename

A fairly obscure error, but ebcdic strings that were used in filenames
would trigger an assert in debug mode, and not normalize a filename
correctly in release.  (It would have had few ill effects.)

Signed-off-by: Gavin Halliday <gavin.halliday@lexisnexis.com>
Add create superfile inside transactions

Creating superfiles now are completely inside transactions,
with unit tests in daregress. Promote superfile was affected,
so a major refactoring in the internal works of transactions
was required, and a promote superfile unit test was also added.
Optimise NORMALIZE to DATASET

This optimisation regards transforming a number of
normalize(row, count, trans(COUNTER)) constructs into
dataset(count, trans(COUNTER)). Tests added.
Allow dataset to have its arguments optimised
Allow dynamic counts
Adding output to avoid premature optimisation on LEFT case
Add comment on how to debug symbols
Merge pull request #2375 from rengolin/dfs-master

Add create superfile inside transactions

Reviewed-By: Jake Smith <jake.smith@lexisnexis.com>
Reviewed-By: Richard Chapman <rchapman@hpccsystems.com>