hpcc-systems/HPCC-Platform — pull request #3726 by rengolin (2012-11-23 15:11:11+00:00)
Title: HPCC-8301 Major Refactory in DFS Remove

WARNING: The 5 commits should be squashed before pulling this patch.

Following previous commits and the need to merge removeEntry with removeSubFile and RemoveSuperFile, many changes were needed to the DFS:

1. Physically removing super-files is illegal
2. Centralize removal type decision (physical/logical)
* Avoid actions to call doRemove[Phys/Entry] directly
3. Allow for actions to call other actions
* Make sure auto-commit doesn't persist changes mid-way through

To implement those features, the following changes were performed:

1. CDelayedDelete class, called only from the end of a transaction, is the sole responsible for deciding which type of delete to use on a file, and all removals must invariably pass through it:
* doDelete() checks if it's super-file and choose logical/physical
* everyone else call addDelayedDelete() instead of doRemove*
* there is no need any more for "remphys" parameter anywhere (all removed)
2. Auto-commit and local transaction logic refactored:
* To avoid the "local" parameter and changing the action (commit/autoCommit), which was error prone, a new concept was added: *transaction depth*
* every time one action calls another, it descend its transaction depth, so when the called method calls autoCommit(), it won't do anything
* at the end of every recursive call, ascend() is called, so autoCommit() can work again
* this allows for complex lists of actions (including delayed deletes) to still be taken atomically, even though the transaction is inactive
3. Final methods using doRemovePhysical/Entry were migrated to addDelayedDelete()
* including removeSubFile() and RemoveSuperFile (which calls the former)

Some tests were created (in daregress) and others were fixed (superfile[7,8].ecl), but some changes were needed to make daregress work properly (and enhance the API).

Tests were not properly catching a lock bug that would occur if the transaction was re-used in auto-commit mode (which is expected).

Whenever adding a super-file to the cache (either when the transaction is active or when the user called lookupSuperFileCached), all its subs (recursively) are also cached