stratosphere/stratosphere — pull request #315 by uce (2013-12-03 13:44:53+00:00)
Title: Add deferred updates to workset iterations (issue #21) and fix chaining of iteration state updates (issue #123)

This pull request enables deferred updates for workset iterations (#21) and interation state updates by chained tasks (#123).

- **Deferred updates**: Before, workset iterations updated both the workset and solution set in the tail task. For connected components, this meant that the iteration tail (update components join) created both the workset for the next iteration *and* updated the solution set.

With this pull request, it will be possible to do the following:

- Intermediate workset update
- Intermediate solution set update
- (Separate) solution set tail update
- (Separate) workset tail update
- Unified solution set/workset tail update (as before)

I extended `ConnectedComponentsNepheleITCase` with all possible variants. @vasia has a use case, where she further wants to test this (sorry, for letting you wait this long :bowtie:).

@StephanEwen: does it also make sense to have an intermediate task, which updates both sets and then has a tail later? It didn't think so and intermediate updates are exclusively workset or solution set updates. While posting this, I'm actually not so sure anymore. ;-)

- **Chained iteration state updates**: If an iteration task has chained tasks, the last operator in the chain will do the state update.

`IterationsWithChainingNepheleITCase` is a test case for this.

@StephanEwen: does it also make sense to have an intermediate chained driver to do the update and not just the last?

---

When creating a Nephele JobGraph, you have to explicitly configure the type of update for the tail (before this the tail was always doing a workset update).

The relevant options are:
- setIsWorksetIteration()
- setIsWorksetUpdate()
- setIsSolutionSetUpdate()
- setIsSolutionSetUpdateWithoutReprobe()

Also the head has to wait for the solution set tail, if there is a separate one:
- setWaitForSolutionSetUpdate()

---

At some point, the overengineering bug bit me and I actually spent quite some time not doing anything productive with the code (aka this was basically finished a couple of weeks ago). There are quite some overlappings between the tail and intermediate task, which could be refactored. But the resulting code might be harder to understand. **Feedback wanted as I feel like not seeing the forest for the trees.**