StephanEwen/stratosphere — pull request #11 by tillrohrmann (2014-03-27 12:20:50+00:00)
Title: Netstack rework

I finished the first cleanup of the ExecutionGraph construction. The RuntimeEnvironment is now completely removed from the ExecutionGroupVertex. The RuntimeEnvironment and thus also the invokable will first be instantiated on the TaskManager. To still compute the input splits and the input split type in the JobManager, the JobInputVertex has now access to the InputFormat. In fact, all JobInputVertices require to have an InputFormat object. Additionally, the JobOutputVertex can have an OutputFormat. If an OutputFormat is set, then the method initialize with the task configuration is called in the JobManager. This hook can be used to set up the distributed filesystem in the case of an FileOutputFormat. The Input and OutputFormat objects are serialized by using the StubWrapper of the TaskConfiguration. To make a long story short, instead of instantiating for every vertex a RuntimeEnvironment and the invokable object only the InputFormat and OutputFormat are instantiated.

These changes imply that some of the sanity checks cannot be executed anymore. For example, the number of generated output and input gates cannot be compared to the number of outgoing and ingoing edges of a vertex, as it was done before. But I assume that this is not a big drawback compared to the stricter separation of task execution and execution graph generation.

Fixed problem of job manager failing logic. The job manager can now also fail if its current state is created or scheduled. This problem arises if the first task, which is supposed to turn to running, fails due to an exception. Then the job's status, which is still scheduled, should turn to failing. Before, this problem didn't occur, because all invokables were instantiated before the deployment on the JobManager and thus all possible exceptions were caught there.

Furthermore, I replaced legacy code such as FileLineReader/Writer and JobFileInput/Output by the current equivalents DataSourceTask/DataSinkTask and JobInputVertex/OutputVertex. The legacy code appeared only in old testing code.