infochimps/swineherd — Issue #3: Swineherd Workflows are too verbose
Author: kornypoet
Created: 2012-02-20 22:26:20+00:00
State: open

Since the main goal of a ```Swineherd::Workflow``` is to chain together the inputs and outputs of successive scripts, so a user does not have to worry about any of the intermediary naming or defining issues. This idea could also be applied to the script definition. The beginning of every ```Swinherd::Workflow``` contains a list of script definitions that usually go like this:

```ruby
flow = Workflow.new(id) do
graph_assembler = PigScript.new(File.join(templates, 'assemble_multigraph.pig.erb'))
degrees_calculator = PigScript.new(File.join(templates, 'degree_distribution.pig.erb'))
pagerank_initializer = PigScript.new(File.join(templates, 'pagerank_initialize.pig.erb'))
pagerank_iterator = PigScript.new(File.join(templates, 'pagerank_iterate.pig.erb'))
join_and_scale = PigScript.new(File.join(templates, 'join_and_scale.pig.erb'))
...
```

And then the definition of these tasks:

```ruby
task :graph_assembler do
graph_assembler.env['PIG_OPTS'] = options[:pig][:options]
graph_assembler.attributes = {
:reduce_tasks => options[:hadoop][:reduce_tasks],
:out => next_output(:assemble_multigraph)
}
graph_assembler.run
end
task :degrees_calculator => :graph_assembler do
...
end
task :pagerank_initializer => :degrees_calculator do
...
end
task :pagerank_iterator => :pagerank_initializer do
...
end
...
```
This appears to be a lot of repeated naming and defining. Since the workflow's only purpose it chain together the execution of scripts, several assumptions can be made:

* The name of the script, its reference variable, and physical name on disk should all be almost identical.
* The various scripts can all be assumed to be template files with no breakage.
* Several things could be condensed and put together behind the scenes of the task method to allow for less verbose workflows.

Here is an example of how a workflow could become more DRY:

```ruby
flow = Workflow.new(:id => id, :template_dir => "/some/place") do

pig_task :graph_assembler do |assembler|
# Assumed there was a template script located at "/some/place/graph_assembler.pig.erb"
# Called PigScript.new("/some/place/graph_assembler.pig.erb")
assembler.attributes = {}
# Instead of access to a graph_assembler variable, the task yields the script object it created behind the scenes
assembler.run!
end

wukong_task :next_step => :graph_assembler do |next_step|
# Assumed there was a template script located as "/some/place/next_step.rb.erb"
...
end
...
```

This shortens the defining steps of a ```Swineherd::Workflow```, and makes each variable reference easier to connect with its scope.