Add an Executor to run flows without a Job

This is based on #914 The goal here is to make it easy to take a function that produces a TypedPipe, and to run that.

Next, we should have a methods that write `TypedPipe[T]` or `ValuePipe[T]` to the temp directory, then reads that in the Future to get a `Iterable[T]` or `T` respectively.

With the above, you can easily write jobs that loop that does some ML algorithm, for instance.
But that's also a lie, since we don't really know or ever care what `C` is.
In fact, `C` seems like an error in cascading's design, since he
backpedaled and uses `Map<Object, Object>` in most places now for the
config.


On Thu, Jun 26, 2014 at 11:20 AM, Jonathan Coveney <notifications@github.com
> wrote:

> In scalding-core/src/main/scala/com/twitter/scalding/Execution.scala:
>
> > +   *
> > +   * Note that the only config considered is in conf.
> > +   * The caller is responsible for setting up the Config
> > +   * completely
> > +   */
> > +  def buildFlow[T](mode: Mode, conf: Config)(op: Reader[ExecutionContext, T]): (T, Try[Flow[_]]) = {
> > +    val newFlowDef = new FlowDef
> > +    conf.getCascadingAppName.foreach(newFlowDef.setName)
> > +    // Set up the uniqueID, which is used to access to counters
> > +    val uniqueId = UniqueID(UUID.randomUUID.toString)
> > +    val finalConf = conf.setUniqueId(uniqueId)
> > +    val ec = ExecutionContext.newContext(newFlowDef, mode, uniqueId)
> > +    try {
> > +      val resultT = op(ec)
> > +
> > +      // The newFlowDef is ready now, and mutates newFlowDef as a side effect. :(
>
> I think if you do a type parameter C and pass it to the return type, it
> will just get filled in with Nothing and then erased. This should work
> fine, no?
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/twitter/scalding/pull/915/files#r14267542>.
>



-- 
Oscar Boykin :: @posco :: http://twitter.com/posco