This handles the case where scalding will save out a tsv and re-use it down stream leading to issues where the types are not strings

I would have taken your word for it. :)


On Thu, Jun 26, 2014 at 8:58 AM, ianoc <notifications@github.com> wrote:

> In scalding-core/src/main/scala/com/twitter/scalding/RichPipe.scala:
>
> > @@ -549,7 +549,16 @@ class RichPipe(val pipe: Pipe) extends java.io.Serializable with JoinAlgorithms
> >     * Write all the tuples to the given source and return this Pipe
> >     */
> >    def write(outsource: Source)(implicit flowDef: FlowDef, mode: Mode) = {
> > -    outsource.writeFrom(pipe)(flowDef, mode)
> > +    /* This code is to hack around a cascading quirk. in a graph
> > +    A -> FlatMap -> write(tsv) -> FlatMap
> > +    in the second flatmap cascading will read from the written tsv for running it. However TSV's use toString and so is not a bijection.
> > +    here we stick in an identity function before the tsv write to keep to force cascading to do any fork/split beforehand.
> > +    */
> > +    val writePipe: Pipe = outsource match {
> > +      case t: Tsv => new Each(pipe, Fields.ALL, new IdentityFunction, Fields.REPLACE)
>
> https://travis-ci.org/twitter/scalding/builds/28528347 should cover this
> without the fix:
>
> https://github.com/twitter/scalding/compare/testCacheBugTest
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/twitter/scalding/pull/913/files#r14259317>.
>



-- 
Oscar Boykin :: @posco :: http://twitter.com/posco