Thanks, I wasn't aware of that. I'll get to work.


On Tue, Jan 7, 2014 at 8:31 PM, Matt Hayes <notifications@github.com> wrote:

> In src/java/datafu/pig/text/opennlp/TokenizeME.java:
>
> > + * define TokenizeME datafu.pig.text.TokenizeME();
> > + *
> > + * -- input:
> > + * -- ("I believe the Masons have infiltrated the Apache PMC.")
> > + * infoo = LOAD 'input' AS (text:chararray);
> > +
> > + * -- output:
> > + * -- ({(I),(believe),(the),(Masons),(have),(infiltrated),(the),(Apache),(PMC),(.)})
> > + * outfoo = FOREACH input GENERATE TokenizeME(text) as tokens;
> > + * }
> > + * </pre>
> > + */
> > +
> > +
> > +
> > +public class TokenizeME extends EvalFunc<DataBag>
>
> Without outputSchema you have to add a cast in order to use the result of
> the UDF, as otherwise the type of the tuple inside the bag is not defined.
>
> For example, I'm testing this on your tokenizeWhitespaceTest test. If you
> describe data2 you get:
>
> data2: {{}}
>
> Now try to do something basic with the data in the bag:
>
> data2 = FOREACH data2 {
>   filtered = FILTER tokens BY $0 == 'is';
>   GENERATE filtered;
> }
>
> You get an error
>
> Projected field [tokens] does not exist in schema: :bag{}.
>
> The workaround is to cast it like this:
>
> data2 = FOREACH data GENERATE TokenizeWhitespace(text) AS tokens:bag{(token:chararray)};
>
> This makes my example work. It also allows the more readable:
>
> data2 = FOREACH data2 {
>   filtered = FILTER tokens BY token == 'is';
>   GENERATE filtered;
> }
>
> But it's a pain to cast like this. Better to define the schema in the UDF.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/linkedin/datafu/pull/84/files#r8715624>
> .
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com