Good call. Yes, those tokenizers should be included as well.


On Mon, Jan 6, 2014 at 12:42 PM, Matt Hayes <notifications@github.com>wrote:

> In src/java/datafu/pig/text/Tokenize.java:
>
> > + * Example:
> > + * <pre>
> > + * {@code
> > + * define Tokenize datafu.pig.text.Tokenize();
> > + *
> > + * -- input:
> > + * -- ("I believe the Masons have infiltrated the Apache PMC.")
> > + * infoo = LOAD 'input' AS (text:chararray);
> > +
> > + * -- output:
> > + * -- ({(I),(believe),(the),(Masons),(have),(infiltrated),(the),(Apache),(PMC),(.)})
> > + * outfoo = FOREACH input GENERATE Tokenize(text) as tokens;
> > + * }
> > + * </pre>
> > + */
> > +public class Tokenize extends EvalFunc<DataBag>
>
> What about the SimpleTokenizer and WhitespaceTokenizer classes? Should
> UDFs for these be included as well? Maybe this particular class should be
> called TokenizeME.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/linkedin/datafu/pull/84/files#r8676695>
> .
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com