Next I'm going to import lots of stuff from Lucene, for string distance,
n-gram generation, etc. Want to ship this stuff first, though.


On Tue, Jan 7, 2014 at 8:33 PM, Matt Hayes <notifications@github.com> wrote:

> In src/java/datafu/pig/text/opennlp/TokenizeME.java:
>
> > + * define TokenizeME datafu.pig.text.TokenizeME();
> > + *
> > + * -- input:
> > + * -- ("I believe the Masons have infiltrated the Apache PMC.")
> > + * infoo = LOAD 'input' AS (text:chararray);
> > +
> > + * -- output:
> > + * -- ({(I),(believe),(the),(Masons),(have),(infiltrated),(the),(Apache),(PMC),(.)})
> > + * outfoo = FOREACH input GENERATE TokenizeME(text) as tokens;
> > + * }
> > + * </pre>
> > + */
> > +
> > +
> > +
> > +public class TokenizeME extends EvalFunc<DataBag>
>
> Thanks! And thanks again for working on this, it'll be a great addition.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/linkedin/datafu/pull/84/files#r8715646>
> .
>



-- 
Russell Jurney twitter.com/rjurney russell.jurney@gmail.com datasyndrome.com