chrishokamp/pignlproc — pull request #1 by pablomendes (2012-06-25 14:54:40+00:00)
Title: pusing analyzer and tokenstream out of exec()

The analyzer is being created and collected every time your UDF is being executed. I suggest moving it out of the "exec" method to a member variable. You might want to take a "language" argument in the constructor so that you can pass it to SnowballAnalyzer(Version matchVersion, String name, Set stopWords). You will also need the language in order to get the correct stopwords file from the jar. [2]
- The token stream is also being created for each tuple. This means that the garbage collector has to deal with these objects for every line in our many gigabytes of Wikipedia. I suggest using the reusableTokenStream method from the analyzer, and also defining it outside of the exec method