apache/spark — pull request #2494 by rnowling (2014-09-22 21:04:39+00:00)
Title: [SPARK-3614][MLLIB] Add minimumOccurence filtering to IDF

This PR for (SPARK-3614)[https://issues.apache.org/jira/browse/SPARK-3614] adds functionality for filtering out terms which do not appear in at least a minimum number of terms.

This is implemented using a minimumOccurence parameter (default 0). When terms' document frequencies are less than minimumOccurence, their IDFs are set to 0, just like when the DF is 0. As a result, the TF-IDFs for the terms are found to be 0, as if the terms were not present in the documents.

This PR makes the following changes:
* Add a minimumOccurence parameter to the IDF and DocumentFrequencyAggregator classes.
* Create a parameter-less constructor for IDF with a default minimumOccurence value of 0 to remain backwards-compatibility with the original IDF API.
* Add tests to the Spark IDFSuite and Java JavaTfIdfSuite test suites
* Updated the MLLib Feature Extraction programming guide to describe the new feature