bstewart/stm — pull request #3 by gtuckerkellogg (2014-12-19 01:53:20+00:00)
Title: changed textProcessor to keep a record of dropped docs.

I noticed that for some collections of relatively short documents, the textProcessor function may silently drop some documents. These have been documents containing only stop words after stemming, punctuation removal, etc. While the DocumentTerm matrix kept all the rows, when converted to the stm internal format inside textProcessor, these documents were silently dropped, so findThoughts on the original document collection raised an error because the number of documents didn't match.

This now returns a docs.removed element in the list, so I can remove them from the original set and proceed. I'm using the results of textProcessor like this: like this:

# processed contains the results of running textProcessor
if (length(processed$docs.removed)) {
documents <- documents[-processed$docs.removed,]
}