utcompling/Scalabha — pull request #2 by eponvert (2012-03-12 06:18:09+00:00)
Title: refactored and commented CondFreqCounterTests

Hey Dan, take a look at this pull request.

I hit some stormy weather trying to move the new naive Bayes classifier to use your frequency distribution stuff, and also found that opennlp.scalabha.tag.support.CondFreqCounterTests were also failing. Like, do

> sbt test-only opennlp.scalabha.tag.support.CondFreqCounterTests

So, I figured out, more-or-less, what's going on with the CondFreqCounter classes, and updated the tests according. The tests pass now, but there's still a couple points of confusion for me:

1. I'm not sure that in-vocabulary items should be ignored if not witnessed with an antecedent (i.e. what is p(a|A) if (a,A) is not witnessed) -- see my comments and the frequency table in the comments on like lines 102ff
2. I'm not sure how to interpret smoothed probabilities when then antecedent is never witnessed (i.e. what is p(a|A) if A itself is never witnessed)

I'm not saying the implementation is wrong, but I'd like to double check.