Quick fix: getGlobalTermWeightFromString was returning 0 much of the time with log entropy weighting (for terms that exist in the corpus). 

I haven't yet got to the bottom of this, but it makes more sense to use getGlobalTermWeight in this context anyhow (as this occurs within a loop across fields).

This fix is likely to improve results when using incremental indexing and weighting together.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@876 483481f0-a63c-0410-a8a8-396719eec1a4
Apologies - broke the stack on my last commit by including a call to an experimental weighted superposition method that does not exist here. This should fix it.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@877 483481f0-a63c-0410-a8a8-396719eec1a4
Improved description of IncrementalDocVectors.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@886 483481f0-a63c-0410-a8a8-396719eec1a4
Improved description of IncrementalTermVectors.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@887 483481f0-a63c-0410-a8a8-396719eec1a4
I suspect the "break" belongs outside the loop here, as this eliminates some odd behavior also (unnecessary builds, peculiar filenames and such).

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@888 483481f0-a63c-0410-a8a8-396719eec1a4
GC appears more inclined to collect last iteration's document vector this way.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@889 483481f0-a63c-0410-a8a8-396719eec1a4
Minor enhancement: calculate global weights of terms to-be-indexed only.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@891 483481f0-a63c-0410-a8a8-396719eec1a4
The "majority rule" is replaced by probabilistic normalization as a normalization method of choice. The performance seems tolerable (mostly as n(superposition) > n(normalization)), though I have yet to do any comparative evaluations of normalization speed. However, this doesn't seem prohibitive, and the advantage of having (x+x+y) = 2x + y rather then (x+x+y) = x seems worth pursuing.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@893 483481f0-a63c-0410-a8a8-396719eec1a4
I think this test change is necessary for probabilistic normalization not to be seen as an error.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@895 483481f0-a63c-0410-a8a8-396719eec1a4
....another possibility for probabilistic normalization would be to render this deterministic such that superposing the same set of vectors will produce the same superposition each time by seeding the normalization pseudorandom number generator. It seems as though this would be desirable in some circumstances, though further experimentation is required to determine the ups and downs of this approach. 

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@906 483481f0-a63c-0410-a8a8-396719eec1a4
Slight modification to previous commit, such that each superposition is seeded with a unique signature derived from the voting record. This way, the same set of vectors should always result in the same superposed product (if weighted the same way), but the same (very biased) set of per-dimension probabilities will not be used in every superposition. 

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@907 483481f0-a63c-0410-a8a8-396719eec1a4
Added local weighting (binary vs. TF vs. log(1+TF)) 

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@908 483481f0-a63c-0410-a8a8-396719eec1a4
Added local weighting to IncrementalDocVectors, and caching of global frequencies to LuceneUtils.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@909 483481f0-a63c-0410-a8a8-396719eec1a4
Faster probabilistic normalization (though still slower than the BSC majority rule)

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@911 483481f0-a63c-0410-a8a8-396719eec1a4
Removed some vestigial code that made it in on the last commit

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@912 483481f0-a63c-0410-a8a8-396719eec1a4
Removed some unnecessary import statements

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@913 483481f0-a63c-0410-a8a8-396719eec1a4
Moved some machinery outside of the inner loop, speeding things up a little further.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@914 483481f0-a63c-0410-a8a8-396719eec1a4
Building a hash table for global term frequencies of all terms, including those that do not pass the frequency threshold test and are therefore never seen again, turns out to be a monumentally bad idea.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@927 483481f0-a63c-0410-a8a8-396719eec1a4
Integrating Lucene43 branch into trunk. We will test this a bit more and then release.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@931 483481f0-a63c-0410-a8a8-396719eec1a4
Adding Dominic's Serendipity affiliation.

git-svn-id: http://semanticvectors.googlecode.com/svn/trunk@935 483481f0-a63c-0410-a8a8-396719eec1a4