CeON/CoAnSys — pull request #223 by pdendek (2014-01-19 18:37:53+00:00)
Title: Document Similarity - Major Changes in Logic

Backward compatibility
* all workflows using document-similarity-tiny-s{1,2,3}.pig will still work without problems

New Features (in Java code and Pig scripts):
1. Removing meaningless terms
1.1. [with ranked-word-count] the top percentage of wc-groups (the top n% of words used, e.g. computations), which act as stopwords can be removed.
1.2. [with ranked-word-count] words used in n documents (e.g. one or two) can be removed. this reduces substantially number of terms, which probably would not match with other documents
1.3. [with min-tfidf] removing terms with tfidf weight lower then a given threshold
2. Similarity calculations
2.1. [Inverted cosine] calculation of similarity between documents which have at least one word in common. The similarity value is the sum of common words tfidfs (where tfidf from both documents are multiplied)
2.2. [Inverted avg] calculation of similarity between documents which have at least one word in common. The similarity value is the average of terms' tfidfs from each documents
3. Normalization
3.1. [Cosine] nominator obtained from [inverted cosine] is divided by (a) left document tfidfs to the power of 2, summed and square rooted (b) multiplied by same value for right document
3.2. [Avg]
3.2.1. [Max local] similarity between 2 documents is divided by max similarity between docA and docB
3.2.2. [Max global] similarity is divided between max similarity level in the collection
3.3. [Unnormalized] similarities are not further normalized
4. Scripts for word-count ranking analysis -- words frequencies, term number, group of frequencies
5. Auxiliary
5.1. Simple full pairwise document similarity
5.2. Removing documents lacking doi, year or title (using Elephant Bird)
5.3. Dump for postgres
5.4. .gitignore