CeON/CoAnSys — pull request #259 by pdendek (2014-05-08 11:10:52+00:00)
Title: Major changes in the Document Similarity Module. Minor issues in other modules

The more flexible structure of "document-similarity-ranked-workflow"
* allows to choose which parts of workflow should be run
* additionally incorporates
** [pig-script] the term selection procedure by tfidf min val
** [pig-script] the procedure for accepting all terms
* the new terms transformation routine
* accepting terms written in latin and non-latin alphabet languages (regexp: string.replaceAll("([^\\u0080-\\uFFFF a-zA-Z_\\-\\d\\s'])+", SPACE) )
** latin/all-langs approach can be chosen
* removing stop-words from the following languages: de,dk,en,es,fi,fr,hu,it,nl,no,pl,pt,ru,se,tr
* serialization to protocol buffers is now optional
* rejected terms are stored (may be used to manually create "collection stop-words")
* use of globals and default parameters in the workflow
* removal of side products
* more comments

The new workflow "document-similarity-recalculate-sim-about-1"
* recalculates similarity of a given pairs of documents (and not the whole collection) on the less strict conditions for a term selection procedure

The new workflow "document-similarity-oap-uberworkflow"
* master workflow connecting "document-similarity-ranked-workflow" and "document-similarity-recalculate-sim-about-1"
* workflow allowing calculation of document-similarity followed by postprocessing and serialization
* bugfixes and small amendments
* removal of side products

Analysis of Document Similarity Module results:
* analysis of selecting threshold for terms on the number of groups of document pairs with the same similarity level
* analysis of selecting threshold for terms on the number of documents represented with the vector of a given length
* other analyses

Python/Jython code for the analysis of UNIBI data

R code for AND