#39: just refactored a little
Merge branch 'master' into shlomo.39.tokenizer_too_greedy
#39: fixed reference removal
it was assuming to match "<" instead of "&lt;" and ">" instead of "&gt;"
#39: fixed greediness of reference removal
#39: fixed greediness of Infobox removal
#39: improved url recognition
much more accurate now, covered by many more unit tests
also all patterns are case invarient now since string gets converted to lowercase before pattern search
#39: improved link handling
first of all it is not greedy anymore. The pattern makes sure that only parts of x in [[x]] can be deleted (only things within the doubl-brakets).
using a regex named group we are able to preserve the shown part of each link, e.g. in [[Brandeis_University|going to Brandeis]] "Brandeis_University" is the link and "going to Brandeis" is the shown part as it appears (underlined) in the article - we keep this shown part and delete the rest.
#39: removed two now obsolete patterns
they also were greedy and probably caused some data loss
#39: introduced nested reference handling
safety in patterns (to avoid greediness) resulted in removing only the inner most match if it is nested inside another potential match. Now this problem is solved by applying the these potentially nested patterns as long as this changes the text.
Closes #39.
Closes #32.
Closes #15.
#39: fixed html removal bug
(was using % instead of & as tag prefix)
#42: introduced optional parameters for filtering
lemmas stored in the index
#42: added double parsing error handling
#42: moved mockito and mrunit libraries to the other libraries
#42: implemented optional caching of PROFESSION_INDEX.
Speeds up the processing a lot.
Closes #42.
#42: fixed issue with optional cacheProbs parameter.
it was not handled as optional in the code logic
#42: removed logging.
Hadoop was throwing org.apache.hadoop.util.Shell$ExitCodeException which indicated permission violations (according to stackoverflow). Apparently the cluster did not like the LOG logic, probably because it was trying to write the log file in some folder where we don't have write permissions.
Commenting out all LOG statemenets and excluding the log4j2-test.xml log configuration file solved this issue.