moved stopword removal and short word removal to tokenizer, added unit tests and explicit option in the config file
added explicit tokenisation options to exp22-25 conf files
more tokenizer unit tests
change min corpus frequency to 10 for experiments 22-25 to filter out infrequent words
fixed bug where the number if nearest neighbours inserted by the bov vectorizer was wrong
updated test_main.py with recently added conf options, enabled one test that was previously disabled (better understand it now) and disabled to others (not sure how they fit in atm)