Simplified __main__ by inlining a method
WIP: Stats are now held in a pandas DataFrame and periodically flushed to disk to minimise memory usage
Updated unit test for StatsRecorder
Stopped extracting statistics from log files (because it's easier and more robust to get data from the Stats objects maintained internally)
Made thesaurus available at train time too. The only reason to do that is to record how many train-time tokens are in it. Added test resources that got left out at some point.
Holding stats in a list in memory and only converting to a pandas data frame when flushing to disk. Also not doing any analysis, just collection, as the experiment is running. This speeds things up a fair bit.
Adapted tests and resources to account for the fact that a Thesaurus now holds DocumentFeature-s instead of strings.
Enabled shelve writeback to have it cache results and reduce disk reads. This considerably speeds things up.
Removed unnecessary information returned by PrecomputedSimilaritiesVectorSource.get_nearest_neighbours to speed it up by an order of magnitude. This breaks compatibiblity between its API and that of CompositeVectorSource, which is ok as the latter should not be used.