Major refactoring to do with introduction of a Token and DocumentFeature classes. This makes the code much easier to read and enables me to add extra attributes to the tokens (previously a tuple) without breaking code that relies on having a specific piece of information stored in a specific location in the tuple. As a result, I was easily able to implement subj-verb-obj and adj-noun document feature extraction based in dependency parsed input
Also:
- added ability to disable n-gram features completely in ThesaurusVectorizer (set n_gram range to (0,0) )
- modified all composers so that their _get_vector method is consistent (takes an iterable of tokens) and their __contain__ takes a document feature
- added a dummy subject-verb-object Oxford composer
- added a convenience method to DocumentFeature, called tokens_as_str
- Added safety checks to the current dummy implementations of Baroni and Oxford SVO composers to ensure they have all the available info they need at creation time
added debug mode to dump_all_composed_vectors.py