First commit
Split SimpleTokenizer into SimpleTokenizer and GapTokenizer.
Added SimpleSentenceSplitter. Removed tokenSplitter from TokenVector.
Reimplemented tokenizeVector using list slicing list_a[1:2] = new_list
Temp for development of tests
another temp commit
mabe fixed the bugs, but still has encoding problems under windows.
Fixed unicode problem with object_hook in JSON. TODO: lots of NoneType errors still present.
Fixed the NoneType errors. Added sample records file for testing.
Sentences not working properly. Added test case for them -- TDD
temp commit
Implemented sentence indices in Token objects and getSentences method in TokenVector to construct vectors for each sentence in the vector.
Added testing for sentences and getIndexLength and extend methods for the TokenVector class. The methods facilitate creating dummie vectors for testing, but can be useful for solving other problems as well.
shift to office
Fixed bug in gap tokenizer. Needed to operate with REs, not just string matching.
development config
Started index tokenizer
fixed testing of faulty token repairs
Added export to file and rasp_file
Before pythonisation