* Fiddle with token features
* Fiddle with token features
* Fix bug with trailing punct on contractions. Reduced efficiency, and slightly hacky implementation.
* Move special tokenization into its own lookup table, away from the cache.
* Fix performance issues by implementing a better cache. Add own String struct to help
* Efficiency tweaks
* Fiddle with declarations, for small efficiency boost
* More performance fiddling, particularly moving the specials into the cache, so that we can just lookup the cache in _tokenize
* Add initial work on simple hash table
* Replaced cache with own hash table. Similar timing
* PointerHash working, efficiency is good. 6-7 mins
* Switch hash interface, using void* instead of size_t, to avoid casts.
* Improve interface for PointerHash
* Remove short-circuiting of initial_size argument for PointerHash
* Move EnglishTokens stuff to Tokens
* Update tokenization tests for new tokenizer rules
* Fix niggling memory error, which was caused by bug in the way tokens resized their internal vector.
* Upd Tokens to use vector, with bounds checking.
* Switch to using a heap-allocated vector in tokens
* Pass only the tokens vector to _tokenize, instead of the whole python object.