Add C interface to libtokenizer.
Implement iterator over tokens.
Add tests for tokenizer.