Added tokenizer.h and unit test
Merge pull request #41 from cesanta/tok_test

Added tokenizer.h and unit test