initial commit
separated token verification to own class
Added FunctionToken, made tokenizer_test.cpp in to string search app for demonstration
Moved header files to root of the repo
Merge branch 'master' of https://github.com/jvstech/tokenizer.git