Token vector functions are now inlined.
Added vector string utility function.
Removed newline.
Added unit test suite for vector_string.
Tokenizer change from single function to type with functions.