implement a next token function with a sensible interface

squeezing it into the iterator trait was not working
add test for handling of end-of-file
some constructors for TokenizerError
basic string lexing