removed tokenize function, moving to a struct based one
test