reworked tokenize to make it more generic