refactor tokenizing code a bit to make implementation of changes that might be needed in the future easier
add another converted test