Adds unit tests for the tokenize function but the unit tests aren't set up correctly so can't actually be run.