split up tokenize into separate methods