clean and tokenize functions refactored