Fully implemented tokenization.