```python
line_iterator = iter(lines)
tokens = tokenize.generate_tokens(lambda: next(line_iterator))
```