andialbrecht/sqlparse — pull request #63 by bittrance (2012-04-21 23:52:28+00:00)
Title: Support for reading from file-like object

This pull request concerns support for parsing from a file-like object so that the whole parse set need never stay in memory. I plan to feed sqlparse multi-gigabyte MySQL data dumps, so loading it as a string is not an option.

It may be noted that the API:s are not as symmetric as one might desire; my suggestion would be to make both parse and parsestream return generators (perhaps even dropping parsestream altogether and detect input).

In performing tests for this task, I found that the groupings module performs very badly when supplied with mysql multi-row insert statements. Such a statement can have hundreds of thousands of tokens in them. I did some tweaks to the tokens module and shaved 10-15% of the total time, but for this case to be really useful, the groupings module need to be rewritten not to depend on list lookups (.remove() and .index()) so much. I estimate that it could be speeded 50-75% at least. I haven't attacked this, because I think it would entail near to a rewrite of the module.