esnme/ultramysql — Issue #6: Lazy result fetching
Author: smurfix
Created: 2012-02-10 21:08:12+00:00
State: open

Right now ResultSet.rows contains the whole result. That's not a good idea, as I really need to process some large queries which doesn't fit in main memory and, even if it would fit, the latency would kill me. :-/

Ideally, ResultSet should be an iterator which fetches datasets as needed. For compatibility, the .rows would still be an array with the results; you can pre-fill it by way of a property.
Of course, that doesn't work as-is with nested queries, so the connection needs to keep a pointer to the last-returned resultset and tell it to fetch+store the rest of its data when the next query is started.

This is fairly straightforward to implement, I did it in ten minutes in the old gevent-mysql code.
See github.com:smurfix/gevent-MySQL.git.

Looking at the ultramysql code, this would mean breaking up Connection::handleResultPacket(). If you create a new connection::readOneRow() instead, and call that from the resultset, that'd allow you to drop the m_capi.resultRowBegin/End() calls, so the whole thing would actually become (slightly) faster.

I hope to be able to get around to implementing that in a few weeks, I have a vacation coming up … if somebody beats me to it, good. ;-)