If memory usage is still a concern, you could try using the chunking approach used by bulk email.  Look at subtasks.py for details.