zopyx/zopyx.plone.migration — pull request #4 by thet (2014-09-30 23:21:07+00:00)
Title: avoid memory leaks by batching

with a 30GB database with lots of big binary data and 4020 content objects to export, i run into an issue where my 4GB development machine run completly out of memory after 1500 objects.
first attempt was to try to fix the memory leak by closing files earlier, flushing and syncing file pointers and dereferencing variables. i had no luck. looks like, python or the operating system (linux, in that case) is not freeing memory, at least with python 2.4.
so i added the ability to batch content exports.
here it is.