tricycle/lesswrong — pull request #57 by johnsoft (2012-12-11 01:58:22+00:00)
Title: Migration update

This is an update to the migration script. It no longer uses generator methods, which I'm hoping will eliminate the ballooning memory usage. Database rows are fetched/committed in batches, instead of one at a time, which will hopefully speed up the execution. It can also be killed while running, and it will resume where it left off, using a SQLite database stored in the current directory.

I don't have a giant data set to test on, so most of these improvements are just educated guesses of the bottlenecks on my part. If there is still a memory leak somewhere, now that the script is resumable, it would probably be easiest to just write a bash script that kills and restarts the process ad nauseum until it finishes.