seatgeek/soulmate — pull request #27 by Whitespace (2012-06-26 01:59:49+00:00)
Title: Extended bin/loader to be more memory efficient for large datasets

I split the `Soulmate::Loader#load` method into two parts: cleanup and load.
By doing this we can keep calling `Loader#load` without having to blow away the
old data.

I then extended the `bin/soulmate` file to take filename and batch_size
arguments. By not waiting for all the data to arrive on STDIN, we can tune
the batch_size to reach a consistent memory usage, which also helps speed up
loading as well.

I ran some tests using `time soulmate load sample.json` with various batch_size
parameters. I inspected memory using `Activity Monitor.app` and took some
averages over 3 runs each. `sample.json` was a 100,000 line file.

batch_size time memory_used
100,000 1:37 219.8
10,000 1:15 58.2
1,000 1:13 46
100 1:14 46.4

As you can see, memory stays stable when the batch size is smaller, and loading
data is faster, too.

I haven't touched the other commands in `bin/soulmate`, since I don't have a
need to use them yet.

Tests have been modified to reflect the new `Loader#cleanup` method (they were
using `Loader#load` as a hack anyway), and they all pass.

Fixes #26