BVLC/caffe — pull request #1175 by jyegerlehner (2014-09-28 21:42:53+00:00)
Title: Extract features in smaller leveldb batches so we don't crash (as often)

I found for what seems like a "normal" scenario, extract_feature was being killed by the kernel presumably due to out-of-memory, after a while of the machine becoming very slow with lots of VM swapping going on. The leveldb::WriteBatch instance is storing up 1000 images' worth of features in memory before committing to the file system (prior to this PR). Which can be a lot of memory. Changing the DB-write-batch-size to 100 allowed extract_feature to complete without any errors, with top showing extract_feature topping out at about 45% memory. This PR changes the DB write batch size to 100.

A bit of arithmetic on the scenario I had:
A machine with 8 GB of RAM (admittedly light). I was writing 4 feature blobs each roughly the same size. One of them was 1024 channels x 12 height x 12 width x 4 bytes/float = 655032 feature bytes per image. With the batch size at 1000, it stores up 1000 of those in the leveldb::WriteBatch, so about 0.65 GB. There were 4 feature blobs (num_features) being exported. And I think leveldb may double buffer what you store, and in any case with other memory being consumed it was over the limit.

Admittedly this isn't a very thorough solution to the problem. I'm merely pushing the tripwire a ways (10x) further out. I don't see any downside to making the change other than since we have more disk operations maybe slightly less efficient. But that doesn't seem significant compared to crashing.