BVLC/caffe — pull request #355 by jeffdonahue (2014-04-24 02:52:03+00:00)
Title: Wrap up SyncedMem resize from @kloudkl; make train/test nets share data blobs

This PR wraps up the SyncedMem changes from @kloudkl in #250 (thanks for doing most of the work on this @kloudkl!). I ended up removing the thrust host/device_vector dependency as I couldn't get them working at all (almost all tests segfaulted), and instead kept the `void*` pointers that we were already using to store the data (and still syncing using explicit `cudaMemcpy` calls). I added private fields `cpu_capacity_` and `gpu_capacity_` which keep track of the current space allocation, and methods `cpu_resize()` and `gpu_resize()` allocate additional memory (copying any current memory) if the current allocation is insufficient.

The final commit makes use of this new functionality and my `ShareData` and `ShareDiff` methods from #332 to share all the *data* blobs between the training and test net (#332 only shared weight blobs). Because of this change I've expanded the imagenet_val batchsize from 50 to 250 (as close to the train batch size of 256 as possible while still evenly dividing 50K...) for slightly faster test time with little memory cost.

I ran some tests of the ImageNet model on a Tesla K40 and found little to no difference in training/test time (not counting the test time improvement due to the expanded batch time), while memory use after the first test iteration improves significantly:

```
@ capacity-aware-memory:

before first test iteration (train batch size 256): 3014 MB / 12287 MB
after first test iteration (test batch size 50): 3053 MB / 12287 MB
100 iter train time: ~135 sec
test time: ~102 sec

before first test iteration (train batch size 256): 3014 MB / 12287 MB
after first test iteration (test batch size 250): 3141 MB / 12287 MB
100 iter train time: ~135 sec
test time: ~99 sec

@ dev:

before first test iteration (train batch size 256): 3014 MB / 12287 MB
after first test iteration (test batch size 50): 3255 MB / 12287 MB
100 iter train time: ~135 sec
test time: ~102 sec

before first test iteration (train batch size 256): 3014 MB / 12287 MB
after first test iteration (test batch size 250): 4143 MB / 12287 MB
100 iter train time: ~135 sec
test time: ~99 sec
```