BVLC/caffe — pull request #1060 by longjon (2014-09-10 03:49:15+00:00)
Title: GPU memory debugging option

Often a net will fail to run due to a CUDA out-of-memory error. In such cases, Caffe currently provides no easy way to see the breakdown of allocated memory, which may include data blobs, gradient blobs, parameter blobs, and buffers.

This PR adds an option to the `caffe` tool, `-debug_gpu_alloc`, which causes allocations of GPU memory from `SyncedMemory` (which is all of them) to print their size and the name of the function which caused the allocation. (It would be nice to have information about exactly what kind of blob is being allocated and what its name is, but such information is not readily available (although it's accessible in theory by finding the calling `Net` in the backtrace). This PR takes a simpler approach which still provides more information than just the size of the allocation.)

The output looks like this:
```
$ caffe train -solver examples/mnist/lenet_solver.prototxt -debug_gpu_alloc
[...]
I0909 20:44:09.574717 4663 syncedmem.cpp:22] debug: gpu alloc 313600 bytes
I0909 20:44:09.575876 4663 syncedmem.cpp:55] debug: from caffe::BasePrefetchingDataLayer::Forward_gpu
I0909 20:44:09.671166 4663 syncedmem.cpp:22] debug: gpu alloc 400 bytes
I0909 20:44:09.672235 4663 syncedmem.cpp:55] debug: from caffe::BasePrefetchingDataLayer::Forward_gpu
I0909 20:44:09.672392 4663 syncedmem.cpp:22] debug: gpu alloc 4608000 bytes (4.39453 MB)
I0909 20:44:09.673388 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.673722 4663 syncedmem.cpp:22] debug: gpu alloc 57600 bytes
I0909 20:44:09.674720 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.675045 4663 syncedmem.cpp:22] debug: gpu alloc 2000 bytes
I0909 20:44:09.676031 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.676441 4663 syncedmem.cpp:22] debug: gpu alloc 2304 bytes
I0909 20:44:09.677489 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.677553 4663 syncedmem.cpp:22] debug: gpu alloc 80 bytes
I0909 20:44:09.678539 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.680603 4663 syncedmem.cpp:22] debug: gpu alloc 1152000 bytes (1.09863 MB)
I0909 20:44:09.681604 4663 syncedmem.cpp:55] debug: from caffe::PoolingLayer::Forward_gpu
I0909 20:44:09.681941 4663 syncedmem.cpp:22] debug: gpu alloc 1152000 bytes (1.09863 MB)
I0909 20:44:09.682940 4663 syncedmem.cpp:55] debug: from caffe::PoolingLayer::Forward_gpu
I0909 20:44:09.683285 4663 syncedmem.cpp:22] debug: gpu alloc 1280000 bytes (1.2207 MB)
I0909 20:44:09.684269 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.684597 4663 syncedmem.cpp:22] debug: gpu alloc 128000 bytes
I0909 20:44:09.685577 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.685900 4663 syncedmem.cpp:22] debug: gpu alloc 100000 bytes
I0909 20:44:09.686889 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.687021 4663 syncedmem.cpp:22] debug: gpu alloc 256 bytes
I0909 20:44:09.687999 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
I0909 20:44:09.688057 4663 syncedmem.cpp:22] debug: gpu alloc 200 bytes
I0909 20:44:09.689028 4663 syncedmem.cpp:55] debug: from caffe::ConvolutionLayer::Forward_gpu
[...]
```

Consider:
* What do you think the output format and parameter name should be?
* Should the debugging code live in `syncedmem.cpp` or somewhere else?
* According to the Internet, this should work on OS X, but someone should check that.