BVLC/caffe — pull request #1362 by jeffdonahue (2014-10-26 08:09:37+00:00)
Title: Prefetching HDF5DataLayer (currently broken)

Posting this hoping someone will have debugging tips or suggestions on how to fix it. I modified the HDF5 reading interface to have two methods: `HDF5PrepareBlob` which pre-shapes a Blob for a particular number of rows (or -1 to indicate all rows in the HDF5 file), and `HDF5ReadRowsToBlob` to read a particular number of rows from an HDF5 file to a Blob, starting at a particular offset of each.

The intention of this interface is to enable reading HDF5 data without ever allocating additional memory -- the main thread calls `HDF5PrepareBlob` in `LayerSetUp` (followed by `mutable_cpu_data`) to pre-allocate the blob data, then the prefetch thread calls `HDF5ReadRowsToBlob` to fill the presized Blob with however many floats it needs.

On my machine, the new interface works perfectly in the first commit where it's all executed in the main thread (passes the unit tests, at least..), but in the second commit, when I use the prefetch thread, there seem to be all sorts of non-deterministic problems, so I must be reading/writing some memory I'm not supposed to be reading/writing in the prefetch thread.

valgrind sometimes points to my use of `c_str()` to access filename in `hdf5_data_layer.cpp:32` -- it often reports an invalid read from `H5Fopen` or the print statement of the filename on line 33.