BVLC/caffe — pull request #294 by longjon (2014-04-05 04:12:47+00:00)
Title: Add a layer for in-memory datasets, and expose it to Python

(This PR is related to but distinct from #196. Together with #286 it addresses #135.)

This PR adds a new layer called `MemoryDataLayer` that accepts two contiguous blocks of memory (for data and labels) and (in `Forward`) updates the top blobs to walk along the provided memory.

* Raw pointers are used by `MemoryDataLayer`; should `shared_ptr`s be used instead?
* `Blob` and `SyncedMemory` both get a `set_cpu_data` method to allow them to point to memory owned by someone else
* As a result, no copying is performed
* As a result, input data length must be a multiple of batch size
* A special method, `Net.set_input_arrays` is added to pycaffe for telling the `MemoryDataLayer` to point to ndarray data
* Labels are passed into this method as a 4d array for uniformity (but could be a 1d array for convenience?)
* Tests do not yet exist for `MemoryDataLayer` (perhaps merging should wait until they do?)

When combined with #286, training in Python with ndarray data is now possible, e.g.,
```python
solver = caffe.SGDSolver('solver.prototxt')
solver.net.set_input_arrays(data, labels)
solver.solve()
```