BVLC/caffe — pull request #1522 by netheril96 (2014-12-03 12:09:24+00:00)
Title: Abstraction and unification of data input

This is a generalization of `IndirectionLayer` in #1414 as discussed with @sguada

## Motivation

There are two problems with the current way of data input into the network:

1. The rigidity imposed by `Datum`. It unnecessarily couples "data" (which are usually images) with a single integer "label". The inflexibility of `Datum` makes it difficult and unnatural to model multi-label classification, attribute classification, face verification and many more cases where "data" and "label" does not have one-to-one correspondence.
2. The code duplication and multiple responsibilities of all kinds of `DataLayer`s. There are `DataLayer`, `ImageDataLayer`, `HDF5DataLayer`, `DummyDataLayer` and many other ones. Each data layer has to handle both the task of reading the data as well as propagating them in the network. They also repeatedly handle the error prone task of multithreaded prefetching despite sharing a common base class.

## Core idea

The first idea is unify data and label under `Blob`. All "data" and "label" are just regular blobs, where the latter is just a special case (1x1x1 integer). So the new layers will just output blob. Whether they should be interpreted as data or label or something else is not these layers' concern.

The second idea is to separate the data access from the act of propagating them.

The data access is abstracted by `DataSource` class. The data source is modeled as a mapping between integer and floating point array (`int32 -> float[]`). The implementation may be

* backed by files, like CSV, JSON, LMDB, HDF5, images
* dynamically generated, such as a counter, a constant stream of zero (analogous to `/dev/zero`) and random numbers (analogous to `/dev/random`)
* modulation of other data sources, such as caching and transformation.

The propagation is handled by two layers, `DataSequenceLayer` and `DataMappingLayer` (the naming is not fixed yet).

* `DataSequenceLayer` outputs data in sequence. It reads from the lowest index to the highest and then cycles back. Because it knows in advance what data to retrieve, it may prefetch in a different thread. The threading code only needs to be implemented once for all sources.
* `DataMappingLayer` outputs data with the indices specified by the bottom blob. This allows filtering, reordering, shuffling and slicing of the data source.

## Example definition

Here is a definition file for `DataMappingLayer`

```protobuf
layer {
type: DATA_MAPPING
bottom: "index"
top: "data"
name: "mapping"
data_mapping_param {
channels: 3
height: 12
width: 12
data_source_param {
type: CSV
filename: "mapping.csv"
}
}
}
```

`DataSequenceLayer` can be similarly declared, except that it does not have bottom input but a `batch_size` parameter.

## Example network

### Original case (image + label)

![](https://netheril96.github.io/images/abstraction_and_unification_of_data_input/img_label.svg)

### Multiple label

![](https://netheril96.github.io/images/abstraction_and_unification_of_data_input/img_label_label.svg)

### Verification

![](https://netheril96.github.io/images/abstraction_and_unification_of_data_input/img_img_label.svg)

### Shuffling

`ImageDataLayer` has a special option to constantly shuffle input. With the new architecture, the similar can be achieved on any data source, by piping random indices into `DataMappingLayer`. The result is not the same, though, because this allows duplication.

![](https://netheril96.github.io/images/abstraction_and_unification_of_data_input/shuffle.svg)

### Selection

If there are many data input, splitting them into training and testing set and maintaining synchronization is tedious and error-prone. Instead, we can group them in the same data source. A single layer that outputs indices will be able to select different part of them in different scenarios. The synchronization is ensured by the fact that each `DataMappingLayer` receives the same index.

![](https://netheril96.github.io/images/abstraction_and_unification_of_data_input/select.svg)

## To do

This PR is a *minimal* functional implementation of the above idea. It only has two data sources: one "pseudo device" that outputs constant, another backed by CSV files. It also lacks prefetching, which could be added later.

The following data sources may be implemented after and if this PR is accepted:

* LMDB
* LevelDB
* HDF5
* protobuf
* JSON
* Gray/color images
* Random integers
* Normal distributed float
* Range slice
* Window