BVLC/caffe — pull request #500 by jeffdonahue (2014-06-13 21:24:36+00:00)
Title: Weight sharing

Built on top of #497 -- I pushed that to a new branch here (`fix-backward-interface`) and PRed this against that, and once #497 is finished and merged I will pull against dev.

This adds the ability to share parameters between layers, which has a number of applications, the canonical one perhaps being recurrent neural network (RNN) training.

To share weights between two or more layers with parameters (currently just `InnerProductLayer`s and `ConvolutionLayer`s), specify the same `blob_name` for all of these layers. (You can also name the biases with a second `blob_name`, as in the `blobs_lr` and `weight_decay` parameters.) You can see a very simple example of this in `src/caffe/test/test_net.cpp`: see the unit test named `InitDiffDataSharedWeightsNet`:

```
layers: {
name: 'innerproduct1'
type: INNER_PRODUCT
inner_product_param {
num_output: 10
bias_term: false
weight_filler {
type: 'gaussian'
std: 10
}
}
blob_name: 'sharedweights'
bottom: 'data'
top: 'innerproduct1'
}
layers: {
name: 'innerproduct2'
type: INNER_PRODUCT
inner_product_param {
num_output: 10
bias_term: false
weight_filler {
type: 'gaussian'
std: 10
}
}
blob_name: 'sharedweights'
bottom: 'data'
top: 'innerproduct2'
}
```

This means layers innerproduct1 and innerproduct2 are sharing the same set of weights as they've both specified `blob_name: 'sharedweights'`, and because they also take the same bottom blob, their outputs, top blobs `innerproduct1` and `innerproduct2`, should be identical (so this is not actually something you'd ever want to do; I do it there just for testing purposes).

Note that in this case we specify only one blob name because we've set `bias_term: false`; if we didn't have `bias_term: false` we'd need to specify two `blob_name`s, but probably the second one should be empty unless we actually want to share biases. (Specifying the empty string as a `blob_name` is equivalent to not specifying a `blob_name` in my implementation.)

```
blob_name: 'sharedweights'
blob_name: ''
```

The entire implementation is in `Net::Init`, `Net::AppendParam`, and `Net::Update`. `Init` figures out which layer will actually "own" the shared param (the first one to list its `blob_name`), and `Update` adds the non-owned layers' computed diffs into the diff of the owner blob, then only actually performs updates on owned blobs. Memory-wise, all shared blobs actually point to the same memory location for the parameter's data, but still have separately allocated diff blobs, as the logic to handle learning rate, weight decay, etc is still handled by the Solver (which is blissfully unaware that parameters can be shared).

Open to hearing feedback on the interface, implementation, etc. I'm not sure I'm happy with `blob_name` as the name of the field, I think it would be less ambiguous to use `param_name` or something, but would be inconsistent with the other per-parameter field `blobs_lr` (and actually to be consistent with that it should be `blobs_name`, but I strongly prefer the singular here..).