Siamese Networks / Distance Learning / Transfer Learning

Hi,

I am working on implementing a siamese network in caffe. The general pipeline for training I am thinking of right now is:

1. train a network on a N-way classification task
2. 
  * remove the top 1 or 2 layers from the network 
  * create a copy of the network (using <code>sharedweights</code>) 
  * add a binary output distance layer that measures the distance between the two outputs

(2a) is where I believe the first change in caffe needs to be made. That is, using the representation learned by one deep network for some other task by changing the top 1 or 2 layers. 

I put together a small hack that allows this in caffe by loading a state file of a trained network and passing an optional <code>int remove_from_top</code> to the function <code>Solver::Restore</code> and <code>Net::CopyTrainedLayersFrom</code>. This changes the behavior to only load the state of the first <code>(total - remove_from_top)</code> layers of the network. The rest of the layers specified in the new network's <code>.prototxt</code> file should initialize normally (because they are initialized before loading from state)

Do you have any suggestions or another preferred approach on how to tackle to this?