This paper presents a new technique for adapting a neural network to a new task for which there is not a lot of training data. The most widely used current technique is that of fine-tuning. The idea in this paper is to instead learn a network that learns features that are complementary to the fixed network. Additionally, the authors consider the setting where the new network/features are “stitched” to the old one at various levels in the hieararchy, rather that it just being a parallel “tower”.   This work is similar in spirit (if not in some details) to the Progressive Nets paper by Rusu et al, as already discussed. The motivations and experiments are certainly different so this submission has merit on its own.  The idea of learning a “residual” with the stitched connnections is very similar in spirit to the ResNet work. It would be nice to compare and contrast those approaches.  I’ve never seen a batch being used 5 times in a row during training, does this work better than just regular SGD?  In Figure 5 it’d be nice to label the y-axis. That Figure would also benefit from not being a bar chart, but simply emulating Figure 4, which is much more readable!  Figure 5 again: what is an untrained model? It’s not immediately obvious why this is a good idea at all. Is TFT-1 simply fine-tuning one more layer than “Retrain Softmax”?  I think that the results at the end of section 3 are a bit weak because of usage of a big network. I would definitely like to see how the results change if using a smaller net.  The authors claim throughout the paper that the purpose of the added connections and layers is to learn *complementary* features and they show this with some figures. The latter are a convinving evidence, but not proof or guarantee that this is what is actually happening. I suggest the authors consider adding an explicit constraint in their loss that encourages that, e.g. by having a soft orthogonality constraing (assuming one can project intermediate features to some common feature dimensionality). The usage of very small L2 regularization maybe achieves the same thing, but there’s no evidence for that in the paper (in that we don’t have any visualizations of what happens if there’s no L2 reg.).  One of the big questions for me while reading the paper was how would an ensemble of 2 pre-trained nets would do on the tasks that the authors consider. This is especially relevant in the cars classification example, where I suspect that a strong baseline is that of fine-tuning VGG on this task, fine-tuning resnet on this task, and possibly training a linear combination of the two outputs or just averaging them naively.  Disappointing that there are no results in figure 4, 5 and 8 except the ones from this paper. It’s really hard to situate this paper if we don’t actually know how it compares to previously published results.   In general, this was an interesting and potentially useful piece of work. The problem of efficiently reusing the previously trained classifier for retraining on a small set is certainly interesting to the community. While I think that this paper takes a good step in the right direction, it falls a bit short in some dimensions (comparisons with more serious baselines, more understanding etc).
Output: en