Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes new idea of using controller modules for increment learning. Instead of finetuning the whole network, only the added parameters of the controller modules are learned while the output of the old task stays the same. Experiments are conducted on multiple image classification datasets. 

I found the idea of using controller modules for increment learning interesting and have some practical use cases. However, this paper has the following weakness:
1) Missing simple baselines. I m curious to see some other multitask learning approach, e.g. branch out on the last few layers for different tasks and finetune the last few layers. The number of parameters won't be affected so much and it will achieve better performance than 'feature' in table 3.
2) Gain of margin is really small. The performance improvements in Table1 and Table3 are very small. I understand the point is to argue with fewer parameters the model can achieve comparable accuracies. However, there could be other ways to design the network architecture to reduce the size (sharing the lower level representations).
3) Presentation of the paper is not quite good. Figures are blurry and too small. 
review_title: Interesting idea but missing some simple baselines.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to adapt convnet representations to new tasks while avoiding catastrophic forgetting by learning a per-task “controller” specifying weightings of the convolution-al filters throughout the network while keeping the filters themselves fixed.


Pros

The proposed approach is novel and broadly applicable.  By definition it maintains the exact performance on the original task, and enables the network to transfer to new tasks using a controller with a small number of parameters (asymptotically smaller than that of the base network).

The method is tested on a number of datasets (each used as source and target) and shows good transfer learning performance on each one.  A number of different fine-tuning regimes are explored.

The paper is mostly clear and well-written (though with a few typos that should be fixed).


Cons/Questions/Suggestions

The distinction between the convolutional and fully-connected layers (called “classifiers”) in the approach description (sec 3) is somewhat arbitrary -- after all, convolutional layers are a generalization of fully-connected layers. (This is hinted at by the mention of fully convolutional networks.)  The method could just as easily be applied to learn a task-specific rotation of the fully-connected layer weights.  A more systematic set of experiments could compare learning the proposed weightings on the first K layers of the network (for K={0, 1, …, N}) and learning independent weights for the latter N-K layers, but I understand this would be a rather large experimental burden.

When discussing the controller initialization (sec 4.3), it’s stated that the diagonal init works the best, and that this means one only needs to learn the diagonals to get the best results.  Is this implying that the gradients wrt off-diagonal entries of the controller weight matrix are 0 under the diagonal initialization, hence the off-diagonal entries remain zero after learning?  It’s not immediately clear to me whether this is the case -- it could help to clarify this in the text.

If the off-diag gradients are indeed 0 under the diag init, it could also make sense to experiment with an “identity+noise” initialization of the controller matrix, which might give the best of both worlds in terms of flexibility and inductive bias to maintain the original representation. (Equivalently, one could treat the controller-weighted filters as a “residual” term on the original filters F with the controller weights W initialized to noise, with the final filters being F+(W\crossF) rather than just W\crossF.)

The dataset classifier (sec 4.3.4) could be learnt end-to-end by using a softmax output of the dataset classifier as the alpha weighting. It would be interesting to see how this compares with the hard thresholding method used here.  (As an intermediate step, the performance could also be measured with the dataset classifier trained in the same way but used as a soft weighting, rather than the hard version rounding alpha to 0 or 1.)


Overall, the paper is clear and the proposed method is sensible, novel, and evaluated reasonably thoroughly.
review_title: -

final_score: rating: 5: Marginally below acceptance threshold
review_content: ----------------- Summary -----------------
The paper tackles the problem of task-incremental learning using deep networks. It devises an architecture and a training procedure aiming for some desirable properties; a) it does not require retraining using previous tasks’ data, b) the number of network parameters grows only sublinearly c) it preserves the output of the previous tasks intact.

----------------- Overall -----------------
The paper tackles an important problem, aims for important characteristics, and does extensive and various experiments. While the broadness of the experiments are encouraging, the main task which is to propose an effective task-incremental learning procedure is not conclusively tested, mainly due to the lack of thorough ablation studies (for instance when convolutional layers are fixed) and the architecture seems to change from one baseline (method) to another.

----------------- Details -----------------
- in the abstract it says: "Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added task, typically as many as the original network."
The linear-combination constraint in the proposed approach is a strong one and can learn a sub-optimal solution for the newly introduced tasks.

- Page 3: R^C → R^{C_o}

- The notation is (probably unnecessarily) too complicated, perhaps it’s better to formulate it without being faithful to the actual implementation but for higher clarity and ease of understanding. For instance, one could start from denoting feature maps and applying the controller/transform matrix W on that, circumventing the clutter of convolutional kernels.

- What is the DAN architecture? 

- In table 1 a better comparison is when using same architecture (instead of VGG) to train it from scratch or fine-tune from ImageNet (the first two rows)

- What is the architecture used for random-weights baseline?

- An experiment is needed where no controller is attached but just the additional fully-connected layers to see the isolated improvements gained by the linear transform of convolutional layers.

- Multiple Base Networks: The assumption in incremental learning is that one does not have access to all tasks/datasets at once, otherwise one would train them jointly which would save parameters, training time and performance. So, finding the best base network using the validation set is not relevant.

- The same concern as above applies to the transferability and dataset decider experiments

review_title: Extensive experiments but inconclusive for the main message (task-incremental learning)

Meta-review: This work tackles an important problem of incremental learning and does so with extensive experimentation. As pointed out by two reviewers, the idea does seem novel and interesting, but the submission would require some rewriting before being potentially accepted at a venue like ICLR. I suggest focusing the paper more on the task-incremental learning aspects, doing the ablation studies (and other changes) as requested by the reviewers, and having a rich appendix with details (with more discussion in the paper itself).
Decision: Reject