Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: After rebuttal:
Authors have addressed all my doubts. I recommend accepting this paper.

=============================
Before rebuttal:
Summary:

This paper proposes a new way to do transfer learning. Specifically, authors first train a big source ConvNet and then for each task, they train a small ConvNet in which each layer subscribes to some k channels in the corresponding layer of the source ConvNet. Authors show that this model works better than methods that fine-tune the last few layers of the source network and performs close to costlier methods like progressive networks but with lesser parameters and higher throughput. Experiments on 5 tasks verify their claim.


My comments:

Overall, this is a very interesting paper.

1. This is an interesting model to do transfer or lifelong learning but only for ConvNet architectures with image data. To avoid overstating the results, I request the authors to highlight this limitation in both the title and the abstract.
2. Page 3, para starting with “In detail”: Is the ResNet50 for delta model pre-trained or not? I know it is not pre-trained based on future paragraphs. But it is good to clarify it here.
3. Sharing the same source network across multiple tasks during inference time is useful only when all the tasks take the same input. This is a very restricted application. This needs to be elaborated and highlighted in the paper.
4. I would like to see the LF results included in the paper even though it has catastrophic forgetting issues.
5. In Figure 4, the x-axis represents training throughput or inference throughput? I guess it is training throughput. Also, are the models trained for all the tasks in parallel (as described in serving all the tasks at once section) or separately? Even though I can guess answers for these, it is better to make these explicit in the paper for the benefit of the readers.
6. It is never a good idea to show test curves for a task. Please remove the test curves from Figure 4. Instead, use a separate validation set and show validation curves.
7. Are the authors willing to release the code to reproduce their results?

Minor comments:

1. Section 1, second para, 1st line: “wee” should be “we”
2. Table 1: Fix grammar in MP description.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: *** Increased to Accept from Weak Accept after author rebuttal and changes to the paper ***

This paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning. SNOW starts with a pre-trained, frozen source model, and trains delta models for target tasks which, at each layer, concatenate a small number of task-specific features with the top-K most useful subset of features in the corresponding layer in the source model. As long as the target tasks are sufficiently related to the source task, it allows for small delta models and a small additional parameter overhead in the form of one weight per source model feature map. While there are (i) some issues with the presentation of results for training efficiency, (ii) some question marks over the sensitivity of the model to hyper-parameters, and (iii) several grammatical errors / typos in the manuscript, if these can be addressed I recommend the paper for acceptance because it seems to strike a superior balance of efficiency (regarding memory usage and inference speed) and accuracy when compared to a number of baselines, and to my knowledge it is a novel approach.

Detailed comments:
* Section 2.2 - How is sigma (the exploratory noise added for feature selection during training) chosen and how sensitive is the approach to its value? It seems like it was fine-tuned, given that a different sigma is chosen for the Action dataset (several orders of magnitude difference). In practice, tuning sigma could significantly increase training time. 
* It seems like the performance of only one run was plotted per hyperparameter setting - it would be informative to see a mean and standard deviation especially since the approach seems like it could be unstable for the wrong hyperparameter settings.
* Related to the previous point, how much do the top-K feature selections change throughout training? One would have thought that this could cause instability during training for a high sigma. If sigma is too low, you could end up with suboptimal feature selection.
* Figure 4 graphs are a bit misleading because the throughput on the x-axis is reported per GPU and the larger models all need 2 or more GPUs. While this is mentioned in the main text, it is still optically deceptive and the results are GPU-dependent - presumably if the GPUs had a larger memory, the larger models would not seems as slow. I think it would be clearer to plot images/sec on the x-axis or to rerun the experiments just using a single GPU.
* It is stated that  “[d]etermining K […] has a critical impact on both size and target accuracy in the target models”, where K is the number of feature maps in the source model that the delta model subscribes to in each layer. How sensitive is the accuracy exactly? Can this be quantified or discussed in more detail?
* Furthermore, how sensitive is the performance to the number of target-model-specific features at each layer?
* Different learning rate schedules were used for SNOW and baselines - initial lr for SNOW is 1.0, while for all other models it is 0.1. Was it checked whether the baselines improve when they are run with an initial lr of 1.0? Was this hyperparameter more heavily tuned for SNOW than for the baselines?
* Since the source model is fixed, the applicability of the approach to lifelong learning is heavily dependent on the usefulness of the source model to subsequent tasks. If it is not, then one will have to incorporate large delta models. Furthermore, there can be no transfer between the tasks trained in the delta models.
 
Grammatical errors / suggestions:
* Page 1, first line: “hallmark” doesn’t make sense in this context - maybe “key objective” or “goal”?
* Page 1, 2nd paragraph, first line: “wee” -> “we”.
* Page 1, 2nd paragraph, line 6: “best top-K” -> either “K best” or “top K"
* Page 2, last paragraph, first line: “three folds” -> “threefold"
* Section 2.1, line 2: “pooing”->”pooling”. Same typo on Page 4, last line.
* Page 6, line 1: “training from the scratch” -> “training from scratch"
* Page 6, line 9: “more 6x than” -> “6x more than"
* Overall, the manuscript needs to be proofread a few times.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper attempts to tackle transfer learning and lifelong learning problem by subscribing to knowledge via channel pooling. The channel pooling is actually selecting the subsect of the feature map according to the way that prediction accuracy from the delta model can be maximized. Experiments show effectiveness of the proposed method.

Pros:
Overall, this paper is well written and easy to follow. The technique is sound and the problem studied in this paper is significant.

Cons:
1.	I do not think that the model proposed in this paper is able to tackle lifelong learning problem. The main reason is that lifelong learning basically requires only one model that will continue to learn from new tasks. After learning several new tasks, people hope this model can still perform well on the previous tasks as well as the current ones. However, in this paper, not only one model is learned. Instead, new models appear when new tasks are given, which does not meet the definition or requirement of lifelong learning. It only meets the requirement of transfer learning. The experimental results also validate my opinion since only one new task is given while in lifelong learning, continuous new tasks will come and the original model should perform well on all of them as well as on the old tasks.
2.	In Figure 4, the legend in the first picture will confuse the readers. I suggest the authors put it outside all the figures. Besides, the proposed method in the last picture is not the best. What do the authors want to convey by this picture?


review_title: Official Blind Review #2

Meta-review: This paper proposes a method, SNOW, for improving the speed of training and inference for transfer and lifelong learning by subscribing the target delta model to the knowledge of source pretrained model via channel pooling.

Reviewers and AC agree that this paper is well written, with simple but sound technique towards an important problem and with promising empirical performance. The main critique is that the approach can only tackle transfer learning while failing in the lifelong setting. Authors provided convincing feedbacks on this key point. Details requested by the reviewers were all well addressed in the revision.

Hence I recommend acceptance.
Decision: Accept (Poster)