Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: - The authors propose a method of learning a consolidated image feature representation from a collection of related task-specific teachers that transfer well on novel recognition tasks.

- To achieve it, the authors utilize multi-teacher multi-task model distillation framework that jointly distills one or several task-specific teachers with a generalist one (trained with Imagenet dataset). 

- Each teacher is set to operate on a different set of classes, and a multi-head student is trained to emulate all teachers.

- Experimental results show that the proposed method doesn’t need to revisit original training data of each teacher, but gains better performance.

main_review: - In this paper, the authors have stated their key difference from the traditional distillation loss item is the generalist teacher item. Thus, I feel the contribution  is not that strong enough, since the generalist teacher is the pre-trained network.

- In addition, the generalist teacher item is just the model pretrained on imagenet dataset. What if in the other situation that the task is in NLP or speech recognition? Does it mean that the method can’t generalize well?

- In order to avoid catastrophic forgetting, do the authors try to add regularization item like the works of incremental learning do?

- The work here aims at transfer learning, but the experiments are set with the classes in the same domain.

- Some advice in the experiments, the chart figures are not very suitable here.

summary_of_the_review: - Besides the generalist teacher, what is the other contribution?

- Please discuss the generalization ability of the proposed method.

- Discuss more about the experiments.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a task-agnostic Knowledge Distillation method that includes a generalist model as an additional teacher, to limit student forgetting and representation collapse. The authors call this method representation consolidation and claim it can benefit both related and unrelated downstream tasks. The authors experiment with ImageNet and iFood image classification tasks, the accuracy of the proposed method is generally higher than using Knowledge Distillation only.
main_review: The main contribution of the paper is the  _representation consolidation_ method to consolidate knowledge from multiple task-specific teacher models into a single expert student, given a generalist teacher is available. This boils down to adding a generalist teacher to the training. 

The main findings are that consolidating a task-specific teacher with a generalist teacher is sufficient to rescue the student and consolidation performs only slightly worse/similar to a multi-task joint training oracle, at least on image classification tasks. Which brings about one of the main issues of this paper - despite proposing a general framework, only the image classification problem has been addressed. I would have liked to see more complex tasks such as dense predictions can benefit from the proposed method (e.g.., optical flow, depth estimation). The main insight here is - if we can get a generalist that works well on most scenarios, could we squeeze additional performance from the proposed method or not? Otherwise, the main claim of this paper would be - keeping a generalist teacher improves _any_ transfer learning. Which is kind of a bold statement for ImageNet / iFood testing only, IMHO.

Furthermore, most of the times few-shot learning is illustrated, the few examples of full fine tuning are less favorable to the proposed method. 

Otherwise, the paper is well written and illustrated, and the experiments support the claims, in the scenario proposed by the authors.


Strenghts:
- general method for a better distilled student
- easy to implement, provided a generalist is available for the proposed task
- no data replaying for the generalist network during fine-tuning

Weaknessses:
- simple training scenario, mostly classification, few-shot learning
- where fine-tuning is employed (Fig 4/ 10) the results are less convincing
summary_of_the_review: I lean towards weakly rejecting the paper, based on the limited applicability/experiments mentioned above. If there is a general claim to be made, I would like to see more evidence and other domains tested, beyond image classification. As it stands, I do not find the technical contribution sufficient for acceptance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a representation consolidation method to properly aggregate the pre-trained knowledge from multiple teachers for transfer learning. It claims that a generalist teacher is necessary for preserving the transferability of distilled representations, and task-specific teachers contribute to better performance in the same-domain downstream tasks. The introduced method performs on par with multi-task training while neglecting the need for teacher datasets.
main_review: Strengths:

1. The problem of how to properly use multiple pre-trained models for downstream transfer learning is interesting and practical in real life.

2. The introduced method achieves better performance than distillation from a single ImageNet-pretrained teacher or task-specfic teacher.

Weaknesses:

1. This paper tells a big story of ``life-long meta-learning'' in the beginning, but fails to verify it in the method and experiment part. Specifically, the authors argue that they would like to automatically enrich the diversity of the pre-trained model zoo by accumulating knowledge from downstream tasks. However, I cannot find any evidence of enriching the diversity in this paper.

2. The authors claim that the generalist teacher is important to the transferability of distilled representations. It should be important to carefully discuss the choice of the generalist teacher in this framework, e.g., how does it perform when adopting different or multiple generalist teachers. I did not find anything about it.

3. Similarly, how to properly select task-specific teachers for corresponding downstream tasks? How much does it affect when choosing different numbers of task-specific teachers? In the experimental part, the authors only show the comparisons when adopting ad-hoc teachers, which is less convincing.

4. As shown in Fig. 1, I find the comparison between *distillation directly to the downstream task* and *distillation first and then finetuning to the downstream task* interesting. It is necessary to conduct ablation studies on this comparison while keeping the teachers (a generalist teacher + multiple task-specific teachers) the same. 


====================

Post-rebuttal:

Thanks for the authors' efforts. But unfortunately, I choose to maintain my original score considering the other reviewers' concerns and the overclaiming problem (the method and experiments fail to support the motivation) as I mentioned in the initial review.
summary_of_the_review: The motivation of this paper (life-long meta-learning) is attractive, but the method and experiments fail to well support it. I tend to reject this paper, and recommend the authors refine it and submit it to the next conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the problem of multi-model consolidation, which aims to merge (consolidate) the knowledge from multiple models into one model so that we can better transfer this merged model for downstream tasks. Different from the traditional knowledge distillation or multi-teacher distillation, the proposed method aims to transferability of merged models for the downstream tasks, whilst achieving good performance on source tasks. The key technique (technical contribution) is to add a generalized model during the knowledge distillation, which can be regarded as s "special" teacher model.  

The authors conduct extensive experiments on multiple datasets and most of the experiments are under a few-shot linear probe transfer learning setting.
main_review: Strengths:

1. This paper is well written and easy to follow. I think the study multi-teacher consolidation problem is interesting and important.  In some practical domains (like healthcare), it is difficult to construct a large dataset to train a model,  how to conduct transfer learning from multiple datasets (models) is critical.

2. The authors conduct extensive analysis on multiple datasets.

Weakness:
1. The technical contribution is limited, as the method is somehow ad-hoc. Different from previous multi-teacher-distillation works, the authors add a generalized model (ImageNet model) during knowledge distillation to prevent representation collapse. However, there is no deep analysis of why this can achieve such a goal. Moreover, this strategy simply treats the generalized model and other teacher models equally, while there should be different roles during the distillation between the generalized model and teacher models.

2. As the technical contribution is limited, I would appreciate the merits of experiments.  Although the authors conducted extensive analysis, most of them are in a few-shot learning setting (i.e., using only several images to train the downstream networks) not a fine-tuning setting. For the few-shot learning setting, the performance of different experimental runs has large perturbations, it is truly difficult to determine the effectiveness of the proposed method. 

3. The authors also conduct the experiments under full fine-tuning in Exp. 3. However, the improvement of the proposed method compared with "imagenet-pretrained + fine-tuning" is quite incremental (see Figure 4). Therefore, it is difficult to sufficiently support the advantage of the proposed multi-teacher consideration method with the existing experiments.

Other comments:

1. In the first paragraph, the authors mention the overall goal is to "enrich the diversity of the expert library by accumulating knowledge from transferred downstream tasks" and in the second paragraph, the authors argue that adding fine-tuned downstream models back into the library is naive and does not scale. However, to achieve the overall goal, the proposed multi-model consideration method still seems to depend on this naive strategy. Maybe the authors can reorganize the motivation part in the introduction to avoid confusion.

2.  It is better if the authors can briefly summarize the size/class distribution of each dataset in the experimental setup section. 

3.  As there are performance perturbations for different experimental runs, especially under the few-shot learning setting. Are the results in the tables the average values of different runs or single runs? If they are the results of different runs, it is better to show the std of the results.
summary_of_the_review: The studied problem is interesting and important and the whole paper is easy to follow, while the technical contribution is limited and the experiments are not very strong to support the advantage of the proposed method.
review_title: null

Meta-review: Authors present an approach to consolidate multiple teachers into a single student model that can be adapted to new tasks. The method involves using a proxy dataset to facilitate distillation to prevent having to replay images from the teacher datasets. A multi-task multi-head objective is utilized, agnostic to the loss function, in which two are studied. Downstream task performance is used as the performance measure.

Pros:
- The problem of how to best leverage multiple teachers for a downstream task is important and interesting.
- Presents a method to generate distilled students that can be finetuned to tasks that demonstrates performance gains over baselines (imagenet alone or task specific teacher). 
- Easy to follow and implement.
- Analysis across multiple datasets.

Cons:
- Multiple reviewers expressed concerns about current level of novelty / contribution. In some sense, it is natural to expect that combinations of task-related and generalist distillation would improve performance.
- Main results demonstrate improvements in performance when teacher and tasks are related to one another. But authors do not address how to select task-specific teachers for distillation. Related tasks and their matching to the target task are assumed to be known. Authors cited related prior works that attempt to do this matching, but do not apply it to their study for a full solution.
- Authors do not study variations of generalist teachers. How does changing the generalist teacher impact performance?
- Some reviewers expressed concern presentation is not clear. In particular, the style of figures may not be appropriate to best convey results and analyses of this type of work. Comparing different approaches is difficult looking at thin lines. Tables are perhaps better suited to convey these results.
- Multiple reviewers expressed concerns full-finetuning results are not convincing (Fig 4), though few-shot results look more convincing

Authors and reviewers had interaction, but reviewers maintained their recommendation of weak reject. All reviews unanimous in their decisions. Authors are encouraged to take into consideration all the comments and submit to another venue.
Decision: Reject