Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors discuss the phenomenon that using a projector on a student’s feature can improve the performance of distillation when the feature dimensions are the same.
They focus on the projector between teachers and students, which is unnoticed in the past.
Then the authors propose a simple ensemble method to further improve the performance.
They conduct comprehensive experiments in this paper and the authors claim the superiority of their proposal.
A complete story, from the phenomenon to the essence, which is simple and effective.

strengths_and_weaknesses: Strengths
1.	The method is simple and easy to implement.
2.	Experiments are detailed and code is provided.
3.	This paper is well written.
Weakness
1.	The novelty of the proposed method is limited. Are you sure that there are no more researchers who have used this method to improve the performance of their model?
2.	the number of projectors in the ensemble seems to be sensitive, how to determine this hyperparameter appropriately in practice?
3.	The ablation study does not provide the effect of different numbers of projectors on distillation when the feature dimensions are different.
questions: 
“the student network benefits from a projector even if the feature dimensions of the student and teacher are the same” , Is this conclusion first discovered and proposed by the author of this paper? The word used in the abstract is “observe”.

Initialization of different projectors is not clear, do different initialization methods have a big impact on the experimental results?

limitations: As mentioned above in the Weakness.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a feature distillation method based on an ensemble of multiple projectors.  The model uses multiple projectors to project the features of a student model, and the Direction Alignment loss are calculated between the multiple projected features and the teacher features. Some analysis are given to explain why projector is helpful. Experimental results show that the proposed method outperforms previous competitors.
strengths_and_weaknesses: Strengths:

1. This paper focuses on the projectors in feature distillation models and proposes a simple but effective ensemble method. This angle is interesting and the proposed model can be widely applicated.

2. The writing is clear and easy to understand. 

3. Experimental results show that the proposed model obtains promising results.

Weaknesses:

1. The ensemble strategy is too simple. The authors simply add all the projected features together.  In fact, ensemble strategy plays a critical role in ensemble learning. In many situations, simply averaging the outputs of weak learners cannot bring improvements.

2. This paper lacks discussions about the projector behaviors, especially their diversity. The authors only give a simple description of "projectors with different initialization would provide different transformed features, which is beneficial to the generalizability of the student".  However, different initialization cannot guarantee enough variety between projectors. Moreover, the different projected features are simply added together to calculate losses. There is no special architecture and training designs to promote the projector diversity. In experiments, there is also no experimental analysis of the projector behaviors.

3. The theoretical analysis is not clear enough. In section 3.1, the authors try to analyze why projector is helpful by comparing the feature gradients with and without projector. The explanation is that the non-linear transformation updated from previous data helps to better capture the global feature distribution. This is not convincing enough to me. Furthermore, this explanation cannot explain why multiple projectors are better.
questions: 1. I suggest the authors give some visualization results or experimental analysis of the projector behaviors during and after training. If different projectors show significant differences, it would be better to give some theoretical analysis about what brings them variety. 
2. The theoretical analysis in section 3.1 needs to be clearer. Also, the authors need to give explanations of why more projectors are better under their feature gradients perspective.
limitations: The authors have addressed the limitations and potential negative societal impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This is an experiment-driven paper. The authors do lots of experiments and comparisons to demonstrate the effectiveness of feature projectors in feature distillation. 
strengths_and_weaknesses: Strengths:

The authors do lots of experiments and comparisons to show the superiority of feature projectors in feature distillation. 

Weaknesses:

1. This is an experiment-driven paper. The authors do lots of experiments and give empirical analyses. However, I still think the contributions are not enough for NeurIPS. 
2. The authors should re-organize the Method section. It is not obvious from one equation to the next. And the notations are somewhat confused. 
    - The notations $s^p$ and $W^p$ is confused. I suggest to use $t$ as the $t$-th iterations and use $W_p$ to represent the parameters of projector. It is more clear. 
    - The Eq. (2), Eq. (3) and Eq. (4) should be more clear. It is not easy to understand, especially for the junior researchers. For example, Eq. (3) can be denoted as $\frac{\partial L_{DA}}{\partial s_i^p}=\frac{\partial L_{DA}}{\partial g(s_i^p)}\frac{\partial g(s_i^p)}{\partial s_i^p}$. The last term of Eq. (4) can be denoted as $\frac{\partial L_{DA}}{\partial g(s_i^p)}\frac{\partial g(s_i^p)}{\partial W^p}$. 
3. The hypothesis (Line 125-126) is a little coarse. I think the main contribution in this paper is the analysis about the effectiveness of the projector. Therefore, the authors should give deeper analyses. 
4. In experiments, table is more suitable and clear to demonstrate the performance than figure. I recommend the authors use tables instead of Fig. 3 and Fig. 4.  
5. Considering the feature distillation, some experiments about CUB-200-2011, Cars-196 and SOP should be included. 
questions: See weaknesses. 
limitations: See weaknesses.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a feature matching-based distillation method that makes use of a set of feature projectors to better align the features of the student and teacher network. 
The authors show that such an ensemble of projectors can improve the distillation performance further even if the student and teacher have the same feature dimension.
Some analysis is conducted to show the reason of introducing a projector can benefit the student network learning.
Experiments were performed on benchmark datasets using various teacher-student network structure pairs to compare with different methods.
strengths_and_weaknesses: Strengths:
- The paper is well written with comprehensive experiments.
Weakness:
- The idea is simple but the authors should explore more options for an ensemble of projectors, e.g. ensemble of various structure projectors.
questions: 1. Line 251, which table is this referring to? Is it table 1?
2. Is there any limitation in the training of an ensemble of projectors? Is the training time impacted by the number of projectors and/or the depth of the projectors?
limitations: The authors have a section for discussing limitations and future works. However, it did not directly address the limitation of the multiple projectors, e.g. training time, memory usage, etc.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposed a new feature distillation method via projector ensemble. In particular, different from the traditional single-projector method, it introduces multiple projectors and computes the average output to perform KD. Further, two different designs including a single-layer projector and MLP projector are discussed and compared. Experiments on popular datasets are conducted to verify the effectiveness of the method.
strengths_and_weaknesses: The strengths are listed below:
(1)	The proposed method achieves significant performance improvement. 
(2)	This proposed method is simple and easy to be reproduced.

The weaknesses are listed below:
(1)	The novelty is limited. The single-layer projector is widely used in KD. Further, a simple extension from single-layer to multi-layer is rather simple. It is easy to come up with this idea. 
(2)	Lack of references in all of the tables. It is recommended to add references of comparing methods in tables.

questions: Please refers to the weaknesses mentioned above.
limitations: As mentioned in the paper, the proposed method is only verified on classification. Its effectiveness on other tasks can be further explored.
review_title: null

Meta-review: metareview: The paper received 5 positive reviews and the reviewers increased/remained their scores after the rebuttal. All the reviewers agree that the proposed method is simple yet effective, and the experiments are comprehensive.

Overall this work proposes an improved feature distillation method via projector ensemble. But I hope the authors will discuss the computational costs brought by multiple projectors clearly, as suggested by the reviewers.
confidence: Certain
Decision: Accept