Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a function-consistent feature distillation method in the computer vision scenario. The motivation of the proposed method is based on the assumption that the widely used l2 loss between features does not take function-consistent into consideration, bearing the consequences that the student model may suffer from the negative transfer. Then the author introduces Function-Consistent Feature Distillation(FCFD), which measures the similarity between student and teacher's outputs generated by the lateral part of the same network. Extensive experiments show the improvement of the proposed method.
strength_and_weaknesses: Strength:
- The paper proposes a novel function-consistent feature distillation method targeting the computer vision scenario.
- The discussion on negative transfer from L2 loss between features is interesting. 
- The experimental evaluation is adequate, and the results somehow support the main claims.

Weakness:
- The motivation is not clear to me. The author claimed in the paper that there are some intermediate Conv and BN modules to map the student features to teacher features into the same size. Some methods, such as Review[1] and MGD[2], all consider multiple projection layers. It seems that the negative transfer from L2 loss is curtailed by these projection modules. I wonder if there is any theoretical analysis that supports the author's claim. A toy example of shallow learning can not provide adequate support for author's claim from a deep learning perspective. 
- The method is complicated. The proposed method considers both the student and teacher's lateral parts of the network, and then the training cost is relatively high. To me, the minor performance improvement can not make up for the additional complexity of the system.
- Some important experiments are missing: 1) The details of the projection module B() is missing. 2) In table 1 and table 2, the results w/o KD are omitted. 

[1] Yang, Zhendong, et al. "Masked Generative Distillation." arXiv preprint arXiv:2205.01529 (2022).
[2] Chen, Pengguang, et al. "Distilling knowledge via knowledge review." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.

I hope the authors can reply to me on the following:
- More mathematical insights on negative aspects of l2 loss.
- The complexity analysis of the proposed distillation method.
- The details on projection module B() and the missing experimental results on table 1 and table2.
clarity,_quality,_novelty_and_reproducibility: I can understand most parts of the paper though some expressions such as ``appearance'' and ``function'' are weird. I am on the border of the novelty. Some mathematical insights or proof sketches are missing to support the author's main claims. Some key details, such as the projection module B() are not sufficiently well-described. However, I choose to trust the results of the paper.
summary_of_the_review: The paper proposes a feature distillation method targeting the computer vision scenario.
Although the author claims that the conventional l2 loss between features contains some unfixed flaws, the motivation and insight are not clear. A toy example of shallow learning is far from supporting the claim. 
On the other hand, the proposed solution is complicated. Considering the improvement in performance is not significant (needless to say, some important baselines are missing), the whole cost is too high to be practical in the real industry environment.
Overall, I think the paper is below the borderline of ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel feature distillation method. Instead of directly restricting features between teacher and student using explicit loss functions like L2, the proposed method, FCFD, utilizes lateral networks to define the feature loss,  and improves over existing feature distillation methods. 
strength_and_weaknesses: strength:
+ A novel distillation method, FCFD, is proposed emphasizing the matching between teacher and student intermediate features not w.r.t pre-defined losses like L2 but w.r.t. function (aka the lateral network).
+ Extensive experiments on image classification and object detection are conducted to investigate the effectiveness and generalization of FCFD. The results show that the proposed FCFD significantly outperforms state-of-the-art knowledge distillation methods.
+ The proposed FCFD is compatible with many advanced knowledge distillation techniques with orthogonal contributions. Experiments verify its effectiveness when combined with other methods.
+ I like the toy example given in Figure 1, it illustrate some limitation of the L2 loss. (I suggest also mentioning other losses like smooth L1).

weakness:
- Even though the random sampler is introduced in Section 2.3, I could not find how it is implemented in experiments. Especially how the \delta values be sampled?
- In Table 4, why SimKD result under WRN-40-2/WRN-16-2 is not given? The values in this row seem to be directly borrowed from the SimKD paper, but without a confidence interval. If the authors reproduced SimKD results, then why not filling all the cells?

clarity,_quality,_novelty_and_reproducibility: The writing is clear, the whole paper is easy to understand. The novelty seems sufficient to me. The experiments are all conducted on public benchmarks.
summary_of_the_review: The idea of distilling using a part of the network seems novel to me. Extensive experiments have been conducted to verify the effectiveness of the proposed method in both image classification and object detection. Moreover, the authors illustrate that the proposed method can be effectively combined with other existing methods, showing a strong potential in application. In general, I recommend acceptance of this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper explores the new way to measure the distance between the teacher and student features, for better feature distillation (FD) performance. Specifically, the authors first point out the commonly used L2-norm feature distance is not good in semantic function, and then present a new FD method called function-consistent feature distillation (FCFD). Besides logits based KD loss at the heads and common FD loss at some intermediate teacher-student layer pairs, FCFD introduces two extra loss terms based on two kinds of lateral network parameter reuses: feeding dimension-aligned student features at a certain layer to the paired teacher layer, and feeding dimension-aligned teacher features at a certain layer to the paired student layer. The effectiveness of the proposed method is validated on both image classification (CIFAR100 and ImageNet) and object detection (MS-COCO) tasks.
strength_and_weaknesses: Strengths.

+ Improving feature matching is a critical research topic in FD research.

+ The basic ideas of the proposed method are easy to understand.

+ Experimental comparisons are performed on both image classification and object detection tasks with diverse teacher-student network pairs.

Weaknesses.

- The method.

The novelty of the proposed method, FCFD, is very limited. Note that the core designs of FCFD are two kinds of lateral network parameter reuses: feeding dimension-aligned student features at a certain layer to the paired teacher layer, and feeding dimension-aligned teacher features at a certain layer to the paired student layer. However, these two ideas have already been well explored in many existing knowledge distillation works such as cross distillation [1], residual distillation [2], explicit connection distillation [3],  softmax regression representation learning [4]. Unfortunately, these works are completely missed by the authors. Given the existence of these works, FCFD has no new technical contribution, to the best of my knowledge.

[1] Few Shot Network Compression via Cross Distillation, AAAI 2020.

[2] Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts, NeurIPS 2020.

[3] Explicit Connection Distillation, ICLR 2020 submission.

[4] Knowledge distillation via softmax regression representation learning, ICLR 2021.

- The experiments.

Experimental comparisons are not convincing enough.  

For experimental comparisons on image classification datasets (CIFAR100 and ImageNet), in Table 4, etc., it seems that the authors directly use the results reported in the papers of Review KD and DKD as the baselines. This is not fair, as for ablation, the comparison of FCFD+DKD/FCFD+SimKD vs. DKD/SimKD needs to run on the same training machines and code settings. 

For experimental comparisons on object detection dataset MS-COCO, experimental settings are also not optimal: 1) for object detection task, mainstream feature distillation methods, such as [1-3], already reuse/share feature pyramid network (FPN, the neck) of the teacher backbone as the FPN for the student. In this context, is it necessary to apply the proposed FCFD? 2) For experimental comparisons, FCFD should be compared to mainstream feature distillation methods for object detection, but not FD methods for image classification as they usually perform much worse.

In the final formulation of FCFD, there are five loss terms, how to set proper weights to them? 

How about the training cost of FCFD? As it will lead to heavy extra memory cost, it is necessary to compare training cost of FCFD with other FD methods. 

[1] Distilling Object Detectors with Feature Richness, NeurIPS 2021.

[2] Instance-Conditional Knowledge Distillation for Object Detection, NeurIPS 2021.

[3] Focal and global knowledge distillation for detectors, CVPR 2022.

**----Update----**

Balancing both positive and negative aspects of the rebuttal from the authors, although I raised the score from 3 to 5 (I would more like to give a score of 4 if there is such a choice), I still think this paper is not good enough to reach the acceptance bar of ICLR.
clarity,_quality,_novelty_and_reproducibility: The presentation of this paper is basically clear. My main concerns are to the limited technical contribution/novelty, and unfair experimental comparisons. Besides, the implementation of FCFD may not easy as there are many hyper-parameters. No code is provided.

Please refer to my comments in 'Strength And Weaknesses' for details.
summary_of_the_review: The novelty of this paper is very limited.

Please refer to my comments in 'Strength And Weaknesses' for details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new knowledge distillation (KD) method to better align student features with teacher features. The authors claim that traditional L2 loss for feature-level KD is not appropriate due to the deviation of functional mapping of similar features. In order to alleviate this problem, a function-consistent feature-level KD is utilized to align the cross-propagated features of both the student and the teacher. Various designs, such as separate batch statistics and path sampling, are incorporated to help the training and inference process. Experiments on both classification and object detection datasets demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: 1. The asymmetric loss design for the student and teacher network requires more ablation studies. Specifically, when “using student’s lateral part as the lens”, the authors think that “randomly initialized bridge” may bring unstable training status, and thus feature-level supervision is not available in "$L_{func’}^{k}$". When considering “teacher’s lateral part”, the bridge module still exists, however, the feature-level supervision is activated in "$L_{func}^{k}$". There had better be ablations like:
    1. Only KL supervision in "$L_{func}^{k}$"
    2. KL plus L2 supervision in "$L_{func’}^{k}$"

    Moreover, the following ablations are also expected:
    1. Only L2 supervision in "$L_{func}^{k}$"
    2. Only L2 supervision in "$L_{func’}^{k}$"
    3. L2 supervision on partial layers (Currently, all layers after the "k-th" one seem to be considered in the loss. Is it the best practice?)

2. For the path sampling scheme used in this paper, the authors simply sample two paths during training. I wonder the effects of different path sampling schemes, such as the number of sampled paths, the sampling probability (uniformly or imposing some priors to different layers) and the relative number of teacher-to-student and student-to-teacher paths if there are more than two paths. By the way, the sampling seems to be canceled for object detection, it might also work for classification to freeze the sampled path during the whole training process. In short, I think more ablations about the sampling scheme should be included in this paper.

3. Although the authors have repeatedly emphasized the extra training cost is acceptable, I think some numerical indicator, such as the training time and GPU memory peak, should be included in the experimental results.

4. Can the proposed method work for segmentation? Intuitively, I still think the propagation of sampled paths would introduce a lot of cost, which might hinder the training of dense prediction tasks, such as semantic segmentation.

5. For the separate batch statistics, which one will be applied during inference, or is there some fusion scheme to mix statistics of different paths during inference?

6. I observe that “FCFD+DKD” is inferior to “FCFD” under the settings of “WRN-40-2/WRN-40-1” and “WRN-40-2/WRN-16-2” in Table 4. Still in Table 4, the result of “SimKD” under “WRN-40-2/WRN-16-2” is missing, and “FCFD+SimKD” is also inferior to “FCFD” under this setting. Is there an explanation for these results?

7. Can the authors provide some feature visualizations to explain the results in Table 6? For example, the PCA of student and teacher features w.r.t. “FCFD” and "$L_{kd}+L_{task}+L_{app}$".

clarity,_quality,_novelty_and_reproducibility: Clarity: 8/10

Quality: 8/10

Novelty: 6/10

Reproducibility: Unknown
summary_of_the_review: Overall, this paper could be accepted if the authors can answer the aforementioned concerns to some extent.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper induces a new loss function and architecture for knowledge distillation. Prior work penalizes a distance metric between student and teacher features. This paper argues that instead a measure of "function (in)consistency" should be penalized. At layer L of the networks, features are extracted from student and teacher nets and sent to the latter layers (L+1 and beyond, called "lateral net") of the _opposite_ net -- teacher and student nets respectively. Then disagreement is penalized in the outputs. This encourages student features to be aligned with the teacher lateral net and vice versa.

Strengths:
* Reasonable idea with good motivation
* Improved performance on a variety of benchmarks

Weaknesses:
* Several prior works have used a similar "swapping" strategy, inputting student features into teacher lateral net (Cross Distillation, Residual Distillation, and SRRL)
* Some concerns about the fairness of the detection experiments
* Mathematical motivation was not convincing to all reviewers

justification_for_why_not_higher_score: Reviewers were split on this paper. After extensive discussion with the authors, most concerns were alleviated, but a few issues remained, including concerns about the novelty, the mathematical motivation, and the fairness of the experiments. I find these concerns to be valid and encourages the authors to continue to address them, but I feel that 1) the deficiencies of L2 are well described in the Section 1.3, and 2) the majority of the experiments are fair and additional improvements can be made for camera ready. The main remaining weakness is the novelty, which I agree is limited, hence I advocate accept at just the poster level.
Decision: Accept: poster