Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors investigate the value of fully connected layers at the end of convolutional neural networks in the small data regime. They demonstrate that the addition of these layers significantly improves model quality in this regime.
strengths_and_weaknesses: Strengths
1. I found the paper to be very clear and well written. It was easy to understand what the authors were doing and why.
2. The results seem significant. This is not a phenomenon that I was aware of previously, although I am not an expert in the use of deep learning in the small data regime.

Weaknesses
1. Distillation is known to improve quality for a constant parameter count so I’m not convinced that the distillation experiments disprove the hypothesis that the quality gains of adding fully connected layers are from increased parameter count. A more convincing argument is that you’re not increasing the parameter count much because of the dimensionality reduction in your FC stack. I’d encourage the authors to include data on the % increase in model parameters with their proposed addition.


questions: 1. I’d like for the authors to include data on the % increase in parameter count with their mechanism in all experiments, as I stated above.
2. I’m confused by the author’s characterization of ResNet and EfficientNets as fully convolutional. These architectures are typically drawn with a single fully connected layer following the convolutional component of the model.
3. Caltech101 in Figure 3 appears to be a counterexample. I’m wondering if you can explain this more. Your explanation claims you see similar behavior on this dataset but that is not what I see looking at Figure 3c
4. One ablation that would be interesting is only training an FC stack on the target task and showing its quality relative to the results in Figure 3. I am curious if it would outperform on very small datasets and then be surpassed as the dataset grows.
5. I was not able to understand the sentence “We show in our experiments that the network with the train time added fully connected layers still significantly outperforms the original architecture, even if both have an equal number of weights”. I’d like for the authors to clarify this in the text.
6. Some numbers called out in the text are very specific when I would expect them to vary depending on the target architecture. For example, the “512” and “64” in Figure 1 and the “42k” in section 2. I think the text might be more clear if the authors described these dimensions in abstract and stated their exact parameterization in the experiment details.

limitations: The above sections detail limitations/questions I’d like to address.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a training method that involves a module plugged into the final features of a backbone as an additional classification head for joint training with the existing classification head. Along with the original head, the newly added head also pass through the softmax for the cross entropy loss; the training is performed with the summation of the two cross-entropy loss. The proposed module is dubbed Feature Refiner (FR), consisting of two fully connected layers followed by the layer normalizations and operating with the gradient gate (GG) right before the classification head. GG acts exactly like a stop-gradient technique, which is widely used in self-supervised learning, to train the original head only based on the output of the frozen backbone, which is the input for the head, and the backbone is trained only with the extra head with FR. At the inference phase, the original backbone with the original classification head is used for the forward propagation. The authors claim that this training method works well on low-data regimes (for me, it is limited to the low-data regimes), which may not be expected. Some experimental results on small datasets, including the CIFAR datasets and the Caltech datasets for supervised learning, active learning (only on CIFAR), and semi-supervised learning (presumably on CIFAR), are provided to show the effectiveness of the proposed method. The authors try to show the universality of the proposed method of training with some backbones, not constrained on ResNets.

strengths_and_weaknesses: ### Strengths
- This paper is easy to follow. 
- The proposed idea looks somewhat interesting.

### Weaknesses
- The main concern with the proposed method is that it seems to have a very similar training pipeline to SimSiam [1], which showed that using the stop-gradient is a key to training a backbone in self-supervised training. Specifically, Feature Refiner (RF) and the classification heads seem to be the predictor and the projector in the method [1, 2], respectively (the order is reversed but would not be a matter from my standpoint). A difference is at the loss, but as the authors claim (in line 83, p.3), assuming the network is trained in a KD way, the proposed method is a supervised SimSiam (with a single-view ). I hardly agree with the authors' claim that the method performs like a KD,  except for using only a single-view image; the training procedure is very close to SimSiam. Therefore, the authors should argue the difference between the proposed method with SimSiam.
- There is no intuition why the proposed method has the benefit of training with small data.  
- The experimental setup is somewhat unconvincing; the setup is inconsistent and does not follow the authors' claim of requiring pre-training for the method (in line 234, page 9). The authors specify that they use an ImageNet-pretrained ResNet for the Caltech datasets training, only providing a seemingly inappropriate reason.
- All the training dataset is small, so the experimental verification of the claim is limited. Using small data for training and a small dataset is technically different. Therefore, it would be better to justify the proposed method on a larger scale dataset such as ImageNet with a small data regime for a stronger claim.
- Comparison of the proposed method with MLPmixer and ViT-B16 in Figure 4 seems unfair. MLPMixer and ViT-B16 have the stem of performing non-overlapping patchification for the input, so training them with 32x32 size images in the CIFAR datasets degrades the model accuracy regardless of the size of training data.
- As aforementioned, the authors claim the proposed method is a joint KD. However, the loss is not a straight KD-based loss (e.g., the KLD loss), and training a backbone with the extra head may not leverage the KD concept, in my opinion. Can the authors elaborate on the concept?


### Pre-rebuttal comment
This work presents a training method using the stop gradient technique with an extra FC head for model training in a supervised manner. Except for using a single-view image and supervised loss, the overall training pipeline is quite similar to the previous self-supervised methods [1, 2], so the authors should elaborate on the difference and provide any intuition why the proposed method could work well. Another concern is that the experiments are not convincing because of the inconsistent experimental setups and small-scale experiments. Therefore, I am leaning towards rejection but would like to see the authors' response and the other reviewers' comments for my final decision.

[1] Chen et al, Exploring Simple Siamese Representation Learning, CVPR 2021
[2] Grill et al, Bootstrap your own latent: A new approach to self-supervised Learning, NeuRIPS 2020

questions: - Why the ImageNet-pretrained model is used only for training on Caltech datasets?

limitations: - Limitation is provided, but any potential negative social impacts do not seme to be provided.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper shows that one can improve accuracy in the low-data regime by adding fully connected layers to CNNs during training and distilling knowledge to a classification head, resulting in a network with the same number of parameters and better generalization performance. 
strengths_and_weaknesses: I found the method proposed in this paper very interesting and the results showed clear improvements over the base network. Considering how easy this is to implement, I can see this paper having a large amount of impact to the community, and the experiments showing improvements in the active learning and semi-supervised learning setting show the diversity of this approach. However, the lack of knowledge distillation baselines and theoretical backing has me questioning whether this is any different than a typical teach student distillation approach. 

For instance, what would happen if one jointly trained ResNet18 and ResNet50, applied a loss similar to [1]? In this case, the test time network would contain no more parameters, so it seems like a reasonable comparison, and it follows a similar intuition of this paper (train with more parameters and drop them during test-time). Although I am not too familiar with current KD methods, I would be surprised if none of them showed a similar gain in performance, so it would be a necessary baseline to compare this papers method to. 

A lesser concern I have is with overfitting. Since this method seems to be training a network with more parameters on a small amount of data, I wonder if performance on robustness benchmarks would be worse than the baseline network. 

Overall I think that this paper has the potential to have high impact as it is well written and has a simple, easy to follow method that is effective under several tasks. I am giving it a reject as I don't believe the current experiments are sufficient to prove that this method is more advantageous than other knowledge distillation methods, but I am eager to have the authors quell my doubts with a more thorough evaluation. 

[1] Zhang et al. "Deep Mutual Learning"
questions: In figure 6f, why does the accuracy of FR-EfficientNet decrease at 10k samples while the accuracy of EfficientNet increases? It seems strange as this effect seems to only happen at this one datapoint. 

In figure 4b, 5(c,d), and 6(c,d,e,f) the FR method has higher accuracy than the baseline for all amounts of data; did you run experiments where these networks are trained on the entire dataset? Is there a general point at which there are diminishing returns in accuracy? 

**Update**

After the authors provided new experiments to show that their method does in fact outperform knowledge distillation methods in low data regimes, I am satisfied with the evaluation and have changed my review from reject to accept. 
limitations: n/a
review_title: null

Meta-review: metareview: The paper shows that using final fully-connected layers helps the generalization of convolutional neural networks in low-data regimes. The addition of these layers significantly improves model quality resulting in a network with the same number of parameters and better generalization performance.

Initially reviewers had mixed evaluation of the paper. All the reviewers saw that the proposed method is simple and easy to follow, at the same time providing clear improvements over baselines. Also agreed that the results are "significant" and "surprising" effect. There were some concerns raised by the reviewers but the author's rebuttal mostly addressed and improved the paper with sufficiently more experiments and analysis supporting the main claim. Reviewer `DX6o` mentioned that there are few updates promised by the authors which can't be validated until camera ready but it does not seem to warrant block publication. 

The Author-Reviewer discussion period was active and the authors did a great job clearing various concerns and questions and all reviewers agreed to support acceptance of the paper. The paper demonstrates a simple yet effective method for small data regime which would be interesting to the broad NeurIPS audience both for practitioners as well as researchers. 
confidence: Certain
Decision: Accept