Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper designs and studies regularization effects for different types of output layers on training deep neural networks, which is very useful in practice. They propose five output layer designs and conduct various experiments including a case study. The results show that their proposed designs can achieve improvements of up to 10% on evaluation datasets. Overall, this paper is well-written and easy to follow. 
main_review: Pros:
1) The authors propose two important factors, neuron dependency and expressivity, for  measuring the regularization effect on different output layer designs. 
2) The authors conduct various experiments and a case study to validate their designs and the results show up to 10% improvements in accuracy for validation datasets. 

Cons:
1) In table 1, I wonder why random fixed layers can achieve such high performance in accuracy, even better than baseline. Is there any explanation? 
2) It is not very clear to me why Gradient⊙Activation can measure the dependency. The authors should give a brief discussion/explanation here. 
3) The novelty of this paper is somewhat limited. The proposed output designs seem very similar to existing approaches, e.g., w_random - random project, w_sparse - dropout.
4)  What is the difference between w_spare and dropout regularization? What is the difference between w_ensemble and simultaneously training multiple classifiers?
5) Figure 2 can be improved as it is a little bit hard to tell the difference by color. I recommend using different groups of colors or different types of line styles. 

limitations_and_societal_impact: This has been discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper investigates the regularizing effect of different output layer designs in deep neural networks. The authors propose several different design choices of the output layers, which are all very simple but effective as the authors demonstrated in the experiments. They also propose two terms named neural dependency and expressivity between the neurons and classes. With these simple designs of the different output layers, the experiments on multiple classification tasks and resnet/densenet CNN models improve the performance for a large margin, which helps avoid the overfitting problem. 

The main contributions:
1. Propose simple yet effective output layer designs for deep neural networks.
2. Introduce neural dependency and expressivity factors to measure the overfitting problem.
3. Experiments on different classification tasks are expressive. 
main_review: This paper studies the overfitting problem and the regularization methods for deep neural networks. Different from previous work, the authors work on the output layer and specifically design different output layers for regularization. They introduce five different simple methods of the output layers. Easy to follow and interesting analysis, the introduced two factors are also some evidence for their analysis. The experiments are also express to see the improvements. The paper is also very clear to follow. 
However, there are also some questions and concerns:
1. The alpha in the scaled output layers is designed to smooth the output distributions as I understand, though the authors connect to the dependencies. Therefore, it is expected alpha has huge effects on the model training, can you plot the curves of the performance change? Btw, the choice of alpha also depends on the different datasets since the data (class) distribution of each task should be different. It is expected to see the comparisons.
2. Similarly, the choice of q is also interesting, could the authors also study the effect of q? Too many zero connections hurt the performance badly. For sparse fixed layers, could you give a clear comparison between this to the dropout and dropconnect method? Strong relationships, is sparse version a static version of dropout?
3. Could you give some clear explanations or visualizations about the 1to1 layer? What's the most important impact of this design? At least, this does not change the output distributions.
4. One major weakness is that the method is only compared with the basic resnet and densenet models, and also the basic training, without comparison to advanced methods/models. This significantly reduces the importance since this paper tries to sell the output design, which should be applied together with other stronger models and regularization methods to see the importance and effectiveness. 
5. Final minor one, In table 2, IN-top1 is only 76.36% for baseline. However, it seems that the Resnet-50 can achieve about 77.15% as shown in https://paperswithcode.com/sota/image-classification-on-imagenet. Is there some difference?

Overall, I like this paper and am pleased to improve the score with answers. 
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: Usually a fully connected layer is used as the final layer of an image classification network. To reduce the overfitting of this layer, this paper studies the use of activation scaling, fixed randomization, sparsity and in-layer ensembling as a regularization technique.
This paper also introduces neuron dependency and expressivity as two factors contributing to overfitting and proposes ways to measure these factors.

main_review: - There are no evaluation results on the popular classification benchmarks such as ImageNet, CIFAR10 or CIFAR-100. Without results on these benchmarks it’s hard to know if the proposed method gives additional benefit in comparison to the current state-of-the-art regularization methods. Specially, it is important that the paper includes results on the large scale dataset of ImageNet as the results may change as we have more training data.
- The current-state-of-the-art results on the provided benchmarks is much higher than the reported results in the paper. It would be helpful if authors provide results on top of stronger networks.
- By adding more regularization to the previous layers of the network, we may need less regularization for the output layer. I think authors should also provide analysis for their proposed method when they apply it on top of other common regularization and also data-augmentation methods.
- The writing of the paper is not clear and needs improvement.
- Table 5 provides a comparison to the dropout and shows that ensemble method which builds multiple heads on top of the network has a better performance in comparison to the dropout. Are the proposed ensemble method and dropout orthogonal and can we combine them together?

limitations_and_societal_impact: There is no discussion on potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This experimental work studies various sparse output-layer types for transfer-learning. Authors report interesting gains through usage of certain learned/fixed output layers. Although I found the experiments interesting and believe that they would spark more research on the topic; the story and results needs a little bit more work.
main_review: # Strengths 
- The correlation between neuron saliency and results are very interesting and novel as far as I am aware.
- Usage of sparse output layers for transfer also novel as far as I know. 

# Limitations/Questions/Recommendations
0. It seems like authors use weight decay value of 0.0001 (seems to be same for backbone and output layer). This choice affects the performance of dense output layers greatly. Therefore needs to be searched (possibly only for the output layer). It would be great to have a curve showing the performance of dense baseline output layer with different regularisation coefficients on x-axis.
1. I think the "scaling head" corresponds to using a temperature parameter in softmax (https://en.wikipedia.org/wiki/Softmax_function). Assuming that it is case, I think this connection must be clear and this variant should be studied under "loss type" used, as this is one parameter one can pick during finetuning (for all different architectures).
2. Figure 2 didn't really helped me to understand the terms introduced neuron dependency/expressivity better. The formulas presented later did though. I recommend authors to use the definition earlier. More importantly I think authors should use existing terms instead of introducing new ones. "neuron dependency" means "saliency" (as cited in text). Similarly I think neuron expressivity measure can be captured with 'per-class saliency' or "class-saliency". I think the introduction can be shorten significantly with these changes helping reading to focus on main results. 
3. Can the ensemble connections be represented by a single sparse layer? It would be nice to discuss connections between random sparse layers, 1-to-1 layers and ensemble layers. To me the later 2 seems to be a specific case of a random sparse layer. Furthermore it is not clear why authors choose to report values for 90% for the sparse layers, despite the fact higher sparsities seem to give better results. 

## Minor Points 
- It would be better if References to accuracy gain (i.e. 10%) in abstract and intro has more detail in them like up and downstream datasets.
- Having 3 input neurons in Figure-5 for 1-to-1 and ensemble layers can make them easier to understand.
- Personally I find the p-scores confusing compared to confidence intervals or variance; which are used more commonly in ML (from what I read/see). Is there a reason this is preferred. If so, I think it would be nice to have a sentence on this in the main text.

## After Rebuttal
I like to thank authors for their response, especially for spending the time to create the weight decay ablation. It is helpful to see and I recommend authors to use the values found in the next version of the paper, even if the conclusions are not changing. Another recommendation is to have an l-1 penalized dense output layer baseline. 

I decided to keep my score as it is, which is a "weak-accept". I think the paper would become a clear accept with further experiments (i.e. more backbones/down-stream tasks like transformers and NLP) and ablation studies (like different levels of sparsity). 
 
limitations_and_societal_impact: Authors talk about the limitations at the end of the paper, which I appreciate.
review_title: null

Meta-review: This work experiments with various strategies for improving addressing overfitting and regularization of the output layer. In particular, the authors explore ideas such as activation scaling, fixed randomization, sparsification, and ensembling as regularization techniques.This paper also introduces two methods, neuron dependency and expressivity, for measuring the degree of overfitting. The authors tested their ideas out on various image classification tasks including STL-10, CUB-200, Cars-196 and Food-101.

The reviewers were interested in the results and the techniques introduced for measuring overfitting. However, the reviewers also identified several concerns about the thoroughness of the experiments, and encouraged the authors to perform additional experiments and ablations to add more evidence for the viability of the methods. Additionally, no reviewer stepped forward to strongly advocate for the acceptance of this work. For all of these reasons, this paper will not be accepted at NeurIPS, but the authors are heavily encouraged to perform additional experiments and submit this to future venues.

Decision: Reject