Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a loss to relax the assumption of using a fixed k for top-k classification learning. The authors use the existing differentiable sorting and ranking operators. Experimental results also achieve a state-of-the-art on ImageNet.
main_review: Strengths: 
	The motivation of this paper is clear to draw k from a probability distribution for training. 
	The idea of this paper is pretty novel and exciting which makes the classification model robust.
	The extensive experiments conducted on five data sets are sufficient to show the advantages of the proposed idea
Weaknesses:
	The details of the differentiable sorting networks is not represented. How to rank the predicted scores of the final classification layer and get the probability distribution?
	In figure 1, the first row (rank1) are multiplied by 1 and the second row(rank2) are multiplied by 0.5. Please explain the reason.

summary_of_the_review: This paper derives a family of top-k cross entropy losses which is a novel practice. The experimental analysis on ImageNet including the impact of the distribution and ranking set size m, etc, is concrete and sufficient.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses Top-k classification learning. Based on the recent progress on differentiable sorting and ranking, the author proposes a loss function for top-k classification where the k is not fixed but follows a given probability distribution. To improve the efficiency, a splitter selection network is proposed so that fewer layers are required for the sorting network. 

The proposed loss function can be combined with different sorting methods. In experiments, the loss function is shown to be effective in training a model from scratch on Cifar10. It can also be used in fine-tuning on ImageNet dataset and has performance gain. 
main_review: strengths
1. The idea of using different probability distributions for k is interesting. The results also demonstrate the effectiveness of this idea.
2. The experiments of incorporating different sorting methods are comprehensive.


weaknesses
1. In my opinion, the P_K is more like a set of weights, rather than a probability distribution. If it is the case, I recommend improving the descriptions to reduce confusion.
2. It would be nice to present an experiment with conditional probability distributions for k of different classes based on their semantic meaning (like person, animal). I think it is also a significant contribution of this paper.

summary_of_the_review: This paper proposes a flexible loss function for top-k classification, providing useful insights for image classification. So it is worth of reading for the researchers in this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method to employ the benefits of differential sorting methods towards top-k classification learning. The presents several experiments with sampling weights for different ranks and presents the results. The loss is used for fine-tuning in most experiments (apart from the CIFAR100 case). The method seems to give minor improvements on the ResNeXt-101 32*48d baseline. 
main_review: Strengths:
- The efforts on optimizing the top-k classification learning through differentiable sorting appear novel to me. The discussion on differential sorting is comprehensive. The paper specifically discusses each of the options and how it is optimized for the studied scenario. 
- Experiments are thorough 
- The work is also interesting because the performance gains come only because of fine-tuning.


Weaknesses

- The second term and eqn 2 would be constant with k=5 (if only the top five rows of the P matrix are constructed). Expanding eqn2 for the given example in Fig1, the loss would be: -log(0.5 *.03 + 0.5 (0.3+0.6)), assuming Panda is the ground truth class. Consider a case of P_K [0.5 0 0 0 0.5], the equation would be -log (0.5 top1 + 0.5 (top1+top2+top3+top4+top5)). If only five columns are reconstructed and if they are column stochastic, then the sum of top1 to top5 would always be 1. Then the second term will always give a constant value. Requesting the authors to clarify this aspect.

- At first, it appears that the distribution would be a sample. However, fixed distribution is used for a set of experiments. For example, it is either [0.5 0 0 0 0.5] or [0.2 0.2 0.2 0.2 0.2] for the entire experiment. Hence, presenting it as "sampled" is confusing. The best results come when you have the top1 and the sum of the top five values. Hence, the initial discussion and intuition can be improved a bit.

- The improvements on Noisy Student EfficientNet-L2 are negligible. 88.35 to 88.36 is certainly not statistically significant. Were experiments for Table1 were also ran 10 times (like table 2?).

- Please mention the number of rows that were reconstructed for each experiment. The number of columns (m) is mentioned in the experiments but not the number of rows.

- Berrada et al. was used to train the model from scratch. It would be worth comparing their loss for fine-tuning purposes as well. I think that would be a fairer comparison. 


summary_of_the_review:  Although the paper brings several novel perspectives, there remain several ambiguities as well. Some additional experiments, clarifications can also strengthen the draft. Overall, in the current form, the paper is a borderline one and the final decision will depend a lot on the discussion during the rebuttal phase.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a differentiable loss for top-K classification based on differentiable sorting networks, i.e. sorting neural networks in which basic min/max operations are replaced by smoothed versions (i.e. softmax/softmin). The main principle is to use the sorting network to estimate the probability of the rank of each class and then filter only the top-k. An extension consists in considering that k can take several possible values at random (e.g. 50% chance of being 1 and  50% chance of being 5). 

The resulting loss is experimented on three datasets (CIFAR-100, ImageNet-1K and ImageNet-21K-P) and with three existing sorting networks. Performances are mainly compared to cross-entropy showing low improvements. 
main_review: Strengths
- Set-valued classification is an important topic to cope with class ambiguity. Few works (only one as far as I know [1]) have proposed a top-k loss for neural networks and there is room for improvements 
- The proposed approach is different from [1] as it relies on sorting networks to determine the set of the most likely classes rather than a purely top-k objective

Weaknesses
- A first weakness is that the contribution is quite incremental and not well justified from a theoretical point of view. Using sorting networks for top-k is an acceptable strategy from a practical point of view but a bit over-kill and not very new from a theoretical point of view. The proposal to use several values of K is also not really justified. The principle of top-K is to predict sets of fixed size contrary to other set-valued classification approaches that attempt to solve other objectives (e.g. adaptive set sizes but equal to K on average). We let the authors refer to [2] for a clear overview of the different objectives. Here, the objective is not really clear. If K is supposed to be a random variable (e.g. 50% chance of being 1 and  50% chance of being 5), that means that for the same image x, the classifier is supposed to return randomly either one class or 5 classes without any consideration with regard to the image content itself. 
- another main weakness is that no significant improvement of the proposed loss over cross-entropy is shown. The reported top-K accuracy gains are not systematic and so low that they may be not statistically significant. As a first step towards a better understanding of the results, the authors should first compute some significance tests (e.g. p-values on several runs and a clear cross-validation procedure for model selection among epochs). But even so, it won’t resolve the fact that the performance gain is observed only for some specific configurations (e.g. a specific sorting network and specific values of K probabilities) and remains very low even in such advantageous conditions. 

[1] Berrada, L., Zisserman, A., & Kumar, M. P. (2018). Smooth loss functions for deep top-k classification. arXiv preprint arXiv:1802.07595.
[2] Chzhen, E., Denis, C., Hebiri, M., & Lorieul, T. (2021). Set-valued classification--overview via a unified framework. arXiv preprint arXiv:2102.12318.

summary_of_the_review: An interesting attempt to improve the top-K classification but consistent limitations:  
(I) an incremental contribution and no clear justification of considering k as a random variable
(ii) no significant improvement of the proposed loss over cross-entropy
review_title: null

Meta-review: The main consensus among the reviewers was that although the approach is interesting, this submission suffers from two main weaknesses:

- The methodology is not very novel, and the proposed parts of the method not well justified (in particular regarding the interplay of a differentiable sorting approach and of the random choice of k)

- The results, compared to a standard cross-entropy loss are not very convincing: there does not seem to be a statistically significant advantage.
Decision: Reject