Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In a nutshell, authors extend confidence estimation in DNNs to semi-supervised learning. To my understanding, the way they have approached the problem is that the more consistent the prediction for a given data sample x_i across different stages of training (including weights) is, the more confident the model will be; and that is independent of y_i, i.e. consistency is the proxy. To some extent it reminds me of this work (https://link.springer.com/article/10.1007/s00521-019-04332-4) where the authors annotated unlabelled data based on uncertainty estimation. The results presented (and visualised, e.g. figure 1) are promising, but half of the paper (especially the first half) tries to mathematically formulate the problem unnecessarily. Even the theorem might be redundant, with the most relevant being eq. 2.
strength_and_weaknesses: Strengths:

a)	A simple but effective method, that works competitively with respect to other methods.
b)	The authors also show some results in medical image segmentation
c)	In theory, it should generalise well in other datasets

Weaknesses:

a)	I think it is a weak formulation and some quantification of the bias of the estimator would be useful – to be honest, though appendix covers me but it is also not an excuse to use the appendix to write a longer paper
b)	I appreciate the use of medical images but results on more datasets would demonstrate the generalisation of the method, as CIFAR and ISIC2017 might be not the best ones for this task
c)	I think most of the formulas and maths in the first half of the paper, mainly pages 2 and 3 are redundant in my opinion. You want to explain something very simple and the maths just makes it more complicated than it is.


clarity,_quality,_novelty_and_reproducibility: Firstly, in related work pg2, in the second sentence from last, there is a missing citation "?".
Secondly, the paper would benefit from removing some redundant formulations, and instead adding more important ones from the appendix. Especially on pages 2,3.
Thirdly, the idea of expanding confidence in semi-supervised learning settings might be kind of novel and original but some ablation studies to understand limitation and stress test it would have been useful.
Reproducibility is at the bare minimum, as it is only the maths available (and might not be everything needed there), and the fact that they have used Resnet-18 for the active learning and other Resnet variants for other parts of the paper. I would have expected to see a pseudo-code/algorithm at the very least.
summary_of_the_review: Good paper, with some useful intuitions around expanding confidence in semi-supervised settings, but the paper lack proper organisation, useful ablations/experiments, and a more compelling provision of the importance of this work. It would require some moderate improvements to make it publishable, but it is nevertheless an important piece of work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a consistency ranking loss that enables the model's softmax output to be a close approximation of consistency. Consistency in turn correlates well with confidence estimation. In other words, if the model's prediction for a sample is consistent as the training progresses, it is likely to have high confidence in its prediction for that sample. Their method can leverage unlabeled data and thereby works well in limited label settings. They show that their method of estimating confidence outperforms other methods for image classification and image segmentation tasks. 
strength_and_weaknesses: Strength(s):
1. Using consistency as a surrogate for the correctness and the model's confidence is an interesting and novel idea. The experiment demonstrating a high correlation between consistency and confidence was quite convincing.
2.  The authors nicely transferred their learning about consistency being a decent surrogate to confidence into a loss function that enables softmax outputs to be as close to consistency as possible.
3. Their method can leverage unlabeled data and thereby works well in limited label settings. This is again novel as prior works have mainly focussed on the supervised training setting.  
4. Their experiment with active learning was again a clever way to demonstrate their method's superiority.

Weakness(es)/Suggestion(s):
1. It seems that results on CIFAR10 and CIFAR 100 should be supported by a real-world dataset(s) where we actually see some uncertainty in the outcomes like predicting the weather or predicting a future event from the current data.  Recent work in deep probability estimation actually addresses this issue and also proposes some datasets [1]. I feel the proposed method is an interesting one and therefore, should be tried on some real-world uncertainty prediction datasets. It would only strengthen the paper. 
2. Additionally, some existing methods like [1], [2], [3], and [4] for the model calibration should be added to the comparisons. This again would strengthen the paper. If you feel that the above methods would obviously not outperform the proposed method, the reasons for the same should be included in the relevant work section.
3. For figure 3, some details like which dataset was used, how many training samples were labeled, etc. are missing. 

[1] Deep probability estimation: https://arxiv.org/pdf/2111.10734.pdf
[2] Deep ensembles: https://arxiv.org/pdf/1612.01474.pdf
[3] Mix and Match: https://arxiv.org/pdf/2003.07329.pdf
[4] MMCE: https://proceedings.mlr.press/v80/kumar18a/kumar18a.pdf

clarity,_quality,_novelty_and_reproducibility: The method is novel and clearly explained. I was not able to find the code in the supplementary material or the main paper. Therefore, authors are encouraged to share the code for reproducibility purposes.
summary_of_the_review: The method proposed in the paper tackle many important issues like confidence estimation in semi-supervised settings which was not adequately addressed in the previous work. Additionally, the proposed method is sound, novel, and elegant. Therefore, my recommendation would be marginally above the acceptance threshold (6).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach to estimate the confidence of the predictions using unlabeled data in the semi-supervised setup. The main idea is that training consistency, i.e., the frequency of training data getting the same prediction in sequential training epochs, can serve as a surrogate for the model’s prediction. 

strength_and_weaknesses: Strength: 

-The problem of overconfident predictions is important and requires major attention from the community. 
-The paper is well-written and easy to understand.  
-Extensive experiments are conducted to evaluate the performance of the model. 



Weaknesses:
MAJOR:
-In Figure 1, the paper attempts to show that the training consistency correlates with the distance to the decision boundary and then claims that the distance to the decision boundary can be interpreted as confidence. This claim does not seem to be correct. Since the output space is not calibrated, the distance to the decision boundary in the output is also overconfident and cannot be used to represent confidence. We can see this more clearly by noting that the linear classifier that performs the classification on top of the representations is also estimating a relative distance to the decision boundary. Therefore, the overconfidence problem would not exist in the first place if the distances in the representation space were correlated with confidence. 
-Another point is that the distance in the output may correlate with confidence only on the samples that are close to the decision boundary as depicted in Figure 1. This is obvious because we also observe in conventional networks that low predicted probability shows low confidence. The challenge is actually on samples that are distant from the decision boundary. Therefore, the assumptions made based on the observation in Figure 1 are not accurate and only consider the samples close to the boundary while there is no clue or intuition for samples not close to the boundary.
-The novelty is limited. The main idea is the simple adaptation of the correctness measure proposed by (Moon et al., 2020) to the semi-supervised training setup. Other choices for the adaptation could be considered to augment the contribution. 
-L_{corr} in Equation 3 is not defined.  

clarity,_quality,_novelty_and_reproducibility: The paper is almost clear and seems reproducible. However, the novelty and the scientific quality in terms of contributions are limited.

summary_of_the_review: The novelty and contribution of the paper are limited. In addition, assumptions made based on the exploratory evaluations are not accurate and require further justification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Authors propose a new method for uncertainty estimation in image classification and segmentation considering a semisupervised setting. Their approach takes advantage of also unlabelled data, which is a novel aspect in uncertainty estimation. They propose so called consistency loss which forces softmax output to consistence (defined as consistency of predictions between training epochs). The proposed approach is validated numerically using CIFAR10&100, image segmentation (ISIC2017) and in a active learning experiment. 
strength_and_weaknesses: Strengths:
- They take advantage of unlabelled sampled. Having unlabelled samples is a common situation in practical problems and especially very natural situation in active learning problems. 
- Numerical results looks good: they reach state-of-art in most of experiments, often with a large margin
- They also provide theoretical considerations supporting their approach

Weaknesses:
- It seems that the approach is only applicable with networks with softmax output (i.e. classification or segmentation), not clear if can be extended to regression etc?
- It is not clear to me that what are actually the metrics used in active learning methods. I guess "entropy" results is computed from softmax output? "MC-dropout" is also based on entropy? Your method is based on softmax or entropy?

Minor weaknesses / comments :
- Baselines are based on quite old studies (the latest is from 2018). There are also several more recent approach. But on the hand, MCdropout is still one of best performing solutions and none are considering semisupervised setup, so this is also just a minor comment. 
- Furthermore, numerical results considers only a single set such that training and evolution sets are quite similar (i.e. I would say those are "same distributions"). Some confidence studies considers out-of-distribution detection, meaning that they study if method can detect a sample which is from a significantly different distribution (e.g. Dirty MNIST introduced by Mukhoti & Gal, https://paperswithcode.com/dataset/dirty-mnist). I would like to see also this aspects of their method. But can also be future study. 
- Another similar study could be if method can separate aleotoric (uncertainty due to noise etc) and epistemic uncertainty (uncertainty due to lack of training data etc). This is meaningful in active learning if different levels of aleatoric noise are present as it is pointless to pick samples with high aleotoric noise. See Yarin Gal's work. But can also be future study.

Questions:
- Double sum consistency loss is expensive to compute as the authors mention. But can it be avoided? Does the loss sense make any sense without taking only difference of consistency and softmax output  (i.e. computing $\sum_s\max\{0,c_s-\kappa_s\}$?
- In training, does minibatch include fixed ratio of labelled and unlabelled samples or is this ratio random? If latter, is it a problem if no labelled samples end up to the mini batch?

clarity,_quality,_novelty_and_reproducibility: The paper is well written. Clarity and originality are sufficient. 
summary_of_the_review: The approach seems novel and relatively simple, and numerical experiments also looks very promising. There are a few points that I would have liked to be see in their study (see "Minor weaknesses / comments / questions" above), but on the hand, I agree that not all can be included into this paper and those could also be part of future study etc. Therefore I would recommend acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper's main contribution is to use training consistency, i.e., the frequency of a training datum getting the same prediction in sequential training epochs, as a surrogate function of the model’s confidence. While the idea is simple and straight forward, the design of the loss function is non-trivial. This is also the main strength of the paper.
The main weakness of the paper is actually the presentation. Some parts of the paper such as the theoretical analysis are redundant. Instead, the authors are suggested to include those results in the supplementary into the main paper.Ther


justification_for_why_not_higher_score: The paper has sufficient novelty for poster, but it is not an outstanding one for oral. 
justification_for_why_not_lower_score: The idea is interesting and effective. The experimental results, especially after rebuttal with additional results, are sufficient for accepting this paper.
Decision: Accept: poster