Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In general, this paper proposes to use the label smoothing method to calibrate the model in open intent classification problem, based on two previously used method logit classifier (LC) and adaptive decision boundary (ADB). The result in low KLR (Known Label Ratios) shows the superiority of the proposed method over previous methods, while the result in high KLR does not. Extensive experiments are taken to demonstrate that the proposed method can achieve improvements in inductive bias, representation landscape and open intent classifier performance. 
strength_and_weaknesses: **Strengths**
1. The proposed method is clear and simple;
2. The analysis and experiments are somewhat comprehensive.

**Weaknesses**
1. The authors claim the benefit of calibration by the proposed method, but fail to show an evaluation of the calibration performance in terms of expected calibration error (ECE).

2. The experiment evaluation should be further strengthened. The following related prior works are missing and should be compared.

a. Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog, AAAI 2020

b. Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models, EACL 2021
clarity,_quality,_novelty_and_reproducibility: see above.
summary_of_the_review: see above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the open intent classification problem. There are standard supervised learning approaches for this task, but the authors hypothesize that cross entropy based objective in the existing open intent classification problem leads to predictions which are overly biased towards known intents, leading to poor performance on open intents. It has been presented with empirical evidence that changing a learning objective in a more calibrated manner can lead to better performance and outperforms the existing state-of-the-art. Authors also investigate the reason for the supremacy of the calibrated model and highlight its connection with higher layers in the model.
strength_and_weaknesses: Strengths

S1: It is an interesting idea to utilize model calibration towards open intent classification.

S2: The proposed approach has been applied to text and image domain state-of-the-art methods.

S3: The paper presents a detailed set of experiments.

S4: The paper is easy to follow, but writing needs improvement.


Weaknesses

W1: The proposed approach has been empirically verified on two state-of-the-art methods, which are Adaptive Decision Boundary (ADB) and Logit-based classifier (LC). It could have been interesting to see if the approach generalizes to a few others out of domain intent classification methods.
W2: It will be interesting to investigate the behavior of other calibration methods like temperature scaling with the proposed approach. This will add the element of completeness in the research question being investigated.

W3: Logit-based classifier should be one of the baselines.

W4: There are various grammatical mistakes and typos in the paper.  i) For example, in section 4.3, “we assume the adequate calibration strength will be differ from the C-LC’s optimal one.”  ii) Similarly, in section 4.2 in results section, “we resulted that calibration itself is beneficial”. iii) Typo ⇒ “optinmal calibration sterngth”

W5: Figure 1 has enough empty space which can be utilized for increasing the subplot sizes. Especially, legend is difficult to read.

clarity,_quality,_novelty_and_reproducibility: This paper presents an interesting dimension to investigate the benefits of model calibration in open intent classification problem. For establishing concrete results, at least one other calibration method should be used for drawing the conclusive evidence that indeed model calibration incorporates the inductive bias into open intent classification setup. One more out of domain intent classification method can be used in addition to ADB and LC.
summary_of_the_review: Overall, this paper investigates an interesting question. For the thorough investigation and establishing the claims, more experimentation is required.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes two open intent classification methods based on the label-smoothing method, C-LC, and C-ADB. More specifically, the label smoothing method takes a weighted average of the ground-true label and the uniform 1/k vector. The experiments 
evaluates the proposed methods on real-world datasets with previous baselines.
strength_and_weaknesses: Strength:

The main idea of the paper is clear. The experimental part is solid with sufficient empirical results and analyses.

Weaknesses:

About the Novelty: the proposed method is more like an empirical trick or a modification to the previous baseline LC and ADB. The motivation and justification to use the label smoothing trick remain unclear. 

About the Experiments: The proposed methods did not significantly outperform the previous baseline. On the OOS dataset, the proposed C-ADB even got worse performance than the original ADB, which makes the effectiveness of the proposed method unconvincing.
clarity,_quality,_novelty_and_reproducibility: The clarity is good. The quality and novelty are not fair enough. The experiments seem reproducible.
summary_of_the_review: The method described in the paper is more like an empirical trick than a well-designed method. The motivation and justification for the method are not clearly stated. The experimental results did not show significant improvement when applying the proposed method.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper presents an approach to open intent classification based on calibration measures. There is an agreement that the paper presents the ideas clearly, and that the problem addressed is interesting. However, there is also an agreement that the weaknesses outweight the strengths of the paper.

Thee is a consensus among reviewers that the main weaknesses stem from a limited novelty and missing comparisons. While the rebuttal provided some promising elements of response, the main questions remain (see e.g., Rev HPLr's reply).
justification_for_why_not_higher_score: A more complete set of baselines and comparisons (or more compelling arguments of why such comparisons are not provided) would be a necessary step to obtain a higher score.
Decision: Reject