Uncovering the Effectiveness of Calibration on Open Intent Classification
Authors: Hyejin Won
Kyung Ho Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: open intent classification
model calibration
label smoothing

Abstract: Open intent classification aims to simultaneously identify known and unknown intents, and it is one of the challenging tasks in modern dialogue systems. While prior approaches are based on known intent classifiers trained under the cross-entropy loss, we presume this loss function yields a representation overly biased to the known intents; thus, it negatively impacts identifying unknown intents. In this study, we propose a novel open intent classification approach that utilizes model calibration into the previously-proposed state-of-the-art. We empirically examine that simply changing a learning objective in a more calibrated manner outperforms the past state-of-the-art. We further excavate that the underlying reason behind calibrated classifier's supremacy derives from the high-level layers of the deep neural networks. We also discover that our approach is robust to harsh settings where few training samples per class exist. Consequentially, we expect our findings and takeaways to exhibit practical guidelines of open intent classification, thus helping to inform future model design choices.

Under review as a conference paper at ICLR 2023
UNCOVERING THE EFFECTIVENESS OF CALIBRATION ON OPEN INTENT CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Open intent classification aims to simultaneously identify known and unknown intents, and it one of the challenging tasks in modemn dialogue systems. While prior approaches are based on known intent classifiers trained under the cross- entropy loss, we presume this loss function yields a representation overly biased to the known intents; thus, negatively impacts identifying unknown intents. In this study, we propose a novel open intent classification approach that utilizes model calibration into the previously-propee examine that simply changing a learning objective in a more calibrated manner outperforms the past state-of-the-an We further excavate that the underlying rea- son behind calibrated classifier's supremacy derives from the high-level layers of the deep neural networks. We also discover that our approach is robust harsh settings where few training samples per class exist. Consequentially, we expect our findings and takeaways exhibit practical guidelines of open intent classifi- cation, thus helping to inform future model design choices.
state-of-the-art We empirically
INTRODUCTION
Background and Motivation Beyond the success of intent classification under the supervised regime, one of the next challenges in the modern dialogue system is open intent classification (Scheirer et al.] 2013). While the number of intents in the training and test sets is the same un- der the supervised setting (known as closed-set setting), an intent classifier in the real world is required to recognize unknown intents well as known intents (Zhang et al. 2021). For example,
supposing the training set includes N intents, the open intent classification solves N + 1 classifica-
tion where the N + th intent is a set of unknown ones (Shu et al.) 2017 Lin & Xu 2019 iZhang itt al. (021). This open intent classification task is also related to open world recognition (Bendale & Bould 20166] Vaze et al. 2021) or out-o--distribution detection studies (Hendrycks & Gimpel. 2016 Liang et al. 2017) which are actively dealt with image domains, but it is specifically denoted open intent classification in natural language processing domain.
Upon previously-propose open intent classification methods, we figure out that most of these works conventionally trained the closed-set classifier with cross-entropy loss (Bendale & Boult 2016 Hendrycks & Gimpel| 2016 Prakhya et al.] 2017 Shu et al. 2017 Lin & [2122099//aha et al, 2021]. However, we doubt whether this use of cross-entropy loss is the utmost learning objec- tive for identifying open intents. Previous open intent classification study highlighted that adequate strength of decision boundaries among known intents is important for detecting unknown intents (Zhang et al., 2021). To interpret, an inductive bias established with known intents should be neither overly biased nor too loosely optimized. Not only in open intent classification but aecentlyppopposee state-of-the-art open world classification study in the computer vision domain also supports this proposition: acquiring adequate representation power correlates to effective open world classifica- tion performance (Vaze et al. 202I). But, several works once pointed out, the cross-entropy loss is known to convey an inductive bias that is excessively biased to the given labels because it enforces the model to select 2016). To this end, we assume the use of cross-entropy loss has room for improvement and aims to provide outperforming open intent classifier.
single label among the given label space (Recht al.) 2019 Zhang al.)
Main Idea and Its Novelty Our work's key proposition is utilizing model calibration during the model training known intents. Model calibration is method that adjusts model's predicted

---

Under review as a conference paper at ICLR 2023
probabilities of outcomes to reflect the true probabilities of those outcomes (Nixon al. 2019
Referring to the calibration studies, the calibrated deep neural networks accomplished robustness against various noises and perturbations (Mäller et al.) 2019 Pereyra et al., 2017). Inspired by this finding, we presume that applying calibration to the cross-entropy loss will improve the induc- tive bias's quality and escalate the open intent classification performance. Accordingly, state-of-the-art open-world classification methods in the text and image domains and simply ap- ply calibration to their training procedure. Throughout our work, firstly showed whether the calibration improves inductive bias compared to the cross-entropy loss. Then, we further examine whether our simple idea can outperform previous open intent classifiers in various problem settings and how calibration changes the representation landscape in the trained model. Although our idea seems to be simple, we highlight that the proposed open intent classifiers are novel because, to the best of our knowledge, our approach is the first attempt to utilize calibration to improve open intent classification performance in the text domain.
select
Key Contributions
As preliminary analysis, we show that model calibration reduces the bias of the con- ventional known intent classifier, as well as escalates the distribution discrepancy between known and unknown intents. We analyze that this large discrepancy would contribute to better open intent classification performances
We propose two novel methods in open intent classification, C-LC and C-ADB, by applying model calibration to the previously proposed state-of-the-an in image and text domains, respectively. Under the particular settings, we discover the proposed methods become new state-of-the-an
We further scrutinize that the supremacy of C-LC and C-ADB derives from the representa- tions at higher layers of the deep neural networks. We interpret that the proposed methods acquire better contextual understandings than the previously-proposed methods.
Lastly, we examine our approaches' stability in extreme settings of the training set. We discover that C-ADB is less stable than C-LC given few training samples per known intent; thus, there should be careful considcration on using C-ADB.
2 RELATED WORKS
Open Intent Classification At first, Scheirer et al. (003)) defined the task of open-set recognition in the computer vision domain and inspired substudies. Fei Liu (006) applied SVM with center- based similarity to solve open set classification. Bendale & Boult (016) preposed OpenMax model, which fits Weibull distribution in the penultimate layer of the network. [Prakhya et al.) (2017) and Shu et al. (2117) adopt the OpenMax model in open intent classification and show that convolutional neural networks are good feature extractors in NLP domain. [eendrycks Gimpel| (016) suggest that predicting cattof-diitrrbuutio example can be distinguished based on softmax probability. Sub- sequently, post-procesd post-prccessingbbee based methods were proposed. [ii & Xu (2019) apply Large Margin Cosine Loss to the post-processing-basee method; the model learns that it maximizes inter-class variance and minimizes intra-class variance. /Zhang et al. (202T) introduce learning adaptive deci- sion Monndar((ADB)) and centroid for open intent classification, which is a post-processing-beee based method. (hhu et al. (2021) use several data augmentation strategies to expand distribution shift ex- amples on ADB. Throughout the prior works, we analyze that key takeaway of the precise open intent classification method is establishing adequate decision boundaries among the known intent samples; it is also usually denoted as appropriate tightness of decision boundaries (Zhang et al., 2021]. As the aforementioned methods commonly employed cross-entropy loss to train the known intent classifier, we hypothesize it is not advantageous in establishing good decision boundaries. Under this motivation, aim to scrutinize optimal decision boundaries' tightness through applying model calibration to the know intent classifier.
Model Calibration Calibration reflects ground truth correctness likelihood in predicted class la- bel. Good calibrated confidence provides suitable information on why the neural network prediction is made. Guo et al. (2017) proposed temperature scaling to calibrate modern neural networks with over-confident problems. Lee et al. (2017) suggests two additional terms on the original objec- tive function for detecting out-of-distribution. Research of calibration has been widely examined
2

---

Under review as a conference paper at ICLR 2023
for computer vision, but natural language processing has only recently begun. Kumar& & Sarawagi (2019) presented that neural machine translation models are miscalibrated, Desai & Durrett (2020) tried temperature scaling on BERT and RoBERTa model, and analyzed their calibration over three tasks. In addition, they show that further increasing empirical uncertainty help out-of-domain clas- sification. Moreover, Müller al. (2019) shows label smoothing behaviors while training network and investigates the effect of label smoothing, which improve model calibration. Among various methods in model calibration, we employed label smoothing in the proposed methods for the fol- lowing reasons: 1) it does not require any validation set, 2) it directly influences the representation power of deep neural networks while temperature scaling does not.
OUR APPROACH
This section illustrates detailed descriptions of the proposed open intent classifiers. First, among numerous studies on open-world classification, we selected two tate-offthe ofate-of-the-ar ti--hhe-attt methods for image and text domains. We selected the method proposed in Vaze et al. (202I) from the image domain denoted as Logit-based Classifier (LC), whereas Adaptive Decision Boundary (ADB) (Zhang et al., 2021) method is employed from the text domain. Note that both approaches utilize cross-entropy loss during the model training. To calibrate these models, we apply label smoothing (Szegedy et al., 2016) with the calibration strength of a and denote them as Calibrated Logit-based Classifier (C- LC) and Calibrated ADB (C-ADB), respectively. We hereby highlight that temperature scaling approaches (Guo et al. 2017). which are other promising calibration methods, are not considered in our work because they only change absolute logit values without any change in inductive bias. We formalize the cross-entropy loss with label smoothing in equation [ Note that K implies the number of known intents, a stands for the calibration strength, Pk and yk means logit vector and label-smoothed ground truth, respectively.
e:
7 wk
LL y)
yk L log(ps) where Pk:
K Ei=
ex
2k
=vc(1 - a)) ax)+ +a/K (1)
Calibrated Logit-based Classifier (C-LC) C-LC is open intent classifier that shares the same motivation with the I.C, recently-proposed open-world classification method (Vaze et al.) 2021).
Its novelty exists in the calculation of confidence, because it illustrates that simply using the logit vector before the softmax layer can surprisingly increase the open intent classification performance, while most prior works used logit vectors after the softmax layer. During the training procedure, trains closed-set classifier with cross-entropy loss. Given a test sample, it LC recognizes unknown intents if a given sample's 's confidence yielded by the trained model is smaller than a preset threshold. Note that this confidence is measured as the maximum value at the logit vector extracted from the layer right before the softmax activation. Lastly, it selects the threshold as a mean confidence at the training samples. On the aforementioend procedures of LC, we establish C-LC by changing the learning objective from simple cross cntsroyy loss into the calibrated one, shown in Equation]]
LC-ADB = LLS + Louunday
(2)
Calibrated ADB (C-ADB) C-ADB is another open-intent classifier proposed in study. The original ADB identifies unknown intents if a given sample locates far from the known clusters' centroids in the latent space. To empower the model to find adequate decision boundaries' tightness, ADB trains the model with cross-entropy loss and boundary loss, where the boundary loss aims to predict the radius of each known intent. We denote this training procedure as post-processing. Given the trained model Φ(x; f), each known intent's 's centroid (C₁ ... Cx) and radius (R1 ..Rx) in the latent space, ADB measures the distance (dg) between given sample's 's representation and th cluster's centroid (C&), and it is denoted as [|0( 0) Ciel. The ADB identifies given test sample as known intent if dę goes smaller than Rq. If every de goes larger than corresponding CR. it regards it as an unknown intent. We say this open intent classification procedure post- processing. While the LC simply predicts a given sample as an open intent when its confidence is less than preset thresdhold, we hereby highlight that ADB utilizes post-proceesor which utilizes estimated distances in the latent space. On the aforementioend establishment procedures of ADB, calbrate the model with applying label smoothing toward the cross-entropy loss as shown in
3

---

Under review as a conference paper at ICLR 2023
Equation 2 We presume calibrated pre-processing procedure and post- prosesrocessing would make
synergy on identifying unknwn intents.
4 EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Dataset and Problem Setting We utilized three public datasets (STACKOVERFLOW (Xu et al, 2013), BANKING (Casanueva al. 2020). OOS (Larson et al. 2019)) which widely utilized in past open intent classification study (Lin & Xu 2019 et al.] 2021). )). We show a brief summary of these datasets in Table Upon these datasets, we postulate three problem settings with different Known Label Ratios (KLR). The KLR implies the ratio of known labels to the total number of labels. Supposing the scenario under the KLR of 25%, we use 25% of total intents as known ones while the other 75% intents are set as unknown ones. Thus. the open intent classifier can only learn 25% of the total intents during the training stage and identify both known and unknown intents (75% of total intents) during the test stage. Our study utilized three KLRs of 25%, 50%, and 75% in the experiments. We utilized accuracy and Fi-score on the test set as evaluation metrics. Given N intents for the known label, the open intent classifier solves N + 1 classification where the added one label implies unknown intents. We report the average performance over five runs of experiments for each known class ratio.
Implementation and Training Details Both C-LC and C-ADB utilizes pre-trained BERT provided in Huggingface as a backbone fea- ture extractor Desai & Durret (2020) once urged that transformer-bases language models are more advantageous open intent classifica- tion. Given a pre-trained BERT, we implement both C-LC and C-ADB by applying calibration to each one's learning objective. For training details, we train these models with batch size of 128, optimized paramters with Adam W optimizer (Loshchilov & Hutter 2017). We set the learning rate as 2e-5 and schedule it with cosine scheduiling. We Please refer to the supplementary materials for codes and more details.
Table Descriptions of the utilized datasets. Note that the OOS dataset has 150 intents in the training and validation set, but one additional in- tent exists in the test set.
STACKOVERF
BANKING
OOS
Number of Intents
201
[50K+1)
Training Set Validation Test Set
9003
12000
15000
1000
3080
6000
5700
Baselines We employed five baseline open intent classification methods to examine our approaches' effectiveness. Brief elaborations on the baseline approaches described as follows. MSP Hendrycks & Gimpel (2016) proposed that the -ftto--diittiiutionn example can be diverged based on the maximum softmax probability when predicting the sample class. DOC (Shu et al. (2017) applied DOC the CNN layer to open intent classification and m 1-vs-rest sigmoid classifier for m known classes instead of the softmax operation the final layer. OpenMax(Bendale (endale & Boult (2016) is computer vision open-set detection method of based. It applicd extreme distribution of distances from the mean in each class distinguish open set examples. Deep Lin & Xu (019) replaces softmax loss with Large Margin Cosine Loss(LMCL) in a feature extractor model (Bi-LSMM) and feeds these features to the Local Outlier Factor(LOF) for discovering open intent examples. LC Vaze et al. (2021) utilizes closed-set classifier's logit values for open intent classification. It claims that a well-trained closed-set classifier's ' inductive bias is sufficient to identify open-world samples. ADB Zhang et al. (202T) proposed the adaptive decision boundary, a post-processing method, to create quality feature representation. Decision boundaries and centroid of each pre-leameed class are required for open classification.
4.2 Do CALIBRATION REDUCE THE BIAS OF CROSS-ENTROPY Loss?
Setup We first and foremost scrutinize whether the calibration contributes to discriminating un- known intents from the known labels. As we presume that the use of cross-entropy loss yields large bias to the known intents, we analyze whether the cross-entropy loss indeed exhibits insuf- ficient discriminative decision boundary between known and unknown intents. Moreover, also aim to validate whether the calibration establishes a more distinct representation between known and unknown intents. To excavate answers to these questions, we trained closed-set intent classifiers
4

---

Under review as a conference paper at ICLR 2023
with various calibration options: No calibration (label smoothing with strength of 0) and label smoothing with the strength of 0.2, 0.5, 0.8, and 0.9. Note that the larger strength implies stronger calibration the model. Given the trained classifiers, at the layer before the softmax activation) from known and unknown test samples and analyze their distributions. We regard the more distinct distribution between known and unknown intents casts better representation quality, because it implies the model posits more appropriate discriminative decision boundaries We perform the analyses on three datasets under the 75% KLR, and visualize the confidence distributions in Figure Please refer the supplementary materials for experiment results under the other KLR options.
extract the confidence scores (extracted
Result Following the results shown in Figure we observe that cross-entropy loss exhibits par- ticular amount adjoined area between known and unknown intents. Regardless of dataset types, closed-set classifier poses particular amount of adjoined confidence distributions between known and unknown intents; thus, we conclude that cross-entropy loss bears improvement avenue for more effective open intent classification. We further scrutinize that calibration separates these two dis- tributions by establishing a representation less biased to the known intents. Interestingly, the more strong calibration exhibits larger discrepancy between known and unknown labels in every KLR. While we could not confidently say the stronger label smoothing creates a better representation for discriminating unknown intents, resulted that calibration is beneficial to establishing discrepancy between known and unknown intents regardless of KLR levels.
OF (c) 0.S
(a)
(b) 0.2
(d) 0.8
(e) 0.9
Figure 1: Distribution of maximum value of logit between known intent (blue) and unknown (or- ange) intent samples under KLR of 75%. From top bottom, each row indicates the dataset type of STACKOVERFLOWW BANKING, and OOS. From left right, each column indicates calibra- tion strengths of 0, 0.2, 0.5, 0.8, and 0.9. Note that blue and orange distribution implies known and unknown intents, respectively.
4.3 WHAT IS OPTIMAL CALIBRATION STRENGTH?
Setup After we confirm the calibration as a presumable solution for the cross-entropy loss's bias, then excavate what is an optimal calibration strength on C-LC and C-ADB. As the C-ADB has additional post-processing procedure, we assume the adequate calibration strength will be differ from the C-LC's optimal one. To examine this question, we train C-LC and C-ADB along with various calibration strengths, and check the open intent classification performances. We regard an optimal calibration sterngth as the one acheive the best performance. The experiment results are shown in Table
Result From Table 2 shows that each C-LC and C-ADB has different adequate calibration strengths. We interpret this phenomenon highlights the importance of adequate decision bound- aries' tightness, that they should not be overly tightened for good open intent classification per- formance. First, C-LC's performance maximized at the strongest calibration strength because closed-set intent classifier without any calibration does not tighten decision boundaries. As conven- tional cross-entropy loss is insufficient in creating tight decision boundaries, it requires substantial
5

---

Under review as a conference paper at ICLR 2023
impact distinguishing known intent clusters for good performance; thus, applying the strongest calibration was beneficial in elevating open intent classification performance. On the other hand, C-ADB's performance is maximized moderate calibration strength. We analyze this phenomenon occurs because the original ADB already bears tightened decision boundaries compared to the L.C. the ADB tightens the decision boundaries with post-processing procedure. Suppose we apply the strongest calibration for C-ADB. In this case, we analyze the decision boundaries would become overly tightened as both post-processing procedure and calibration simultaneously tighten the de- cision boundaries. As the prior study (Zhang et al. (02T) once urged, overly tightened decision boundary degrades the open intent classification performance. We presume a combined influence from post-processing procedure and strong calibration creates extreme tightness of decision bound- aries, which leads to inferior performance. Conversely, we presume an adequate tightness of deci- sion boundaries is accomplished by combining post-processor and moderate calibration; therefore, C-ADB with moderate calibration strength achieves the best performance. In a nutshell, we discover that C-LC and C-ADB requires different calibration strenghs for effective open intent classifcation. Furthermore, we confirm that open intent classifier indeed requires an adequate decision boundaries' tightness level and re-assured that everly-tightene decision boundary degrades the performance.
Table 2: Our C-LC and C-ADB's open intent classification performances under various calibration strengths. Each method acquires different calibration strengths for the best performance.
Dunaset Smoothing
STACKOVERFLEE
STACKOVERFLOW
BANKING
KLK
Accuracy
[1 Scure
Accuracy
score
Aucurac
[]] scure
Accuracy
Accuracy
scure
Avcuracy
score
0.2
69125
45:20
76.95
30.13
57.19
62.44
39.395
83:35
$2.54
7.4.48
91.11
R1.04
25%
0.8
0.5
72.41
90
$4.05
65.65
73.89
61.15
63.18
57.78
58.
58.82
66.03
65.11
90.07
15.45
83.85
19:77
82.02
83.58
73:46
75.46
91.65
91.52
81.45
80.82
0.9
84.55
71.82
83.69
68.59
90.12
71.11
13.69
15.61
75.71
67.63
90.98
79.52
0.2
76.12
05:04
73.105
02.76
79,22
501.2
37.09
R5.31
$1.32
31.43
S9.23
B4E11
50%
0.5
76.51
69.01
71.33
79.08
75.79
75.31
68.61
$1.45
64.00 64.98
72.70
87.10
83.04
85:47
$1.66
50.85
81.14
79.94
58.23
K3.54
84.15
77.34
76.55
83.33
57.56
70.17
71.95
77.66
68.72 72
73.92
N3.99
77.90
70.45
73.27
75.81
79.30
83.04
75.13
$3.90
84.06
4.4 COMPARISON WITH BASELINES
Setup After we scrutinize that C-LC and C-ADB requires different calibration strenghts, we then analyze whether the proposed methods outperform baseline approaches described in section 4.1 and compare the performances in various KLR levels. For the calibration strength of C-L.C and C-ADB, we used the ones which accomplish the best accuracy in Table at each KLR under each dataset. The results are shown in Table
upproaches. We implement
Calibration is advantageous in low KLR We discover that the proposed methods accomplish promising open intent classification performances compared to the baselines. Especially, the pro- posed C-ADB outperforms the baselines KLR of 25% and 50%, while it accomplises competitive performance in 75% KLR. Unfortunately. C-LC achives promising performances but fail to outper- form the prior methods. For the C-ADB's ' competitive performances in 75% KLR, we hypothesize that large KLR setting is more difficult to sensitively discover an optimal calibration strength. The larger number of intents implies larger number of known intent clusters; thus, we expect the model cannot easily establish effective decison boundaries between the known clusters. As Zhang et al. (202)) once urged, especially under the highly-complex decision boundaries, the calibration should have to be carefully applied into the model maximize the open intent classification. In other words, when many known intents exist, careless calibration has higher risk of demolishing repre- sentations on given data. However, we simply applied heuristically -ahhcceelll-hosen chosen calibration strength (one among 0, 0.2, 0.5, 0.8, 0.9); thus, we expect this careless use of strength is related to the competitive performance of C-ADB.
To experimentally excavate our hypothesis's 's validity, we visualized the representations yielded from the models different KLRs. We visualized the representations extracted at the last layer of BERT with t-SNE (Van der Maaten & Hinton 2008 which is conventionally utilized method of dimen- sionality reduction, and the results are shown in Figure 2 These results show that a model under the large KLR exposes less eess-qualified representation clusters of known intents which implies that model's 's representation insufficient to discriminate various known intents clearly. Referring to the representation distribution under low KLR,, known intents are clearly discriminanted regardeless.

---

Under review as a conference paper at ICLR 2023
Table 3: Comparative study the proposed methods with baselines
STACKOVERFLOW
BANKING
008
KLR
Model
Accuracy
Fi-score
Accuracy
Fi-score
Accuracy
Fi-score
MSP
28.67
75.89
43.67
50.09
47.02
47.62
OpenMax
40.28
77.45
49.94
54.14
68.5
61.99
DeepUnk
LC
78.09
58.61
70.86
37.18
85.00
49.25
ADB
86.72
81.08
78.85
71.62
77.19
C-LC (OURS)
84.55
71.12
83.69
68, 68.59
90.12
71.81
(QURS)
MSP
$2.42
63.01
59.73
71.18
62.96
70.41
DOC
OpenMax
60.35
68.18
65.31
74.24
80.11
80,56
30%
DeepUnk
58.98
68.01
72.73
77.53
83.35
82.16
LC
80.26
72.28
64.22 22
43.03
77.29
54,13
C.LC (OURS)
80.50
76.18
79.99 98
77.34
85.52
76.55
C-ADB (OURS)
87.10
85.47
$1,66
81.14
88.23
84.61
MSP
72.17
77.95
75.89
83.6
74.07
82.38
DOC
68.91
75.06
76.77
83.34
78.73
83.59
OpenMax
74.42
79.78
84.07
76.8
73.16
75%
DeepUnk
72.33
78.28
78.52
84.31
83.71
86,23
LC
71.09
70,25
52.49
46.19
66.82
52.25
ADB
COUR))
75.25
77.90
70.45
73.27
75.81
68.75
C-ADB (OURS)
X2.24
85.41
78.82
83.66
84.60
85,18
Consequentially, we figure out that calibration is effective in escalating open intent classification performances in low KLRs, but there should be more appropriate manner of selecting the calibra- tion strength. One presumable solution utilizing a learnable calibration strength for C-LC and C-ADB, but leave this point improvement avenue.
(25%, 0.0)
(25%, 0.9)
(50%, 0.0)
(50%, 0.9)
(75%, 0.0)
(75%, 0.9)
Figure 2: The representation distributions between known intent (colored) and unknown intent (gray) samples in STACKOVERFLOW where the representations are yielded by C-LC. Note that the left element the bracket implies KLR, and the right one means calibration strength.
(25%, 0.0)
(25%, 0.9)
(50%, 0.0)
(50%, 0.9)
(75%, 0.0)
(75%, 0.9)
Figure The representation distributions between known intent (colored) and unknown intent (gray) samples in STACKOVERFILOWW where the representations are yielded by C-ADB. Note that the left element the bracket implies KIR,, and the right one means calibration strength.
We figured out that the proposed C-LC could not perform open in- tent classification better than the original ADB and C-ADB. We analyze ADB-based methods' effec- tiveness stems from the post-processing procedure it establishes tighter decision boundaries between known intents. Conversely, in C-LC, the lack of this post-processor might contribute to discover adequate tightness of decision boundaries among known in- tents; therefore, could not accomplish comparable performances with ADB-based methods. To improve this limit, we aim to excavate further how we can elevate C-LC's performances. We presume improving a threshold of identifying unknown intents would additionally escalate the performance; thus, utilized validation sets for choosing this threshold and denoted this method as C-LC- Val. The proposed C-LC-Va uses validation set consisting of samples from both known and unknown intents. While naive C-LC sets threshold as
Validation set can improve C-LC
Table Open intent classification perfor-
mances under the the validation set
370 SOVEITT 776
BANSINI
EXIS
Ansrnc
AccRrayy
25%
CADS
94.87
$3.85
33.55
73.45
91.65
$1.45
C-CC//
83.55
79.25
31.35
14,34
$9.23
90.15
15%
C-ADB
85.17
81.66
$1.14
$6.55
87.33
88.55
81.31
81.53
$5.00
96.05
CLC.V
86.35
$0.37
75.42
$1.61
85.01
95.28

---

Under review as a conference paper at ICLR 2023
mean confidence in the training samples, C-LC- Val chooses decision threshold set by Youden's J statistics (Youden] [950) (which is commonly used to set optimal threshold in ROC curve, as well as in prior works (Schisterman et al. 2005 Powers 201T). Note that the calibration strength for C-LCC Val is selected as the one that accomplished the best accuracy in Table We compared the C-LC-Val's performance with other options in Table 4 From Table 4 we scrutinize that C-L.C- Val improves naive C-LC's performances in several cases; thus, we conclude that utilizing a validation set can be one presumable method escalate C-LC's performance. Compared to naive C-LC, We highlight that this assumption of validation set is limited setting as real-world practitioners cannot always acquire unknown intent samples a priori. But, we aim to show that the practitioners can use C-LC-Val if they can acquire particular amount of unknown intent samples.
4.5 STABILITY UNDER A FEW TRAINING SAMPLES
Setup We aim to further excavate whether the proposed methods robustly sustain their performances under the low number of labeled training samples. We define the portion of labeled training samples as Training Data Ratio (TDR), and configure the experiments with TDRs of 0.2, 0.4, 0.6, 0.8, and 1.0. Under the TDR of 0.2, it means there exist only 20% of labeled training samples per each intent. The results are shown in Table 5 and Table 6 Note that or means the calibration strength and a bold number implies the best performance at a given TDR setting.
Table 5: The proposed methods' performances measured in accuracy under various TDR levels.
STACKINERILOW
BANKING
D05
TER
C-DD6
CAD6
003.3
3.00
016.3
03
36.77
18.79
2006
01.77
$6.84
$1:81
85.53
09.1
2007
98.35
$1.51
90.42
15-22
05.88
07.28
67.00
16.72
85
26-1
56.55
04:3
89.14
85.11
5555
8-:11
73.79
27.8
0.53
08:11
01.37
0115
03.08
83:34
08.32
72.43
16.2
12.27
13.04
$5.33
15.05
$6.40
7510
$2.00
87.33
90.96
11.51
5781
55.5
011
86/0
20.33
53:00
14.55
10
7um
19.03
34
$1.20
20:1
21.3
01.44
56
12.41
25.23
99:1
81.35
5034
$7.56
17.32
12.0
73.38
79.02
79.48
$1.90
70.8
79.46
18777
17:07
17.40
12.3
17.00
85.55
1307
82.55
$359
83.12
$3.55
03
73.22
77.06
7443
76.88
93.07
41.8
65.45
78.34
14.1
06.6
6641
72.05
7430
7645
53.32
$3.95
16.0
77.5
83.56
3351
85
26:88
15
6655
16:1
1131
$9.99
76.55
77.06
Nu0
7111
71.5
13.0
75:66
1551
5141
51.30
15.00
14.55
$3.9
15-0
11
600
1635
11.33
$1.55
$1.0
7641
31.0
05.66
7100
15.36
$1.0
$1.06
13.9
$391
Table 6: The proposed methods' performances measured in F1 score under various TDR levels.
STRERIRTEOT TWW
0.:55
01.11
$3301
10.11
14.30
15.34
60.40
54.99
02.15
14,99
1471
3100
310
71.01
02.47
13.54
06.38
13.35
18.55
2018
5099
78.33
83.01
$3.30
14.04
37.42
13.34
55.55
86.3
14.35
15.35
15.23
0523
12:35
18:35
36.22
06.13
12.38
05:33
12.22
30.9
11.34
63.00
1022
14.2
1933
T130
71:31
1520
.99
83
1000
15:35
03:33
$1.25
05:17
12.00
13.35
59.05
33.05
12.07
96.0
00.00
11:15
74.85
12.34
11.99
01.37
11.6
$1.22
1270
65.55
12.50
15.0
15:00
150
013
6315
11.0
7000
0.4
3006
04.04
$2.98
05.44
14.7
300
12.48
25.2
66.1
$4.77
$3.2
11.29
6148
68.44
12.4
12.34
11.79
6332
1439
08.1
$3.55
8457
85.1
6159
68.33
14.4
Result We discover that key findings presented in the section 4.4 still exist unless there are fewer
training samples per class. Nevertheless, we observe that C-ADB is less stable than C-LC under the various training samples per class; thus, analyze that there should be careful configuration of C-ADB, especially under a few training samples. Following the results, C-LC's performances do not significantly change under various TDR levels, while C-ADB's 's performances bear such high variability. Particularly, C-ADB's s performance experiences a harsh drop under low KLR (i.e., 25%) and high calibration strength. We presume the underlying reasons behind these phenomena are also the tightness of decision boundaries. Supposing low TDR, a model would learn insufficient knowledge during the training; thus, we expect risk overfitting and disqualified understanding on known intents. Under the overfitted, disqualified representations, we presume applying calibra- tion or post-procee post-processors (which escalated open intent classification performance under large training samples) would not contribute to acquiring better decision boundaries. Accordingly, it degrades the open intent classification performance. Therefore, we urge that applying calibrations or post- processor is not a golden key to escalating performance in every circumstance. Still, we also dis- cover that the landscape of performance change along with various KLR and TDR levels depends on the dataset. We yield this analysis as an improvement avenue as it is not core component of our study. As a naive direction of analysis, we expect there should be a method of quantifying the decision boundaries" tightness for the open intent classification task. In a nutshell, recommend
8

---

Under review as a conference paper at ICLR 2023
that NLP practitioners use the proposed C-ADB carefully. We highlight that C-ADB might yield inconsistent performance if a few training samples exist on known intents.
5 WHY DO CALIBRATION CONTRIBUTE TO OPEN INTENT CLASSIFICATION?
Setup Lastly, we aim to excavate how the model calibration contributes to the escalation of open intent classification performances. We hypothesize the answer would exist in the learned represen- tations of the model; thus, we measured representation similarities between various models trained under different calibration strengths. We establish five pairs consisting of models trained under dif- ferent calibration strength as follows: (0 v. 0), (0 v. 0.2), (0 v.0.5). (0 V. 0.8) and (0 v. 0.9). Given models in a pair by applying Cen- tered Kernel Alignment (CKA) (Kornblihh et al. 2019). CKA measures representation similarity between two layers from different models and returns the similarity score between and (where means the highest similarity). Among various methods in quantifying representation similarities, such as CCA or SVCCA, we utilize CKA as accomplished state-of-the-art performances their domain's 's benchmarks (Kornblith et al.) 2019 Wu et al.) 2020 Sridhar & Sarah 2020). Following the prior work (Kornblith et al., 2019). we compared representation similarities at every LayerNorm layer between two models in given pair. Due to the page limits, we show representation similarities
these pairs, we quantitatively measure the similarity between
among various models trained under STACKOVERFLOW dataset in Figure Please refer to the supplementary materials for representation similarities on the other datasets.
0.0
1.0
0.0
1.0
10
10
10
10
Laber Smoshing
0.8
Lazel Smootiing
0.8
Label Smoothing
0.8
Label Smoothng
0.8
6
0.6
0.6
0.6
0.6
2
8
0.4
0.4
0.4
10
0.4
Lebel Smoothing 0.0
Label Smoothing 0.9
Label Smoothing 0.0
Label Smoothing 0.9
(a) C-LC (0,0)
(b) C-LC (0,0.9)
(c) C-ADB (0,0)
(d) C-ADB (0,0.9)
Figure 4: Representation similarities between models with and without calibration
Result Upon the Figure for both C-LC and C-ADB, we scrutinize that model calibration yields different high-level representations from the model without any calibration. While overall land- scapes of representation similarities look similar regardless calibration strengths, similarity very last layer (11th (1thh layer) becomes less similar as long the calibration strength increases. We interpret this phenomenon implies that calibration makes the model's high-level representation be- comes different. Along with prior work's amalysis (Hao et al. 2019 Wu et al. 2020), further hypothesize that calibrated model bears a different contextual understanding of given text input. We analyze calibrated model interprets the text input less biased to the known intents; thus, it dis- criminates unknown intents based on this qualified understanding of the text data. Note that this analysis supports that calibrated model acquires 'different' representation from the not-calibrated one, but it does not justify how the calibrated representation yields better open intent classification performance. We let this point as an improvement avenue.
CONCLUSION
In this study, we propose novel open intent classification methods that utilize label smoothing on prior state-of-the-arr methods. We experimentally show this simple idea improves prior approaches' performance in particular settings as calibrated representation makes an adequate tightness on deci- sion boundaries among known intents; thus, the proposed C-ADB becomes novel state-of-the-an benchmark settings. Furthermore, we analyze the proposed methods" supremacy derives from high-level representations, which implies model calibration contributes to acquiring more quali- fied contextual understanding of text inputs. Last but not least, we also scrutinize that C-ADB is less stable than C-LC under few training samples; thus, highly recommend that practitioners care- fully utilize it on their own datasets. Nevertheless, several improvement avenues exist as proposed in the prior sections. We expect practitioners in real world use the proposed methods, especially C-ADB, to establish effective NL. applications in their domains.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Abhijit Bendale and Terrance E Boult. Towards open set deep networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563-1572, 2016.
Inigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint arXiv:2003.04807, 2020.
Shrey Desai and Greg Durrett. arXiv:2003.07892 2020.
Calibration of pre-trained transformers.
arXiv preprint
Geli Fei and Bing Liu. Breaking the closed world assumption in text classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 506-514, 2016.
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1332 PMLR, 2017.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. Visualizing and understanding the effectiveness of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 4153-4152, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/DI1 1/249-1424. URL
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and outto--distribution examples in neural networks. arXiv preprint arXiv:1610.02136, 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR, 2019.
Aviral Kumar and Sunita Sarawagi. Calibration of encoder decoder models for neural machine translation. arXiv preprint arXiv: /903:/993.00000 00802, 2019.
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld, Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv:1909.0202: 2019.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratee classifiers for detecting out-of-distribution samples. arXiv preprint 11X.::11111.09355. 2017.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv: 17X1v:1006.02690, 2017.
Ting- En Lin and Hua Xu. arXiv: 00434, 2019,
Deep unknown intent detection with margin loss. arXiv preprint
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0101, 2017.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? Ad- vances in neural information processing systems, 32, 2019.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Measur- ing calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv: 1701.0560
2017.
David Powers. Evaluation: From precision, recall and f-measure to roc, informedness, markedness & correlation. Journal of Machine Learning Technologies, 2(1))37-63, 2011.
10

---

Under review as a conference paper at ICLR 2023
Sridhama Prakhya, Vinodini Venkataram, and Jugal Kalita. Open set text classification using CNNs. In Proceedings of the 14th International Conference on Natural Language Processing (ICON. 2017), pp. 466-475, Kolkata, India, December 2017. NLP Association of India. URL https: //acl Maatannnhoollg. org/w17-7557
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, pp. 5389-
PMLR, 2019.
Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 35(7):
1757-1772, 2013. doi: 10.1 2..10///.MM..222...55.
Enrique F Schisterman, Neil J Perkins, Aiyi Liu, and Howard Bondell. Optimal cut-point and its cor- responding youden index to discriminate individuals using pooled blood samples. Epidemiology, pp. 73-81, 2005.
Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. arXiv preprint arXiv:1709.087/11 2017.
Lei Shu, Yassine Benajiba, Saab Mansour, and Yi Zhang. Odist: Open world classification via distributionally shifted instances. Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3751-3756, 2021.
Sharath Nittur Sridhar and Anthony Sarah. Undivided attention: Are intermediate layers necessary for bert? arXiv preprint arXiv:2012.11881, 2020.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigmiew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. arXiv preprint arXiv:2110.0622 2021.
John M Wu, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Simi- larity analysis of contextual word representation models. arXiv preprint 2005::2005.0117,, 2020.
Jiaming Xu, Peng Wang. Guanhua Tian, Bo Xu, Jun Zhao, Fangyuan Wang, and Hongwei Hao.
Short text clustering via convolutional neural networks. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 62 69, 2015.
William J Youden. Index for rating diagnostic tests. Cancer, 3(1):32-35, 1950.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR,
2016. URL http:
//arxiv. www.aii...g.aas/a//ool.aaaaaaaaa abtpg//bbsllee 611 03530
Hanlei Zhang, Hua Xu, and Ting-En Lin. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp- 14374-14388 2021.
11


Reviews:
[{"review_id": "roR6v90lCU", "reviewer": "Reviewer_UJRs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes two open intent classification methods based on the label-smoothing method, C-LC, and C-ADB. More specifically, the label smoothing method takes a weighted average of the ground-true label and the uniform 1/k vector. The experiments \nevaluates the proposed methods on real-world datasets with previous baselines.", "summary_of_the_review": "The method described in the paper is more like an empirical trick than a well-designed method. The motivation and justification for the method are not clearly stated. The experimental results did not show significant improvement when applying the proposed method.", "strength_and_weaknesses": "Strength:\n\nThe main idea of the paper is clear. The experimental part is solid with sufficient empirical results and analyses.\n\nWeaknesses:\n\nAbout the Novelty: the proposed method is more like an empirical trick or a modification to the previous baseline LC and ADB. The motivation and justification to use the label smoothing trick remain unclear. \n\nAbout the Experiments: The proposed methods did not significantly outperform the previous baseline. On the OOS dataset, the proposed C-ADB even got worse performance than the original ADB, which makes the effectiveness of the proposed method unconvincing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xyj4ujIoj5j", "reviewer": "Reviewer_HPLr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In general, this paper proposes to use the label smoothing method to calibrate the model in open intent classification problem, based on two previously used method logit classifier (LC) and adaptive decision boundary (ADB). The result in low KLR (Known Label Ratios) shows the superiority of the proposed method over previous methods, while the result in high KLR does not. Extensive experiments are taken to demonstrate that the proposed method can achieve improvements in inductive bias, representation landscape and open intent classifier performance. ", "summary_of_the_review": "see above.", "strength_and_weaknesses": "**Strengths**\n1. The proposed method is clear and simple;\n2. The analysis and experiments are somewhat comprehensive.\n\n**Weaknesses**\n1. The authors claim the benefit of calibration by the proposed method, but fail to show an evaluation of the calibration performance in terms of expected calibration error (ECE).\n\n2. The experiment evaluation should be further strengthened. The following related prior works are missing and should be compared.\n\na. Likelihood ratios and generative classifiers for unsupervised out-of-domain detection in task oriented dialog, AAAI 2020\n\nb. Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models, EACL 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "IahYvj2XtS", "reviewer": "Reviewer_HHjh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the open intent classification problem. There are standard supervised learning approaches for this task, but the authors hypothesize that cross entropy based objective in the existing open intent classification problem leads to predictions which are overly biased towards known intents, leading to poor performance on open intents. It has been presented with empirical evidence that changing a learning objective in a more calibrated manner can lead to better performance and outperforms the existing state-of-the-art. Authors also investigate the reason for the supremacy of the calibrated model and highlight its connection with higher layers in the model.", "summary_of_the_review": "Overall, this paper investigates an interesting question. For the thorough investigation and establishing the claims, more experimentation is required.\n", "strength_and_weaknesses": "Strengths\n\nS1: It is an interesting idea to utilize model calibration towards open intent classification.\n\nS2: The proposed approach has been applied to text and image domain state-of-the-art methods.\n\nS3: The paper presents a detailed set of experiments.\n\nS4: The paper is easy to follow, but writing needs improvement.\n\n\nWeaknesses\n\nW1: The proposed approach has been empirically verified on two state-of-the-art methods, which are Adaptive Decision Boundary (ADB) and Logit-based classifier (LC). It could have been interesting to see if the approach generalizes to a few others out of domain intent classification methods.\nW2: It will be interesting to investigate the behavior of other calibration methods like temperature scaling with the proposed approach. This will add the element of completeness in the research question being investigated.\n\nW3: Logit-based classifier should be one of the baselines.\n\nW4: There are various grammatical mistakes and typos in the paper.  i) For example, in section 4.3, “we assume the adequate calibration strength will be differ from the C-LC’s optimal one.”  ii) Similarly, in section 4.2 in results section, “we resulted that calibration itself is beneficial”. iii) Typo ⇒ “optinmal calibration sterngth”\n\nW5: Figure 1 has enough empty space which can be utilized for increasing the subplot sizes. Especially, legend is difficult to read.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject