Closed Boundary Learning for Classification Tasks with the Universum Class
Authors: Hanzhang Zhou
Zijian Feng
Kezhi Mao
Venue: EMNLP 2023 Findings (2023)
Keywords: classification tasks
representation learning
the miscellaneous class

Abstract: The Universum class, often known as the *other* class or the*miscellaneous* class, is defined as a collection of samples that do not belong to any class of interest. It is a typical class that exists in many classification-based tasks in NLP, such as relation extraction, named entity recognition, sentiment analysis, etc. The Universum class exhibits very different properties, namely heterogeneity and lack of representativeness in training data; however, existing methods often treat the Universum class equally with the classes of interest, leading to problems such as overfitting, misclassification, and diminished model robustness. In this work, we propose a closed boundary learning method that applies closed decision boundaries to classes of interest and designates the area outside all closed boundaries in the feature space as the space of the Universum class. Specifically, we formulate closed boundaries as arbitrary shapes, propose the inter-class rule-based probability estimation for the Universum class to cater to its unique properties, and propose a boundary learning loss to adjust decision boundaries based on the balance of misclassified samples inside and outside the boundary. In adherence to the natural properties of the Universum class, our method enhances both accuracy and robustness of classification models, demonstrated by improvements on six state-of-the-art works across three different tasks. Our code is available at https://github.com/hzzhou01/Closed-Boundary-Learning.

Closed Boundary Learning for Classification Tasks with the Universum
Class
Hanzhang Zhou 1.3, Zijian Fengl. Kezhi Mao 2.3+
"Inttitute of Catastrophe Risk Management, Interdisciplinary Graduate Programme, Nanyang Technological University, Singapore School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore Future Resilient Systems Programme, Singapore-ETH Centre, CREATE campus, Singapore (hanzhang@01, (eng@119)@e.cr ntu. edu. ekzmaoθntu. edu. sg
Abstract
designate all such classes as the Universum class (U). Universum class exits in various classification- based problems NLP, such as relation extrac- tion (RE) (Zhang et al., 2017), named entity recog- nition (NER) (Tjong Kim Sang and De Meulder, 2003), sentiment analysis (SA) (Tjong Kim Sang and De Meulder, 2003), and natural language infer- ence (NLI) (Bowman et al., 2015). To distinguish the Universum class and the rest of the classes, we call the classes of interest as targer classes (T). The set of all classes (A) in the data can be expressed as = U UT
The Universum class, often known as the other class or the miscellaneous class, is defined as a collection of samples that do not belong to any class of interest. It is a typical class that ex- ists many classification-bass such relation extraction, named entity recog- nition, sentiment analysis, etc. The Universum class exhibits very different properties, namely heterogeneity and lack of representativeness in training data; however, existing methods of- ten treat the Universum class equally with the classes of interest, leading to problems such as overfitting, misclassification, and diminished model robustness. In this work, we propose closed boundary learning method that applies closed decision boundaries classes of inter- est and designates the area outside all closed boundaries in the feature space as the space of the Universum class. Specifically, we for- mulate closed boundaries as arbitrary shapes, propose the inter-class rule-based probability estimation for the Universum class cater to its unique properties, and propose a bound- ary learning loss to adjust decision boundaries based the balance of misclassified samples inside and outside the boundary. In adherence the natural properties of the Universum class, our method enhances both accuracy and ro- bustness of classification models, demonstrated by improvements six state-of-the-art works across three different tasks. Our code is avail- able at Boundary-Learning
tasks NLP,
Universum class: A collection of samples that do not belong any class of interest.
Target class: class of interest in the task, i.e.. one the classes other than the Universum class.
The sample compositions of the Universum class and target classes are usually very different. Figure 1(a) provides some samples of target class (entity- destination) and the Universum class (other) relation extraction. Intuitively, we can observe that the -estiddestination samples adhere to an intra- class pattern: an entity goes somewhere. However, the three examples of the other relation type are vastly dissimilar and do not exhibit any intra-class pattern. In fact, the Universum samples are labeled according to an inter-class pattern: they do not belong to any of the predefined target classes.
1 Introduction
We further highlight the differences between the Universum class and target classes in two proper- ties. (1) Heterogeneity:
classification-bart tasks, quite often we en- counter a class named as other class, miscellaneous class, neutral class or outside (O) class. Such a class is collection of samples that do not be- long any class of interest, such as samples of relation class in relation extraction task. We adopt the terminology in (Weston et al., 2006) to
The Universum class is com- posed of heterogeneous samples, which may form multiple clusters in the feature space of the test set, illustrated by the green samples in Figure I(c). This is because the Universum class, as the class name "other" implics, contains all potential implicit classes that are not explicitly defined in the
"Corresponding author.

---

Patigy-Devtiiatiion Relation
Other Relation
INNSIALM
actress airport.
captain and are grateful for the support.
NISC
OBG
Quake survivors moved into makeshift houses.
The room was filled with huge canvases.
The reseurch team going the jungle.
stories narrated through dance.
PSR
120
163
X06
Moas Sugrre Dikance
(a)
(b)
00
(c)
(d)
Figure I: Illustration of distinction between the Universum class. (a) Samples selected from the SemEval 2010 Task 8 dataset on relation extraction. (b) Compactness comparison between the test data of the Universum class (green) and target classes (blue). (c) The distribution of target classes (class 1, 2, 3) and the Universum class (class 4). In particular, the gray samples represent Universum samples test set that are not represented by the training data. (d) The open decision boundaries obtained by traditional classifiers. (c) The arbitrary closed boundaries obtained by our proposed method.
task. For example, in samples of the other class in Figure ((a), implicit classes may include the entity- parallel relationship, the entity-fill fill relationship, and the entity-narrative relationship.
ity discern the genuine predictive rule for the Universum class can also compromise the model's robustness.
(2) Lack of Representativenes in Training Data: The Universum class is the complementary set predefined target classes in the task. Therefore, it contains all possible implicit classes, i.c., classes not explicitly defined in the task but may appear in the real world. In this case, Universum samples in the training data unable to sufficiently represent all possible patterns of the genuine distribution of the Universum class. As depicted in Figure (c)) gray samples represent Universum samples in the test set that are not represented by the training data. Classifiers with open boundaries are prone to mis- classifying unseen samples in the test set that is not represented by the training data.
Although such heterogeneous samples are easily mapped into a compact cluster for the training set, it is problematic for the test set. This is because the inherent predictive rule of the Universum class follows unique inter-class pattern: sample is labeled as Universum if it does not belong any target classes. This sharply contrasts with the con- ventional intra-class patterns seen in target classes. Considering human annotation practices, an entity is labeled as Location when aligns with estab- lished patterns of Location entities. In contrast, a sample is labeled as Others not due to intra-class patterns specific to the Others class, but because it fails to conform to the patterns of Location, Per- son, Organization. Consequently, when current classification models treat the Universum class and target classes in the same manner, they tend to over- the noise the Universum class by memorizing various peculiarities of the heterogeneous samples rather than recognizing the general predictive rule. Given the variations in data distributions between the test and training sets, only memorizing various peculiarities casily lead to overfitting, causing decline in accuracy. Furthermore, this inabil-
Additionally, we provide a quantitative compari- son of the average compactness between the Uni- versum class and the target classes within the test data for the NER task (Fu et al., 2021), as de- picted in Figure 1(b). Notably, even though the Universum class is the class with the most samples, it exhibits significantly poorer compactness in its learned representations. This empirical observation aligns with our earlier theoretical analysis. Both the inability to discern the genuine inter-class pre- dictive rule of the Universum class and the lack of

---

representativeness in the training data contribute to this compromised compactness for the Universum class. Experiment details are in Appendix A.
sum class, our method improves both aceuracy and robustness of classification models, which is vali- dated on six state-of-the-art (SOTA) works across three different tasks.
Despite the substantial difference between the target classes and the Universum class, this issue has long been neglected by the research commu- nity. The majority of works (Zhu and Li, 2022; Ye al., 2022; Wan al., 2022: Fu al., 2021; Li al., 20) treat the Universum class and target classes equally. Typically, a linear layer and soft- max function are applied at the end of the model to generate open decision boundaries, which we believe are inappropriate for tasks containing the Universum class.
2 Related Works
2.1
Classification Tasks with the Universum Class
The Universum class widely exists in classifica- tion based tasks in NLP, such as relation extraction (RE) (Zhang ct al., 2017), named entity recognition (NER) (Tjong Kim Sang and De Meulder, 2003), and aspect category sentiment analysis (ACSA) (Jiang ct al., 2019), as summarized in Table 1. It should be noted that the span-based methods (Zhu and Li, 2022; Li et al., 2021a) a) enumerate all pos- sible spans for classification, which introduces extra other class. Despite the heterogeneity and lack of representativennss of the Universum class, current works (Zhu and Li, 2022; Wan et al., 2022; Fu et al., 2021; Tian et al., 2021; Chen et al., 2021; Li et al., 202 lb, 2020; Yu et al., 2020) solve the classification problems containing the Universum class normal multi-class classification problems and treat the Universum class and target classes equally.
How can account for the distinct properties of the Universum class and target classes to derive better representations and classifiers? We think the key lies conforming to the inherent prop- erties of the Universum class. In this work, we propose a closed boundary learning method for classification- tasks with the Universum class. Traditional methods often employ open boundary classifiers and constrain the representations of Uni- versum samples to be distributed into compact cluster. However, the open decision boundaries can easily misclassify Universum samples, as illus- trated in Figure (dd). In addition, the restriction on compact space violates the inherent inter-class pattern of the Universum class. Therefore, we pro- pose use closed boundary classifiers as shown in Figure (e).. We constrain the space of target classes be closed spaces and designate the area outside all closed boundaries in the feature space as the space of the Universum class. The treatment perfectly fits the nature of the Universum class: a sample is marked as the Universum if it does not belong to any target class during labeling.
2.2 Closed Boundary Learning Methods
Closed boundaries are often adopted in research fields of out-of-distribution (OOD) detection (Gomes et al., 2022; Ren et al., 2021; Chen et al., 2020), open recognition (Zhang et al., 2021; Liu et al., 2020), anomaly detection (Zong et al., 2018), and outlier detection (Sharan al., 2018: 10 Sugiyama and Borgwardt, 2013). We borrow the term "gener- alized OOD detection" from (Yang et al., 2021b) to encapsulate these problems. Furthermore, we sum- marize the differences between the classification with the Universum problem and the generalized OOD detection, focusing both problem setting and methodology. The detailed analysis are pro- vided in Appendix
The main contributions of this work summa- rized as follows: (1) We address an understudied problem in this paper. The Universum class widely exists in many NLP tasks and general machine learning tasks, but hasn't 't received significant atten- tion in these contexts. (2) Methodologically, we generate closed boundaries with arbitrary shape, propose the inter-class rule-based probability es- timation for the Universum class to cater to the inherent properties of the Universum class, and propose a boundary learning loss to learn the deci- sion boundary based on the balance of misclassi- fied samples inside and outside the boundary. (3) In adherence to the natural properties of the Univer-
2.3 Universum Learning Methods
Although we adopt the terminology of Universum (Weston et al., 2006), the problem setting of our work is entirely different from that of previous stud- ies on Universum learning (Weston et al., 2006; Chapelle et al., 2007; Qi et al., 2012; Zhang and Le- Cun, 2017). The idea of Universum learning stud- ies is to exploit external, unlabeled Universum data

---

Label
Task
Dataset
Proportion
Name
SemEval 2010 Task (Hendrickx et al., 2019)
Other
17.4%
TARCED (Zhang et al., 2017)
relation
79.5%
CoNLL--2003 (Tjong Kim Sang and Meulder, 2003)
Miscellancous
14.6%
NER (span based method) ACSA
CoNI.1-2003 (Tjong Kim Sang and Meulder, 2003)
Other
>90%
MAMS (Jiang et al., 2019)
Neutral
43.4%
Table 1: The tasks and datasets that the Universum class exists.
to improve the accuracy of supervised tasks. How- ever, in our problem, the Universum class is one of the internal, labeled classes of multi-class classifi- cation problems and we propose closed boundary learning conform to its unique properties dur- ing learning. Furthermore, methodologically, the Universum learning method either employs open- boundary classifiers (Zhang and LeCun, 2017) or is incapable of distinguishing the Universum sam- ples from labeled samples (Weston et al., 2006; Chapelle et al., 2007; Qi et al., 2012), neither of which are appropriate to the problem we presented.
of Universum samples during the pretraining.
3.2 Generating Closed Boundary of Arbitrary Shape for Target Classes
Existing closed boundary classification methods mainly use spherical shape boundaries (Zhang et al., 2021; Liu et al., 2020); however, we argue that the spherical shape may not be the optimal solution because data samples are unlikely to per- feetly fit into sphere, and spherical shape bound- ary may produce misclassifications. We adopt the Gaussian mixture model (GMM) and the threshold value to generate boundaries with arbitrary shapes.
3 Method
3.2.1 Gaussian Mixture Model
The goal of our proposed method is to learn closed decision boundaries for target classes and meanwhile, jointly classify the Universum samples and target samples. We first give detailed description of how to recognize the Universum class in Appendix C. In order to make our proposed method compatible with most existing classification methods, the starting point of our method is the representations of the final layer classification models, which is a linear layer that maps data from high-dimensional feature space to a power-dimensinn space. We denote the sample representations of the final linear layer as H (hn, h,,....... hN-13 E R Nxl where N is the number of samples, and is the output dimension the linear layer.
Problem Definition:
We apply GMM with m components to estimate the class conditional probability distribution for each target class C₁ and further derive the joint probability estimation for each class.
m
p(hx (.) = - TWN (haimss E,)
(1)
i=1
phh: C2) = phh C.)((C))
(2)
where h2 denotes the input feature vector of the kith sample, μ4 and E, are the estimated mean vector and covariance matrix of the ith Gaussian compo- nents, respectively. Tij is the non-negative mixture weight under the constraint that Σ Maj 1. μ4, E, and Tij all learnable parameters in the model.
According to Bayes Theorem, the posterior prob- ability p(C: hę) = p(h (C)pCC)) Since we are p(b)))
3.1 Pretraining
Our method estimates the probability distribution of target classes based on their sample distribu- tions. In order to avoid estimation based on ran- domly initialized weight and speed up the learning process, employ N-pair loss (Sohn, 2016) for pretraining, making sample representations be of small intra-class distance and large inter-class dis- tance. Notably, in accordance with the nature the Universum class, we make a change that does not require the model to reduce the intra-class distance
interested in argmax p(bg/Ca)p((4) the decision
p(h:)
can be made based on joint probability p(hs, Ct).
3.2.2 Arbitrary Shape Boundary
Geometrical View: Inspired by the DENCLUE al- gorithm in generating arbitrary shape clusters (Hin- neburg and Keim, 1998), we introduce a threshold value & for each target class. closed boundary of arbitrary shape formulated by points satisfy- ing p(h, C.) = & Figure is an illustration

---

An intuitive way to incorporate the above rules is two-step method that consists of Universum class detection and target class classification. However, such a pipeline method has the issue of error prop- agation. In addition, general probability estimation methods exploit intra-class sample distributions, which fail to overcome the natural inter-class prop- erty of the Universum class and do not conform to Rule 1. Therefore, strategy need be devised to convert Rule 1 and 2 into probability expression, while simultaneously facilitates the learning of the neural network.
(a)
(b)
Figure 2: Illustration of generating arbitrary shape boundaries.
formulating an arbitrary shape boundary in a two- dimensional space. A sample is assigned to class C it is located inside the closed boundary. If the number of components of the GMM is set to one, then the shape of the boundary becomes spherical, with its center and covariance matrix specified by No and Eo, respectively. In this sense, the com- monly used spherical shape boundary (Zhang et al.. 2021; Liu al., 2020) is a special case of our method.
For compliance with Rule 1, the estimated proba- bility of the Universum class must satisfy the fol- lowing two conditions: for Universum class sam- ples: Vi : p(hg, U) > p(hg, C₁) and for target class samples: ži p(h,, U) p(h,, Ct). We leverage the relationship between & and p(h; C+) defined in Equation 3 to construct the estimation of p (gg U) that satisfies the above two conditions. In addition, to enhance the learning of neural net- works, the gradient obtained from an Universum sample should move this sample away from its clos- est target class boundary. Therefore, we also in- volve max(p(he, Cj)), the probability of the clos- est target class of Universum sample, to guide the Universum sample move away from target class boundaries. We propose to estimate the probability distribution of the Universum class as follows:
Notably, the threshold values = E1, E2, .. Sn-1 are a learnable parameters, which eliminates the la- borious process of hyperparameter tunning, Specif- ically, they are learned based on the balance of miselassified samples inside and outside the bound- ary through our proposed boundary learning loss, which is introduced later.
Probabilistie View: The above geometrical pro- cess can be described as:
E p(h U) = p(h C.)
&
h& E C if Ci) ()>{ &
+ (1- p(h C.)
(4)
(3)
h4 & C, p(h:: C3) &
3.3 Inter-Class Rule-Based Probability Estimation for the Universum Class
1, p(bs; Cu) > Eu
where
(5)
0, p(hi Cu) S En
The main obstacle to properly addressing the issue of the Universum class to estimate the probabil- ity of the Universum class based on its inherent inter-class property rather than intra-class sample distributions. We propose an inter-class rule- based probability estimation method to address this issue.
argmax, pha,C.) & argmax; p(h Ci)
(6)
& is the threshold value of target class and u. v {1, 2..... 1}.
3.3.2 Analysis of the Proposed Estimation For estimated probability of the Universum class in Equation 4, two cases are possible. Case p(h C.) > Eus i.e., sample h2 is located inside at least one closed boundary. In this case, we have
3.3.1 Motivation and the Estimation
We classify samples of Universum class and target classes based on the following rules we defined:
Rule I: A sample is assigned to the Universum
class it is not located inside any of the closed boundaries of target classes.
Eu
p(h, U) En "(hee C.)
< Ea < p(he, C.)
Rule A sample is assigned to the target
class with the highest p(hg, Ct) if is located inside at least one closed boundary.
Since p(hg, U) < p(hg Ca), the model will select the target class with the highest p(h,, Ct), which

---

fits perfectly with Rule
samples, the boundary learning loss forces misclas- sified samples to be distributed in the proper region, which works well with cross-entropy loss.
Case 2: p(hg C.) S Eu. i.e. the sample hy dis- tribute outside every closed boundary.
Combining the condition of case 2 and Equation 6, we have
3.5 Framework Overview
p(hn, Ci)
p(hg, C.)
vi E (1,2,...,,.
21
To provide better clarity of our method, we pro- vide a succinct breakdown of the closed boundary learning framework's 's workings:
En
i.., Vi E n 1
phg,, Ci) & (7)
Initialization Post-Pretraid Post-Pretraining: After pre- training, we employ the GMM for each target class. The Inpectation-Maxinizztiee algorithm is employed to set the initial values for the GMM parameters μ4; E, and Tij- As we transition to the training phase, the param- eters of GMM are treated as learnable parame- ters. Contrary to traditional methods using the EM algorithm for continuous updates, these parameters are dynamically updated by the neural network throughout the training pro- cess.
Combining Equation 6 and Equation 7, we can derive that Vi (1.2.....
1}:
(EM)
&
2 2 plhe, C.) 2 p(hs. Ci) p(hg, U) = p(he, C.)
In case 2, from Equation and Equation 3, we can learn that sample hg is located outside all closed boundaries of target classes. In this case, the probability of Universum class p(hu, U) obtains the largest value. Therefore, Rule is perfectly expressed by the proposed probability estimation of the Universum class.
Probability Estimation: Probability distri- butions of target classes are estimated using GMM, as articulated in Equation 2. The prob- ability distribution of the Universum class is computed through our Inter-Class Rule- -ased Probability Estimation method, which is rep- resented in Equation 4.
3.4 Boundary Learning Loss
To facilitate the learning of the closed decision boundaries, we propose a boundary learning loss below. Our intuition is that the decision boundary should be adjusted to the balance of misclassified samples inside and outside the boundary. For ex- ample, if samples of class distribute inside the boundary of class i, then the boundary should con- tract to exclude such samples and vice versa.
Training Optimization: During the train- ing process, we use a combined loss func- tion, summing the cross-entropy loss with the boundary learning loss for optimization.
1 M
E, p(hg, Ct)
Lbi
wp log
KCD
Experiments
p(h, C.)
wi log
4.1 Experimental Methodology
&
TeT
We demonstrate the efficacy of our method on six different SOTA models on three datasets of different NLP tasks, including SemEval 2010 Task (Hendrickx et al., 2019), MAMS (Jiang et al., 2019), and CoNLL-2003 (Tjong Kim Sang and De Meulder, 2003). The proportion of Universum samples in the SemEval 2010 Task 8, MAMS, and CoNLL-2003 datasets are 17.4% (highest in 19 classes), 90%, and 43.4%. respectively. It is noteworthy that the ratio of Universum class in the NER task is not calculated from the miscellaneous samples in the dataset but from the other samples which are introduced by the span-based method (Zhu and Li, 2022; Fu et al., 2021).
M is the total number of misclassified samples for all boundaries, n is the number of classes, O and II denote the set of training samples misclas- sified outside and inside the decision boundary i, respectively. The weights in the loss function CC), ue Ch)+C W/- C and they are detached and cut off the gradient. Weights w) and wy have smaller values for samples located far from the boundary, enabling the boundary to be adjusted primarily on the basis of easily and semi-hard neg- atives instead of hard negatives.
During training, we sum the cross-entropy loss and boundary learning loss for optimization. In addi- tion to balancing inside and outside misclassified
We evaluate the effectiveness of our proposed

---

Task
Method
Fl/accuracy
p-value
SpanNER (Fu al., 2021)
92.0910.16
(Fu SpanNER
al., 2021) + ADB
77.22 + 0.49
0.001
SpanNER (Fu al., 2021) + OFCC
91.22 + 0.12
NER
SpanNER (Fu
2021) COOLU BS (Zhu and Lt, 2022)
93.50+0..13
92.53+0.02
BS (Zhu and Li, 2022) + ADB
75.32 1 0.55
<.0.1
BS (Zhu and Li, 2022) + OECC
91.88 + 0.15
BS (Zhu and Li, 2022) COOLU
93. 93.1710.13
A-GCN (Tian ct al., 2021)
88.67+0.18
A-GCN (Tian ct al., 2021) + ADB
85.99 + 0.23
<.011
A-GCN (Tian et al., 2021) ) + OECC
88.00 + 0.09
A-GCN (Tian et al., 2021) COOLU
89.33 10.20
TaMM (Chen et al., 2021)
88.76+0.23
TaMM (Chen et al., 2021) ADB
85.08 1 0.26
0.01
TaMM (Chen et al., 2021) OECC
88.17 + 0.18
TaMM (Chen et al., 2021) COOLU
89.47+0.21
AC-MIMLLN (Li al., 2020)
76.13=0.22%
AC-MIMLLN (Li al., 2020) ADB
71.78 1 0.85%
0.01
AC-MIMLLN (Li al., 2020) OECC
74.02 0.68%
ACSA
AC-MIMLLNN (Li et al., 2020) COOLU
77.35
SCAPT (Li al., 2021b) SCAPT al., 2021b) ADB
81.13=0.19%
79.67 0.44%
< 0.01
SCAPT (Li al., 2021b) OECC
83.36 + 0.27%
SCAPT al., 2021b)+ COOLU
85.06+00.23%
Table 2: The overall performance of applying closed boundary learning on baseline models.
C1Osed bOundary Learning for classiciation with the Universum class (COOLU) on SOTA works, including SpanNER (Fu et al., 2021), BS (Zhu and Li, 2022), A-GCN (Tian et al., 2021), TaMM (Chen et al., 2021), AC-MIMLLN (Li et al., 2020), and SCAPT (Li et al., 2021b). The implementation details are in the Appendix D.
about by our COOLU method statistically signif- icant. The above experimental results answer RQ1 in positive.
4.3
A Closer Look at the MicroF1, Precision and Recall
Another question is does COOLU enhance classifi- cation accuracy for all classes or just the Univer- sum class? (RQ2) We show the micro F1 score of each class in applying closed boundary learn- ing SpanNER (Fu et al., 2021) in Table 3. The micro Fl score for the Universum (other) class, introduced by the span-based method, is excluded from the overall Fl score calculation as per the task requirements. The micro F1 score is improved in all classes, with the absolute improvement of 0.01, 2.43, 0.29, 1.31, and 4.02, respectively. The Fl improvement of the Universum class is very small compared to target classes because its sam- ple size is more than 100 times larger than other classes, making the denominator very large when calculating. The results answer RQ2 positively: the improvement of overall performance not only attributed to the improvement of the Universum class, but also to the improvement of all classes as whole.
4.2 Overall Experimental Results
Our first research question (RQ) is can COOLU achieve a "free" accuracy gain on tasks with the Universum class? (RQ1) Table 2 shows the overall results for all 6 models. The reported results are the average of three runs. Models with our proposed closed boundary learning outperform the original models with open classifiers on NLP tasks contain- ing the Universum class. The overall accuracy or Fl score is improved on six models we evalu- ated, with the largest improvement from 92.09 to 93.50 in Fl score. We also notice that the improve- ment on the RE task is not significant as on the NER and the ACSA tasks (0.66 against 1.41 and 1.22). This may be due to the fact that Universum samples only account for 17.1% of the SemEval 2010 Task 8, which is considerably less than the other datasets. In addition, statistical tests between the accuracy/FI score of the baseline models and our method indicate that the improvement brought
The third research question is how does COOLU improve classification model's perfor- mance? (RQ3) We find that our proposed closed

---

ORG
PER
LOC
MISC
Other
Method
P
R
SpanNER
89.92 89.81 89.87
97.74
96.47
97.11
93.05 93.88
93.
78.99
82.83
80.87
99.87
99.83 99.85
SpanNER
90.2990.30 92.30
98.5496.2
97.40
96.1793.41
94.77
88.0281.97
84.89
99.84 99.8999.86
COOLU
Table 3: The micro Fl score of SpanNER (Fu et al., 2021) with and without closed boundary learning.
boundary learning significantly improves the pre- cision score of all target classes, with the largest absolute gain being 9.03 in Table 3. By analyzing the change in precision and recall, we can derive the following findings: Firstly, the misclassification of Universum samples as target samples results in low precision scores for target classes and low re- call score for the Universum class in the baseline method, which proves our claim that the Universum class easily misclassified if its unique properties are neglected. In addition, our proposed closed boundary learning method can effectively prevent the misclassification of the Universum class into target classes, which significantly improve the pre- cision score of target classes at the expense of very slight reduction in recall and similarly im- prove the recall score of the Universum class at the expense of a slight reduction in precision. The second finding answers RQ3.
data. Such knowledge boundary can avoid the mis- classification of unseen non-target class samples as target class. The above two mechanism would contribute to better robustness of the model.
We evaluate the robustness of the model based on TextFlint (Wang et al., 2021), robustness evalua- tion toolkit for NLP tasks. Specifically, TextFlint generate perturbations of the test data, and the ro- bustness of the model evaluated using the trans- formed test dataset. The terms such as "CrossCate- gory", "OOV", and "SpellingError" in Table 4 are different ways of transforming the test data. Detail information of these transformation methods are illustrated in Appendix D.3.
It can be learned from the Table 4 that the robust- ness of SpanNER improved significantly by apply- ing our proposed COOLU method, with the im- provement of absolute Fl score of 1.44, 8.01, 5.03. and 2.55, respectively. In addition, although the improvement of the F1 score in A-GCN is less than 1, the improvement in robustness of the model is considerably large. The absolute FI score on ro- bustness evaluation datasets are improved at 2.33, 1.89, 1.35, and 1.63, respectively. Considering lots of studies theoretically identify trade-off be- tween robustness and accuracy (Tsipras et al., 2019; Zhang et al., 2019; Raghunathan et al., 2020), the improvement of both model's accuracy and robust- ness provides positive evidence for RQ4: COOLU can provide a more reasonable way of learning representations and classifiers.
4.4 Model Robustness Evaluation
We are more interest in the research question that In adherence to the natural properties of the Uni- versum class, does COOLU provide more reason- able way of learning by improving both model's accuracy and robustness? (RQ4) We attempt to demonstrate RQ4 by theoretical analysis and exper- imental evaluation of the robustness of the model. Theoretically, since the natural predictive rule of the Universum class is an inter-class pattern that does not belong to any target classes, traditional models are more likely to fit the noise in the Uni- versum class by memorizing various peculiarities of intra-class heterogeneous samples rather than finding the general predictive rule. However, our method can identify the inter-class predictive rule of the Universum class and hence classify out-of- distribution Universum samples more accurately. In addition, the closed decision boundaries we learned are analogous to model's knowledge bound- ary of each target class: the space inside the bound- ary represents what the model knows about a cer- tain class, i.e., the recognized patterns, whereas the space outside the boundary represents what the model doesn't tt know about this class from training
4.5 Ablations
We evaluated two recent OOD detection methods, ADB (Zhang et al., 2021) and OECC (Papadopou- los et al., 2021), to demonstrate the inappropri- ateness of OOD detection approaches for our pro- posed problem. The experimental results reveal that integrating ADB and OECC with classifica- tion models considerably weakens the performance of the original models, as illustrated in Table 2. Given that both ADB and OECC were primarily designed for OOD detection and not the unique problem presented in this work, their unsuitabil- ity is anticipated. Specifically, due to the inher-

---

CrossCategory 77.06
OOV
SpellingError 76.09
AppendIrr
SpanNER
75.14
87.34
SpanNER COOLU
81.39
83.15
81.12
89.89
InsenClausee
SwapFnt
SpellingFrror 77.36
Appendlr
A-GCN
77.81
86.8
86.59
A-GCN + COOLU
80.
88.69
78.71
88.22
Table 4: Comparison of model's robustness with and without closed boundary learning.
Conclusion
Method
Fl Fl/accuracy
SpanNER
92.09
In this work, we highlight an understudied prob- lem in dassification-basee tasks that the Univer- sum class is treated equally with target classes de- spite their significant differences. As a solution, we propose closed boundary learning method COOLU, which conforms the natural properties of the Universum samples. Specifically, we generate closed boundarics with arbitrary shapes, develop an inter-class rule-based strategy to estimate the probability of the Universum class, and propose boundary learning loss to adjust decision bound- aries. COOLU offers easy integration with most classification model, given that operates on repre- sentations of the final layer of classification models. Our method not only boosts the accuracy of SOTA models but also significantly enhances their robust- ness.
SpanNER N-pair pretraining
92.05
SpanNER COOLU (N-pair pretraining)
93 23.50
SCAPT
84.13%
SCAPT + N-pair pretraining
84.16%
SCAPT + COOLU (N-pair pretraining)
85.06%
Table 5:
The effect of pretraining on SpanNER (Fu etuetal al.,
2021) and SCAPT (Li et al., 2021b).
ent difference in the problem settings, the ADB method cannot utilize the label information of the Universum class, leading to a significant decline in accuracy, especially when the Universum samples constitute large portion in the span-based method. In addition, while OECC's problem setting, outlier exposure, aligns more closely with ours compared other general OOD detection methods, it still shows inappropriateness, diminishing the accuracy original classification models. This diminished accuracy can be attributed to: (1) error propagation in its two-step approach, and (2) the instability that arises from manually set thresholds.
Limitations
As a limitation, our method is not suitable for zero- shot or few-shot settings because the accuracy of GMM estimation is positively related to the number of samples used (Psutka and Psutka, 2019). Simi- larly, due the inherent low-dimension constraints of the GMM, an extensive increase in the number of classes could pose challenges to the efficacy of our framework. Nevertheless, given that the last layer dimension, i.e. class number, in our method is usually small and the initialized GMM parameters will be fine-tuned by the neural network, most clas- sification tasks are not limited by these constraints. Since the Universum class widely exists in NLP tasks and many general MIL tasks, method is applicable to most of these tasks.
We also conduct an ablation study on N-pair loss pretraining. In our method, N-pair loss is adopted for pretraining to learn initial representations and to speed up the training process. To demonstrate the effectiveness of our closed boundary learning method and to rule out the possibility that the im- provement of our model is due to the N-pair loss pretraining process, add an additional pretrain- ing step baseline models of SpanNER (Fu et al., 2021) and SCAPT (Li al., 2021b). Table 5 indi- cates that the pretraining process alone cannot im- prove the accuracy of the original baseline models. The improvement brought about by our proposed closed boundary learning is not result of the pre- training step but the result of the entire system.
Acknowledgements
The authors would like thank Edmond Lo, Lihui Chen, Xiyu Zhang, Zixiao Zhu, and the anonymous reviewers for their constructive comments and sug- gestions. The research was conducted at the Fu- ture Resilient Systems at the Singapore-EE Cen-
In addition, an ablation study on final layer di- mension is illustrated Appendix E.

---

tre, which was established collaboratively between ETH Zurich and the National Research Foundation Singapore. This research is supported by the Na- tional Research Foundation Singapore (NRF) under its Campus for Research Excellence and Techno- logical Enterprise (CREATE) programme.
Weiming Hu, Jun Gao, Bing Li, Ou Wu, Junping Du, and Stephen Maybank. 2018. Anomaly detection using local kernel density estimation and context- based regression. IEEE Transactions on Knowledge and Data Engineering, 32(2):218-233.
Gordon Hughes. 1968. On the mean accuracy of sta- tistical pattern recognizers. IEEE transactions on information theory, 14(1):55-63.
References
Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019. challenge dataset and effec- tive models for aspect-hased sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 6280- 6285, Hong Kong, China. Association for Computa- tional L Linguisties.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015, A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1508.05336.
Olivier Chapelle, Alekh Agarwal, Fabian Sinz, and Bernhard Schölkopf. 2007. An analysis of inference with the universum. Advances in neural information processing systems, 20.
Julian
Julia B Nakhleh, Robert Nowak, and Yixuan Li. 2022. Training ood detectors in their natural habitats. In International Conference on Ma- chine Learning, pages 19848-10865.. PMLR.
Guimin Chen, Yuanhe Tian, Yan Song, and Xiang Wan.
2021. Relation extraction with type-aware map mem- ories of word dependencies. Findings of the Asso- ciation.for Computational Linguistics: ACL-IJCNLP 2021, pages 2501-2512, Online. Association for Computational Linguistics.
Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji 2021a. A span-based model for joint overlapped and discontinuous named entity recognition. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 481 4828.
Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. 2020. A boundary based out-of of--iittrbutton classifier for generalized zero-shot learning. In Eit ropean Conference on Computer Vision, pages 572- 588. Springer.
Yuncong Li. Cunxiang Yin. Sheng- -ueng-huaaLu Zhong, and Xu Pan. 2020. Multi-instance multi-labe learning networks for aspect-category sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 3550-3560, Online. Association for Computa- tional Linguistics.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. 2021. SpanNER: Named entity re-frecognition as span pre- diction. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 7183-7195, Online. Association for Computa- tional Linguisties.
Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang. and Zhongyu Wei. 2021b. Learning implicit senti- ment in aspect-based sentiment analysis with super- vised contrastive pre-training. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 246-256, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Eduardo Dadalto Camara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. 2022. Igeood: An information geometry approach to out-of- distribution detection. In International Conference on Learning Representations.
Iris Hendrickx, Su Nam Kim, Zornissa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebastian Pad6, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2019. Semeval-2011 task 8: Multi- way classification of semantic relations between pairs of nominals. urXiv preprint arXiv: 19t.:19911.022....
Bo Liu, Hao Kang, Haoxiang Li. Gang Hua, and Nuno Vasconcelos. 2020. Few-shot open-set recog- nition using meta-learning In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- lern Recognition, pages 8798-8807.
Yifei Ming, Ying Fan, and Yixuan Li. 2022. Poem: duttof--iittibuttonn detection with posterior sampling. In International Conference on Machine Learning, pages 15650-15665. PMLR.
Dan Hendrycks, Mantas Mazeika, and Thomas Diet- terich. 2018 Deep anomaly detection with outlier exposure. In International Conference on Learning Representations.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. 2020. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5216-5223.
Alexander Hinneburg and Daniel A Keim. 1998. An efficient approach to clustering in large multimedia databases with noise. In Knowledge Discovery and Datamining (KDD (9D)) pages 58-65.

---

Aristotelis-Anee Angelos Papadopoulos. Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. 2021. Out- lier exposure with confidence control for out-of- distribution detection. Neurocomputing, 441:138- 150.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of- distribution detection with deep nearest neighbors. arXiv preprint urXtv:2204.06000
Sumil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel, Gopinath Chennupati. Tanmoy Bhattacharya, and Jeff Bilmes. 2021. An effective baseline for robust- ness to distributional shift. In 2021 20th IEEE In- ternational Conference Machine Learning and Applicutions (ICMLA), pages 278-285. IEEE.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. 2018. Generative probabilistic novelty de- tection with adversarial autoencoders. Advances in neural information processing systems, 31.
Yuanhe Tian, Guimin Chen, Yan Song, and Xiang Wan. 2021. Dependency-driver relation extraction with attentive graph convolutional networks. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4458 4471, Online. Association for Computational Linguistics.
Josef V Psutka and Josef Psutka. 2019. Sample size for maximum-likehine likelihood estimates of gaussian model de- pending on dimensionality of pattern space, Pattern Recognition, 91:25-33.
Zhiquan Qi, Yingjie Tian, and Yong Shi. 2012. Twin support vector machine with universum data. Neural Networks, 36:1 16:112-1199 119.
Erik F. Tjong Kim Sang and Fien De Meulder, 2003. Introduction to the CoNLL-2003 shared task: Language-independer independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang. John C Duchi, and Percy Liang. 2020. Understanding and mitigating the tradeoff between robustness and accuracy. In Proceedings of the 37th International Conference Machine Learning, pages 7909-7919,
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. 2021. Panda: Adapting pretrained features for anomaly detection and segmentation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2806-2814,
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2019. Ro- bustness may be at odds with accuracy. In Inter- national Conference on Learning Representations, 2019.
Jie Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayaman. 2021. simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint arXiv:2106.0002
Juncheng Wan, Dongyu Ru, Weinan Zhang, and Yong Yu. 2022. Nested named entity recognition with span- level graphs. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 892 903, Dublin, Ireland. Association for Computational Linguisties.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexan- der Binder, Emmanuel Müller, and Marius Kloft. 2018. Deep one-class classification. In International conference on machine learning. pages 4393-4402 PMLR.
Xiao Wang, Qin Liu, Tao Gui, Qi Zhang. Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang. Qinzhuo Wu, Zhengyan Li. Chong Zhang, Ruotian Ma. Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng, Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu. and Xuanjing Huang. 2021. TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pages 347-355, Online. Association for Computational Linguistics.
Vatsal Sharan, Parikshit Gopalan, and Udi Wieder. 2018.
Efficient anomaly detection via matrix sketching. Ad- vances in neural information processing systems, 31.
Sughash Sharma. 1996. Applied multivariate tech- niques.
Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. 2020. P-odn: Prototype-based open deep network for open set recognition. Scientific reports, 10(1):1-13.
Jason Weston, Ronan Collohert, Fabian Sinz, Léon Bot- tou, and Vladimir Vapnik. 2006. Inference with the universum. In Proceedings of the 23rd international conference on Machine learning, pages 1009-1016.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29.
Xuanli Lisa Xie and Gerardo Beni. 1991. A validity measure for fuzzy clustering. IEEE Transactions Pattern Analysis & Machine Intelligence, (3(0))8841- 847.
Mahito Sugiyama and Karsten Borgwardt. 2013. Rapid distance-basse outlier detection via sampling. Ad- vances in neural information processing systems, 26.

---

Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. 2021a. Semantically coherent out-of-distribution de- tection. In Proceedings of the IEEE/CVF Interna- lional Conference on Compuler Vision, pages 8301- 8309.
for unsupervised anomaly detection. In International conference on learning representations.
A Compactness of the Universum Class of the Test Set
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. 2021b. Generalized out-of-distribution detec- tion: A survey, arXiv preprint arXiv:21 10. 1134.
We evaluate the compactness of the Universum class and target classes on the test data and de- pict the result in Figure 3. The representation of test samples after learning with open bound- ary classifiers by SpanNER (Fu et al., 2021) is used for evaluation. We evaluate the compactness based on the root-mean-square standard deviation (RMSSD) (Sharma, 1996), and the mean square distance (MSD) (Xie and Beni, 1991), which are commonly used in clustering studies to evaluate compactness of a cluster. The smaller the RMSSD or MSD, the better the compactness. It is illustrated in Figure 3 that the compactness of the "OTHER" class is significantly worse than target classes. No- tably, the class with the second compactness is the "MISC" class, i.e., the "miscellaneous" class, which also type of the Universum class.
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4904 4917, Dublin, Ireland. Association for Computational Linguistics.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Compulational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
Alireza Zaeemzadeh, Niccolo Bisagno, 7eno Sam- bugaro. Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. 2021. Out-of-distribution detection using union of 1 I-dimensional subspaces. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
Given that the Universum class has the highest number of samples in the datasets we examined, it should be represented best through training and hence formulate the most compact cluster in the representation space for the test set. Yet, our em- pirical findings paint a different picture. Then, interesting question was raised: Why does the Uni- versum class, despite being the largest, exhibit the worst compactness in its learned representa- tions?
Hanlei Zhang, Hua Xu, and Ting-En En Lin. 2021. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14374- 14382.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. 2019. Theo- retically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482. PMLR.
Our research indicates that the answer is rooted in its inherent inter-class pattern. The Universum class is defined as a cluster of samples that do not belong to any of the predefined target classes. Con- sidering human annotation practices, entity is labeled as Location when it aligns with established patterns of Location entities. In contrast, a sample is labeled as Others not due to intra-class patterns specific to the Others class, but because it fails to conform to the patterns of Location, Person, or Organization. Consequently, current classification models are designed to recognize intra-class pat- terns and unable to discern the inherent inter-class predictive rule of the Universum class, which will result in the poorer compactness of the Universum class.
Xiang Zhang and Yann LeCun. 2017. Universum pre- scription: Regularization using unlabeled data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Dosition-awars attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017), pages 35-45.
Enwei 7hu and Jimpeng Li. 2022. Boundary smooth- ing for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Compustaional Linguistics (Volume I: Long Papers), pages 7096-7108 108, Dublin, Ireland. Association for Computational Linguistics.
Additionally, from the lens of representation learning, the Universum class essentially encap- sulates "everything else" in a given task. Thus, no
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. 2018. Deep autoencoding gaussian mixture model

---

OTHER
OTНER
MISC
MISC
Type
Typ
LOC
LOC
PER
PER
ORG
CRS
Mean Siuurne Distance
Rost-mman--squee standard devation
(a)
(b)
Figure 3: The compactness evaluation of the Universum class and target classes of the test data of NER task.
matter the volume this class, it's implausible to capture every nuance and pattern inherent to such a broadly defined class. This insight also sheds light on why, despite its significant size, the Uni- versum class demonstrates the least compactness in its representations.
isting Universum samples is important to generate accurate decision boundaries in our problem.
B.1.2 Difference in Methodology
By definition, the OOD detection problem assumes that the training data do not contain any OOD samples. However, a branch the OOD studies, known as outlier exposure (Katz-Sumuels et al., 2022; Ming et al., 2022; Yang et al., 2021a; Thulasi- dasan et al., 2021; Mohseni et al., 2020; Hendrycks et al., 2018), introduces auxiliary outlier data dur- ing training. The introduced auxiliary data makes it close to the format of our raised classification prob- lems with the Universum class. However, outlier exposure methods are not suitable for our prob- lem. The outlier exposure method mostly adopts a two-step approach that consists of multi-class clas- sification and OOD identification. Such two-step approach will suffer from error propagation prob- lem. In addition,the OOD identification step dis- tinguishes OOD and ID samples based on score obtained by cross entropy or energy function. How- ever, both cross entropy and energy function are monotonically varying. As a result, the decision boundary derived from threshold score of the monotonically varying function is an open bound- ary, which leaves the heterogeneity and represen- tativeness issues pointed out in this paper still unresolved.
B Related Works
Continuing from previous discussions, we delve further into the related works on closed boundary learning methods here.
B.1 Closed Boundary Learning Methods
Closed boundaries are often adopted in research fields of out-of-distribution (OOD) detection (Gomes et al., 2022; Ren et al., 2021; Chen et al., 2020), open set recogmition (Zhang et al., 2021; Liu et al., 2020), anomaly detection (Zong et al., 2018), and outlier detection (Sharan et al., 2018; Sugiyama and Borgwardt, 2013). We borrow the term "gen- eralized OOD detection" from (Yang al., 2021b) encapsulate these problems and discern their dif- ferences from our proposed classification with the Universum class problem.
B.1.1 Difference in Problem Setting
Classification tasks can categorized into prob- lems based on closed-world assumption and open- world assumption (Yang et al., 2021b). The general- ized OOD detection is treated under the open- world assumption, while the classification problem with the Universum class is treated under the closed- world assumption. In addition, the OOD samples are not available in the training data in generalized OOD detection, whereas a considerable number Universum samples are included in the training data our problem setting. The information of ex-
From methodological point of view, our work is also different from the works in generalized OOD using closed boundaries. In generalized OOD stud- ies, the closed boundaries are formulated by the classic density-based method (Pidhorskyi al., 2018: Hu al., 2018), one-class classification method (Reiss et al., 2021: Ruff et al., 2018), or distance- based method (Gomes et al., 2022; Sun

---

al., 2022; Zhang et al., 2021: Zaeemzadeh et al.. 2021; Shu et al., 2020). The distance-based meth- ods are limited to spherical boundary shapes but our method can generate arbitrary shape bound- aries. The one-class classification method formu- lates only one closed boundary between positive and negative samples while our work generates closed boundaries for all target classes. Finally, only positive samples are used to learn decision boundaries in density-basee method, while both target class samples and Universum samples are used in our work.
D.2 Training Process
During pretraining process, all parameters of the original model 0 are learned. We employ GMM estimation training data after pretraining and obtain the initial value of pi, Σ, and Tij. where E 21...... n - 1]. {1, 2....... m). n is the number of classes, and m is the number of GMM components. Through preliminary experiments. we observed that the number of GMM components has minimal impact on our model's performance. Thus, we typically select = 4 in our experiments. The threshold values is initialized around the a quantile of p(h,, C.) values (k {0, 10... .N; 1}). where a is the accuracy or Fl score of the orig- inal model. With our inter-class rule-based proba- bility estimation for the Universum class, we obtain [p(h;, Ci), phh,, Cn-1). U)]- Then, the original model parameters 0, GMM parame- ters μ4: E, Trj and threshold values & are learned by cross-entropy loss and our proposed boundary learning loss.
Defining the Universum Class
Universum class exists in many tasks and datasets as we summarized in Table 1. Notably, the Uni- versum class has various names such as other and miscellaneous, etc. In sentiment analysis, the neu- tral class can be considered as the Universum class because the word neutral is defined "having no strongly marked or positive characteristics fea- tures", which means the neutral class collection of all samples without strong emotions. Similarly, the no relation class in the relation extraction task can be considered as the Universum class.
We use NVIDIA RTX A5000 GPUs to run the experiments and the model parameters are mostly follow the original baseline models.
D.3 Robustness Evaluation
We evaluate the robustness of the model based on TextFlint (Wang et al., 2021), robustness evalu- ation toolkit for NLP tasks. There are two kinds of transformations provided by TextFiiit to gen- erate the robust evaluation dataset, namely univer- sal transformation and task-specific transformation. We adopt two universal transformations and two task-specific transformations to the test set of NER and RE task and generate four robustness evalua- tion datasets for each task. The terms of different transformations are explained below.
D Implementation Details
D.1 Baseline Models
We reproduce the baseline models based the officially released source code, and apply closed boundary learning on the source code. All reported results are the average of three runs. It should be noted that some results of baseline models are slightly different from those given in the original papers due to the variations in random seeds and package versions when reproducing baseline models from their officially released codes. Nevertheless, baseline models and models with closed boundary learning are fairly compared in our work under the same random seed and deep learning environment.
"SpellingError": Universal transformation.
Brings slight errors to words in the test sam- ples.
"AppendIr":: Universal transformation. Add irrelevant information to test samples.
In the six baseline models we selected, different results based on multiple language models are often reported in one work. We choose one of the pretrained models used in each work and reproduce the baseline models. The pretrained language model we used each baseline and their reported results are summarized Table 6.
Task-specific transforma- tion for NER. Replace the entity spans with substitutions from different category.
"CrosSCategory"":
"OOV": Task -psk-specific transformation for NER. Replace the entity spans with substitutions out of vocabulary.

---

Method
Pretrained Model
Reported Fl/accuracy
SpanNER (Fu al., 2021)
BERT-base
92.28
BS (Zhu and Li, 2022)
RoBERTa
93.65
A-GCN (Tian et al., 2021)
BERT-base
89.16
TaMM (Chen et al., 2021)
BERT-base
89.18
AC-MIMLLN (Li et al., 2020)
Glove
76.42
SCAPT (Li et al., 2021b)
BERT-base
85.24
Table 6: The pretrained models chosen for each baseline model and the corresponding Fl score/accuracy reported in the original paper.
on the accuracy of the model on the SpanNER (Fu et al., 2021) with closed boundary learning and present the result in Figure 4. The FI score of the test grows with increasing of dimensions and reaches a maximum value of 93.88 when the dimension is seven, and then decreases with the dimension. The trend fits well with the Hughes phenomenon (Hughes, 1968). Our method is quite robust with the dimension and the overall result of SpanNER + COOLU reported in Table 2 is set ten rather than the optimal value.
93.8
Macro 13 Soore
93.6
93.2
10
15
20
25
30
the Loght Space
Figure 4: Impact the last layer dimension on the accuracy of the model.
"InsertClause": Task-specific transformation for RE. Change sample sentences by append- ing adjuncts from the aspect of dependency parsing.
"SwapEnn":: Task-specific transformation for RE. Swap the named entities in sentence into entities of the same type.
Specifically, the models are trained and validated on the original training set and validation set, but the test set transormed into the robustness eval- uation dataset by the transformations proposed by TextFlint. Then, modelw are tested on the trans- formed test set.
E
The Impact of the Final Layer Dimension
The last layer's dimensionality can affect the perfor- mance of the model. Recalling the classic Hughes phenomenon (Hughes, 1968) that the model ac- curacy is monotonically increasing first and then monotonically decreasing with the dimension of data, the dimension of the final layer may cho- sen to boost model performance.
We investigate the effect of last layer dimension


Reviews:
[{"review_id": "dLdjZqTjgt", "reviewer": "Reviewer_G4Yh", "summary": "", "questions": "A. How the gradients are propagated through loss function, GMM etc. to train the model?\nB. Why do you think that comparison only with the baseline (you treat OECC as an ablation) is sufficient to show the usefulness of your approach?\nC. To what extent training the model with your method is more computational expensive than the training of the baseline?\nD. What statistical test were used?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer G4Yh,\n\nWe sincerely appreciate your thorough review and valuable feedback on our paper. Your insights have been instrumental in helping us improve the quality of our work. Please find below a summary of our responses to your comments and concerns.\n\n---\n**[C1]: it is difficult to understand how actually the whole framework works**\n\n**[A1]: We genuinely appreciate your feedback, emphasizing the need for a clearer and more integrated presentation of our method. We recognize this gap and are committed to enhancing the clarity in the revised manuscript.**\n\nTo address this, we provide a succinct breakdown of our framework's workings: \n* **Initialization Post-Pretraining**: After pre-training, we employ the GMM for each target class. The Expectation-Maximization (EM) algorithm is employed to set the initial values for the GMM parameters $\\mu_i$, $\\Sigma_i$, and $\\pi_{ij}$. As we transition to the training phase, the parameters of GMM are treated as learnable parameters. Contrary to traditional methods using the EM algorithm for continuous updates, these parameters are dynamically updated by the neural network throughout the training process.\n* **Probability Estimation**: Probability distributions of target classes are estimated using GMM, as articulated in Eq. 2. The probability distribution of the Universum class is computed through our Inter-Class Rule-Based Probability Estimation method, which is represented in Eq. 4.\n* **Training Optimization**: During the training process, we use a combined loss function, summing the cross-entropy loss with the boundary learning loss for optimization.\n\nGiven your feedback, we realize that while each component of our framework was detailed in the paper, the absence of an overall description might have fragmented the reader's understanding. To rectify this, we commit to enrich the revised version with a schematic representation of our framework.\n\n**In response to your question on gradient propagation**: updating mechanics for the GMM parameters are introduced in the \"Initialization Post-Pretraining\" above, where we emphasize that these parameters are dynamically updated by the neural network rather than the EM algorithm. Addressing the non-differentiability points in Eq. 4, similar to the ReLU function at 0, in practical applications, exact non-differentiable points, like 0 for ReLU or the boundary in Eq. 5, are not likely reached due to precision constraints. Moreover, near this boundary, the gradients of the components in Equation 4 exhibit close resemblances, which curbs the potential for the model to be steered in suboptimal directions near the boundary.\n\n---\n**[C2]: Questions and concerns with experiments.**\n\n**[A2]: We greatly value your insightful feedback on our experiments. Please find our detailed responses to each of your points below:**\n\n* Comparison with Other Methods:\n\nWe appreciate your suggestion to broaden our experimental comparisons. In light of this, we have incorporated **additional baselines** for evaluation. Specifically, we assessed both the ADB [1] and KNN-contrastive learning [2] methods. Experimental findings consistently demonstrate that our method achieves superior performance compared to these baselines.\n\n| | SpanNER | A-GCN | AC-MIMLLN |\n|---------|----------|----------|----------|\n| ADB | 87.81 | 85.99 | 71.78 |\n| KNN-Contrastive | 89.08 | 86.32 | 72.16 |\n| COOLU |   **93.50** | **89.33** | **77.35** |\n\n***Table 1**: Performance comparison with additional baselines.*\n\n[1] Zhang, Hanlei, Hua Xu, and Ting-En Lin. \"Deep open intent classification with adaptive decision boundary.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 16. 2021.\n\n[2] Zhou, Yunhua, Peiju Liu, and Xipeng Qiu. \"KNN-contrastive learning for out-of-domain intent classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n* Statistical Test:\n\nWe used the paired t-test for our statistical evaluation and ensured its assumptions were met.\n\n* Definition of \"reasonable way\":\n\nThank you for pointing out this ambiguous phrasing. By \"reasonable way,\" we aimed to convey that COOLU offers a method that is capable to improve both the accuracy and robustness of classification models. We will refine this statement in the revised version to better communicate our intention.\n\n* Ablation study on N-pair loss:\n\nThe ablation study of N-pair loss is provided in Appendix E. Our method relies on the pre-trained feature space for GMM initialization, thus, removing the pretraining step isn't feasible. However, to transparently demonstrate the effectiveness of our approach, we integrate N-pair loss pretraining into the baseline method. This is done to clarify that the observed performance improvements stem from the unique attributes of our proposed method, rather than merely the addition of the pretraining step.\n\n* Experimental details:\n\nWe appreciate the emphasis on clarity of experimental details. The choice of GMM components was informed by our preliminary experiments. We observed that the number of GMM components had a minimal impact on our model's results. Moreover, there wasn't a need for an excessive number of components. Thus, we set four as the choice for our method. In terms of training time, our method took additional pre-training step (~20% of the total training epochs) and one more epoch specifically for the GMM parameters' initialization, in contrast to the baseline. We will ensure these details are clearly presented in the revised version.\n\n---\n**[C3]: Concern on motivation**\n\n**[A3]: We apologize for any confusion stemming from our paper's presentation and value your insight on this matter. The crux in addressing your concern lies in recognizing the inherent inter-class pattern of the Universum class. To elucidate this further, we would like to address your concern through experimental results and theoretical explanations.**\n\nFrom an experimental standpoint, we understand your view that a larger sample size would yield better representativeness of the class.   Given that the Universum class is the most populous in the datasets we examined, it should be represented best through training and hence formulate the most compact cluster in the representation space for the test set. **Yet, our empirical findings paint a different picture.** As depicted in Figure 1 (b) of the paper, the averaged compactness of the Universum class is significantly worse, being over two times less compact than most target classes. The experimental details can be found in Appendix A.\n\nThen, an interesting question was raised \"**Why does the Universum class, despite being the largest class, exhibit the worst compactness in its learned representations?**\" Our research indicates that the answer is rooted in its **inherent inter-class pattern**. The Universum class is defined as a cluster of samples that do not belong to any of the predefined target classes. Considering how humans recognize this class, during dataset annotation, an entity is labeled as \"*Location*\" because it aligns with the patterns of \"*Location*\" entities. In contrast, another sample is labeled as \"*Others*\", not because it follows the inner patterns of the \"*Others*\" entities, but because it doesn't match patterns of \"*Location*\", \"*Person*\", and \"*Organization*\". Consequently, neural networks are better suited to recognize this class by its lack of alignment with target classes (its inter-class pattern) rather than by seeking specific intra-class patterns from its samples. \n\nFrom the lens of representation learning, the Universum class essentially encapsulates \"**everything else**\" in a given task. Thus, no matter the volume of this class, it's implausible to capture every nuance and pattern inherent to such a broadly defined class. This insight sheds light on why, despite its significant size, the Universum class demonstrates the least compactness in its representations.\n\nFurthermore, we delve into the consequences of not giving the Universum class specialized treatment, such as overfitting and diminished robustness between lines 86 to 107 in our paper. Thank you for your feedback, We will refine our motivation for greater clarity in the revised revision\n\n---\n**[Respond to Questions and typos]**\n\nThank you for your informative questions. These questions are already addressed in our previous responses. \n\nThank you for your thorough examination of our work and for highlighting the typos. We commit to conducting a meticulous review of the manuscript and rectifying all typos in our revised version. \n\n---\nWe deeply appreciate your insights, which have significantly enriched the quality of our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- interesting idea and relatively new problem\n- formulation of a new loss function producing closed decision boundaries (unfortunately, some parts of it are not clear -see Reasons to Reject)", "reasons_to_reject": "- Overall, it is difficult to understand how actually the whole framework works (lack of a pseudocode or a schema). After pretraining, GMMs is estimated (with Expectation-Maximization? = it is not specified). Then the probabilities from GMM are used in a loss function to learn thresholds ξ. Later, somehow cross-entropy loss is applied on class probabilities which are computed basing on GMM with hard thresholds (Eq. 4) to update the models' parameters. How the gradients are propagated through loss function with probabilities computed using hard thresholds (for Universum class) and GMM (trained by EM?) to finally get to the classification model?\n- Experiments. \n\t- The authors compare their approach by extending with it several models. However, apart from the baseline only one other method for outlier exposure (which is unsuitable for universum classes according to the authors) is taken into account. Other related approaches are not studied.\n\t- The authors performed some statistical tests, but they do not specify which one. Were the test assumptions properly checked?\n\t- The authors formulate a research question \"does COOLU provide a more reasonable way of learning representations and classifiers\". What \"reasonable way\" of learning classifiers actually means? This statement is very imprecise.\n\t- Lack of some ablation studies, e.g. how well the method works without N-pair loss pretraining?  \n\t- Some experimental details are not clear. E.g. How the number of GMM components was selected? What is the time complexity of this approach? (how the training time of the new method compare with the training time of the baseline?)\n\n- Motivation\n\tThe authors claim the lack of representativeness of Universum class in training data. However, in most datasets used in the experimental study such class is quite large, hence having abundant representation in the data. That would mean that basically all the classes are not represented well enough in the training data. The authors probably have in mind the fact that Universum class is an aggregation of \"all possible implicit classes\" but still this claim about lack of representativeness is only supported by a visualization (which is not based on any theoretical model or real data).", "questions_to_authors": "A. How the gradients are propagated through loss function, GMM etc. to train the model?\nB. Why do you think that comparison only with the baseline (you treat OECC as an ablation) is sufficient to show the usefulness of your approach?\nC. To what extent training the model with your method is more computational expensive than the training of the baseline?\nD. What statistical test were used?", "paper_topic_and_main_contributions": "The paper points out a previously underexplored problem of the Universum class (\"other\", \"miscellaneous\") in multi-class classification problems. New approach based on N-pair loss pretraining, GMM and new loss function is proposed, which produced closed decision boundaries around remaining (not Universum) classes. The Universum class is specially handled (lack of closed decision boundary) and treated as a remaining (\"out-of-distribution\") class. The experiments on NER, relation extraction and sentiment analysis demonstrate that the framework offers approx. 1% improvement in F1/acc measures.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- \"Figure 2 is an illustration of formulating an arbitrary shape boundary in a three- dimensional space.\" = it is a 2d shape. The third dimension is the value of prob. density.\n- grammar \"An intuitive way to incorporate the above rules is a two-step method consists of Universum class detection and target classes classification\n- (6) presents two separate symbols,  remove the bracket\t\n\n\nIn 3.3.1 authors introduce 2 rules to estimate probabilities. Later, they claim that using this rules (which work in a mutually exclusive conditions) would actually result in a pipeline method which has the issue of error propagation. In fact, both rules can be easily rephrased into only one rule and their direct application does not have to lead to error propagation. But I agree with the motivation about facilitating the learning of the\nneural network."}}, {"review_id": "7sIWaYm0sA", "reviewer": "Reviewer_gG1E", "summary": "", "questions": "1) How does the COOLU improving NER and ACSA tasks compare to the method improving RE tasks? More improvement can find the NER and ACSA tasks.  \n2) I will suggest a more detailed experimental setting such as a case study and visualization. It may be better to analyze how to discriminate the multi-hop relations (universum or targe class) and their heterogeneity and diversity in RE.  \n\n ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer gG1E,\n\nWe sincerely appreciate your thorough review and valuable feedback on our paper. Your insights have been instrumental in helping us improve the quality of our work. Please find below a summary of our responses to your comments and concerns.\n\n---\n**[C1]: GMM is not an ideal module, which is constraint on the definite and low dimensionality of the class setting.**\n\n**[A1]**: Thank you for drawing attention to the potential constraints of GMM, especially in the context of definite and low-dimensionality class settings.\n\nOur framework is applied after the final layer of classification models, which is a linear layer that maps data from a high-dimensional space down to a dimensionality equal to the number of classes of the task. Consequently, the input dimension for GMM is essentially fixed by the number of classes. Additionally, given that most classification tasks involve a relatively small number of classes, GMM's limitations regarding low dimensionality become less critical for our framework. Finally, in our framework, the neural network dynamically updates both the GMM parameters and the data distribution of the GMM input in the feature space throughout the training process, which may also enhance the performance of GMM.\n\nHowever, we concur that as the number of classes grows significantly, GMM's inherent constraints could challenge our framework's effectiveness. Your feedback highlights the importance of exploring better alternatives in future works. We appreciate this perspective.\n\n---\n**[C2]: A lack clarity about the mapping between the threshold value, the scale of data, task types, number of classes and the performance.**\n\n**[A2]**: Thank you for highlighting this important point. We understand that these factors' effect on performance needs a clearer exposition in our work. Here's a deeper analysis:\n\n* **Threshold Value**: Contrary to a manually selected parameter, the threshold value is a learnable parameter in the model. This treatment prevents us from distinctly evaluating its impact on performance.\n* **Scale of Data**: Our current experiments utilize the full data supervised training setting, aligning with the classification tasks we evaluated. However, we would like to incorporate experiments with varied data proportions (e.g., 25%, 50%, 75%) in our revised version, anticipating insightful findings.\n* **Task Types**: As shown in Table 1 of the paper, the proportion of the Universum class fluctuates across tasks. Notably, the RE task has the lowest proportion at 17.4%, compared to 90% and 43.4% in other tasks. This disparity might primarily account for the least improvement we observed on the RE task.\n* **Number of Classes**: As the number of classes increases, GMM's inherent dimensionality constraints become more pronounced, potentially impacting performance. This may be validated in the RE task, which, with the largest number of classes, shows less improvement with our method.\n\nWe notice that while most of the above information is present in the paper, it hasn't been systematically collated and analyzed. **Your feedback is insightful, and in response, we will present a clearer mapping between these factors, aiding readers in understanding their influence on the model's performance better**.\n\n---\n**[Q1]: How does the COOLU improving NER and ACSA tasks compare to the method improving RE tasks? More improvement can find the NER and ACSA tasks.**\n\n**[A1]**: Thank you for the question. We have provided an analysis of this issue in \"Task Types\" and \"Number of Classes\" in answers of **C2**. \n\n---\n**[Q2]: I will suggest a more detailed experimental setting such as a case study and visualization.**\n\n**[A2]**: Thank you for your constructive feedback. We will integrate the visualisation and case study in the revised manuscript.\n\n---\n**[Response to Typos and Presentation Improvement]**\n\nThank you for your thorough examination of our work and for highlighting the typos and suggestions on the presentation. We will enhance the clarity and presentation of our paper following your suggestions.\n\n---\nWe deeply appreciate your insights, which have significantly enriched the quality of our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)  The paper is clear and well-written. I appreciate the comprehensive elaboration with four research questions and arguments about the beneficial classification performance and robustness of COOLU for universum class and other classes.\n2)  This paper shows the previously ignored connections between the universum class with inherent properties and the misclassification of samples of interest in diverse classification tasks.\n3)  The idea of using closed decision boundaries for classification tasks is well-motivated, and this paper is easy to follow.", "reasons_to_reject": "1) I understand the insight of GMM with inter-class rule-based probability estimation for unversum classification. But it seems GMM is not an ideal module, which is constraint on the definite and low dimensionality of the class setting. \n2) A lack clarity about the mapping between the threshold value, the scale of data, task types, number of classes and the performance. \n", "questions_to_authors": "1) How does the COOLU improving NER and ACSA tasks compare to the method improving RE tasks? More improvement can find the NER and ACSA tasks.  \n2) I will suggest a more detailed experimental setting such as a case study and visualization. It may be better to analyze how to discriminate the multi-hop relations (universum or targe class) and their heterogeneity and diversity in RE.  \n\n ", "paper_topic_and_main_contributions": "This paper formulates a new question of distinguishing the Universum class and the interest classes in classification tasks. The authors aim to leverage closed decision boundaries with a rule-based probability estimation strategy to classify the universum and target class. The experiments on the proposed COOLU demonstrate the comprehensive improvement over the BS, A-GCN, TaMM, AC-MIMLLN, SpanNER, and SCAPT benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) The authors could abbreviate the contributions of this work.\n2) The detailed dataset descriptions could be shown in the experimental methodology.\n3) Minor: the full stop is missing in L1123.\n\n"}, "extra_text": {}}, {"review_id": "8tHUd0QzMm", "reviewer": "Reviewer_RZMD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer RZMD, \n\nWe sincerely appreciate your thorough review, constructive feedback, and valuable support on our paper. Your insights have been instrumental in helping us improve the quality of our work. Please find below a summary of our responses to your comments and concerns.\n\n---\n**[C1]: I am surprised that a closed boundary method can do well. Is it because the NER and RE tasks are particularly amenable to such a method? For supervised classification, I am not sure. Can you include some traditional supervised classification methods.**\n\n**[A1]**: Thank you for raising this question. To address your concern, we provide a detailed explanation below.\n\nThe six classification methods we evaluated, spanning the RE, NER, and sentiment analysis tasks, are all supervised classification methods. Specifically:\n* The NER task classifies entity spans into different entity classes.\n* The RE task classifies the entity pairs into various relation classes.\n* The sentiment analysis task classifies sentences into different sentiment classes. \n\n**While the strategy for representation learning differs among these methods, they adhere to the standard procedure typical of supervised classification methods**: they learn representation, then map representation space into logits, subsequently pass logits through a softmax function to derive probabilities for each class, and finally employ the cross-entropy loss for optimization.\n\nTo answer your question about **why a closed boundary method can do well**, we believe the answer lies in two aspects:\n* The Universum class exhibits very different properties from the target classes. Target class samples follow intra-class patterns: considering the human annotation process, an entity is labeled as \"*Location*\" because it aligns with the common patterns of \"*Location*\" entities. **In contrast, the Universum samples follow a distinct inter-class pattern: an entity is labeled as \"*Others*\" not because it aligns with common patterns of \"*Others*\" entities, but because this entity doesn't align with patterns of \"*Location*\", \"*Person*\", and \"*Organization*\" classes**. Overlooking the special properties of the Universum class can lead to problems such as overfitting, diminished robustness, and misclassification of unseen samples, as we illustrated between lines 86 to 121 in our paper.\n* On the other hand,  the question arises: Why do closed boundary methods in OOD detection fail to improve the performance of these classification tasks? The primary reason is that they are originally designed for a different problem setting. This mismatch leads to problems such as error propagation inherent to two-step methods, instability from manually set thresholds, and the underutilization of Universum class samples, all leading to poor performance. In contrast, our framework introduces the \"Inter-Class Rule-Based Probability Estimation for the Universum Class\", which allows for the joint learning of both the Universum class and target classes. Additionally, we propose the boundary learning loss to facilitate the learning of closed boundaries. We also transform the manually set threshold value into a learnable parameter of the neural network. \n\n---\n**[C2]: Include some OOD detection methods as baselines.**\n\n**[A2]**: We highly value your constructive suggestion. In light of this, we have incorporated **additional baselines** for evaluation. Specifically, we assessed both the ADB [1] and KNN-contrastive learning [2] methods. Experimental findings consistently demonstrate that our method achieves superior performance compared to these baselines. We are willing to include more baseline experiments in the revised manuscript.\n\n| | SpanNER | A-GCN | AC-MIMLLN |\n|---------|----------|----------|----------|\n| ADB | 87.81 | 85.99 | 71.78 |\n| KNN-Contrastive | 89.08 | 86.32 | 72.16 |\n| COOLU |   **93.50** | **89.33** | **77.35** |\n\n***Table 1**: Performance comparison with additional baselines.*\n\n[1] Zhang, Hanlei, Hua Xu, and Ting-En Lin. \"Deep open intent classification with adaptive decision boundary.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 16. 2021.\n\n[2] Zhou, Yunhua, Peiju Liu, and Xipeng Qiu. \"KNN-contrastive learning for out-of-domain intent classification.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n---\nWe deeply appreciate your insights, which have significantly enriched the quality of our paper. We look forward to hearing your feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed approach of building closed boundary classifier to deal with Universum class is interesting. \n\nThe proposed method is novel. ", "reasons_to_reject": "I am surprised that a closed boundary method can do well. Is it because the NER and RE tasks are particularly amenable to such a method? For supervised classification, I am not sure. Can you include some traditional supervised classification methods. \n\nOOD detection researchers have tried different types of approaches to build closed boundary classifiers. In NLP, there are many papers, e.g., DOC: Deep Open Classification of Text Documents. EMNLP-2017; Deep open intent classification with adaptive decision boundary, AAAI-2021; KNN-contrastive learning for out-of-domain intent classification. ACL2022; Adversarial Self-Supervised Learning for Out-of-Domain Detection, NAACL-2021.  \n\nYour baselines should include some SOTA OOD detection methods. I think kNN with a pre-trained feature extractor should do quite well too. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a closed-boundary classification method to deal with the miscellaneous class with many clusters. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the issue of distinguishing the "Universum class" from interest classes in classification tasks. It introduces a closed-boundary classification method with a rule-based probability estimation approach to classify the Universum and target classes. The proposed COOLU method outperforms several benchmarks like BS, A-GCN, TaMM, AC-MIMLLN, SpanNER, and SCAPT. The approach handles the miscellaneous class effectively, offering approximately a 1% improvement in F1/accuracy measures in NER, relation extraction, and sentiment analysis experiments.

The problem addressed is important in many real-world scenarios, and the results presented in the paper are promising. However, the main issue with the work lies in its clarity, as indicated in the discussion, which required several additional details to fully appreciate the paper and this research.
Decision: Accept-Findings