Closed Boundary Learning for NLP Classification Tasks with the Universum Class
Authors: Hanzhang Zhou
Zijian Feng
Kezhi Mao
Edmond Y.M. Lo
Lihui Chen
Venue: Submitted to ICLR 2023 (2023)

Abstract: The Universum class, often known as the other class or the miscellaneous class, is defined as a collection of samples that do not belong to any class of interest. It is a typical class that exists in many classification-based tasks in natural language processing (NLP), such as relation extraction, named entity recognition, sentiment analysis, etc. During data labeling, a significant number of samples are annotated as Universum because there are always some samples that exist in the dataset but do not belong to preset target classes and are not of interest in the task. The Universum class exhibits very different properties, namely heterogeneity and lack of representativeness in training data; however, existing methods often treat the Universum class equally with the classes of interest. Although the Universum class only contains uninterested samples, improper treatment will result in the misclassification of samples of interest. In this work, we propose a closed boundary learning method that treats the Universum class and classes of interest differently. We apply closed decision boundaries to classes of interest and designate the area outside all closed boundaries in the feature space as the space of the Universum class. Specifically, we formulate the closed boundaries as arbitrary shapes, propose a strategy to estimate the probability of the Universum class according to its unique property rather than the within-class sample distribution, and propose a boundary learning loss to learn decision boundaries based on the balance of misclassified samples inside and outside the boundary. We evaluate our method on 6 state-of-the-art works in 3 different tasks, and the performance of all 6 works is improved. Our code will be released on GitHub.

Under review as a conference paper at ICLR 2023
CLOSED BOUNDARY LEARNING FOR NLP CLASSIFICA- TION TASKS WITH THE UNIVERSUM CLASS
Anonymous authors Paper under double-blind review
ABSTRACT
The Universum class, often known as the other class or the miscellaneous class, is defined as collection of samples that do belong to any class of interest. It is typical class that exists in many classification based tasks in natural language processing (NLP), such as relation extraction, named entity recognition, sentiment analysis, etc. During data labeling, significant number of samples are annotated as Universum because there are always some samples that exist in the dataset but do not belong to preset target classes and are not of interest in the task. The Universum class exhibits very different properties, namely heterogeneity and lack of representativeness in training data; however, existing methods often treat the Universum class equally with the classes interest. Although the Universum class only contains uninterested samples, improper treatment will result in the misclassification of samples of interest. In this work, we propose a closed boundary learning method that treats the Universum class and classes of interest differently. We apply closed decision boundaries to classes of interest and designate the area outside all closed boundaries in the feature space the space of the Universum class. Specifically, we formulate the closed boundaries as arbitrary shapes, propose strategy to estimate the probability of the Universum class according its unique property rather than the within-class sample distribution, and propose boundary learning loss to learn decision boundaries based on the balance of misclassified samples inside and outside the boundary. By conforming to the natural properties the Universum class, our method improves both accuracy and robustness classification models. We evaluate method on 6 state-of-the-ar works in 3 different tasks, and the F1 score/accuracy of all 6 works is improved. Experimental results also indicate that method has significantly enhanced the robustness of the model, with the largest absolute FI score improvement of over 8% on the robustness evaluation dataset. Our code will be released GitHub.
INTRODUCTION
In classification based tasks of NLP, quite often we encounter a class named as other class, miscel- laneous class, neutral class or outside (O) class. Such a class is collection of samples that do not belong to any class of interest, such as samples of no relation class in relation extraction task. We adopt the terminology in (Weston et al.) 2006) to designate all such classes as the Universum class (U). Universum class exits various classification- based problems in NLP, such relation extraction (RE) (hhng et al.) (011)). named entity recognition (NER) (Tjong Kim Sang & De Meulder 2003), sentiment analysis (SA) (Tjong Kim Sang & De Meulder 2003), and natural language inference (NLI) (Bowman et al.) (201)). To distinguish the Universum class and the rest of the classes, we call the classes of interest as target classes (T). The set of all classes (A) in training and testing data can be expressed as = UT
Universum class: collection of samples that do not belong to any class of interest.
Target class: A class of interest in the task, i.e., one of the classes other than the Universum class.
The sample compositions of the Universum class and target classes are usually very different. Figure [(a] provides some samples of target class (entitydestination) and the Universum class (other) in

---

Under review as a conference paper at ICLR 2023
relation extraction. From the examples, an intrinsic patter: an entity goes somewhere. However, the three examples of the other relation type are vastly dissimilar and do not exhibit any common intrinsic pattem. In fact, the Universum samples are labeled according to an extrinsic pattern: they do not belong to any of the predefined target classes.
observe that the entity-destination samples adhere to
We further highlight the differences between the Universum class and target classes in two properties. (1) Heterogeneity: The universum class is composed of heterogeneous samples, which may form multiple clusters in the feature space of the test set, as illustrated by the green samples in Figure [(b] This is because the Universum class, as the class name "other" implies, contains all potential implicit classes that are not explicitly defined in the task. For example, in the other samples given in Figure [[a] implicit classes may include the entity-parallel relationship, the entity-fill relationship,
and the entity-narrative relationship. Although such heterogeneous samples are easily mapped into a compact cluster for the training set, it is problematic for the test set. Since the natural predictive rule of the Universum class is an extrinsic pattern that does not belong to any target classes, the model is more likely to fit the noise in the Universum class by memorizing various peculiarities of intrinsic heterogeneous samples rather than finding the general predictive rule. Considering the data distribution of the test set differs from the training set, only memorizing various peculiarities can easily lead to overfitting and result in an accuracy drop. Moreover, the lack of robustness is another consequence of inability to find the extrinsic predictive rule for the Universum class.
(2) Lack of Representativeness in Training Data: The Universum class is the complementary set of predefined target classes in the task. Therefore, it contains all possible implicit classes, i.c., classes not explicitly defined in the task but may appear in the real world. In this case, Universum samples in the training data are unable to sufficiently represent all possible patterns of the genuine distribution of the Universum class. As depicted in Figure [(b] gray samples represent Universum samples in the test set that are not represented by the training data. Classifiers with open boundaries are prone to misclassifying unseen samples in the test set that is not represented by the training data.
Despite the substantial difference between the target classes and the Universum class, this issue has long been neglected by the NLP research community. The majority of works (Zhu & Li 2022 Ye ct al. 2022 Wan et al. 2022 Pouran Ben Veyseh et al. 2022 Tian et al. 2021 Fu et al. 202 I Li et al. 2021b) treat the Universum class and target classes equally. Typically, a linear layer and a softmax function are applied at the end of the model to generate open decision boundaries, which believe are inappropriate for tasks containing the Universum class.
How take into account the different properties of the Universum class and target classes in classifier design? In this work, we propose a closed boundary learning method for classification-basee tasks with the Universum class. Traditional methods often adopt open boundary classifiers since the problem is under the closed-world assumption. However, the open decision boundaries can easily misclassify Universum samples, as illustrated in Figure I(c) Therefore, we propose to use closed boundary classifiers as shown in Figure I(d). We constrain the space of target classes to be closed spaces and designate the area outside all closed boundaries in the feature space as the space of the Universum class. This treatment perfectly fits the nature of the Universum class: sample is marked as the Universum if does not belong to any target class during labeling. The aforementioned two properties are also well addressed in this way.
The main contributions of this work are summarized as follows:
We bring attention to important issue that is frequently neglected in classification- based NL. tasks like NER, RF, SA, and NLI: the Universum class, such as the other class, exhibits very different properties from target classes and should be treated differently.
Methodologically, we generate closed boundaries with arbitrary shape, which include the commonly used spherical-shaped boundary as special case. In addition, we leverage the information of both target classes and the Universum class to learn the decision boundary. We propose a boundary learning loss to generate the boundary based on the balance of misclassified Universum samples and the target class samples. In contrast to the intuitive two-step pipeline method, where Universum identification and multi-class classification may lead to eиror propagation, we develop strategy to estimate the probability of the Universum class without relying on its intrinsic sample distribution and learn the classification of the Universum class and target classes jointly.
2

---

Under review as a conference paper at ICLR 2023
Entit- Destination Relation
Other Relation
The famous actress arrived the airport.
The captain and crews grateful for the support.
Quake survivors moved into makeshift houses.
The room was filled with huge canvases.
The research team is going into the deep jungle.
The stories are narrated through dance.
(a)
00
00
(b)
(c)
(d)
Figure 1: Illustration of distinction between the Universum class. (a) Samples selected from the
SemEval 2010 Task dataset on relation extraction. 3) and the Universum class (class 4). In particular, the gray samples represent Universum samples in the test set that are not represented by the training data. (c) The open decision boundaries obtained by traditional classifiers. (d) The arbitrary closed boundaries obtained by our proposed method.
(b) The distribution of target classes (class
We apply closed boundary learning framework to 6 state-of-the-art (SOTA) works in 3 different tasks with Universum class. In all 6 experiments, the accuracy or F1-score is improved over the original work. Experimental results further show that the enhanced accuracy is attributed to the improvement of both the Universe class and the target class.
By conforming to the natural properties of the Universum class, our method provides more reasonable way of learning and enhance the robustnessof the model well. Experimental results indicate that the model's robustness has been significantly enhanced by our method, with a largest absolute FI score improvement over 8% on the robustness evaluation dataset.
2 RELATED WORKS
2.1 CLASSIFICATION TASKS WITH THE UNIVERSUM CLASS
The Universum class widely exists in classification based tasks in NLP, such as relation extraction (Zhang al. 2017), ) named entity recognition (Tjong Kim Sang & De Meulder 2003), and sentiment analysis (Jiang et al. 2019), as summarized in Table It should be noted that the span-based methods (Zhu & 2022 Li et al., (021aa) enumerate all possible spans and classify the entity from the whole set of spans, which introduces an extra other class. Despite the heterogeneity and lack of
representativeness of the Universum class, current works (Zhu & 2022 Wan al. 2022, Fu
et al. 2021] Tian et al.) [021]: Chen et al. 2021]] Li et al., [021b;] 2020 Yu et al. 2020) solve the
classification problems containing the Universum class as normal multi-class classification problems
and treat the Universum class and target classes equally.
2.2 CLOSED BOUNDARY LEARNING METHODS
Closed boundaries are often adopted in research fields of out-of-distribution (OOD) detection Gomes
et al. 2022 Ren et al. 2021: Chen et al. 2020), open set recognition (Zhang et al. 202 Liu et al.
2020 anomaly detection Zong al. 2018), and outlier detection (Sharan et al.) 2018 Sugiyama &
Borgwardt 2013). We adopt the terminology of generalized OOD detection Yang et al. 202 to encompass these problems and analyze the differences.
3

---

Under review as a conference paper at ICLR 2023
2.2.1 DIFFERENCE IN PROBLEM SETTING
The problem we raised in this paper lies in the intersection of open-world and closed-world assumptions. Classification tasks can be categorized into problems based on closed-world assumption and open-worte assumption (Yang et al. 2021b). The classification problem with the Universum class shares similarities and differences with both assumptions. The tasks highlighted in this paper are treated under the closed-we world assumption. Nevertheless, the Universum samples in these tasks are analogous to OOD samples in open-world assumption. On the other hand, the problem we raised differs significantly from open-world problems. This because the OOD samples are not available in the training data in generalized OOD detection problems, whereas a considerable number of Universum samples are included in the training data in our problem setting. The information of existing Universum samples is important to generate accurate decision boundaries in our problem.
2.2.2 DIFFERENCE IN METHODOLOGY
By definition, the OOD detection problem assumes that the training data do not contain any OOD samples. However, a branch of the OOD studies, known as outlier exposure (Katz-Samuels et al.
2022 Ming et al. 2022 Yang et al. 2021a Thulasidasan et al. 2021 Mohseni et al. [0200] Hendrycks al. 2018p. introduces auxiliary outlier data during training. The introduced auxiliary data makes it close to the format of our raised classification problems with the Universum class. However, outlier exposure methods are not suitable for our problem. The outlier exposure method mostly adopts two-step approach that consists of multi-class classification and OOD identification. The OOD identification step distinguishes OOD and ID samples based on a score obtained by cross entropy (Yang al. 202 Thulasidasan al., 2021 Mohseni et al. 2020 Hendrycks et al. 2018). and energy function KKaz-Sammue et al. [002]] Ming et al.) 2022). However, both cross entropy and energy function are monotonically varying. As result, the decision boundary derived from a threshold score of the monotonically varying function is an open boundary, which leaves the heterogeneity and representativeness issues pointed out in this paper still unresolved. In addition, the two-step approach will also suffer from error propagation.
From a methodological point of view, work is also different from the works in generalized OOD using closed boundaries. In generalized OOD studies, the closed boundaries are formulated by the classic density-based method (Pidhorskyi al. 2018 Hu et al. 2018), one-class classification method
(Reiss et al. 202T: Ruff et al., 2018). or distance- based method (Gomes et al.) 2022] Sun et al. 2022
Zhang 2021 Zaeemzadeh et al. [021[Shor et al.) (020). The distance- basee methods are limited
to spherical boundary shapes but our method can generate arbitrary shape boundaries. 'he one-class classification method formulates only one closed boundary between positive and negative samples while our work generates closed boundaries for every target class. Finally, only positive samples are used to learn decision boundaries in density-basee based method, while both target class samples and Universum samples are used in our work.
Table 1: The tasks and datasets that the Universum class exists.
Task
Dataset
Label Name
Proportion
Relation Extraction
SemEEval 2010 Task (Hendricks
17.4%
Relation Extraction
TARCED (Zhing et al. 2017)
No relation
79.5%
Named Entity Recognition
CoNLLL22033
Tiong Kim Sang & De Meulder
2003
Miscellancous
14.6%
Named Entity Recognition (span
CONLL-2003
Tjong Kim Sang De Meulder 2003
Other
2-90%
based method)
Aspect Calegory Sentiment Analysis
MAMS Jiang al.
2019
Neutral
43.4%
METHOD
Problem Definition: The goal of our proposed method is to learn closed decision boundaries for target classes and jointly classify the Universum samples and target samples. In order to make our proposed method compatible with most existing classification methods, the starting point of our method is the representations of the final layer of classification models, which a linear layer that maps data from siggh-iimnssinaal feature space to lower-dimension space. We denote the sample representations of the final linear layer as H = (hn. hy.....y-.11. } E IR" Nxt where N is the number of samples, and l is the output dimension of the linear layer.
4

---

Under review as a conference paper at ICLR 2023
3.1 DEFINING THE UNIVERSUM CLASS
Universum class exists in many tasks and datasets as we summarized in Table 1 Notably, the Universum class has various names such as other and miscellaneous, etc. In sentiment analysis, the neutral class can be considered as the Universum class because the word neutral is defined as "having no strongly marked or positive characteristics features", which means the neutral class is a collection of all samples without strong emotions. Similarly, the no relation class in the relation extraction task can be considered as the Universum class.
3.2 PRETRAINING
Our method estimates the probability distribution of target classes based on their sample distributions. In order to avoid estimation based on randomly initialized weight and speed up the learning process, employ N- N-pair loss (Sohn 2016) for pretraining. making sample representations be of small intra-class distance and large inter-class distance. Notably, in accordance with the nature of the Universum class, we make change that does not require the model reduce the intra-class distance of Universum samples during the pretraining.
3.3 GENERATING CLOSED BOUNDARY OF ARBITRARY SHAPE FOR TARGET CLASSES
Existing closed boundary classification methods mainly use spherical shape boundaries (Zhang et al. 2021] Liu et al.] 2020); however, argue that the spher- ical shape may not be the optimal solu- tion because data samples perfectly fit into a sphere, and a spher- ical shape boundary may produce mis- classifications. We adopt the Gaussian mixture model (GMM) and the threshold value to generate boundaries with arbi- trary shapes.
unlikely
(b)
Figure 2: Illustration of generating arbitrary shape bound- aries.
3.3.1 GAUSSIAN MIXTURE MODEL
We apply GMM with m components estimate the class conditional probability distribution for each target class C₁, and further derive the joint probability estimation for each class.
m
phh C.) =
NWN (hu; p4, E,)
p(hg, Ca) = p(h Ci)p(C,)
where h4 denotes the input feature vector of the kth sample, μi; and E, are the estimated mean vector and covariance matrix of the Gaussian components, respectively. Ti the non-negative mixture
m weight under the constraint that ΣØ= Tij 1. pii E,, and oıj are all learnable parameters in the model.
should be noted that the accuracy of GMM estimation positively related to the number of samples used (Psutka & Psutka 2019), which should be at least equal to the dimension of the data. Therefore, our method is not suitable for zero-shot or few-shot settings. Nevertheless, given that the dimension I in our method is very small and the initialized GMM parameters will be fine-tuned by the neural network, constraints on sample size can be easily overcome on the most classification tasks.
According to Bayes Theorem, the posterior probability p(C, h&) p(h, (C.)p(C.). Since we are interested in argmax
p(bę)
p(h. (CC.)( the decision can be made based on joint probability p(h:, C.). p(hx)
3.3.2 ARBITRARY SHAPE BOUNDARY
Geometrical View:
Inspired by the DENCLUE algorithm in generating arbitrary shape clusters (Hinneburg & Keim 1998), introduce threshold value & for each target class. A closed boundary
5

---

Under review as a conference paper at ICLR 2023
of arbitrary shape is formulated by points satisfying p(h, C2) E.. Figure is an illustration of formulating an arbitrary shape boundary in a dinee-dimesiionl space. A sample is assigned to class C2 if it is located inside the closed boundary. If the number of components of the GMM is set one, then the shape of the boundary becomes spherical, with its center and covariance matrix specified by 140 and Eg, respectively. In this sense, the commonly used spherical shape boundary (Zhang et al.) 2021 Liu et al., (2200) is a special case of our method.
Notably, the threshold values = Eτ, f2. : E are a learnable parameters, which eliminates the laborious process of hyperparameter tunning. Specifically, it is learned based on the balance of misclassified samples inside and outside the boundary through our proposed boundary learning loss,
which is introduced later.
Probabilistic View: The above geometrical process can be described as:
Th E C. if ph2; C,) > & ha & C, p(u,, C.) S &
3.4 EXTRINSIC RULE-BASED PROBABILITY ESTIMATION FOR THE UNIVERSUM CLASS
Our method learns the Universum class and target classes jointly to prevent error propagation by intuitively classifying target classes and the Universum class in two steps. However, unlike target classes that estimated by intrinsic sample distribution, estimating the probability distribution of the Universum class must be based on an extrinsic rule that does not belong to any target classes. We propose an extrinsic rule-based probability estimation method to address this issue.
3.4.1 MOTIVATION AND THE ESTIMATION
We classify the samples of Universum class and target classes based on the following rules in this work:
Rule 1: A sample is assigned to the Universum class if it is not located inside any of the closed boundaries of target classes.
Rule 2: A sample is assigned the target class with the highest p(hk: C.) if it located inside at least one closed boundary.
An intuitive way incorporate the above rules is two-step method consists of Universum class detection and target classes classification. However, such a pipeline method has the issue of error propagation. Another way is to use the distance relationship between the sample point and centers of target classes. However, the distance, e.g., Mahalanobis distance, is only a simple special case obtained from Section 3.4. In addition, the distance-based method is incompatible with cross-entropy loss, which has the benefit of being directly related to accuracy. Finally, general probability estimation methods, such as the GMM model, exploit intrinsic sample distributions, which fail to overcome the special properties of the Universum class and do not conform to Rule 1. Therefore, a strategy need to be devised to convert Rule into a probability expression, while simultaneously facilitates the learning of the neural network.
For compliance with Rule 1, the estimated probability of the Universum class must satisfy the following two conditions: for Universum class samples: Vi: p(h,, U) > p(hg, C.) and for target class samples: Zi p p(h,, (/) < p(h,, C2). We can leverage the relationship between & and p(h,, C,) defined in Equation 3 to construct the estimation of p(hç, U) that satisfies the above two conditions. In addition, to enhance the learning of neural networks, the gradient obtained from an Universum sample should move this sample away from its closest target class boundary. Therefore, we also involve max (p( (h2, Ct)), the probability of the closest target class of a Universum sample, to guide the Universum sample move away from target class boundaries. We propose to estimate the probability distribution of the Universum class as follows:
p(h,, U) p(h,, C.)
X
+(1 +1- X)
(4)
p(h,, C.)
A 1, p( hgg Cw) > &u
argmax, Ci) & 0 = argmax, p (h2, Ci)
(5)
0, p( hh, C.) S &
& is the threshold value of target class i, and u, (1.2....,1 1).

---

Under review as a conference paper at ICLR 2023
3.4.2 ANALYSIS OF THE PROPOSED ESTIMATION
For estimated probability of the Universum class in Equation cases are possible. Case 1: p(k,, C.) > Eai i.e., sample h2 is located inside at least one closed boundary. In this case, have
Eu ph, U) = Eu p(hs, C.)
< Eu < p(hye C.)
Since p(h,, U) < p(h,, C.), the model will select the target class with the highest p(h Ct). which fits perfectly with Rule 2.
Case 2: p(h,, C.) S Eu. i.e., the sample h2 distribute outside every closed boundary. Combining the condition of case 2 and Equation 5 we have
p(ha, C.). S p(he, < & i.e., Vi E (1,2,... ... 1}: p(ha, S
Vi E {1,2, ................................
Eu
(6)
Combining Equation 5 and Equation 6 we can derive that:
Eo
Vi E {1,
... 21, 13 p(hg U) = Eu p(he, Co)
C.) 2 phas, C2)
(7)
In case from Equation and Equation B] we can learn that sample h2 is located outside all closed boundaries of target classes. In this case, the probability of Universum class p(ha, U') obtains the largest value, as shown Equation 7 Therefore, Rule perfectly expressed by the proposed
probability estimation of the Universum class.
3.5 BOUNDARY LEARNING LOSS
To facilitate the learning of the closed decision boundaries, we propose boundary learning loss below. The decision boundary should be adjusted to the balance of misclassified samples inside and outside the boundary. For example, if samples of class j distribute inside the boundary of class i, then the boundary should contract to exclude such samples and vice versa.
LaI
Σ 1=1
We. log p(hs, C.)
&
+
Σ wi log IET
p(hi, Cj) &,
M
REO
M is the total number of misclassified samples for all boundaries, n is the number of classes, O and I denote the set of training samples misclassified outside and inside the decision boundary i, p(ha.C,)
& respectively. The weights in the loss function are wk p(hs CJ . wh phh +5, and they are detached and cut off the gradient. Weights wy and wy have smaller values for samples located far from the boundary, enabling the boundary to be adjusted primarily the basis of easily and semi-hard negatives instead of hard negatives.
C.)
During training, sum the cross-entropy loss and boundary learning loss for optimization. The boundary learning loss forces misclassified samples to be distributed in the correct region, which works well with cross-entropy loss. Moreover. the boundary learning loss defined the sum of each boundary's 's the false positive (FP) samples and false negative (FN) samples. For instance, if there are considerably more FN samples than FP samples, the loss caused by FN samples will be larger, causing the boundary to adjust to eliminate more FN samples and balance the number of FN and FP samples. Hence, the boundary learning loss can further improve Fl score.
EXPERIMENTS
4.1 DATASETS AND BASELINES
Our proposed method is evaluated six different SOTA models three classic datasets in NLP, including SemEval 2010 Task 8 (Hendrickx et al.) 2019). MAMS (Jiang et al.) 2019). and CoNLI-2003 (Tjong Kim Sang & De Meulder 2003). The proportion of Universum samples in the
7

---

Under review as a conference paper at ICLR 2023
Table 2: The overall performance of applying closed boundary learning baseline models.
Task
Method
Fl/accuracy
p-value
al. SpanNER
2021
F2.09
SpanNER Fu ct al.
2021
+ OECC
91.22 + 0.12
0.000
SpanNER [Fu el al.
2021
+ COOLU
93.5010..3
NER
BS (Zuu & Li
2022)
92.53.0.02
BSC Zhu Li
2022
OECC
91.88 1 0.15
<.001
Zhu L1,
2022
COOLU
93.17
A-GCN
Tian cl al.
2021)
88.67.0.11
A-GCN
Tan a.
2021
OECC
88.00 1 0.09
<.01
RE
A-GCN
Tian
2021
COOLU
89.33
TaMM
Chen al.
2021
88.7610.23
TaMM
Chen cl al.
2021
OECC
88.17 + 0.18
< 0.01
TaMM CChnn et al.
2021
COOLU
89. 89.47+0.21
AC-MIMLLN [L
2020
76.1320.29%
AC-MIMLLN
2020
OECC
74.02 0.68%
0.01
AC-MIMLLNN Li et al.
2020
+ COOLU
07.35=0.42%
ACSA
SCAPT (Li ct al.
20216
84.13+0.19%
SCAPT L et al.
20216
OFCC
83.36 + 0.27%
< 0.01
SCAPT ct
2021b
COOLU
85.06-0.23%
three datasets are 17.4% (highest in 19 classes), 90%, and 43.4%. respectively. is noteworthy that the ratio of Universum class in the NER task is not calculated from the miscellaneous samples in the dataset but from the other samples which are introduced by the span-based method (Zhu & 2022 Fu et al. 2022I.
We evaluate the effectiveness of our proposed ClOsed bOundary Learning for classiciation with the Universum class (COOLU)
6 SOTA works, including SpanNER (Fu al. 202T). BS (Zhu & Li 2022). A-GCN (Tian et al. 2021). TaMM (Chen et al. 2021), AC-MIMLLN (Li et al., 2020). and SCAPT (Li et al. 202lb. Moreover, OECC (Papadopoulos al. 202T). recent outlier exposure method for OOD, is also applied to these works for comparison with our method. The implementation details are in the Appendix.
4.2 OVERALL EXPERIMENTAL RESULTS
Table shows the overall results for all 6 models. Models with our proposed closed boundary learning outperform the original models with open classifiers on NLP tasks containing the Universum class. The overall accuracy Fl score improved on all six models we evaluated, with the largest improvement from 92.09 to 93.50 in F1 score. In addition, statistical tests between the accuracy/F1 score of the baseline models and our method. indicate that the improvement brought about by our COOLU method is statistically significant.
Our proposed COOLU method also perform better than OECC Papadopoulos et al. 2021). To our knowledge, there has been no work special designed for Classifying with the Universum class. OECC as an outlier exposure method, although its problem setting is the closest to ours, is still not suitable. The decrease in accuracy caused by applying OECC is due to: (1) error propagation resulting from outlier detection and multi-class classification in two steps and (2) instability of manually setting thresholds.
4.3 A CLOSER LOOK AT THE MICROF1, PRECISION AND RECALL
The improvement of overall performance not only attributed to the improvement of the Universum class, but also to the improvement of all classes whole. We show the micro Fl score of each class by applying closed boundary learning on SpanNER in Table The micro Fl score of other class, which is introduced by the span- based method, is not listed in the table because the overall Fl score of the task depends on the Fl score of the four entity types below. The micro Fl score is improved in all classes, with the absolute improvement of 2.43, 0.29, 1.31, and 4.02, respectively.
8

---

Under review as a conference paper at ICLR 2023
Table 3: The micro F1 score of SpanNER (Fu et al.
2021) with and without closed boundary learning.
Mothod
ORG
PER
LOC
MISC R
P
F1
P
R
F1
P
R
F1
F1
SpanNER
89.92
59.81
89.87
97.74
96.47
97.11
93.05
93.85
93.46
80.87 88.02
78.99
82.83
SpanNER + COOLU
94.29
90.30
92.30
98.54
96.25
97.40
96.17
93.41
94.77
81.97
84.89
Table 4: Comparison of model's robustness with and without closed boundary learning.
CrusCategury
O0V
SpellingError
Appenillrr
SpanNER
77.06
75.14
76.09
87.31
SpinNER- COOLU
81.
83.15
81.12
89.89
InserCllasse
Swaplini
SpellingEiro
Appendler
A-GON
77.
86.8
77.30
86.39
A-GCN + COOLU
80.17
88.69
78.71
88 22
In addition, our proposed closed boundary learning significantly improves the precision score of all classes, with the largest absolute gain being 9.03. We also observe a slight drop in recall as result of replacing open decision boundaries with closed ones. But the increase in precision considerably outweighs the decrease in recall. The relatively poor performance of the baseline method in precision score further proves our argument that the Universum class is easily misclassified if its special properties neglected. The improvement in precision score indicates that the proposed closed decision boundaries of target classes can effectively avoid misclassification of the Universum class into target classes. Simultaneouslyy as fewer Universum samples are misclassified, more Universum samples are correctly identified.
4.4 MODEL ROBUSTNESS EVALUATION
What's the relationship between accuracy and robustness? Some studies theoretically identify trade-off between robustness and accuracy (Tsipras et 2019 et 2019 Raghunathan et al., 2020), another finding shows no inherent trade-off (Yang et al. 2020). Regardless, challenging to improve both accuracy and robustness of the model, which is achieved by our proposed method,
We evaluate the robustness of the model based on TextFlint (Wang et (021]. p a robustness evaluation toolkit for NLP tasks. Specifically, TextFlint generate perturbations of the test data by universal transformations and task-specific transformations, and the robustness of the model is evaluated using the transformed test dataset. The terms "CrossCategory" "OOV", "SpellingError", etc. in Table are different ways of transforming the test data. Detail information of these transformation methods are illustrated in Appendix B.3. It can be observed from the Table that the robustness of SpanNER (Fu et al. 2021) improved significantly by applying our proposed COOLU method, with the improvement of absolute Fl score of 4.44, 8.01, 5.03, and 2.55, respectively. In addition, although the improvement of the Fl score in A-GCN (Tian et al. (021] is less than 1, the robustness of the model is considerably improved. The absolute FI score on robustness evaluation datasets are improved at 2.33, 1.89, 1.35, and 1.63, respectively. The improvement in robustness validates our argument that our method conform to the natural properties of the Universum class and successfully teach the model the extrinsic predictive rule rather than overfitting to various peculiarities of heterogeneous Universum samples.
CONCLUSION
In this work, we highlight an overlooked issue in classification- based NLP tasks that the Universum class is treated equally with target classes despite their significant differences. We propose closed boundary learning method COOLU as solution. which conforms the natural properties of the Universum class. Specifically, we generate closed boundaries with arbitrary shapes, develop an extrinsic rule-based strategy to estimate the probability of the Universum class, and propose a boundary learning loss to adjust decision boundaries based on the balance of misclassified samples inside and outside the boundary. Experimental results demonstrate that our proposed COOLU method improves the accuracy of all SOTA baseline models and simultaneously enhances model's robustness.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508 05326, 2015.
Guimin Chen, Yuanhe Tian, Yan Song, and Xiang Wan. Relation extraction with type-aware map memories of word dependencies. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2501-2512, Online, August 2021. Association for Computational Linguis- tics, doi: 10. 100.18655/112000 Linniiin acl.221. URL https: /tt.://ps///a.aaaooooooooo//////
Findings acl. 221
Xingyu Chen, Xuguang Lan, Fuchun Sun, and Nanning Zheng. boundary based out-of-distribttion classifier for generalized zero-shot learning. In European Conference Computer Vision, pp. 572-588. Springer, 2020.
Jinlan Fu, Xuanjing Huang, and Pengfei Liu. SpanNER: Named entity re-Irecognition span predic- tion. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7183-7195, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 acl--lng..553 URL httpt:ttt////tttt httpa///ss///aaaaaaaoooooo/////a wwt.::/...cloooooo.. 558
Eduardo Dadalto Camara Gomes, Florence Alberge, Pierre Duhamel, and Pablo Piantanida. Igeood: An information geometry approach to diitr--dittribution detection. In International Conference on Learning Representations, 2022.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebas- tian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422, 2019.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2018.
Alexander Hinneburg and Daniel A Keim. An efficient approach to clustering in large multimedia databases with noise. In Knowledge Discovery and Dutamining (KDD '98), pp. 58-65, 1998.
Weiming Hu, Jun Gao, Bing Li, Ou Wu, Junping Du, and Stephen Maybank. Anomaly detection using local kernel density estimation and context-based regression. IEEE Transactions on Knowledge and Data Engineering. 32(2):218 2-33,,233-2100 2018.
Gordon Hughes. On the mean accuracy of statistical pattern recognizers. IEEE transactions on information theory, 14(1):55-6) 63, 1968.
Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. A challenge dataset and effective models for basect-based sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pp. 6280 180-6225, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v Mttp///DD1---1000 URL https:77
wwwp:i/ooooyyooo/11111111110 ac acc aaanhhooooyy.... www.oogg///9/.. D0////1991133
Julian Katz-Samuels, Julia B Nakhleh, Robert Nowak, and Yixuan Li. Training ood detectors in their natural habitats. In International Conference on Machine Learning, pp. 10848-10865. PMLR,
2022.
Fei Li, ZhiChao Lin, Meishan Zhang, and Donghong Ji. A span-based model for joint overlapped and discontinuous named entity recognition. In Proceedings of the 59th Annual Meeting of the f Computational Linguistics and the 1 1thh International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4814-482 2021a.
Association
Yuncong Li, Cunxiang Yin, Sheng-hua Zhong, and Xu Pan. Multi-instance multi-label learning networks for aspect-category sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3550-3560, Online, November 2020. Association for Computational Linguistics. doi: 18.18653/v w2000.emmll.....mmm main.287. URL https: //acl antho ogy..og//........oo emn p- 287
10

---

Under review as a conference paper at ICLR 2023
Zhengyan Li, Yicheng Zou, Chong Zhang. Qi Zhang, and Zhongyu Wei. Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training of the 2021 Conference
Proceedings Empirical Methods in Natural Language Processing, pp. 246-256, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics. doi: 10. http5///1/1211.mmlppoooo 22. URLhttts:: //ac 2021. emnlp 22
http:///////aaaaaaoaooooooo wwt.lllthohooy.... org/
Bo Liu, Hao Kang, Haoxiang Li, Gang Hua, and Nuno Vasconcelos. Few-shot open-set recognition using meta-learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7798-8807, 2020.
Yifei Ming, Ying Fan, and Yixuan Li. Poem:
detection with posterior sampling. In International Conference on Machine Learning, pp. 15650-15665. PMLR, 2022.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Aristotelis-Angllos Papadopoulos. Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for dittr--distribution detection. Neurocomputing, 441: 138-150, 2021.
Stanislav Pidhorskyi, Ranya Almohsen, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. Advances in neural information processing systems, 31, 2018.
Amir Pouran Ben Veyseh, Minh Van Nguyen, Franck Dernoncourt, Bonan Min, and Thien Nguyen. Document-level event argument extraction via optimal transport. In Findings of the Associ- ation for Computational Linguistics: ACL 2022, pp. 1658-1658, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/w1/0022 findings- 130. URL
http://cc..aatoooooooooo//22222/ wttp:/02.....nnnngnee....a.. 130
Josef V Psutka and Josef Psutka. Sample size for maximun-likellhood estimates of gaussian model depending on dimensionality of pattern space. Pattern Recognition, 91:25-33, 2019.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John C Duchi, and Percy Liang. Understand- ing and mitigating the tradeoff between robustness and accuracy. In Proceedings of the 37th International Conference on Machine Learning, pp. 7909-7919, 2020.
Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 2806-2814 2021.
Jie Ren, Stanislay Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshmi- narayanan. simple fix mahalanobis distance for improving near-ood detection. arXiv preprint 1rXiv:2106.09022. 2021.
Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference on machine learning, pp. 4393 4402. PMLR, 2018.
Vatsal Sharan, Parikshit Gopalan, and Udi Wieder. Efficient anomaly detection via matrix sketching. Advances in neural information processing systems, 31, 2018
Sughash Sharma. Applied multivariate techniques. 1996.
Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific reports, 10(1):1-13, 2020.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016.
Mahito Sugiyama and Karsten Borgwardt. Rapid distance-based outlier detection via sampling. Advances in neural information processing systems, 26, 2013.
11

---

Under review as a conference paper at ICLR 2023
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribun detection with deep nearest neighbors. arXiv preprint arXiv:2204.06507 2022.
Sunil Thulasidasan, Sushil Thapa, Sayera Dhaubhadel. Gopinath Chennupati, Tanmoy Bhattacharya, and Jeff Bilmes. An effective baseline for robustness to distributional shift. In 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 778-285. IEEE, 2021.
Yuanhe Tian, Guimin Chen, Yan Song, and Xiang Wan. Dependency-driven relation extraction with attentive graph convolutional networks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4458 4471, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/20222 118666//1/22211.accoogg.... URL https
//aclanthology. wro. 2021. acl .oallloon........ 344
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CONILL-2003 shared task: Language-independeen named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https://
aclanth wwg/wogg//30 0419
Dimitris Tsipras. Shibani Santurkar, Logan Engstrom. Alexander Turner, and Aleksander Madry, Ro- bustness may be at odds with accuracy. In International Conference on Learning Representations, number 2019, 2019.
Juncheng Wan, Dongyu Ru, Weinan Zhang, and Yong Yu. Nested named entity recognition with span-level graphs. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 892-903, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ acw.oooo......oo........ URL http://ww..oooooooo////2.....--.
Xiao Wang, Qin Liu, Tao Gui, Qi Zhang, Yicheng Zou, Xin Zhou, Jiacheng Ye, Yongxin Zhang, Rui Zheng, Zexiong Pang, Qinzhuo Wu, Zhengyan Li, Chong Zhang, Ruotian Ma, Zichu Fei, Ruijian Cai, Jun Zhao, Xingwu Hu, Zhiheng Yan, Yiding Tan, Yuan Hu, Qiyuan Bian, Zhihua Liu, Shan Qin, Bolin Zhu, Xiaoyu Xing, Jinlan Fu, Yue Zhang, Minlong Peng. Xiaoqing Zheng, Yaqian Zhou, Zhongyu Wei, Xipeng Qiu, and Xuanjing Huang. TextFlint: Unified multilingual robustness evaluation toolkit for natural language processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations, pp. 347-355, Online, August 2021. Association for Computational Linguistics. doi: (0.18653/v1/2022 acl-demo.41 | https
//aclanthology htg.e//ho..ooo....oo../..aaeeaae 41
Jason Weston, Ronan Collobert, Fabian Sinz, Léon Bottou, and Vladimir Vapnik. Inference with the universum. In Proceedings of the 23rd international conference on Machine learning, pp.
1009-1016, 2006.
Xuanli Lisa Xie and Gerardo Beni. A validity measure for fuzzy clustering. IEEE Transactions Pattern Analysis & Machine Intelligence, 13(08):841 847, 1991.
Jingkang Yang, Haoqi Wang, Litong Feng. Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out- distriiiittiiitiinnn detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8301-8309, 2021a.
Jingkang Yang. Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. XrXiv preprint 2rXiv:2110.113344 2021b.
Yuo-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. Advances in neural information processing systems, 33:8588-8601, 2020.
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. Packed levitated marker for entity and relation extraction. Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguisties (Volume 1: Long Papers), pp. 4904-4917, Dublin, Ireland. May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 act.:22.ccll-oo.33171333333 URL https
7/ //aclanthol ogy wgw.oogg..02......//.ogo aall2a..cl111
12

---

Under review as a conference paper at ICLR 2023
Nan Ye, Kian Ming A Chai, Wee Sun Lee, and Hai Leong Chieu. Optimizing f-measures: A tale of two approaches. In Proceedings of the 29th International Conference on Machine Learning. ICML 2012, volume 1, pp. 289-296, 2012.
Juntao Yu. Bernd Bohnet, and Massimo Poesio. Named entity recognition dependency parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6470-6476, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18633/11/220.. acl-main.577. URL
org/ .ccl
Alireza Zaeemzadeh, Niccolo Bisagno, Zeno Sambugaro, Nicola Conci, Nazanin Rahnavard, and Mubarak Shah. distrod-dtttrbbtionn detection using union of 1-dimensional subspaces. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9452-9461, 2021.
Hanlei Zhang, Hua Xu, and Ting-En Lin. Deep open intent classification with adaptive decision boundary. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14374-14382 2021.
Hongyang Zhang. Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing (EMNLP 2017), pp. 35 2017. URL
staforrd edu/pubs /hang2017aacree.
Enwei Zhu and Jinpeng Li. Boundary smoothing for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pp. 7096-7108, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: lon..ll...-........
10.18653/v1/2022..ad http://22222...looo.o... 490. URL https: ht///allannte httpp::/sc..antholog..oooo www.olloyy.rg/2222222 acl -lng
490
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In International conference on learning representations, 2018.
A COMPACTNESS OF THE UNIVERSUM CLASS OF THE TEST SET
We evaluate the compactness of the Universum class and target classes on the test data and depict the result in Figure 3 The representation of test samples after learning with open boundary classifiers by SpanNER (Fu al. 202T) is used for evaluation. We evaluate the compactness based on the root-mean-square standard deviation (RMSSD) (Sharma 1996). and the mean square distance (MSD) (Xie & Beni 1991), which commonly used in clustering studies to evaluate compactness of cluster. The smaller the RMSSD or MSD, the better the compactness. It is illustrated in Figure that the compactness of the "OTHER" class is significantly worse than target classes. Notably, the class with the second-worse compactness is the "MISC" class, i.e., the "miscellaneous" class, which is also a type the Universum class.
The compactness evaluation results reflect our view that although the heterogeneous samples are easily mapped into a compact cluster for the training data, is problematic for the test data. Since the natural predictive rule of the Universum class is an extrinsic pattern that not belong to any target classes, the model is more likely to the noise in the Universum class by memorizing various peculiarities of intrinsic heterogeneous samples rather than finding the general predictive rule. In this case, it is hard for the model to map all test samples of the Universum class into the compact space learned from training. The Universum samples that deviate from the original compact space are easily misclassified. On the other hand, target classes do not encounter this problem because the general predictive rule for target classes is intrinsic.
13

---

Under review as a conference paper at ICLR 2023
Table 5:
The pretrained models chosen for each baseline model and the corresponding F1 score/accuracy reported in the original paper.
Method SpanNER Fu et al.
Pretrained Model
Reported F1/accuracy
2021
BERT-base
92.28
BS (Zuu & Li 2022)
RoBERTa
93.65
A-GCN Tian el al.
2021
BERT-base
89.16
TaMM Chen
Z02T
BERT-base
89.18
AC-MIMLLN [Li ei
2020
Glove
76,42
SCAPT (L al.
2021b
BERT-base
85.24
OTHER
OTHER
MISC
MISC
Cts Type
Cliic Type
LOC
PER
ORG
GRS
50
100
150
203
(b)
Figure 3: The compactness evaluation of the Universum class and target classes of the test data of NER task.
B IMPLEMENTATION DETAILS
B.1 BASELINE MODELS
We reproduce the baseline models based on the officially released source code, and apply closed boundary learning on the source code. All reported results are the average of three runs. It should be noted that some results baseline models are slightly different from those given the original papers due to the variations in random seeds and package versions when reproducing baseline models from their officially released codes. Nevertheless, baseline models and models with closed boundary learning fairly compared in work under the same random seed and deep learning environment. In the six baseline models we selected, different results based on multiple language models are often reported in work. We choose of the pretrained models used in each work and reproduce the baseline models. The pretrained language model we used in each baseline and their reported results are summarized in T
B.2 TRAINING PROCESS
During pretraining process, all parameters of the original model 0 are learned. We employ GMM estimation on training data after pretraining and obtain the initial value of μ4, E,, and Tijs where i E (1,2, n 1). E (1,2,... m). n is the number of classes, and m is the number of GMM components. We typically select m 4 in our experiments. The threshold values & is initialized around the quantile of p(ha. Ci) values (k E {0, 1, .N 1}}). where is the accuracy or Fl score the original model. With our extrinsic rule- based probability estimation for the Universum class, obtain ip( he, C1),p(h,, C2), p(h,, he, Cn he, U']. Then, the original model parameters 0, GMM parameters μú, Σ, nrj and threshold values & are learned by cross-entropy loss and our proposed boundary learning loss.
14

---

Under review as a conference paper at ICLR 2023
Table 6: The comparison of our method and the method optimizing F1 score.
Task
Method
F1
SpanNER
Fuu et al.
2021
92.09
NER
SpanNER Fu et al.
2021
+ ML R.
92.43
SpanNER Fu et al.
2021
+ COOLU
93.50
A-GCN Tian et al.
2021
88.67
RE
A-GCN
Tian et al.
2021
+ MLE COOLU
88.61
A-GCN
Tian al.
2021
89.33
AC-MIMLLN (L1 et al.
2020)
76.13%
ACSA
AC-MIMLLN (L et al.
2020
+ ML E
AC-MIMLLNN
2020 + COOLU
77.35%
B.3 ROBUSTNESS EVALUATION
We evaluate the robustness of the model based on TextFlint (Wang al. 2021 robustness evaluation toolkit for NLP tasks. There are two kinds of transformations provided by TextFlint to generate the robust evaluation dataset, namely universal transformation and task- transformation. We adopt two universal transformations and two task-specific transformations to the test set NER and RE task and generate four robustness evaluation datasets for each task. The terms of different transformations are explained below.
"SpeolingError": Universal transformation. Brings slight errors to words in the test samples.
"Appendlrr": Universal transformation. Add irrelevant information to test samples.
Task-specific transformation for NER. Replace the entity spans with substitutions from different category.
"CrossCategory":
"OOV": Task-specific transformation for NER. Replace the entity spans with substitutions out of vocabulary.
"InseriClause":
Task-specific transformation for RE. Change sample sentences by appending adjuncts from the aspect of dependency parsing.
Task speciffe transformation for RE. Swap the named entities sentence into entities of the same type.
"SwapEnt"::
Specifically, the models are trained and validated on the original training set and validation set, but the test set is transormed into the robustness evaluation dataset by the transformations proposed by TextFlint. Then, modelw are tested on the transformed test set.
C COMPARISON WITH THE METHOD OPTIMIZING F-MEASURES
We also compare our method with the F1 score optimization method. Specifically, we adopt the ML method Ye et al. 2012) baseline models and make comparison with our proposed COOLU method. The result is summarized Table 6 Although method optimizing Fl score is unrelated to the Universum class, it is another way boost model's performance. Our model also out performs the MI.E method both NER and RE tasks. In addition, the MT.E method is cannot be employ on sentiment analysis task, which is evaluated by accuracy. It is another limitation of the Fl score optimization method compared to our work.
D ABLATION STUDY ON N-PAIR LOSS PRETRAINING
In method, N- N-pair loss is adopted for pretraining to learn initial representations and to speed up the training process To demonstrate the effectiveness of our closed boundary learning method and to rule out the possibility that the improvement of our model is due the N-pair loss pretraining
15

---

Under review as a conference paper at ICLR 2023
Table 7: The effect of pretraining SpanNER (Fu et al. 2021
and SCAPT
Li al.
202
Method
Fl/aceuracy
SpanNER
92.09
SpanNER + N-pair pretraining
92.05
SpanNER COOLU (N-pair pretraining)
93.50
SCAPT
84.13%
SCAPT N-pair pretraining
84.16%
SCAPT COOLU (N-pair pretraining)
85.06%
94.0
93.8
Macro F1 Score
93.6
93.2
93.0
10
15
35
Dimension of Logit Space
Figure 4: Illustration of generating arbitrary shape boundaries.
process, we add an additional pretraining step to baseline models SpanNER (Fu et al. 2021) and SCAPT (Li al. 2021b). Table indicates that the pretraining process alone cannot improve the accuracy of the original baseline models. The improvement brought about by our proposed closed boundary learning is not result of the pretraining step but the result of the entire system.
E THE IMPACT OF THE FINAL LAYER DIMENSION
The last layer's dimensionality can affect the performance of the model. Recalling the classic Hughes phenomenon (Hughes) 1968) that the model accuracy is monotonically increasing first and then monotonically decreasing with the dimension of data, the dimension of the final layer may be chosen to boost model performance.
We investigate the effect of last layer dimension the accuracy of the model on the SpanNER (Fu et al. 2021) with closed boundary learning and present the result in Figure 4 The Fl score of the test set grows with increasing of dimensions and reaches maximum value of 93.88 when the dimension is seven, and then decreases with the dimension. The trend fits well with the Hughes phenomenon Hughes 1968). Our method is quite robust with the dimension and the overall result of SpanNER + COOLU reported in Table 2 is set as ten rather than the optimal value.
16


Reviews:
[{"review_id": "mWHmcYIFkW", "reviewer": "Reviewer_CgAh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to learn closed boundaries on top of the final layer of a classification model. The authors propose to optimize the classifier with an appropriate loss adjusted by the miss-classified examples within and outside the boundaries. The authors, extend state-of-the-art approaches in three datasets. The results show some improvement in all the cases.", "summary_of_the_review": "Generally the paper is easy to read and flows well. I have some difficulty to assess the improvements that this approach brings. Based on the presented results the proposed approach seems to help improving the models that it extends.", "strength_and_weaknesses": "- The paper is easy to follow and intuitive. Explanations of the different parts is adequate.\n- The authors present several related works also and do some comparisons.\n- Results seem to be ok even though only on three datasets.\n\nRegarding weakness one of the first comments is the lack of other approaches that treat the same problem. The authors cite several related works in related cases (OOD detections, anomalies etc) but do not do any comparisons. I mean, the fact that they enhance few SOTA approaches is already good but I cannot assess how this pushes the current state-of-the-art methods in the specific problem.\n\nLack of statistical tests to evaluate significance. In some cases improvements are quite small. Maybe you could add them?\n\nI am curious if any other baseline like for example to consider the universum class as a side task would work here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mN9Csw6GCE", "reviewer": "Reviewer_sUU6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the universum class (negative class) problem in NLP tasks, by modelling the class boundaries with Gaussian mixture models, and learning thresholds with a boundary learning loss on misclassified examples. They showed that their approach outperforms vanilla classifiers for 3 different NLP tasks.\n\n\n", "summary_of_the_review": "I would not recommend acceptance as I think the arguments made by the authors are not supported by sufficient analysis of the results. The authors could also compare against approaches that directly optimises for f1-measures.", "strength_and_weaknesses": "Strengths:\n- the approach shows that it improves performance on 3 different NLP tasks, by simply learning GMM and threshold values of the positive classes.\n\nWeaknesses:\n- There is little analysis to show how the learned boundaries can better model class boundaries. A detailed analysis of precision/recall would be useful, and whether the approach improves multi-class decision boundaries, or only the boundary with the negative class.\n- The proposed approach emphasises on misclassified positive samples, which could have a similar effect on work that optimises F1-measures, see e.g., [1]. \n\nUpdate after author rebuttal:\n- The authors have addressed some of my concerns mentioned above. Hence I have increased the score to 6.\n\n[1] Ye, N., Chai, K. M., Lee, W. S., & Chieu, H. L. (2012). Optimizing F-measures: A tale of two approaches. In Proceedings of the 29th International Conference on Machine Learning (pp. 289-296). Omnipress.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "htBMgFkApX", "reviewer": "Reviewer_EeCa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies how to appropriately handle the universum (or miscellaneous) class in several common NLP classification problems, such as the \"non-entity\" label in NER, \"no-relation\" label in RE. The motivation of the paper is that these universum classes can have a wide coverage of cases whose semantics vary a lot and thus should not be treated as one category as a whole. That is to say, the decision boundary of the universum class should be in different shapes from other target classes with specific semantics. The solution proposed by the paper is to learn closed boundaries for target classes, and regard the remaining points that lie out of the boundaries of any target class as the universum case. The concrete implementation is to leverage GMM to estimate the class boundaries, and use thresholds over class-conditioned generation probabilities to classify whether a data point belongs to any target class or it should be put into the universum label. The method is evaluated on three different tasks instantiated with six SOTA models and demonstrates improvements over conventional open-boundary learning.", "summary_of_the_review": "While the paper studies an interesting issue in several NLP problems regarding how to appropriately handle the universum (or miscallaneous) class, the major motivation seems not well-justified in the context of using DNN encoder models. There are also concerns regarding the stability/generalization ability of the model due to the additionally introduced parameters and hyperparameters, and the lack of studies for these. Many technical details have not been sufficiently discussed and thus remain confusing. The empirical advantage over standard classification baseline is quite marginal.", "strength_and_weaknesses": "Pros:\n* The motivation of the paper is clear. The proposed solution is simple and intuitive.\n* The method is empirically shown to improve different models on several different tasks.\n\nCons:\n* The first major concern is regarding the correctness of the motivation. While the motivation for special treatments of the universum class is clear and intuitive, it's not convincing to me whether the limitations of conventional multi-class classification as pointed out in this paper actually matter that much. Figure 1 illustrates that the decision boundaries learned by conventional classification models are \"open\", which makes it hard to distribute the universum class instances accurately. However, I doubt if this is really the case -- in the context of using deep neural nets (DNNs) as the feature encoder, the data representations (i.e., $\\boldsymbol{h}$) and decision boundaries ($f(\\boldsymbol{h})$) are jointly learned, so even if the universum class instances have quite different semantics, it should not be hard for DNNs to map them into one compact cluster in the last layer's representation space. In other words, the limitations illustrated in Figure 1 (c) could be a problem for simple models like logistic regression or SVM, but do not seem to be a challenge for DNNs which can learn arbitrary non-linear functions for data representations.\n* The second major concern is the additional parameters and hyperparameters introduced by the proposed algorithm. Firstly, the GMM parameters need to be fit using target-class data -- when the amount of labeled data is small for some classes (e.g. few-shot learning or imbalanced learning), the GMM obtained may not be accurate/stable, and this limitation has not been discussed in the paper. Secondly, a threshold value ($\\xi$) needs to be set for each class. It seems to be a hyperparameter that cannot be automatically learned, but the paper also didn't discuss how it can be appropriately set. The authors also didn't show how sensitive the model is to these threshold values. This is quite concerning because it's impossible to manually set threshold values for each class, especially when the number of classes is large (e.g., in fine-grained NER/RE). \n* The paper is unclear in many technical details. For example, Section 3.3 mentions using N-pair loss for pretraining, but does not explain the motivation/reason. Table 4 seems to show that the pretraining step is not useful, making the purpose for pretraining very confusing. Figure 3 studies the impact of last layer dimension and reports that the optimal value is 7. This is quite confusing because the common dimensionality of pretrained models are over 700; a dimension of 7 seems way too small for the last layer dimension. Did you use another linear layer on top of the last BERT/RoBERTa encoder layer to convert the high-dimensional vector into lower-dimenisonal ones (e.g., 7-10 dimensions) before performing classification?\n* The performance improvements of the proposed method over conventional classification setting are quite marginal (in most cases less than 1 point). The paper also didn't report error bars/stds or significant testing results to validate whether the improvements are meaningful. \n\n---\n**Post-Rebuttal Updates**:  \nI'd like to thank the authors for their provided responses and paper updates to address my concerns raised above. The original paper version had many unclear technical details which made assessing the contribution of the paper quite difficult. However, in the authors' rebuttal and updated paper version, many details have been further clarified, and I found them largely strengthened the contribution. Detailed comments are as below:\n* The difficulty of learning with heterogeneous samples in the universum class has been clarified. I see that the miscellaneous patterns from the universum class that do not appear in training but in test time could be an issue for traditional classification models. However, I feel that some concrete case studies would help (e.g., some embedding plots of the actually learned classification boundaries by the proposed method); Figure 1 does not seem to be the actual plots but looks more like conceptual illustrations.\n* The learning process of GMM (especially how the threshold values are learned) and its limitations (e.g., hard to adapt to few-shot learning) have been better articulated.\n* The motivation for pretraining looks clearer to me.\n* Glad to see the added significant tests and the new OECC baseline.\n\nOverall, I found the paper to have studied an interesting problem (i.e., handling the universum class in classification problems), and the method proposed to be reasonable, resulting in good effectiveness. However, I feel that the application scenarios of the method are somewhat narrow (i.e., there are many classification problems in NLP that do not have universum classes at all) and the method also has inherent limitations of not being applicable to low-data regimes (e.g., few-shot learning). Although it does not appear to be a very strong and suitable paper for ICLR, I also don't want to gatekeep it. Hence, I'm recommending a weak accept.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject