Venue: NAACL 2016

# Breaking the Closed World Assumption in Text Classification

Geli Fei and Bing Liu

Department of Computer Science

University of Illinois at Chicago

gfei2@uic.edu, liub@cs.uic.edu

###### Abstract

Existing research on multiclass text classification mostly makes the _closed world_ assumption, which focuses on designing accurate classifiers under the assumption that all test classes are known at training time. A more realistic scenario is to expect unseen classes during testing (_open world_). In this case, the goal is to design a learning system that classifies documents of the known classes into their respective classes and also to reject documents from unknown classes. This problem is called _open_ (_world_) _classification_. This paper approaches the problem by reducing the open space risk while balancing the empirical risk. It proposes to use a new learning strategy, called _center-based similarity_ (CBS) _space learning_ (or _CBS learning_), to provide a novel solution to the problem. Extensive experiments across two datasets show that CBS learning gives promising results on multiclass open text classification compared to state-of-the-art baselines.

## 1 Introduction

With the rapid growth of online information, text classifiers have become one of the most important tools for people to track and organize information. And the emergence of social media platforms has brought increasing diversity and dynamics to the Web. Many social science researchers rely on the collected online user generated content to carry out research on different social phenomenon. In this case, multiclass text classifiers are widely used to gather information of several topics of interest. However, most existing research on multiclass text classification makes the _closed world_ assumption, meaning that all the test classes have been seen in training. However, in a more realistic scenario where people use a multiclass classifier to collect information of several topics from a data source that covers a much broader range of topics, it is normal to break the closed world assumption and to see the arrival of documents from unknown classes that have never been seen in training. In this case, a multiclass classifier should not always assign a document to one of the known classes. Instead, it should identify unknown classes of documents and label them as unknown or reject. This is called _open_ (_world_) classification.

More precisely, in the traditional multiclass classification setting, the learner assumes a fixed set of classes \(Y=\{C_{1}\), \(C_{2}\), ..., \(C_{m}\}\), and the task is to construct a \(m\)-class classifier using the training data. The resulting classifier is tested/applied on the data from only the \(m\) classes. While in _open_ classification, we allow the classifier to predict labels/classes from the set of \(C_{1}\), \(C_{2}\), ..., \(C_{m}\), \(C_{m+1}\) classes, where the \(\left(m+1\right)^{\text{th}}\) class \(C_{m+1}\) represents the unknown which covers documents of all unknown or unseen classes or topics. In other words, every test instance may be predicted to belong to either one of the known classes \(y_{i}\in Y\), or \(C_{m+1}\) (unknown).

It is thus not sufficient for a classifier to just return the most likely class label among the \(m\) known classes. An option to reject must be provided. An obvious approach to predicting the class label \(y\in Y\cup\{C_{m+1}\}\) for an \(n\)-dimensional data point \(x\in R^{n}\) is to incorporate a posterior probability estimator \(p(y|x)\) and a decision threshold into an existing multiclass learning algorithm [35, 40, 6]. There are many reasons this technique would not achieve good results in open classification. As we will discuss in the following sections, one of the most important reasons is that the underlying classifier is not robust or is not in

formed enough to reject unseen classes of documents due to its significant open space risk.

Traditional multiclass learners optimize only on the known classes under the closed world assumption, while a potential learner for open classification has to optimize for both the known classes and for the unknown classes. Some recent research in the field of computer vision studied the problem, which they call _open set recognition_[2013, 204, Jain et al., 2014] for facial recognition. Classic learners define and optimize over empirical risk, which is measured on the training data. For open classification, it is crucial to consider how to extend the model to capture the risk of the unknown by preventing overgeneralization or overspecialization. In order to tackle this problem, Scheirer et al. (2013) introduced the concept of _open space risk_ and formulated an extension of existing one-class and binary SVMs to address the open classification problem. However, as we will see in section 3, their proposed method is weak as the positively labeled open space is still an infinite area.

In this work, we propose a solution to reduce the open space risk while also balancing the empirical risk for open classification. Intuitively, given a positive class of documents, our open space for the positive class is considered as the space that is sufficiently far from the center of the positive documents. In the multiclass classification setting, each of the target classes is surrounded by a ball covering the positively labeled (the target class) area, while any document falling outside of all the \(m\) balls is considered belonging to the unknown class.

Recent work by Fei and Liu (2015) proposed a new learning strategy called _center-based similarity space learning_ (_CBS learning_) to deal with the problem of covariate shift in binary classification. We found that it is also suitable for open classification. Instead of conducting learning in the traditional document space (or _D-space_) with n-gram features, CBS learning learns in a similarity space. Unlike SVM learning in D-space that bounds the positive class only by an infinite half-space formed with the decision hyperplane, which has a huge open space risk, CBS learning finds a closed boundary for the positive class covering only a finite area, which is a spherical area in the original D-space and thus reduce the open space risk significantly. While discussing CBS learning, we will also describe the underlying assumptions made by it which were not stated in our earlier paper [Fei and Liu, 2015]. Our final multiclass classifier is called _cbsSVM_ (based on SVM).

To the best of our knowledge, this is the first attempt to study multiclass open classification in text from the open space risk management perspective. Our experiments show that _cbsSVM_ for multiclass open classification produces superior classifiers to existing state-of-the art methods.

## 2 Related Work

Compared to research on multiclass classification with the closed world assumption, there is relatively less work on open classification. In this section, we review related work on one-class classification, SVM decision score calibration, and others.

One-class classifiers, which only rely on positive training data, are natural starting solutions to the multiclass open classification task. One-class SVM [7] and SVDD [69] are two representative one-class classifiers. One-class SVM treats the origin in the feature space as the only member of the negative class, and maximizes the margin with respect to it. SVDD tries to place a hypersphere with the minimum radius around almost all the positive training points. It has been shown that the use of Gaussian kernel makes SVDD and One-class SVM equivalent, and the results reported in [8] demonstrate that SVDD and One-class SVM are comparable when the Gaussian kernel is applied. However, as no negative training data is used, one-class classifiers have trouble producing good separations. We will see in Section 4 that their results are poor.

This work is also related to using thresholded probabilities for rejection. As the decision score produced by SVM is not a probability distribution, several techniques have been proposed to convert a raw decision score to a calibrated probability output [15, 16, 17, 18]. Usually a parametric distribution is assumed for the underlying distribution, and raw scores are mapped based on the learned model. A variation of Platt's (2000) approach is the most widely used probability estimator for SVM score calibration. It fits a sigmoid function to the SVM scores during training. Provided with a threshold, a test instance can be rejected if the highest probabil 

ity of this instance belonging to a class is lower than the threshold in multiclass open classification settings.

Recently, researchers in computer vision (Scheirer et al., 2013; 2014; Jain et al., 2014) made some attempts to solve open classification (which they call _open set recognition_) for visual learning from new angles. Scheirer et al. (2013) introduced the concept of open space risk, and defined it as a relative measure. The proposed model reduces the open space risk by replacing the half-space of a binary linear classifier with a positive region bounded by two parallel hyperplanes. While the positively labeled region for a target class is reduced compared to the half-space in the traditional linear SVM, their open space risk is still infinite. In (Jain et al., 2014), the authors proposed to use Extreme Value Theory (EVT) to estimate the unnormalized posterior probability of inclusion for each class by fitting a Weibull distribution over the positive class scores from a \(1\)-vs-rest multiclass RBF SVM classifier. Scheirer et al. (2014) introduced the Compact Abating Probability (CAP) model, which explains how thresholding the probabilistic output of RBF One-class SVM manages the open space risk. Using the probability output from RBF one-class SVM as a conditioner, the authors combine RBF One-class SVM and a Weibull-calibrated SVM similar to the one in (Jain et al., 2014). For both methods (Jain et al., 2014; Scheirer et al., 2014), decision thresholds need to be chosen based on the prior knowledge of the ratio of unseen classes in testing, which is a weakness of the methods.

Dalvi et al. (2013) proposed Exploratory Learning in the multiclass semi-supervised learning (SSL) setting. In their work, an "exploratory" version of expectation-maximization (EM) is proposed to extend traditional multiclass SSL methods, which deals with the scenario when the algorithm is given seeds from only some of the classes in the data. It automatically explores different numbers of new classes in the EM iterations. The underlying assumption is that a new class should be introduced to hold an instance \(x\) when the probability of \(x\) belonging to the existing classes is close to uniform. This is quite different from our work. First, it works in the semi-supervised setting and assumes that test data is available during training. Second, it only focuses on improving accuracy on the classes with seed examples.

## 3 Proposed Method

In this section, we propose our solution for the open classification problem. First we discuss our strategy to reduce the open space risk while balancing the empirical risk of the training data. Then we apply a recently proposed SVM-based learning strategy (Fei and Liu, 2015), which yields the same risk management strategy. We will also discuss its underlying assumptions, which was not addressed in the original paper of Fei and Liu (2015). Lastly, we will show why the proposed solution works for open classification.

### Open Space Risk Formulation

Consider the risk formulation by Scheirer et al. (2013), where apart from the empirical risk, there is risk in labeling the open space (space away from positive training examples) as "positive" for any known class. Due to lack of information on a classification function on the open space, open space risk is approximated by a relative Lebesgue measure (Shackel, 2007). Let \(S_{o}\) be a large ball of radius \(r_{o}\) that contains both the positively labeled open space \(O\) and all of the positive training examples; and let \(f\) be a measurable classification function where \(f_{y}(x)=1\) for recognition of class \(y\) of interest and \(f_{y}(x)=0\) otherwise. The probabilistic open space risk \(R_{o}(f)\) of function \(f\) for a class \(y\) is defined as the fraction (in terms of Lebesgue measure) of positively labeled open space compared to the overall measure of positively labeled space (which includes the space close to the positive examples).

\[R_{o}(f)=\frac{\int_{O}~{}f_{y}(x)dx}{\int_{S_{o}}~{}f_{y}(x)dx}\]

The above definition indicates that the more we label open space as positive, the greater open space risk is. However, it does not suggest how to specify the positively labeled open space \(O\).

In this work, we formulate \(O\) as the positively labeled area that is sufficiently far from the center of the positive training examples. Let \(B_{r_{y}}(cen_{y})\) be a closed ball of radius \(r_{y}\) centered around the center of positive class \(y\)\((cen_{y})\), which ideally contains all positive examples of class \(y\); \(S_{o}\) be a larger ball \(B_{r_{O}}(cen_{y})\) of radius \(r_{o}\) with the same 

center \(cen_{y}\). Let classification function \(f_{y}(x)=1\) when \(x\in B_{r_{0}}(cen_{y})\), and \(f_{y}(x)=0\) otherwise. Also let \(h\) be the positive half space defined by a binary SVM decision hyperplane \(\Omega\) obtained using positive and negative training examples, and let the size of ball \(B_{r_{0}}\) be bounded by \(\Omega\), \(B_{r_{0}}\bigcap h=B_{r_{0}}\). We define open space as

\[O=S_{o}-B_{r_{y}}\big{(}cen_{y}\big{)}\]

where radius \(r_{0}\) needs to be determined from the training data for each known positive class.

This open space formulation greatly reduces the open space risk compared to traditional SVM and 1-vs-Set Machine in (Scheirer et al., 2013). For traditional SVM, whose classification function \(f^{SVM}_{y}(x)=1\) when \(x\in h\), and positive open space being approximately \(h-B_{r_{y}}\big{(}cen_{y}\big{)}\), which is only bounded by the SVM decision hyperplane \(\Omega\). For 1-vs-Set Machine in (Scheirer et al., 2013), whose classification function \(f^{1-ys-set}_{y}(x)=1\) when \(x\in g\), where \(g\) is a slab area with thickness \(\delta\) bounded by two parallel hyperplanes \(\Omega\) and \(\Psi\) (\(\Psi\parallel\Omega\)) in \(h\). And its positive open space is approximately \(g-B_{r_{y}}\big{(}cen_{y}\big{)}\). Given open space formulations of traditional SVM and 1-vs-Set Machine, we can see that both methods label an unlimited area as positively labeled space, while our formulation reduces it to a bounded spherical area.

Given the above open space definition, the question is how to estimate radius \(r_{0}\) for the positive class. We show that the center-based similarity space learning (CBS learning) recently proposed in (Fei and Liu, 2015) is suitable for the purpose. It was original proposed to deal with the negative covariate shift problem in binary text classification.

Below, we first introduce CBS learning and then discuss why it is suitable for our problem, as well as its underlying assumptions.

### Center-Based Similarity Space Learning

We now discuss CBS learning for binary text classification. Let \(D=\{(\textbf{d}_{1},\,y_{1}),\,(\textbf{d}_{2},\,y_{2}),\,...,\,(\textbf{d}_{ n},\,y_{n})\}\) be the set of training examples, where \(\textbf{d}_{i}\) is the feature vector (e.g., with unigram features) representing a document \(d_{i}\) and \(y_{i}\in\{1,\,-1\}\) is its class label. This feature vector is called a document space vector (_ds-vector_). Traditional classification directly uses \(D\) to build a binary classifier. CBS learning transforms each _ds_-vector \(\textbf{d}_{i}\) (no change to its class) to a center-based similarity space feature vector (CBS vector) **obs-v\({}_{i}\)**. Each feature in the CBS vector is a similarity between a center \(c_{j}\) of the positive class documents and \(d_{i}\). CBS learning can use multiple document space representations or feature vectors (e.g., one based on unigrams and one based on bigrams) to represent each document, which results in multiple centers for the positive documents. There can also be multiple document similarity functions used to compute similarity values. The detailed learning technique is as follows.

For a document \(d_{i}\) in \(D\), we have a set \(R_{i}\) of \(p\)_ds_-vectors \(R_{i}=\{\mathbf{x}_{1}^{i},\,\mathbf{x}_{2}^{i},\,\,...,\,\mathbf{x}_{p}^{i}\}\). Each _ds_-vector \(\mathbf{x}_{j}^{i}\) denotes one document space representation of the document \(d_{i}\), e.g., unigram representation or bi-gram representation. Then the center of positive training documents can be computed, which is represented as a set of \(p\) centroids \(C=\{\textbf{c}_{1},\,\textbf{c}_{2},\,...,\,\textbf{c}_{p}\}\), each of which corresponds to one document space representation in \(R_{i}\). Rocchio method in information retrieval (Rocchio, 1971; Manning et al., 2008) is used to compute each center \(\textbf{c}_{j}\) (a vector), which uses the corresponding _ds_-vectors of all training positive and negative documents.

\[\textbf{c}_{j}=\frac{\alpha}{|D_{+}|}\sum_{d_{i}\in D_{+}}\frac{\mathbf{x}_{ j}^{i}}{\left\lVert\mathbf{x}_{j}^{i}\right\rVert}-\frac{\beta}{|D-D_{+}|}\sum_{ d_{i}\in D-D_{+}}\frac{\mathbf{x}_{j}^{i}}{\left\lVert\mathbf{x}_{j}^{i} \right\rVert}\]

where \(D_{+}\) is the set of documents in the positive class and \(|.|\) is the size function. \(\alpha\) and \(\beta\) are parameters, which are usually set empirically. It is reported that using _tf-idf_ representation, \(\alpha=16\) and \(\beta=4\) usually work quite well (Buckley et al., 1994). The subtraction is used to reduce the influence of those terms that are not discriminative (i.e., terms appearing in both classes).

Based on \(R_{i}\) for any document \(d_{i}\) in both training and testing and the previously computed set \(C\) of centers using the training data, we can transform a document \(d_{i}\) from its document space representations \(R_{i}\) to one center-based similarity vector \(\textbf{cbs-v}_{i}\) by applying a similarity function \(Sim\) on each element \(\textbf{x}_{j}^{i}\) of \(R_{i}\) and its corresponding center \(\textbf{c}_{j}\) in \(C\).

\[\textbf{cbs-v}_{i}=Sim(R_{i},\,C)\]

\(Sim\) has a set of similarity measures. Each measure \(m_{j}\) is applied to \(p\) document representations \(\textbf{x}_{j}^{i}\) in \(R_{i}\) and their corresponding centers \(\textbf{c}_{j}\) in \(C\) to generate \(p\) similarity features (_cbs_-features) in \(\textbf{cbs-v}_{i}\).

 

For _ds_-features, we use unigrams and bigrams with _tf-idf_ weighting as two document representations. We also adopt the five similarity measures in Fei and Liu (2015) to gauge the similarity of two vectors. Based on these measures, we produce 10 CBS features to represent a document in the CBS space.

### Why does CBS learning work?

Given the open space definition in Section 3.1, our goal is to estimate the radius \(r_{O}\) of the positively labeled space for the positive class. Now we explain how CBS learning gives an estimate of \(r_{O}\).

Due to learning in the similarity space with similarities as features, CBS learning generates a boundary based on similarities to separate the positive and negative training data in the similarity space, which is essentially a ball encompassing the positive training data in the original document space. In other words, instead of explicitly minimizing the positively labeled open space risk, CBS learning approximates the radius \(r_{O}\) by learning a score based on similarities in the similarity space, which is equivalent to a limited spherical area in the original document space. The generated model thus not only limits the positively labeled open space on the positive side of \(\Omega\) (SVM decision hyperplane), but also balances the empirical risk from the positive and negative training examples. In fact, \(r_{O}\) is approximately the distance from the center of positive class to \(\Omega\) measured in similarities. Figure 1 illustrates the point. The positively labeled/classified region produced by CBS learning is the circle in the original document space, while SVM learning produces a half space bounded by its decision line, which is approximately the tangent line of the circle. Note that as multiple similarity features are used, the spherical area is formed by an integrated similarity produced by SVM, which combines all similarity features.

In order for the method to work well for our multiclass classification, ideally two assumptions should be made about the data. First, the target classes of documents are generated by a mixture model, where each mixture component is responsible for each class of documents. Secondly, after feature normalization each target class of documents is generated by a Gaussian distribution, where the Gaussian mean resides at the center of the class, and its \(n\times n\) covariance matrix has equal eigenvalues so that the positive class can have a spherical shape boundary or a ball. Note that we do not make any assumptions about data from non-target classes.

### Multiclass Open Classification

The preceding discussion is based on binary open classification. We follow the standard technique of combining a set of 1-vs-rest binary classifiers to perform multiclass classification with a rejection option for unknown. The SVM scores for each classifier are first converted to probabilities based on a variation of Platt's Platt (2000) algorithm, which is supported in LIBSVM Chang Lin (2011). Let \(P(y|x)\) be a probably estimate, where \(y\in Y\) is a class label and \(x\) is a feature vector, and let \(\lambda\) be the decision threshold (usually 0.5). Let \(Y\) be the set of known classes, \(C_{m+1}\) be the unknown class, and \(y^{*}\) is the final predicted class for \(x\). The final classifier (called _cbsSYM_) uses this following for classification.

\[y^{*}=\begin{cases}argmax_{y\in Y}P(y|x)&if~{}P(y^{*}|x)\geq\lambda\\ C_{m+1}&otherwise\end{cases}\]

## 4 Experiments

In this section, we show the results of the proposed method _cbsSYM_ and compare it extensively with state-of-the-art baselines across two datasets.

### Baselines

**1-vs-Rest multiclass SVM (_1-vs-rest-SVM_).** This is the standard 1-vs-Rest multiclass SVM with Platt Probability Estimation Platt (2000), and it is implemented based on LIBSVM1 version 3.20) Chang Lin (2011). It works in the same way as the proposed _cbsSYM_ (Section 3.4) except that it uses the document space classification. Linear kernel is used as it is shown by many researchers that linear SVM performs the best for text classification

Figure 1: CBS learning reduces open space risk.

 

(Joachims, 1998; Colas and Brazdil, 2006).

**1-vs-Set Machine (_I-vs-set-linear_)**. For this baseline (Scheirer et al., 2013), we use all the default parameter settings in the original paper. That is, the near and far plane pressures are set at \(p_{A}=1.6\) and \(p_{\Omega}=4\) respectively; regularization constant \(\lambda_{\tau}=1\) and no explicit hard constraints are used on the training error (\(\alpha=0,\beta=1\)).

**W-SVM (_wsvm-linear_ and _wsvm-rbf_)**. These two baselines combine RBF one-class SVM with binary SVM (Scheirer et al., 2014). Both linear kernel and RBF kernel are tested. For thresholding the output, two parameters \(\delta_{\tau}\) and \(\delta_{R}\) are required. We set \(\delta_{\tau}=0.001\), which is used to adjust what data the one-class SVM considers to be related. \(\delta_{R}\) is a required decision threshold not only for W-SVM, but also for the next two baselines (P\({}_{\Gamma}\)-SVM, P\({}_{\Gamma}\)-OSVM). Two ways of setting \(\delta_{R}\) were suggested by the authors. We set it as the prior probability of the number of unseen classes during evaluation (testing). An alternative way is to set it based on an openness score computed using the number of training and testing classes. We tried both methods and found the former gave better results.

**P\({}_{\text{i}}\)-Svm (_P-svm-linear_ and _P-svm-rbf_)**. This baseline is from (Jain et al., 2014), which estimates the probability of inclusion based on the output of binary SVMs. Two kernels are tested. As stated above, the threshold \(\delta\) is set as the prior probability of the number of unseen classes in test.

**P\({}_{\text{i}}\)-OSVM (_P-osvm-linear_ and _P-osvm-rbf_)**. Similar to P\({}_{\Gamma}\)-SVM, P\({}_{\Gamma}\)-OSVM (Jain et al., 2014) uses a multiclass one-class SVM before fitting an Extreme Value Theory distribution to estimate the probability of inclusion. Again, two kernel functions are tested and the prior probability of the number of unseen classes is used to set \(\delta\). As P\({}_{\Gamma}\)-OSVM is a variant of the traditional one-class SVM, we do not use one-class SVM as a baseline.

**Exploratory Seeded K-Means (Exploratory-EM).** In (Dalvi et al., 2013), three well-known multiclass semi-supervised learning methods were extended under the exploratory EM framework. We compare with exploratory version of Seeded K-Means due to its superior performance on 20newsgroup dataset. We also applied the criteria that work the best in the original paper for creating new classes and for model selection, i.e., the MinMax criterion and the AICc criterion. Note that ExploratoryEM works in the semi-supervised setting and uses both the training and test data as labeled and unlabeled data in training. As more than one new class can be introduced during training, for comparison we lump together all instances assigned to new classes as being rejected (unknown). In the experiments, we set the max number of iterations to be 50. Little changes in results are shown after 50 iterations.

All documents use _tf-idf_ term weighting scheme with no feature selection. Source code for different baselines (1-vs-Set Machine2, W-SVM and P\({}_{\Gamma}\)-SVM3, and Exploratory learning4) was provided by the authors of their original papers.

Footnote 2: [https://github.com/Vastlab/liblinear.git](https://github.com/Vastlab/liblinear.git)

Footnote 3: [https://github.com/jlain2/libsm-openest](https://github.com/jlain2/libsm-openest)

Footnote 4: [http://www.cs.crmu.edu/~bbd/ExploreEM_package.zip](http://www.cs.crmu.edu/~bbd/ExploreEM_package.zip)

### Datasets

We perform evaluation using two publically available datasets: 20-newsgroup (Rennie, 2008) and Amazon reviews (Jindal and Liu, 2008). The 20-newsgroup data contains 20 non-overlapping classes with a total of 18828 documents. The Amazon reviews dataset has review documents of 50 types of products or domains. Each type of product has 1000 reviews. For each class in both datasets, we randomly sampled 70% of documents for training, and the rest 30% for testing. Although product reviews are used for experiments, we do not perform sentiment classification. Instead, we still perform the traditional topic based classification. That is, given a review, the system decides what type of product the review is about.

### Experiment settings

Following that in (Jain et al., 2013) and (Dalvi et al., 2013), we conduct open world cross-validation style analysis, holding out some classes in training and mixing them back during testing, and varying the number of training and test classes. Since for a given dataset, the number (percentage) of training classes \(m\) and the number of test classes \(n\) can vary, there are many ways to generate a train-test partition. We report our results using 10 random train-test partitions for each dataset. We vary the number of test classes for Amazon (10, 20, 30, 40, 50), and for 20-newsgroup (10, 20). We use 25%, 

50%, 75% and 100% of the test classes in training.

When 100% of test classes are used in training, the problem reduces to the closed world classification. As most of our baselines such as W-SVM, P\({}_{\text{r}}\)-OSVM and P\({}_{\text{t}}\)-SVM all use prior knowledge to set decision threshold to 0 in the closed world setting, for fair comparison, we also set the threshold to 0 for both 1-vs-rest-SVM and our proposed _cbsSVM_ for closed world classification. By doing this, we always assign a known class label to a test instance. For Exploratory Seeded K-Means, we use an option supported in the exploratory learning package that does not allow any new classes to be introduced in learning.

For each train-test partition, we first compute precision, recall and F1 score for each class and then macro-average the results across all classes. Final results are given by averaging the results of 10 random train-test partitions. Due to space limits, we will only show F1 scores in the paper.

For all the methods that use the RBF kernel, the parameters are tuned via cross validation on the training data, yielding \((C=5,\gamma=0.2)\) for Amazon and \((C=10,\gamma=0.5)\) for 20-newsgroup.

### Results and Discussion

We now show all the results. Results for Amazon is given in Tables 3, 4, and for 20-newsgroup are given in Tables 6 and 8. As we can see, in most situations (23 of 28 settings) our proposed _cbsSVM_ method performs the best. Even when 100% of the test classes are used for training (the traditional closed world classification), _cbsSVM_ still performs 

the best in almost all settings (6 out of 7) except for 20-newsgroup with 20 classes. In this case, it lost to ExploratoryEM by 1.12%. In fact, it is unfair to compare _cbsSVM_ with ExploratoryEM because ExploratoryEM uses the test data in training.

We also analyzed the cases where our technique does not perform well. By comparing Table 1 and Table 6, we see that our method loses to 1-vs-set-linear, wsvm-linear and P\({}_{i}\)-svm-linear on both datasets when training on 2 classes (25%) and testing on 10 classes, though in other cases training on 25% known classes can still yield good results. By inspecting the results, we found that in both settings our technique achieves very high recall but low precision on the known classes, while achieves high precision but low recall on the unknown classes. After careful investigation, we found this is caused by the relatively poor approximation of radius \(r_{O}\) when positive and negative training examples are far apart.

To verify the cause, we conducted more experiments on the 20-newsgroup data using the same setting (10 classes for test and 2 for training). The 10 classes are listed in Table 8. We show the results for two sets of experiments. In each set of the experiments, we keep one known class unchanged in training and select different classes as the second class. We show how the results change on the unchanged class as well as the unknown (reject) classes. Table 9 gives the precision, recall, and F1 score for _comp.windows.x_ and for the unknown classes. Similarly, Table 10 gives the results for _rec.motorcycles_ and for the unknown classes. The first column in both tables are the different second classes used in training. We can see that in both sets of experiments, the precision and F1 score on the unchanged known classes (_comp.windows.x_ and _rec.motorcycles_) are better when a more similar class (closer in distance) is selected in training. In particular, _comp.windows.x_ achieves the best result when _comp.os.ms-windows.misc_ is the second known class, and _rec.motorcycles_ achieves the best result when _rec.autos_ is the second known class. This is because the radius \(r_{O}\) for each positively labeled space is determined based on the distance between the positive and negative training examples. As related classes are closer in distance, a tighter boundary with smaller \(r_{O}\) can be learned. However, our results show in the cases when only 2 known classes are available, a tight boundary is harder to achieve for either class for _cbsSVM_.

## 5 Conclusion

In this paper, we proposed to study the problem of multiclass open text classification. In particular, we investigated the problem via reducing the open space risk, and proposed a solution based on cen

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline  & \multicolumn{3}{c|}{**rec.motorcycles**} & \multicolumn{3}{c}{**Unknown (reject)**} \\ \hline  & Prec. & Recall & F1 & Prec. & Recall & F1 \\ \hline
**comp.sys.mac.hardware** & 0.284 & 0.956 & 0.438 & 0.962 & 0.198 & 0.328 \\ \hline
**rec.autos** & 0.459 & 0.892 & _0.606_ & 0.974 & 0.470 & _0.634_ \\ \hline
**comp.windows.x** & 0.260 & 0.963 & 0.410 & 0.972 & 0.168 & 0.287 \\ \hline
**comp.graphics** & 0.289 & 0.953 & 0.444 & 0.964 & 0.177 & 0.299 \\ \hline
**comp.sys.ibm.pc.hardware** & 0.284 & 0.953 & 0.438 & 0.958 & 0.169 & 0.288 \\ \hline
**alt.atheism** & 0.194 & 0.973 & 0.324 & 0.980 & 0.333 & 0.498 \\ \hline \end{tabular}
\end{table}
Table 10: Results on rec.motorcycles and unknown classes.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline  & \multicolumn{3}{c|}{**comp.windows.x**} & \multicolumn{3}{c}{**Unknown (reject)**} \\ \hline  & Prec. & Recall & F1 & Prec. & Recall & F1 \\ \hline
**rec.motorcycles** & 0.260 & 0.963 & 0.410 & 0.972 & 0.168 & 0.287 \\ \hline
**comp.graphics** & 0.380 & 0.850 & 0.525 & 0.966 & 0.482 & 0.643 \\ \hline
**comp.sys.mac.hardware** & 0.286 & 0.972 & 0.442 & 0.977 & 0.356 & 0.522 \\ \hline
**comp.os.ms-windows.misc** & 0.418 & 0.877 & _0.567_ & 0.976 & 0.513 & _0.672_ \\ \hline
**misc.forsale** & 0.244 & 0.959 & 0.389 & 0.966 & 0.201 & 0.334 \\ \hline
**rec.autos** & 0.226 & 0.979 & 0.367 & 0.976 & 0.162 & 0.277 \\ \hline \end{tabular}
\end{table}
Table 8: 10 domains for testing.

\begin{table}
\begin{tabular}{l|c|c|c|c|c|c} \hline  & \multicolumn{3}{c|}{**rec.motorcycles**} & \multicolumn{3}{c}{**Unknown (reject)**} \\ \hline  & Prec. & Recall & F1 & Prec. & Recall & F1 \\ \hline
**comp.sys.mac.hardware** & 0.284 & 0.956 & 0.438 & 0.962 & 0.198 & 0.328 \\ \hline
**rec.autos** & 0.459 & 0.892 & _0.606_ & 0.974 & 0.470 & _0.634_ \\ \hline
**comp.windows.x** & 0.260 & 0.963 & 0.410 & 0.972 & 0.168 & 0.287 \\ \hline
**comp.graphics** & 0.289 & 0.953 & 0.444 & 0.964 & 0.177 & 0.299 \\ \hline
**comp.sys.ibm.pc.hardware** & 0.284 & 0.953 & 0.438 & 0.958 & 0.169 & 0.288 \\ \hline
**alt.atheism** & 0.194 & 0.973 & 0.324 & 0.980 & 0.333 & 0.498 \\ \hline \end{tabular}
\end{table}
Table 9: Results on comp.windows.x and unknown classes.

 

ter-based similarity space learning. The solution reduced the positive labeled area from an infinite space to a finite space compared to previous work. This markedly reduces the open space risk. With extensive experiments across two public multiclass datasets, we demonstrated that the proposed solution is highly promising. Our future work includes designing a more robust solution that still works well when the number of known classes is small.

## Acknowledgments

This work was supported in part by a grant from National Science Foundation (NSF) under grant no. IIS-1407927, a NCI grant under grant no. R01CA192240, and a gift from Bosch. The content of the paper is solely the responsibility of the authors and does not necessarily represent the official views of the NSF, NCI, or Bosch.

## References

* Bravo et al. [2008]Bravo, C., Lobato, J.L., Weber, R., L'Huillier, G. 2008. A hybrid system for probability estimation in multiclass problems combining svms and neural networks. In: Hybrid Intelligent Systems. pp. 649-654
* Chang and Lin [2011]Chang, C-C. and Lin, C-J. 2011. LIBSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, [http://www.csie.ntu.edu.tw/~cjlin/libsvm](http://www.csie.ntu.edu.tw/~cjlin/libsvm)
* Colas and Brazdil [2006]Colas, F. and Brazdil. P. 2006. Comparison of SVM and some older classification algorithms in text classification tasks. Artificial Intelligence in Theory and Practice. IFIP International Federation for Information Processing, pp. 169-178.
* Dalvi et al. [2013]Dalvi, B., Cohen, W. W. and Callan, J. 2013. Exploratory learning. In ECML.
* Duan and Keerthi [2005]Duan, K.B. and Keerthi, S.S. 2005. Which is the best multiclass SVM method? An empirical study. In: Proceedings of the 6th International Conference on Multiple Classifier Systems. pp. 278-285
* Fei and Liu [2015]Fei, G. and Liu, B. 2015. Social Media Text Classification under Negative Covariate Shift. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisboa, Portugal, 17-21 September.
* Fumera and Roli [2002]Fumera, G., and Roli, F. 2002. Support vector machines with embedded reject option. In: International Workshop on Pattern Recognition with Support Vector Machines (SVM2002). pp. 68-82
* Hastie and Tibshirani [1996]Hastie, T. and Tibshirani, R. 1996. Classification by pairwise coupling. In: Annals of Statistics. pp. 507-513. MIT Press
* Huang et al. [2006]Huang, T.K., Weng, R.C., and Lin, C.J. 2006. Generalized bradley-terry models and multi-class probability estimates. Journal of Machine Learning Research 85-115
* Jain et al. [2014]Jain, L. P., Scheirer, W. J., and Boult, T. E. 2014. Multiclass open set recognition using probability of inclusion. In Proc. ECCV, pages 393-409. Springer.
* Jindal and Liu [2008]Jindal, N. and Liu, B. 2008. Opinion Spam and Analysis. Proceedings of the ACM Conference on Web Search and Data Mining.
* Joachims [1998]Joachims, T. 1998. Text categorization with support vector machines: Learning with many relevant features. ECML.
* Khan and Madden [2014]Khan, S. and Madden, M. 2014. One-Class Classification: Taxonomy of Study and Review of Techniques. The Knowledge Engineering Review, 1-30.
* Kwok [1999]Kwok, J.T.Y. 1999. Moderating the outputs of support vector machine classifiers. IEEE Transactions on Neural Networks 10(5), 1018-1031
* Manning et al. [2008]Manning, C. D., Prabhakar R., and Hinrich, S. 2008. Introduction to Information Retrieval. Cambridge University Press.
* Shackel et al. [2007]Shackel, N., Bertrand's Paradox and the Principle of Indifference. 2007. Philosophy of Science, vol. 74, no. 2, pp. 150-175.
* Platt [2000]Platt, J. C. 2000. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. In Advances in Large Margin Classifiers, A. Smola, P. Bartlett, B. Scholkopf, and D. Schuurmans, Eds. MIT Press, Cambridge, MA.
* Rennie [2008]Rennie, J. 20-newsgroup dataset. 2008
* Rocchio [1971]Rocchio, J. 1971. Relevant feedback in information retrieval. In G. Salton (ed.). The smart retrieval system: experiments in automatic document processing.
* Scheirer et al. [2013]Scheirer, W., Rocha, A., Sapkota A., and Boult T. E. Towards open set recognition. 2013. IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 36, no. 7, pp.1757 -1772
* Scheirer et al. [2014]Scheirer, W. J., Jain, L. P., and Boult, T. E. 2014. Probability models for open set recognition, IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 36, no. 11, pp.2317 -2324
* Scholkopf et al. [2001]Scholkopf, B., Platt, J. C., Shawe-Taylor, J. C., Smola, A. J., Williamson, R. C. 2001. Estimating the support of a high-dimensional distribution. Neural Computation 13, 1443-1471
* Tax and Duin [2004]Tax, D.M.J., Duin, R.P.W. 2004. Support vector data description. Machine Learning 54, 45-66
* Vapnik [1998]Vapnik, V.N. 1998. Statistical Learning Theory. Wiley-Interscience
* Zadrozny and Elkan [2002]Zadrozny, B. and Elkan, C. 2002. Transforming classifier scores into accurate multiclass probability estimates. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 694-699 (2002)