Venue: NAACL 2019

# Integrating Semantic Knowledge to Tackle Zero-shot Text Classification

 Jingqing Zhang \({}^{*}\)

Data Science Institute

Imperial College London

London, UK

&Piyawat Lertvittayakumjorn \({}^{*}\)

Department of Computing

Imperial College London

London, UK

{jingqing.zhang15,pl1515,y.guo}@imperial.ac.uk

Yike Guo

Data Science Institute

Imperial College London

London, UK

###### Abstract

Insufficient or even unavailable training data of emerging classes is a big challenge of many classification tasks, including text classification. Recognising text documents of classes that have never been seen in the learning stage, so-called _zero-shot text classification_, is therefore difficult and only limited previous works tackled this problem. In this paper, we propose a two-phase framework together with data augmentation and feature augmentation to solve this problem. Four kinds of semantic knowledge (word embeddings, class descriptions, class hierarchy, and a general knowledge graph) are incorporated into the proposed framework to deal with instances of unseen classes effectively. Experimental results show that each and the combination of the two phases achieve the best overall accuracy compared with baselines and recent approaches in classifying real-world texts under the zero-shot scenario.

+
Footnote †: Píyawat Lertvittayakumjorn and Jingqing Zhang contributed equally to this project.

## 1 Introduction

As one of the most fundamental problems in machine learning, automatic classification has been widely studied in several domains. However, many approaches, proven to be effective in traditional classification tasks, cannot catch up with a dynamic and open environment where new classes can emerge after the learning stage Romera-Paredes and Torr (2015). For example, the number of topics on social media is growing rapidly, and the classification models are required to recognise the text of the new topics using only general information (e.g., descriptions of the topics) since labelled training instances are unfeasible to obtain for each new topic Lee et al. (2011). This scenario holds in many real-world domains such as object recognition and medical diagnosis Xian et al. (2017); World Health Organization (1996).

Zero-shot learning (ZSL) for text classification aims to classify documents of classes which are absent from the learning stage. Although it is challenging for a machine to achieve, humans are able to learn new concepts by transferring knowledge from known to unknown domains based on high-level descriptions and semantic representations Thrun and Pratt (1998). Therefore, without labelled data of unseen classes, a zero-shot learning framework is expected to exploit supportive semantic knowledge (e.g., class descriptions, relations among classes, and external domain knowledge) to generally infer the features of unseen classes using patterns learned from seen classes.

So far, three main types of semantic knowledge have been employed in general zero-shot scenarios Fu et al. (2018). The most widely used one is semantic attributes of classes such as visual concepts (e.g., colours, shapes) and semantic properties (e.g., behaviours, functions Lampert et al. (2009); Zhao et al. (2018). The second type is concept ontology, including class hierarchy and knowledge graphs, which represents relationships among classes and features Wang et al. (2018); Fergus et al. (2010). The third type is semantic word embeddings which capture implicit relationships between words thanks to a large training text corpus Socher et al. (2013); Norouzi et al. (2013). Nonetheless, concerning ZSL in text classification particularly, there are few studies exploiting one of these knowledge types and none has considered the combinations of them Pushp and Srivastava (2017); Dauphin et al. (2013). Moreover, some previous works used different datasets to train and test, but there is similarity between classes in the training and testing set. For example, in Dauphin et al. (2013), the class "imbb.com" in the training set naturally corresponds to the class "Movies" in

the testing set. Hence, these methods are not working under a strict zero-shot scenario.

To tackle the zero-shot text classification problem, this paper proposes a novel two-phase framework together with data augmentation and feature augmentation (Figure 1). In addition, four kinds of semantic knowledge including word embeddings, class descriptions, class hierarchy, and a general knowledge graph (ConceptNet) are exploited in the framework to effectively learn the unseen classes. Both of the two phases are based on convolutional neural networks Kim (2014). The first phase called **coarse-grained classification** judges if a document is from seen or unseen classes. Then, the second phase, named **fine-grained classification**, finally decides its class. Note that all the classifiers in this framework are trained using labelled data of seen classes (and augmented text data) only. None of the steps learns from the labelled data of unseen classes.

The contributions of our work can be summarised as follows.

* We propose a novel deep learning based two-phase framework, including coarse-grained and fine-grained classification, to tackle the zero-shot text classification problem. Unlike some previous works, our framework does not require semantic correspondence between classes in a training stage and classes in an inference stage. In other words, the seen and unseen classes can be clearly different.
* We propose a novel data augmentation technique called topic translation to strengthen the capability of our framework to detect documents from unseen classes effectively.
* We propose a method to perform feature augmentation by using integrated semantic knowledge to transfer the knowledge learned from seen to unseen classes in the zero-shot scenario.

In the remainder of this paper, we firstly explain our proposed zero-shot text classification framework in section 2. Experiments and results, which demonstrate the performance of our framework, are presented in section 3. Related works are discussed in section 4. Finally, section 5 concludes our work and mentions possible future work.

## 2 Methodology

### Problem Formulation

Let \({\cal C}_{\mathcal{S}}\) and \({\cal C}_{\mathcal{U}}\) be disjoint sets of seen and unseen classes of the classification respectively. In the learning stage, a training set \(\{(x_{1},y_{1}),\ldots,(x_{n},y_{n})\}\) is given where \(x_{i}\) is the \(i\)-th document containing a sequence of words \([w^{i}_{1},w^{i}_{2},\dots,w^{i}_{t}]\) and \(y_{i}\in{\cal C}_{\mathcal{S}}\) is the class of \(x_{i}\). In the inference stage, the goal is to predict the class of each document, \(\hat{y}_{i}\), in a testing set which has the same data format as the training set except that \(y_{i}\) comes from \({\cal C}_{\mathcal{S}}\cup{\cal C}_{\mathcal{U}}\). Note that _(i)_ every class comes with a class label and a class description (Figure 1(a)); _(ii)_ a class hierarchy showing superclass-subclass relationships is also provided (Figure 1(b)); _(iii)_ the documents from unseen classes cannot be observed to train the framework.

### Overview and Notations

As discussed in the Introduction, our proposed classification framework consists of two phases

Figure 1: The overview of the proposed framework with two phases. The coarse-grained phase judges if an input document \(x_{i}\) comes from seen or unseen classes. The fine-grained phase finally decides the class \(\hat{y}_{i}\). All notations are defined in section 2.1-2.2.

 

(Figure 1). The first phase, coarse-grained classification, predicts whether an input document comes from seen or unseen classes. We also apply a data augmentation technique in this phase to help the classifiers be aware of the existence of unseen classes without accessing their real data. Then the second phase, fine-grained classification, finally specifies the class of the input document. It uses either a traditional classifier or a zero-shot classifier depending on the coarse-grained prediction given by Phase 1. Also, feature augmentation based on semantic knowledge is used to provide additional information which relates the document and the unseen classes to generalise the zero-shot reasoning.

We use the following notations in Figure 1 and throughout this paper.

* The list of embeddings of each word in the document \(x_{i}\) is denoted by \(v^{i}_{w}=[v^{i}_{w_{1}},v^{i}_{w_{2}},\ldots,v^{i}_{w_{t}}]\).
* The embedding of each class label \(c\) is denoted by \(v_{c}\), \(\forall c\in\mathcal{C}_{\mathcal{S}}\cup\mathcal{C}_{\mathcal{U}}\). It is assumed that each class has a one-word class label. If the class label has more than one word, a similar one-word class label is provided to find \(v_{c}\).
* As augmented features, the relationship vector \(v^{i}_{w_{j},c}\) shows the degree of relatedness between the word \(w_{j}\) and the class \(c\) according to semantic knowledge. Hence, the list of relationship vectors between each word in \(x_{i}\) and each class \(c\in\mathcal{C}_{\mathcal{S}}\cup\mathcal{C}_{\mathcal{U}}\) is denoted by \(v^{i}_{w,c}\)\(=[v^{i}_{w_{1},c},v^{i}_{w_{2},c},\ldots,v^{i}_{w_{t},c}]\). We will explain the construction method in section 2.4.1.

### Phase 1: Coarse-grained Classification

Given a document \(x_{i}\), Phase 1 performs a binary classification to decide whether \(\hat{y}_{i}\in\mathcal{C}_{\mathcal{S}}\) or \(\hat{y}_{i}\notin\mathcal{C}_{\mathcal{S}}\). In this phase, each seen class \(c_{s}\in\mathcal{C}_{\mathcal{S}}\) has its own CNN classifier (with a subsequent dense layer and a sigmoid output) to predict the confidence that \(x_{i}\) comes from the class \(c_{s}\), i.e., \(p(\hat{y}_{i}=c_{s}|x_{i})\). The classifier uses \(v^{i}_{w}\) as an input and it is trained using a binary cross entropy loss with all documents of its class in the training set as positive examples and the rest as negative examples.

For a test document \(x_{i}\), this phase computes \(p(\hat{y}_{i}=c_{s}|x_{i})\) for every seen class \(c_{s}\) in \(\mathcal{C}_{\mathcal{S}}\). If there exists a class \(c_{s}\) such that \(p(\hat{y}_{i}=c_{s}|x_{i})>\tau_{s}\), it predicts \(\hat{y}_{i}\in\mathcal{C}_{\mathcal{S}}\); otherwise, \(\hat{y}_{i}\notin\mathcal{C}_{\mathcal{S}}\). \(\tau_{s}\) is a classification threshold for the class \(c_{s}\), calculated based on the threshold adaptation method from Shu et al. (2017).

#### 2.3.1 Data Augmentation

During the learning stage, the classifiers in Phase 1 use negative examples solely from seen classes, so they may not be able to differentiate the positive class from unseen classes. Hence, when the names of unseen classes are known in the inference stage, we try to introduce them to the classifiers in Phase 1 via augmented data so they can learn to reject the instances likely from unseen classes. We do data augmentation by translating a document from its original seen class to a new unseen class using analogy. We call this process _topic translation_.

In the word level, we translate a word \(w\) in a document of class \(c\) to a corresponding word \(w^{\prime}\) in the context of a target class \(c^{\prime}\) by solving an analogy question "\(c\):: \(c^{\prime}\)?". For example, solving the analogy "company:firm :: village:??" via word embeddings Mikolov et al. (2013), we know that the word "firm" in a document of class "company" can be translated into the word "hamlet" in the context of class "village". Our framework adopts the 3CoSMul method by Levy and Goldberg (2014) to solve the analogy question and find

Figure 2: Illustrations of semantic knowledge integrated into our framework: (a) class labels and class descriptions (b) class hierarchy and (c) a subgraph of the general knowledge graph (ConceptNet).

 

candidates of \(w^{\prime}\):

\[w^{\prime}=\operatorname*{argmax}_{x\in V}\frac{\cos(x,c^{\prime})\cos(x,w)}{\cos (x,c)+\epsilon}\]

where \(V\) is a vocabulary set and \(\cos(a,b)\) is a cosine similarity score between the vectors of word \(a\) and word \(b\). Also, \(\epsilon\) is a small number (i.e., 0.001) added to prevent division by zero.

In the document level, we follow Algorithm 1 to translate a document of class \(c\) into the topic of another class \(c^{\prime}\). To explain, we translate all nouns, verbs, adjectives, and adverbs in the given document to the target class, word-by-word, using the word-level analogy. The word to replace must have the same part of speech as the original word and all the replacements in one document are 1-to-1 relations, enforced by replace_dict in Algorithm 1. With this idea, we can create augmented documents for the unseen classes by topic-translation from the documents of seen classes in the training dataset. After that, we can use the augmented documents as additional negative examples for all the CNNs in Phase 1 to make them aware of the tone of unseen classes.

```
Input : a document \(x_{i}=[v^{1}_{1},w^{2}_{4},\ldots,w^{i}_{t}]\), an original class label \(c\), a target class label \(c^{\prime}\) Output : translated document \(x^{\prime}_{i}\) replace_dict = _dict_(\(t\)); \(x^{\prime}_{i}\) = []; foreach\(w\in x_{i}\)do ifis_valid_pos(\(w\))then if\(w\)<|place_dict then  each s = solve_analogy(\(w\), \(c\), \(c^{\prime}\), top_k=20); for\(j\)=0 to len(cands)-1 do ifcalls[\(j\)]\(\notin\)replace_dict.values() \(\wedge\) pos.of(\(w\)) \(\in\) pos.list(cands[\(j\)])  Helen replace_dict[\(w\)] = cands[\(j\)]; break; if\(j\)==len(cands)then \(x^{\prime}_{i}\),append(\(w\)); continue; \(x^{\prime}_{i}\),append(replace_dict[\(w\)]); else \(x^{\prime}_{i}\),append(\(w\)); return\(x^{\prime}_{i}\)
```

**Algorithm 1**Document-level topic translation

### Phase 2: Fine-grained Classification

Phase 2 decides the most appropriate class \(\hat{y}_{i}\) for \(x_{i}\) using two CNN classifiers: a traditional classifier and a zero-shot classifier as shown in Figure 1. If \(\hat{y}_{i}\in\mathcal{C}_{\mathcal{S}}\) predicted by Phase 1, the traditional classifier will finally select a class \(c_{s}\in\mathcal{C}_{\mathcal{S}}\) as \(\hat{y}_{i}\). Otherwise, if \(\hat{y}_{i}\notin\mathcal{C}_{\mathcal{S}}\), the zero-shot classifier will be used to select a class \(c_{u}\in\mathcal{C}_{\mathcal{U}}\) as \(\hat{y}_{i}\).

The traditional classifier and the zero-shot classifier have an identical CNN-based structure followed by two dense layers but their inputs and outputs are different. The traditional classifier is a multi-class classifier (\(|\mathcal{C}_{\mathcal{S}}|\) classes) with a softmax output, so it requires only the word embeddings \(v^{i}_{w}\) as an input. This classifier is trained using a cross entropy loss with a training dataset whose examples are from seen classes only.

In contrast, the zero-shot classifier is a binary classifier with a sigmoid output. Specifically, it takes a text document \(x_{i}\) and a class \(c\) as inputs and predicts the confidence \(p(\hat{y}_{i}=c|x_{i})\). However, in practice, we utilise \(v^{i}_{w}\) to represent \(x_{i}\), \(v_{c}\) to represent the class \(c\), and also augmented features \(v^{i}_{w,c}\) to provide more information on how intimate the connections between words and the class \(c\) are. Altogether, for each word \(w_{j}\), the classifier receives the concatenation of three vectors (i.e., \([v^{i}_{w_{j}};v;v^{i}_{w_{j},c}]\)) as an input. This classifier is trained using a binary cross entropy loss with a training data from seen classes only, but we expect this classifier to work well on unseen classes thanks to the distinctive patterns of \(v^{i}_{w,c}\) in positive examples of every class. This is how we transfer knowledge from seen to unseen classes in ZSL.

#### 2.4.1 Feature Augmentation

The relationship vector \(v_{w_{j},c}\) contains augmented features we input to the zero-shot classifier. \(v_{w_{j},c}\) shows how the word \(w_{j}\) and the class \(c\) are related considering the relations in a general knowledge graph. In this work, we use ConceptNet providing general knowledge of natural language words and phrases Speer and Havasi (2013). A subgraph of ConceptNet is shown in Figure 1(c) as an illustration. Nodes in ConceptNet are words or phrases, while edges connecting two nodes show how they are related either syntactically or semantically.

We firstly represent a class \(c\) as three sets of nodes in ConceptNet by processing the class hierarchy, class label, and class description of \(c\). (1) _the_class_nodes_ is a set of nodes of the class label \(c\) and any tokens inside \(c\) if \(c\) has more than one word. (2) _superclass_nodes_ is a set of nodes of all the superclasses of \(c\) according to the class hierarchy. (3) _description_nodes_ is a set of nodes of all nouns in the description of the class \(c\). For example, if \(c\) is the class "Educational Institution", according to Figure 1(a)-1(b), the three sets of Con 

ceptNet nodes for this class are:

(1) educational_institution, educational, institution (2) organization, agent

(3) place, people, ages, education.

To construct \(v_{w_{j},c}\), we consider whether the word \(w_{j}\) is connected to the members of the three sets above within \(K\) hops by particular types of relations or not1. For each of the three sets, we construct a vector with \(3K+1\) dimensions.

Footnote 1: In this paper, we only consider the most common types of positive relations which are _RelatedTo, IsA, PartOf_, and _AttLocation_. They cover \(\sim\)60% of all edges in ConceptNet.

* \(v[0]=1\) if \(w_{j}\) is a node in that set; otherwise, \(v[0]=0\).
* for \(k=0,\ldots,K-1\):
* \(v[3k+1]=1\) if there is a node in the set whose shortest path to \(w_{j}\) is \(k+1\). Otherwise, \(v[3k+1]=0\).
* \(v[3k+2]\) is the number of nodes in the set whose shortest path to \(w_{j}\) is \(k+1\).
* \(v[3k+3]\) is \(v[3k+2]\) divided by the total number of nodes in the set.

Thus, the vector associated to each set shows how \(w_{j}\) is semantically close to that set. Finally, we concatenate the constructed vectors from the three sets to become \(v_{w_{j},c}\) with \(3\times(3K+1)\) dimensions.

## 3 Experiments

### Datasets

We used two textual datasets for the experiments. The vocabulary size of each dataset was limited by 20,000 most frequent words and all numbers were excluded. (1) **DBpedia** ontology dataset (Zhang et al., 2015) includes 14 non-overlapping classes and textual data collected from Wikipedia. Each class has 40,000 training and 5,000 testing samples. (2) The **20newsgroups** dataset 2 has 2 topics each of which has approximately 1,000 documents. 70% of the documents of each class were randomly selected for training, and the remaining 30% were used as a testing set.

Footnote 2: [http://qwone.com/~jason/20Newsgroups/](http://qwone.com/~jason/20Newsgroups/)

Footnote 3: Code: [https://github.com/JingqingZ/KG4ZeroShotText](https://github.com/JingqingZ/KG4ZeroShotText).

### Implementation Details 3
Footnote 3: See Appendix A for details on the results.

In our experiments, two different rates of unseen classes, 50% and 25%, were chosen and the corresponding sizes of \(\mathcal{C}_{S}\) and \(\mathcal{C}_{\mathcal{U}}\) are shown in Table 1. For each dataset and each unseen rate, the random selection of (\(\mathcal{C}_{\mathcal{S}}\), \(\mathcal{C}_{\mathcal{U}}\)) were repeated ten times and these ten groups were used by all the experiments with this setting for a fair comparison. All documents from \(\mathcal{C}_{\mathcal{U}}\) were removed from the training set accordingly. Finally, the results from all the ten groups were averaged.

In Phase 1, the structure of each classifier was identical. The CNN layer had three filter sizes [3, 4, 5] with 400 filters for each filter size and the subsequent dense layer had 300 units. For data augmentation, we used gensim with an implementation of 3CosMul(Rehurek and Sojka, 2010) to solve the word-level analogy (line 5 in Algorithm 1). Also, the numbers of augmented text documents per unseen class for every setting (if used) are indicated in Table 1. These numbers were set empirically considering the number of available training documents to be translated.

In Phase 2, the traditional classifier and the zero-shot classifier had the same structure, in which the CNN layer had three filter sizes [2, 4, 8] with 600 filters for each filter size and the two intermediate dense layers had 400 and 100 units respectively. For feature augmentation, the maximum path length \(K\) in ConceptNet was set to 3 to create the relationship vectors4. The DBpedia ontology5 was used to construct a class hierarchy of the DBpedia dataset. The class hierarchy of the 20newsgroups dataset was constructed based on the namespaces initially provided by the dataset. Meanwhile, the classes descriptions of both datasets were picked from Macmillan Dictionary6 as appropriate.

Footnote 4: Based on our observation, most of the related words stay within 3 hops from the class nodes in ConceptNet.

Footnote 5: [http://mappings.dbpedia.org/server/ontology/classes/](http://mappings.dbpedia.org/server/ontology/classes/)

Footnote 6: [https://www.macmilladictionary.com/](https://www.macmilladictionary.com/)

For both phases, we used 200-dim GloVe vectors7 for word embeddings \(v_{w}\) and \(v_{c}\)(Pennington et al., 2014). All the deep neural networks were implemented with TensorLayer (Dong et al., 2017a) and TensorFlow (Abadi et al., 2016).

Footnote 7: [https://www.macmillandictionary.com/](https://www.macmillandictionary.com/) 



### Baselines and Evaluation Metrics

We compared each phase and the overall framework with the following approaches and settings.

**Phase 1:** Proposed by Shu et al. (2017), **DOC** is a state-of-the-art open-world text classification approach which classifies a new sample into a seen class or "reject" if the sample does not belong to any seen classes. The DOC uses a single CNN and a 1-vs-rest sigmoid output layer with threshold adjustment. Unlike DOC, the classifiers in the proposed Phase 1 work individually. However, for a fair comparison, we used DOC only as a binary classifier in this phase (\(\hat{y}_{i}\in\mathcal{C}_{\mathcal{S}}\) or \(\hat{y}_{i}\notin\mathcal{C}_{\mathcal{S}}\)).

**Phase 2:** To see how well the augmented feature \(v_{w,c}\) work in ZSL, we ran the zero-shot classifier with **different combinations of inputs**. Particularly, five combinations of \(v_{w}\), \(v_{c}\), and \(v_{w,c}\) were tested with documents from unseen classes only (traditional ZSL).

**The whole framework:** (1) **Count-based model** selected the class whose label appears most frequently in the document as \(\hat{y}_{i}\). (2) **Label similarity** Sappadla et al. (2016) is an unsupervised approach which calculates the cosine similarity between the sum of word embeddings of each class label and the sum of word embeddings of every n-gram (\(n=1,2,3\)) in the document. We adopted this approach to do single-label classification by predicting the class that got the highest similarity score among all classes. (3) **RNN AutoEncoder** was built based on a Seq2Seq model with LSTM (512 hidden units), and it was trained to encode documents and class labels onto the same latent space. The cosine similarity was applied to select a class label closest to the document on the latent space. (4) **RNN+FC** refers to the architecture 2 proposed in Pushp and Srivastava (2017). It used an RNN layer with LSTM (512 hidden units) followed by two dense layers with 400 and 100 units respectively. (5) **CNN+FC** replaced the RNN in the previous model with a CNN, which has the identical structure as the zero-shot classifier in Phase 2. Both RNN+FC and CNN+FC predicted the confidence \(p(\hat{y}_{i}=c|x_{i})\) given \(v_{w}\) and \(v_{c}\). The class with the highest confidence was selected as \(\hat{y}_{i}\).

For Phase 1, we used the accuracy for binary classification (\(y,\hat{y}_{i}\in\mathcal{C}_{\mathcal{S}}\) or \(y,\hat{y}_{i}\notin\mathcal{C}_{\mathcal{S}}\)) as an evaluation metric. In contrast, for Phase 2 and the whole framework, we used the multi-class classification accuracy (\(\hat{y}_{i}=y_{i}\)) as a metric.

### Results and Discussion

**The evaluation of Phase 1** (coarse-grained classification) checks if each \(x_{i}\) was correctly delivered to the right classifier in Phase 2. Table 3 shows the performance of Phase 1 with and without augmented data compared with DOC. Considering test documents from seen classes only, our framework outperformed DOC on both datasets. In addition, the augmented data improved the accuracy of detecting documents from unseen classes clearly and led to higher overall accuracy in every setting. Despite no real labelled data from unseen classes, the augmented data generated by topic translation helped Phase 1 better detect documents from unseen classes. Table 4 shows some examples of augmented data from the DBpedia dataset. Even if they are not completely understandable, they contain the tone of the target classes.

Although Phase 1 provided confidence scores for all seen classes, we could not use them to predict \(\hat{y}_{i}\) directly since the distribution of scores of positive examples from different CNNs are different. Figure 3 shows that the distribution of confidence scores of the class "Artist" had a noticeably larger variance and was clearly different from the class "Building". Hence, even if \(p(\hat{y}_{i}=``\text{Building''}|x_{i})>p(\hat{y}_{i}=``\text{Artist''}|x_{i})\), we cannot conclude that \(x_{i}\) is more likely to come from the class "Building". This is why **a traditional classifier in Phase 2 is necessary**.

Regarding Phase 2, fine-grained classification is in charge of predicting \(\hat{y}_{i}\) and it employs two classifiers which were tested separately. Assuming Phase 1 is perfect, the classifiers in Phase 2 should be able to find the right class. The purpose of Table 5 is to show that **the traditional CNN classifier in Phase 2** was highly accurate.

Figure 3: The distributions of confidence scores of positive examples from four seen classes of DBpedia in Phase 1.



Besides, given test documents from unseen classes only, the performance of **the zero-shot classifier in Phase 2** is shown in Table 6. Based on the construction method, quantified the relatedness between words and the class but, unlike \(v_{w}\) and \(v_{c}\), it did not include detailed semantic meaning. Thus, the classifier using \(v_{w,c}\) only could not find out the correct unseen class and neither \([v_{w};v_{w,c}]\) and \([v_{c};v_{w,c}]\) could do. On the other hand, the combination of \([v_{w};v_{c}]\), which included semantic embeddings of both words and the class label, increased the accuracy of predicting unseen classes clearly. However, the zero-shot classifier fed by the combination of all three types of inputs \([v_{w};v_{c};v_{w,c}]\) achieved the highest accuracy in all settings. It asserts that the integration of semantic knowledge we proposed is an effective means for knowledge transfer from seen to unseen classes in the zero-shot scenario.

Last but most importantly, we compared **the whole framework** with four baselines as shown in Table 2. First, the count-based model is a rule-based model so it failed to predict documents from seen classes accurately and resulted in unpleasant overall results. This was similar to the label similarity approach even though it had higher degree of flexibility. Next, the RNN Autoencoder was trained without any supervision since \(\hat{y}_{i}\) was predicted based on the cosine similarity. We believe

\begin{table}
\begin{tabular}{|r||l||l|l|l|l|} \hline Dataset & \multicolumn{3}{c|}{DBpedia} & \multicolumn{3}{c|}{20news} \\ \hline Inputs \textbackslash{} Unseen rate & 50\% & 25\% & 50\% & 25\% \\ \hline \hline Random guess & 0.143 & 0.333 & 0.100 & 0.200 \\ \hline \(v_{w,c}\) & 0.154 & 0.443 & 0.104 & 0.210 \\ \hline \([v_{c};w_{c}]\) & 0.163 & 0.400 & 0.099 & 0.215 \\ \hline \([v_{w};v_{w,c}]\) & 0.266 & 0.460 & 0.122 & 0.307 \\ \hline \([v_{w};v_{c}]\) & 0.381 & 0.711 & 0.274 & 0.431 \\ \hline \([v_{w};v_{c};v_{w,c}]\) & **0.418** & **0.754** & **0.302** & **0.500** \\ \hline \end{tabular}
\end{table}
Table 6: The accuracy of the zero-shot classifier in Phase 2 given documents from unseen classes only.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}||p{56.9pt}||p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline Dataset & \multirow{2}{*}{\(y_{i}\)} & \multirow{2}{*}{DOC} & \multicolumn{2}{c|}{Ours} & \multicolumn{1}{p{56.9pt}|}{Ours} & \multicolumn{1}{p{56.9pt}|}{Ours} \\  & & & & w/o aug. & w/ aug. & \multicolumn{1}{p{56.9pt}|}{W/ aug.} \\ \hline \hline DBpedia & seen & 0.980 & **0.982** & **0.982** & **0.982** \\
25\% & unseen overall & 0.471 & 0.388 & **0.536** \\
20 & overall & 0.871 & 0.855 & **0.886** \\ \hline DBpedia & seen & 0.983 & 0.986 & **0.987** \\
50\% & unseen overall & 0.384 & 0.345 & **0.512** \\
20news & seen overall & 0.684 & 0.666 & **0.749** \\ \hline
20news & seen & 0.800 & **0.838** & 0.831 \\
25\% & unseen overall & 0.573 & 0.431 & **0.577** \\
20news & seen overall & 0.745 & 0.754 & **0.770** \\ \hline
20news & seen & 0.824 & **0.856** & 0.843 \\
50\% & unseen overall & 0.662 & 0.419 & **0.603** \\ \hline \end{tabular}
\end{table}
Table 2: The accuracy of the whole framework compared with the baselines.

\begin{table}
\begin{tabular}{|p{56.9pt}|p{56.9pt}||p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|p{56.9pt}|} \hline Dataset & \multirow{2}{*}{\(y_{i}\)} & \multirow{2}{*}{COUT-based} & \multirow{2}{*}{Coun-based} & Label & RNN + FC & \multirow{2}{*}{CNN + FCN} & \multirow{2}{*}{Ours} \\  & & & Similarity & RNN & Pushy and Srivastava, & & FC \\ \cline{5-7}  & rate & & & (Sapporla et al., 2017) & Autoencoder & Srivastava, & & FC \\ \hline \hline DBpedia & \multirow{2}{*}{25\%} & seen unseen overall & 0.322 & 0.377 & 0.250 & 0.895 & **0.985** & 0.975 \\  & & unseen overall & 0.372 & **0.426** & 0.267 & 0.046 & 0.204 & 0.402 \\  & & overall & 0.334 & 0.386 & 0.254 & 0.713 & 0.818 & **0.852** \\ \cline{2-7}  & \multirow{2}{*}{50\%} & seen seen unseen overall & 0.358 & 0.401 & 0.202 & 0.960 & **0.991** & 0.982 \\  & & unseen unseen & 0.304 & **0.369** & 0.259 & 0.044 & 0.069 & 0.197 \\  & & overall & 0.333 & 0.386 & 0.230 & 0.502 & 0.530 & **0.590** \\ \hline \multirow{4}{*}{20news} & \multirow{2}{*}{25\%} & seen unseen unseen overall & 0.205 & 0.279 & 0.263 & 0.614 & **0.792** & 0.745 \\  & & unseen & 0.201 & **0.287** & 0.149 & 0.065 & 0.134 & 0.280 \\ \cline{1-1}  & & overall & 0.204 & 0.280 & 0.236 & 0.482 & **0.633** & **0.633** \\ \cline{1-1} \cline{2-7}  & \multirow{2}{*}{50\%} & seen unseen unseen overall & 0.219 & 0.293 & 0.275 & 0.709 & 0.684 & **0.767** \\ \cline{1-1}  & & unseen unseen & 0.196 & **0.266** & 0.126 & 0.052 & 0.126 & 0.168 \\ \cline{1-1}  & & overall & 0.207 & 0.280 & 0.200 & 0.381 & 0.405 & **0.469** \\ \hline \end{tabular}
\end{table}
Table 3: The accuracy of Phase 1 with and without augmented data compared with DOC .

\begin{table}
\begin{tabular}{|p{56.9pt}||p{56.9pt}|p{56.9pt}|} \hline Dataset & \multicolumn{1}{p{56.9pt}|}{Differential} & \multicolumn{1}{p{56.9pt}|}{Differential} & \multicolumn{1}{p{56.

 

the implicit semantic relatedness between classes caused the failure of the RNN Autoencoder. Besides, the CNN+FC and RNN+FC had same inputs and outputs and it was clear that CNN+FC performed better than RNN+FC in the experiment. However, neither CNN+FC nor RNN+FC was able to transfer the knowledge learned from seen to unseen classes. Finally, our two-phase framework has competitive prediction accuracy on unseen classes while maintaining the accuracy on seen classes. This made it achieve the highest overall accuracy on both datasets and both unseen rates. In conclusion, by using integrated semantic knowledge, the proposed two-phase framework with data and feature augmentation is a promising step to tackle this challenging zero-shot problem.

Furthermore, another benefit of the framework is high flexibility. As the modules in Figure 1 has less coupling to one another, it is flexible to improve or customise each of them. For example, we can deploy an advanced language understanding model, e.g., BERT Devlin et al. (2018), as a traditional classifier. Moreover, we may replace ConceptNet with a domain-specific knowledge graph to deal with medical texts.

## 4 Related Work

### Zero-shot Text Classification

There are a few more related works to discuss besides recent approaches we compared with in the experiments (explained in section 3.3). Dauphin et al. dauphin2013distributed predicted semantic utterance of texts by mapping class labels and text samples into the same semantic space and classifying each sample to the closest class label. Nam et al. nam2016learned the embeddings of classes, documents, and words jointly in the learning stage. Hence, it can perform well in domain-specific classification, but this is possible only with a large amount of training data. Overall, most of the previous works exploited semantic relationships between classes and documents via embeddings. In contrast, our proposed framework leverages not only the word embeddings but also other semantic knowledge. While word embeddings are used to solve analogy for data augmentation in Phase 1, the other semantic knowledge sources (in Figure 2) are integrated into relationship vectors and used as augmented features in Phase 2. Furthermore, our framework does not require any semantic correspondences between seen and unseen classes.

### Data Augmentation in NLP

In the face of insufficient data, data augmentation has been widely used to improve generalisation of deep neural networks especially in computer vision Krizhevsky et al. (2012) and multimodality Dong et al. (2017b), but it is still not a common practice in natural language processing. Recent works have explored data augmentation in NLP tasks such as machine translation and text classification Saito et al. (2017); Fadaee et al. (2017); Kobayashi (2018), and the algorithms were designed to preserve semantic meaning of an original document by using synonyms Zhang and LeCun (2015) or adding noises Xie et al. (2017), for example. In contrast, our proposed data augmentation technique translates a document from one meaning (its original class) to another meaning (an unseen class) by analogy in order to substitute unavailable labelled data of the unseen class.

### Feature Augmentation in NLP

Apart from improving classification accuracy, feature augmentation is also used in domain adaptation to transfer knowledge between a source and a target domain Pan et al. (2010); Fang and Chiang (2018); Chen et al. (2018). An early research paper applying feature augmentation in NLP is Daume III daumeIII (2007) which targeted domain adaptation on sequence labelling tasks. After that, feature augmentation was used in several NLP tasks such as cross-domain sentiment classification Pan et al. (2010a), multi-domain machine translation Clark et al. (2012), semantic argument classification Batubara et al. (2018), etc. Our work is different from previous works not only that we applied this technique to zero-shot text classification but also that we integrated many types of semantic knowledge to create the augmented features.

## 5 Conclusion and Future Work

To tackle zero-shot text classification, we proposed a novel CNN-based two-phase framework together with data augmentation and feature augmentation. The experiments show that data augmentation by topic translation improved the accuracy in detecting instances from unseen classes, while feature augmentation enabled knowledge transfer from seen to unseen classes for zero-shot learning. Thanks to the framework and the integrated semantic knowledge, our work achieved the highest overall accuracy compared with all the 

baselines and recent approaches in all settings. In the future, we plan to extend our framework to do multi-label classification with a larger amount of data, and also study how semantic units defined by linguists can be used in the zero-shot scenario.

## Acknowledgments

We would like to thank Douglas McIlwraith, Nontawat Charoenphakdee, and three anonymous reviewers for helpful suggestions. Jingqing and Piyawat would also like to thank the support from the LexisNexis(r) Risk Solutions HPCC Systems(r) academic program and Anandamahidol Foundation, respectively.

## References

* Abadi et al. (2016) Martin Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In _12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16)_, pages 265-283.
* Batubara et al. (2018) Dina Khaira Batubara, Moch Arif Bijaksana, and Adiwijava. 2018. On feature augmentation for semantic argument classification of the quran english translation using support vector machine. _Journal of Physics: Conference Series_, 971(1):012043.
* Chen et al. (2018) Zitian Chen, Yanwei Fu, Yinda Zhang, Yu-Gang Jiang, Xiangyang Xue, and Leonid Sigal. 2018. Semantic feature augmentation in few-shot learning. _CoRR_, abs/1804.05298.
* Clark et al. (2012) Jonathan H Clark, Alon Lavie, and Chris Dyer. 2012. One system, many domains: Open-domain statistical machine translation via feature augmentation. In _Proceedings of the Conference of the Association for Machine Translation in the Americas_.
* Daume III (2007) Hal Daume III. 2007. Frustratingly easy domain adaptation. In _Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics_, pages 256-263. Association for Computational Linguistics.
* Dauphin et al. (2013) Yann N Dauphin, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2013. Zero-shot learning for semantic utterance classification. _arXiv preprint arXiv:1401.0509_.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Dong et al. (2017a) Hao Dong, Akara Supratak, Luo Mai, Fangde Liu, Axel Oehmichen, Simiao Yu, and Yike Guo. 2017a. Tensorlayer: a versatile library for efficient deep learning development. In _Proceedings of the 2017 ACM on Multimedia Conference_, pages 1201-1204. ACM.
* Dong et al. (2017b) Hao Dong, Jingqing Zhang, Douglas McIlwraith, and Yike Guo. 2017b. I2t2i: Learning text to image synthesis with textual data augmentation. In _Image Processing (ICIP), 2017 IEEE International Conference on_, pages 2015-2019. IEEE.
* Fadaee et al. (2017) Marzieh Fadaee, Arianna Bisazza, and Christof Monz. 2017. Data augmentation for low-resource neural machine translation. _arXiv preprint arXiv:1705.00440_.
* Fang and Chiang (2018) Wen-Chieh Fang and Yi-Ting Chiang. 2018. A discriminative feature mapping approach to heterogeneous domain adaptation. _Pattern Recognition Letters_, 106:13-19.
* ECCV 2010_, pages 762-775, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Fu et al. (2018) Y. Fu, T. Xiang, Y. Jiang, X. Xue, L. Sigal, and S. Gong. 2018. Recent advances in zero-shot recognition: Toward data-efficient understanding of visual content. _IEEE Signal Processing Magazine_, 35(1):112-125.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. _arXiv preprint arXiv:1408.5882_.
* Kobayashi (2018) Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 452-457. Association for Computational Linguistics.
* Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In _Advances in neural information processing systems_, pages 1097-1105.
* Lampert et al. (2009) Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In _IEEE Conference on Computer Vision and Pattern Recognition_, pages 951-958. IEEE.
* Lee et al. (2011) Kathy Lee, Diana Palsetia, Ramanathan Narayanan, Md Mostofa Ali Patwary, Ankit Agrawal, and Alok Choudhary. 2011. Twitter trending topic classification. In _Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on_, pages 251-258. IEEE 

Omer Levy and Yoav Goldberg. 2014. Linguistic regularities in sparse and explicit word representations. In _Proceedings of the eighteenth conference on computational natural language learning_, pages 171-180.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. _arXiv preprint arXiv:1301.3781_.
* Nam et al. (2016) Jinseok Nam, Eneldo Loza Mencia, and Johannes Furnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In _Thirtieth AAAI Conference on Artificial Intelligence_.
* Norouzi et al. (2013) Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings. _arXiv preprint arXiv:1312.5650_.
* Pan et al. (2010a) Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010a. Cross-domain sentiment classification via spectral feature alignment. In _Proceedings of the 19th international conference on World wide web_, pages 751-760. ACM.
* Pan et al. (2010b) Sinno Jialin Pan, Qiang Yang, et al. 2010b. A survey on transfer learning. _IEEE Transactions on knowledge and data engineering_, 22(10):1345-1359.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Pushp and Srivastava (2017) Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. 2017. Train once, test anywhere: Zero-shot learning for text classification. _arXiv preprint arXiv:1712.05972_.
* Rehurek and Sojka (2010) Radim Rehurek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In _Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks_, pages 45-50, Valletta, Malta. ELRA.
* Romera-Paredes and Torr (2015) Bernardino Romera-Paredes and Philip Torr. 2015. An embarrassingly simple approach to zero-shot learning. In _International Conference on Machine Learning_, pages 2152-2161.
* Saito et al. (2017) Itsumi Saito, Jun Suzuki, Kyosuke Nishida, Kugatsu Sadamitsu, Satoshi Kobashikawa, Ryo Masumura, Yuji Matsumoto, and Junji Tomita. 2017. Improving neural text normalization with data augmentation at character-and morphological levels. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_, volume 2, pages 257-262.
* Seppadla et al. (2016) Prateek Veeranna Sappadla, Jinseek Nam, Eneldo Loza Mencia, and Johannes Furnkranz. 2016. Using semantic similarity for multi-label zero-shot classification of text documents. In _Proc. European Symposium on Artifical Neural Networks, Computational Intelligence and Machine Learning_.
* Shu et al. (2017) Lei Shu, Hu Xu, and Bing Liu. 2017. Doc: Deep open classification of text documents. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2911-2916. Association for Computational Linguistics.
* Volume 1_, NIPS'13, pages 935-943, USA. Curran Associates Inc.
* Speer and Havasi (2013) Robert Speer and Catherine Havasi. 2013. Conceptnet 5: A large semantic network for relational knowledge. In _The People s Web Meets NLP_, pages 161-176. Springer.
* Thrun and Pratt (1998) Sebastian Thrun and Lorien Pratt. 1998. Learning to learn: Introduction and overview. In _Learning to learn_, pages 3-17. Springer.
* Wang et al. (2018) Xiaolong Wang, Yufei Ye, and Abhinav Gupta. 2018. Zero-shot recognition via semantic embeddings and knowledge graphs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 6857-6866.
* Organization (1996) World Health Organization. 1996. Infectious diseases kill over 17 million people a year. _Malaria Weekly_, 3:11-16.
* Xian et al. (2017) Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. 2017. Zero-shot learning-a comprehensive evaluation of the good, the bad and the ugly. _arXiv preprint arXiv:1707.00600_.
* Xie et al. (2017) Ziang Xie, Sida I Wang, Jiwei Li, Daniel Levy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data noising as smoothing in neural network language models. _arXiv preprint arXiv:1703.02573_.
* Zhang and LeCun (2015) Xiang Zhang and Yann LeCun. 2015. Text understanding from scratch. _arXiv preprint arXiv:1502.01710_.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Advances in neural information processing systems_, pages 649-657.
* Zhao et al. (2018) Bo Zhao, Yanwei Fu, Rui Liang, Jiahong Wu, Yonggang Wang, and Yizhou Wang. 2018. A large-scale attribute dataset for zero-shot learning. _arXiv preprint arXiv:1804.04314_.