SlowGuess/ACL_2020_OCR huggingface dataset:
Zhiquan Ye $^{1,2}$ , Yuxia Geng $^{1,2}$ , Jiaoyan Chen $^{4}$ , Xiaoxiao Xu $^{3}$ , Suhang Zheng $^{3}$ , Feng Wang $^{3}$ , Jingmin Chen $^{3}$ , Jun Zhang $^{3}$ , Huajun Chen $^{\dagger,2}$ $^{1}$ College of Computer Science, Zhejiang University  
 $^{2}$ AZFT Joint Lab of Knowledge Engine  $^{3}$ Alibaba Group  
 $^{4}$ Department of Computer Science, Oxford University  
{yezq, gengyx, huajunsir}@zju.edu.cn  
{jiaoyan.chen}@cs.ox.ac.uk, {xiaoxiao.xuxx}@alibaba-inc.com  
{suhang.zhengsh, wf135777, jingmin.cjm, zj157077}@alibaba-inc.com
Zero-shot learning has been a tough problem since no labeled data is available for unseen classes during training, especially for classes with low similarity. In this situation, transferring from seen classes to unseen classes is extremely hard. To tackle this problem, in this paper we propose a self-training based method to efficiently leverage unlabeled data. Traditional self-training methods use fixed heuristics to select instances from unlabeled data, whose performance varies among different datasets. We propose a reinforcement learning framework to learn data selection strategy automatically and provide more reliable selection. Experimental results on both benchmarks and a real-world e-commerce dataset show that our approach significantly outperforms previous methods in zero-shot text classification.
Zero-shot learning (ZSL) is a challenging task as no labeled data is available for unseen classes during training. There are extensive works proposed in zero-shot image classification task. The main focus of these works is how to transfer knowledge from seen classes to unseen classes. To associate unseen classes with seen classes, they usually resort to semantic information such as visual attributes (Lampert et al., 2009), word embeddings of class names (Norouzi et al., 2013) and class hierarchy (Socher et al., 2013). For example, if the model has not seen any instances of "humpback whale" in the training stage, it could still make predictions at testing stage since "humpback whale" is semantically close to "killer whale" and "blue whale" in the seen class set *, so the model is capable of transferring knowledge from seen
classes to unseen classes. These methods assume that semantically similar classes share similar image features, however, they may fail in the cases where classes share low similarities.
This problem becomes even more salient in typical NLP tasks such as text classification. For example, let us consider a 10-class emotion classification task (Yin et al., 2019), in which the model is trained on class "sadness" while makes predictions on instances from class "joy". Notice that most emotions are relatively independent, which means the way we express certain emotion is pretty different from other emotions. As a result, for an unseen class we can hardly find a similar class in the seen class set. Transferring from seen classes to unseen classes can be extremely hard as matching patterns that can be shared among classes are rare.
Essentially, ZSL methods aim to learn a matching model between feature space and semantic space, which refers to text and label in text classification task respectively. Matching patterns between text and label can be roughly classified as class-invariant patterns and class-specific ones. The former refers to the patterns that are shared among classes, while the latter is dependent on a certain class. Table 1 shows an example to illustrate this definition. The string match of label and text, which is highlighted with red color, indicates a simple matching pattern that can be shared among classes. On the contrary, the words that are highlighted with blue color indicate a matching pattern that is specific to a certain class and cannot be transferred among classes easily. Imagine if the model is trained on sentence 1, it can make a correct prediction on sentence 2 while failing on sentence 3 probably.
There are mainly two ways to deal with this troublesome zero-shot learning situation, including (1) integrating more external knowledge to
$^{\dagger}$ Corresponding Author.  
*This example is taken from awa2 dataset, https://cvml.est.ac.at/AwA2/.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3014-3024 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Label	Sentence
fear	1. One day, when I realized that I was alone, I felt fear of loneliness.
guilty	2. I felt guilty when I lied to my parents.
guilty	3. I wished secretly and lied to a friend because I didn’t want her to stay in my house.
Table 1: Illustration of class-invariant and class-specific matching pattern.
better describe class and build more sophisticated connections between classes (Rios and Kavuluru, 2018; Zhang et al., 2019); (2) integrating the unlabeled data to improve the generalization performance. Generally, existing works mainly adopt the former solution, while little attention is paid to the latter one. In this paper, we focus on the latter one and propose a self-training based method to leverage unlabeled data. The basic idea of self-training (McClosky et al., 2006; Sagae, 2010) is to select unlabeled instances that are predicted with high confidence and add them into the training set. It is straightforward to consider that if we add sentence 2 to training set, the model is capable of learning class-specific pattern as sentence 2 and sentence 3 share the intra-class similarity. In this way, we can mine class-specific feature through class-invariant feature.
However, directly applying traditional self-training method to zero-shot learning may encounter some problems: (1) traditional self-training methods use manually designed heuristics to select data, so manual adjustment of selection strategy is costly (Chen et al., 2018). (2) due to the severe domain shift (Fu et al., 2015), traditional self-training method may not provide reliable selection. To alleviate these problems, we present a reinforcement learning framework to learn data selection policy, which can select unlabeled data automatically and provide more reliable selection.
- We propose a self-training based method to leverage unlabeled data in zero-shot text classification. Our method is capable of alleviating the domain shift problem and enabling transferring between classes sharing low similarities and connections.
- We propose a reinforcement learning framework to learn data selection policy automatically instead of using manually designed heuristics.
- Experimental results on both benchmarks and a real-world e-commerce dataset show that our method outperforms previous methods with a large margin of  $15.4\%$  and  $5.4\%$  on average in generalized and non-generalized ZSL respectively.
2.1 Zero-shot Learning
Zero-shot learning has been widely studied in image classification, in which training classes and testing classes are disjoint (Lampert et al., 2013; Larochelle et al., 2008; Rohrbach et al., 2011). The general idea of zero-shot learning is to transfer knowledge from seen classes to unseen classes (Wang et al., 2019). Most methods focus on learning a matching model between image feature space and class semantic space, such as visual attributes (Lampert et al., 2009), word embeddings of class names (Socher et al., 2013), class hierarchy (Socher et al., 2013).
For zero-shot text classification, similar methods have been adopted. (Dauphin et al., 2013) associated text with class label through semantic space, which is learned by deep neural networks trained on large amounts of search engine query log data. (Nam et al., 2016) proposed an approach to embed text and label into joint space while sharing word representations between text and label. (Pushp and Srivastava, 2017) proposed three neural networks to learn the relationship between text and tags, which are trained on a large text corpus. (Rios and Kavuluru, 2018) incorporated word embeddings and hierarchical class structure using GCN (Kipf and Welling, 2016) for multi-label zero-shot medical records classification. (Zhang et al., 2019) proposed a two-phase framework together with data augmentation and feature augmentation, in which four kinds of semantic knowledge (word embeddings, class descriptions, class hierarchy, and knowledge graph) were incorporated.
These works benefit from large training corpus
and external semantic knowledge, however, none of these works have tried to leverage unlabeled unseen data in zero-shot text classification, namely transductive zero-shot learning (Xian et al., 2018). There exists some work to utilize unlabeled data in image classification to alleviate domain shift problem, including (Fu et al., 2012; Rohrbach et al., 2013; Li et al., 2015; Fu et al., 2015), etc. As far as we know, our work is the first to explore transductive zero-shot learning in text classification.
2.2 Self-training
Self-training is a widely used algorithm in semi-supervised learning (Triguero et al., 2015). The basic process of self-training is to iteratively select high-confidence data from unlabeled data and add these pseudo-labeled data to training set. Self-training has shown its effectiveness for various natural language processing tasks, including text classification (Drury et al., 2011; Van Asch and Daelemans, 2016), name entity recognition (Kozareva et al., 2005), parsing (McClosky et al., 2006, 2008; Huang and Harper, 2009). However, there are two main drawbacks of self-training. Firstly, its data selection strategy is simply confidence-based, which may not provide reliable selection (Chen et al., 2011) and cause error accumulation. Secondly, self-training relies on pre-defined confidence threshold which varies among datasets and manual adjustment is costly.
2.3 Reinforcement Learning for Data Selection
There have been some works applying reinforcement learning to data selection in semi-supervised learning, including active learning (Fang et al., 2017), self-training (Chen et al., 2018), co-training (Wu et al., 2018). These works share a similar framework which uses deep Q-Network (Mnih et al., 2015) to learn a data selection strategy guided by performance change of model. This process is time-consuming as the reward is immediate which means the classifier is retrained and evaluated after each instance is selected. Reinforcement learning has also been applied in relation extraction to alleviate the noisy label problem caused by distant supervision. (Feng et al., 2018; Qin et al., 2018) proposed a policy network to automatically identify wrongly-labeled instances in training set. Earlier, (Fan et al., 2017) proposed an adaptive data selection strategy, enabling to dynamically choose different data at different train
Figure 1: Illustration of the traditional classifier and standard ZSL model.
ing stages.
3.1 Problem Formulation and Overview
Here we first formalize the zero-shot text classification problem. Let  $\mathcal{Y}^s$  and  $\mathcal{Y}^u$  denote seen and unseen class set respectively, where  $\mathcal{Y}^s \cap \mathcal{Y}^u = \emptyset$ ,  $\mathcal{Y}^s \cup \mathcal{Y}^u = \mathcal{Y}$ . Suppose there is  $\mathcal{D}^s = \{(x_i^s, y_i^s)\}_{i=1}^N$  for seen classes and  $\mathcal{D}^u = \{x_i^u, y_i^u\}_{i=1}^M$  for unseen classes, where  $x_i$  represents  $i$ -th text and  $y_i$  represents the corresponding label. As shown in Figure 1, ZSL method turns a classification problem into a matching problem between text and class label. During training, we learn a matching model  $f(x, y; \theta)$  from seen classes  $\mathcal{D}^s$  and then make predictions on unseen classes:
$$
\hat {y} = \underset {y \in \mathcal {Y}} {\arg \max } f (x, y; \theta), \tag {1}
$$
where  $\theta$  refers to the parameter of  $f$ . For transductive ZSL, both labeled seen data  $\mathcal{D}^s$  and unlabeled unseen data  $\mathcal{D}^u = \{x_i^u\}_{i=1}^M$  are available during training.
To tackle zero-shot text classification, a reinforced self-training framework is developed in this work. Figure 2 shows an overview of our reinforced self-training framework for zero-shot text classification. The goal of our framework is to select high quality data from unseen classes automatically by agent and use these data to augment the performance of the base matching model. Specifically, we first train the base matching model on seen class data and make predictions on unseen class data. To make it more efficient, the agent performs data selection from a subset of unlabeled data instead of all unlabeled data at each iteration. We rank the instances by prediction confidence and take a certain ratio of instances from
Figure 2: Overview of our reinforced self-training framework for zero-shot text classification.
it at each iteration. The agent is responsible for selecting data from this subset and filter negative instances. The reward is determined by the performance of matching model in validation set. We will introduce the details of our method in the following subsections.
3.2 The Base Matching Model
Our RL-based data selection framework is model-agnostic, which means any matching model is compatible. Here we adopt the widely recognized pre-trained model BERT (Devlin et al., 2018) as the base matching model. For seen classes, given text  $x$  and label  $y$ , we generate  $\{(x,y') | y' \in \mathcal{Y}^s\}$  as training instances, in which  $(x,y')$  is a positive training instance if  $y' = y$ . We take the text as premise and transform the label into its corresponding hypothesis provided in (Yin et al., 2019). Therefore, the input sequence of BERT is packed as "[CLS]  $x$  [SEP] hypothesis of  $y'$  [SEP], where [CLS] and [SEP] are special start and separator tokens, as shown in Figure 3. BERT encoder is composed of multi-layer bidirectional transformers (Vaswani et al., 2017). We use the hidden vector  $c_{x,y'} \in \mathbb{R}^H$  corresponding to [CLS] in the final layer as the aggregate representation. We add a linear layer and compute loss as below:
$$
p _ {x, y ^ {\prime}} = \sigma \left(W ^ {T} c _ {x, y ^ {\prime}} + b\right), \tag {2}
$$
$$
\mathcal {L} = \left\{ \begin{array}{l l} - l o g (p _ {x, y ^ {\prime}}) & y ^ {\prime} = y \\ - l o g (1 - p _ {x, y ^ {\prime}}) & y ^ {\prime} \neq y \end{array} \right., \tag {3}
$$
where  $W$  and  $b$  are parameters of the linear layer,  $W\in \mathbb{R}^{H}$ $b\in \mathbb{R}$ $H$  is the hidden dimension size, and  $p_{x,y^{\prime}}$  indicates the matching score between  $x$  and  $y^\prime$ $\sigma (\cdot)$  is sigmoid function.
3.3 Reinforcement Learning for Self-training
The conventional self-training method simply selects data predicted with high confidence, which
Figure 3: BERT as the base matching model.
is confidence-based. We formalize the data selection as a sequential decision-making process and introduce a RL framework to combine confidence-based strategy and performance-driven strategy. We describe the whole process in Algorithm 1. The details of the RL modules are described below.
3.3.1 State
For each text  $x$ , we get prediction scores  $\{p_{x,y'}|y' \in \mathcal{Y}^u\}$ . The label  $y^*$  with maximum matching score is considered as the pseudo label. For time step  $t$ , the current state  $s_t$  consists of 2 parts: the prediction confidence  $p_{x,y^*}$ , the representation of arriving instance  $c_{x,y^*}$ . We take the hidden vector corresponding to [CLS] as the representation of current instance  $(x,y^*)$ . The policy network takes  $p_{x,y^*}$  and  $c_{x,y^*}$  as input and outputs the probability whether to select or not.
3.3.2 Action
At each step, the agent is required to take action for the current instance  $(x,y^{*})$  - whether to select it or not. At time step  $t$ ,  $a_{t} = 1$  means the agent accepts the current instance and adds it to training set;  $a_{t} = 0$  means rejection. The action value is obtained through sampling from the policy network's output  $P(a|s_t)$ .
3.3.3 Reward
If wrongly-labeled instances are added into training set, it will degrade the performance of the matching model. Therefore the function of reward is to guide the agent to select the instances that are consistent with training set. The reward is determined by the performance of the matching model on validation set, which consists of 2 parts: seen validation set  $\mathcal{D}_{dev}^{s}$  and unseen validation set  $\mathcal{D}_{dev}^{u}$ .  $\mathcal{D}_{dev}^{u}$  comes from the pseudo labeled data, which guides newly-selected data to be consistent with previously-selected data. More specifically, after each batch of selection, we train the matching model using the selected instances,
and evaluate on validation set. We use macro-F1 as the evaluation metric. Assume there are  $N_{3}$  batches in one episode, we get two  $F$  sequences  $F^{s} = \{F_{1}^{s}, F_{2}^{s}, \dots, F_{N_{3}}^{s}\}$  for seen validation set and  $F^{u} = \{F_{1}^{u}, F_{2}^{u}, \dots, F_{N_{3}}^{u}\}$  for unseen validation set. For batch  $k$ , the reward is formulated as:
$$
r _ {k} = \frac {\left(F _ {k} ^ {s} - \mu^ {s}\right)}{\sigma^ {s}} + \lambda \cdot \frac {\left(F _ {k} ^ {u} - \mu^ {u}\right)}{\sigma^ {u}}, \tag {4}
$$
where  $\lambda$  controls the weight of seen class and unseen class,  $\mu$  and  $\sigma$  represent the mean and standard deviation of  $F$ , respectively.
3.3.4 Policy Network
We adopt a multi-layer perceptron (MLP) as the policy network. The policy network receives states: the prediction confidence  $p_{x,y^*}$  and the representation of arriving instance  $c_{x,y^*}$ , then output the probability for each action.
$$
z _ {t} = \operatorname {R e L U} \left(W _ {1} ^ {T} c _ {x, y ^ {*}} + W _ {2} ^ {T} p _ {x, y ^ {*}} + b _ {1}\right), \tag {5}
$$
$$
P (a \mid s _ {t}) = \operatorname {s o f t m a x} \left(W _ {3} ^ {T} z _ {t} + b _ {2}\right). \tag {6}
$$
We use ReLU as the activation function,  $W_{1}, W_{2}, W_{3}, b_{1}, b_{2}$  are the parameters of MLP, and  $P(a|s_t)$  is the probability of actions.
3.3.5 Optimization
To learn an optimal data selection policy, we aim to maximize the expected total reward, which can be formulated as:
$$
J (\phi) = E _ {P _ {\phi} (a \mid s)} [ R (s, a) ], \tag {7}
$$
where  $R(s,a)$  is the state-action value function and  $\phi$  is the parameter of policy network. We update the  $\phi$  via policy gradient (Sutton et al., 2000),
$$
\phi \leftarrow \phi + \eta \nabla_ {\phi} \tilde {J} (\phi), \tag {8}
$$
where  $\eta$  is the discount learning rate. For a batch  $B_{k}$ , we sample an action  $a_{t}$  for each state  $s_t$  according to policy  $P_{\phi}(a|s)$ . After one episode, we compute rewards  $\{r_k\}_{k=1}^{N_3}$  by Equation 4. The gradient can be approximated by
$$
\nabla_ {\phi} \tilde {J} (\phi) = \frac {r _ {k}}{| B _ {k} |} \sum_ {t = 1} ^ {| B _ {k} |} \nabla_ {\phi} \log P (a _ {t} | s _ {t}), \tag {9}
$$
where  $|B_k|$  is the number of instances in one batch,  $r_k$  is the reward of batch  $B_k$ , the parameter of policy network is updated after each episode.
We use two kinds of datasets for our experiments. The first comes from the recently released benchmarks for zero-shot text classification (Yin et al., 2019), including 3 datasets: topic, emotion and situation classification. Considering that some texts in situation dataset has multiple labels, we remove texts with multiple labels and keep single-label texts. To keep consistent with Equation 1, "none" type is not included in unseen classes. Datasets are prepared with two versions of partitions with non
Seen class	Unseen class
#Train	#Valid	#Test
Topic	I	650000	5000	50000
II	650000	5000	50000
Emotion	I	20465	2405	5101
II	14204	1419	8901
Situation	I	2428	240	689
II	1747	173	1102
E-commerce	I	9000	1000	5000
II	9000	1000	5000
Table 2: Statistics of text classification Datasets, where I and II refer to two ways of partitions respectively described in (Yin et al., 2019).
overlapping labels so as to get rid of the models over-fitting on one of them.
To further evaluate our method in real-world scenario, we construct a new dataset from e-commerce platform, where texts consist of user search queries. For seen classes  $\mathcal{Y}^s$ , it consists of the categories of product that users click on after searching. For unseen classes  $\mathcal{Y}^u$ , it consists of the pre-defined user preference classes. User preference refers to the product's attribute that users prefer, such as the efficacy of cosmetic products, the style of furniture. The user preference and product category are disjoint so it can be formalized as a zero-shot learning problem. We annotate 10-class user preference dataset for evaluation and there is 1000 instances for each class. Following (Yin et al., 2019), we created two versions of unseen classes each with 5 classes that do not overlap. The statistics of datasets are shown in Table 2.
We use the BERT-Base (Devlin et al., 2018) as our base matching model, with 12-layer transformer blocks, 768-dimension hidden state, 12 attention heads and total 110M parameters. We use the pre-trained BERT-Base-Uncased* for the English benchmarks and BERT-Base-Chinese† for e-commerce dataset. For training stage, we use Adam (Kingma and Ba, 2014) for fine-tuning with  $\beta_{1}$  as 0.9,  $\beta_{2}$  as 0.999. The max sequence length of BERT input is set to 64. For other hyperparameters, we set learning rate as 5e-5, ratio  $\delta = size(\Omega) / M$  as 0.2, iteration number  $N_{1}$  as 5 and episode number  $N_{2}$  as 20. We select weight  $\lambda$
among  $\{1,2,5,10\}$ . For baselines, we adopt 300-dim GloVe vectors (Pennington et al., 2014) for English words and 300-dim word vectors from (Li et al., 2018) for Chinese words.
Policy network pre-train is widely used by reinforcement learning based methods to accelerate the training of RL agent (Silver et al., 2016; Xiong et al., 2017; Qin et al., 2018). We use seen class data to pre-train the agent, enabling the agent to distinguish negative instances. We set early stop criteria to avoid overfitting to seen class data.
We compare our method with the following baselines: (1) Word2vec measures how well a label matches the text by computing cosine similarity of their representations. Both the representations of text and labels are average of word embeddings. (2) Label similarity (Veeranna et al.) uses word embeddings to compute semantic similarity as well, which computes the cosine similarity between class label and every n-gram  $(n = 1,2,3)$  of the text, and takes the max similarity as final matching score; (3) FC and RNN+FC refers to the architecture 1 and architecture 2 proposed in (Pushp and Srivastava, 2017).
We also compare multiple variants of our models: (1) BERT refers to the base matching model without self-training and RL; (2) BERT+self-training refers to the traditional self-training method, which selects instances with high confidence. However, confidence threshold has great impact on performance. With different thresholds, the number of selected instances differs, resulting in performance change of the model. To provide a fair comparison, we record the number of instances  $k$  selected in every iteration in RL selection process. For self-training, we select top  $k$  instances for every iteration. (3) BERT+RL refers to full model of our methods.
We use macro-F1 as evaluation metric in our experiments since datasets are not well balanced. We report the results in two ZSL setting: generalized and non-generalized. In non-generalized ZSL, at test time we aim to assign an instance to unseen class label  $(\mathcal{Y}^u)$ . While in generalized ZSL, class label comes from both unseen and seen classes  $(\mathcal{Y}^s \cup \mathcal{Y}^u)$ . The harsh policy in testing (Yin et al., 2019) is not adopted in our experiments.
\*https://storage.googleapis.com/bert_models/2018_10_18 /uncased_L-12_H-768_A-12.zip  $\dagger$  https://storage.googleapis.com/bert_models/2018_11_03 /chinese_L-12_H-768_A-12.zip
Topic	Emotion	Situation	E-commerce
I	II	I	II	I	II	I	II
Word2vec	35.50	35.33	4.77	11.45	40.67	36.33	53.09	55.47
Label similarity	34.62	36.14	10.63	16.89	54.56	37.45	59.04	55.89
FC	19.45	22.46	27.36	8.31	24.33	25.01	26.40	22.45
RNN+FC	9.68	13.41	15.45	3.15	15.58	14.09	25.76	18.15
BERT	57.07	45.50	16.86	10.21	60.23	34.15	58.05	66.47
BERT+self-training	72.21	62.90	31.96	19.72	69.00	49.30	65.14	76.72
BERT+RL	73.41	65.53	36.98	19.38	73.14	52.44	70.63	80.32