SlowGuess/ACL_2020_OCR huggingface dataset:
Qi Qin $^{1,2,*}$ , Wenpeng Hu $^{3,*}$ , Bing Liu $^{2,\dagger}$
<sup>1</sup> Center for Data Science, AAIS, Peking University
$^{2}$  Wangxuan Institute of Computer Technology, Peking University
$^{3}$  Department of Information Science, Peking University
{qinqi, wenpeng.hu, dcsliub}@pku.edu.cn
In classification, there are usually some good features that are indicative of class labels. For example, in sentiment classification, words like good and nice are indicative of the positive sentiment and words like bad and terrible are indicative of the negative sentiment. However, there are also many common features (e.g., words) that are not indicative of any specific class (e.g., voice and screen, which are common to both sentiment classes and are not discriminative for classification). Although deep learning has made significant progresses in generating discriminative features through its powerful representation learning, we believe there is still room for improvement. In this paper, we propose a novel angle to further improve this representation learning, i.e., feature projection. This method projects existing features into the orthogonal space of the common features. The resulting projection is thus perpendicular to the common features and more discriminative for classification. We apply this new method to improve CNN, RNN, Transformer, and Bert based text classification and obtain markedly better results.
Text classification is an important task in natural language processing and text mining. It has a very wide range of applications, such as sentiment classification (Liu, 2012), question classification (Li and Roth, 2002), and deception detection (Liu, 2012; Feng et al., 2012). In recent years, deep learning models have been shown to outperform traditional classification methods (Kim, 2014; Iyyer et al., 2015; Tang et al., 2015; Dai and Le, 2015; Jin et al., 2016; Joulin et al., 2017; Shen et al., 2018). Given the input document, the system applies a mapping function (e.g., averaging or summation, a
convolution neural network (CNN), recurrent neural network (RNN), and so on) to learn a dense representation of the document and then uses this representation to perform the final classification. Representation learning is one of the key strengths of deep learning.
In this paper, we propose to further improve the representation learning, i.e., to make the representation more discriminative for classification. Note that throughout the paper we will use sentence sentiment classification as an example to explain different ideas, but in our experiments, non-sentiment classification datasets are also used to show the generality of the proposed method. For text classification, many neural networks and embedding techniques have been devised and applied, e.g., RNN, CNN, Transformer (Vaswani et al., 2017) and Bert (Devlin et al., 2018). For example, RNN can model the whole sentence and also capture the long-term dependencies within the sentence. However, modeling the entire sequence may neglect some key local contexts that are important for classification (Yin et al., 2017). CNN is able to extract more local and position-invariant features (Scherer et al., 2010; Collobert et al., 2011). However, these methods may not give enough weights to some special or discriminative words. To solve this problem, the attention mechanism was introduced. For example, by exploiting attention, Transformer and Bert (which maximizes Transformer's ability to extract sentence semantic information) can achieve even better results than both CNN and RNN on many tasks. We will see some other related methods to produce effective representations in the related work section.
Although the existing models are already able to produce excellent representations, we will show that these representations can still be improved. This paper explores in an entirely different direction, i.e., feature projection. In a typical sentence or
8161
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8161-8171 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
document, there are usually some words or features that are correlated with some class labels, but there are also many other common features that cannot distinguish different classes. For example, in sentiment classification, words like Good and Nice are indicative of the positive sentiment, and words like Bad and Terrible are indicative of the negative sentiment. Words like picture, price, and battery are not indicative of any sentiment, i.e., they are not discriminative. However, they may still interfere the representation learning to produce sub-optimal feature representations for the final classification. Even though the attention mechanism can alleviate this problem to some extent by giving higher weights to words associated with classes and lower weights to the other words that are not indicative of any specific classes. However, due to the idiosyncrasy of the data and the inaccuracy of the attention mechanism, the problem remains.
In this paper, we propose a novel feature projection method to improve feature representation learning to make it more discriminative for classification. The proposed method is called Feature Purification Network (FP-Net). Specifically, FP-Net consists of two sub-networks, a common feature learning network referred to as the C-net and a projection network referred to as the P-net. C-net uses a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2014; Zhang et al., 2019) to extract common features  $\vec{b}$  (i.e., invariant features (Zhang et al., 2019)) that are shared by multiple classes and have little discriminative power for classification. At the same time, P-net uses a traditional feature extractor to learn the feature vector  $\vec{a}$  for the input sentence or document. Then the feature (or representation) vector  $\vec{a}$  is projected onto the vector of the common features  $\vec{b}$  (i.e., vector  $\vec{b}$ ) to get a projection vector  $\vec{c}$ , which represents the input sentence's own common features. Then, we project the feature vector  $\vec{a}$  onto the orthogonal direction of the vector of the common features  $\vec{c}$  to produce the final purer features for classification. It is quite clear and intuitive that this orthogonal project is to get rid of the common features and make the system focusing on those discriminative features only. We will explain why two projections are used in Section 3.
In summary, the key contribution of this paper is the improvement to representation learning through feature vector projection. To the best of our knowledge, this is the first such technique. Specifically,
an Orthogonal Projection Layer (OPL) is proposed to map the features obtained by a traditional feature extractor to the classification-specific semantic space, which is orthogonal to the common features such that we obtain a more relevant and discriminative (or purer) feature representation from the original document for classification.
Extensive experiments have been conducted to verify the effectiveness of the proposed method on two sentence sentiment classification datasets MR and SST2, a natural language inference dataset SNLI, and a question classification dataset TREC. The results show that the proposed method can improve the classification accuracy of RNN, CNN, Transformer and Bert based classification methods markedly, which shows that feature projection is a highly promising direction to explore.
It is well known that one of the key strengths of deep neural networks is their superb ability to learn highly effective representations or features from the raw data, which have been shown to be very successful for all kinds of applications including natural language processing tasks such as text classification (Jin et al., 2016), machine translation (Bahdanau et al., 2014; Vaswani et al., 2017) dialogue (Wang and Jiang, 2016), etc. Previous work on learning representations broadly falls in two main categories: supervised and unsupervised methods. Our work focuses on improving the representation of text for supervised classification.
Supervised methods: These methods improve data utilization efficiency and discriminative feature distillation as they can obtain better training signals from the labeled data. Sequence models such as recurrent neural networks (RNN), Long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) and gated recurrent unit (GRU) (Chung et al., 2014) networks are suitable for handling text because a sentence or document can be regarded as a sequence. Therefore, a large amount of work based on RNN and its variants for feature extraction and downstream tasks has been done (Tang et al., 2015; Wang and Tian, 2016; He et al., 2016). Unlike RNN's sequence modeling approach, CNN (Convolutional Neural Network) uses different sized windows to capture local correlations and position-invariant information (Kim, 2014; Conneau et al., 2016; Lai et al., 2015; Xiao and Cho, 2016; Wang, 2018). A common approach
8162
of these methods is to create an instance-level representation by using the final hidden state of the RNN, the maximum (or average) pooling of the RNN hidden states, or convolutional n-grams. However, they may ignore the importance of special words that are highly discriminative for classification. After Bahdanau et al. (2014) introduced the attention mechanism in machine translation, attention mechanism has been exploited in many natural language processing tasks including text classification to solve the above problem. For example, Yang et al. (2016) introduced attention as an integral part of the model for text classification. Lin et al. (2017) proposed a new model for extracting interpretable sentence embeddings using self-attention. Ma et al. (2018) showed that attention mechanism is also effective for sentiment classification. Vaswani et al. (2017) further illustrated that they can get a stronger sentence-level representation by stacking multiple blocks of self-attention. Bert (Devlin et al., 2018) combines Transformer and a large corpus to produce an even more complete and better sentence-level representation. Some other studies improved the representation of sentences from the perspective of language structures (e.g., parse trees and dependency trees) (Tai et al., 2015; Mou et al., 2015). Subramanian et al. (2018) utilized a single multi-task framework to combine the benefits of diverse sentence representation learning objectives. However, to the best of our knowledge, these existing works and others have not used feature projection to improve (or purify) representations for supervised learning, which we believe is a promising direction to explore.
Unsupervised methods: These methods utilize a large unlabeled text corpus to learn word representations which are then composed into sentence and document representations. For example, Kiros et al. (2015) constructed sentence representations by trying to reconstruct neighbouring sentences. Hill et al. (2016) proposed a log-linear bag-of-words models for sentence representation. The unsupervised smooth inverse frequency method in (Ethayarajh, 2018) built on this but used a weighted average of word embeddings and principal component removal for sentence representations. Our work is again clearly different from these unsupervised methods as the proposed method works under supervised learning. Existing unsupervised methods also do not use feature projection.
Some other works have also been done for semi-
supervised representation learning (Kevin Clark, 2018) and transfer learning (Tamaazousti et al., 2018). Jason Phang (2019) also proposed to use some data-rich intermediate supervised tasks for pre-training to help produce better representation for the end task. To the best of our knowledge, all these previous studies tried to improve representations using external data or knowledge, which are quite different from our method as we don't use any external information. Also, the philosophy of our approach is entirely different as we try to eliminate commonalities among classes through feature projection, which is orthogonal to existing representation learning approaches.
Finally, our work is related to several other works. Ganin and Lempitsky (2014) introduced the gradient reverse layer (GRL) for extracting common features in the context of domain adaptation. It embeds domain adaptation into the process of learning representations so that the final classification decision has more discriminative and invariant characteristics for domain changes. We also use GRL to extract irrelevant or common features. However, we do not work on domain adaptation and they do not use feature projection. Belinkov et al. (2019) used adversarial learning to encourage models to learn representations free of hypothesis-only biases in the SNLI dataset. Zhang et al. (2019) combined GRL and aspect attention to study cross-domain sentiment classification. They found common features across domains and then extracted information from the aspects (which are product features) with the help of common features to do classifications. Our work is clearly different because none of these existing works improve representation learning through feature projection.
3 Feature Purification Network
The overall framework of our model is shown in Figure 1. The whole model consists of two parts, the first part is the projection network (i.e., P-net) and the other is the common feature learning network (i.e., C-net). As mentioned earlier, the goal of C-net is to extract common features and the goal of P-net is to compute the purified features for classification, which is done by projecting the learned full information vector of the input document into a more discriminative semantic space to eliminate the influence of the common features.
P-net consists of four parts: the input layer  $X$ , the feature extractor  $F_{p}$ , Orthogonal Projection
8163
Figure 1: The architecture of FP-Net
Layer (OPL), and the final classification layer  $C_p$ . C-net is also composed of four parts: the input layer  $X$ , the feature extractor  $F_c$  ( $F_p$  and  $F_c$ 's parameters are not shared)<sup>1</sup>, the Gradient Reverse Layer (GRL) and the classification layer  $C_c$ . The key idea of the proposed technique is as follows: The feature vector  $f_p$  computed by the feature extractor  $F_p$  is projected to the orthogonal direction of the feature vector  $f_c$  extracted by  $F_c$  of the C-net. That is,  $f_p$  (the full information extracted from the input document) is projected to the discriminative semantic space to be purified for the final classification. However, in order to perform the orthogonal projection, two operations are required, which we will explain shortly. Next we use CNN as an example feature extractor to detail each component of the proposed FP-Net.
CNN Extractor: Given a dataset  $D = \{(x_{i},y_{i})\}_{i = 1}^{N}$ , where  $x_{i}$  is an input document with the length  $L$  (after padding or cut) and  $y_{i}$  is the label corresponding to the sample  $x_{i}$ . Let  $V_{ij}\in \mathbf{R}^k$  be the word vector corresponding to the  $j$ th word of the document  $x_{i}$ .  $X_{i}\in \mathbf{R}^{L\times k}$  is the embedding matrix of  $x_{i}$ . Recall our FP-Net model consists of two sub-networks, i.e., P-net and C-net, with the same input  $x_{i}$ . The two sub-networks also have the same structure for the feature extractor CNN, but there are no shared parameters between them. The feature extractors of P-net and C-net are  $F_{p}$ ,  $F_{c}$ .
We use  $F_{c}$  as an example to introduce the working of CNN. When the feature extractor  $F_{c}$  receives  $X_{i}$  from the input layer,  $F_{c}$  extracts the advanced features  $f_{c}$  from  $X_{i}$  in the form of n-grams, which is:
$$
f _ {c} = \left[ c _ {1}, c _ {2}, \dots , c _ {l - n + 1} \right] = \left[ c _ {j} \right] _ {j = 1} ^ {l - n + 1}, \tag {1}
$$
where  $c_{j}$  represents the output produced by CNN's filter on  $X_{i}[j:j + n - 1,:]$ . Mathematically, a convolution operation consists of a filter  $W\in \mathbf{R}^{n\times k}$  and a bias  $b\in \mathbf{R}$ . Then  $c_{j}$  can be expressed as:
$$
c _ {j} = g \left(W \cdot X _ {i} [ j: j + n - 1,: ] + b\right), \tag {2}
$$
where  $g$  is a nonlinear activation function such as Relu. We use a Maxpooling operation over the feature map and take the maximum value  $f_{c} = \max \{f_{c}\}$  as the feature corresponding to this particular filter. The same feature extractor  $F_{p}$  will also get the advanced features  $f_{p}$  from the input layer. We refer to the features of the P-net and C-net respectively as
$$
f _ {p} = \operatorname {C N N} _ {\mathbf {p}} (X), \tag {3}
$$
$$
f _ {c} = \operatorname {C N N} _ {\mathbb {C}} (X). \tag {4}
$$
Other details of C-net will be introduced in C-net Module, and likewise, additional details about P-net will be introduced in P-net Module.
C-net Module: The goal of C-net is to extract the common features, which are the semantic information of the input example that is not discriminative for the classification task. As mentioned earlier, common features are those shared by all classes of the problem. The classifier  $C_c$  should not use them to distinguish different classes. To obtain common features, we add a Gradient Reverse Layer (GRL) (Ganin and Lempitsky, 2014; Ganin et al., 2016) after the feature extractor  $F_c$  to reverse the gradient direction. Through this training module, we can obtain the common features that are shared among classes.
Without loss of generality, we can think of the gradient reverse layer as a "pseudo-function" defined by two incompatible equations describing its forward and back-propagation behaviors:
$$
G R L (x) = x, \tag {5}
$$
$$
\frac {\partial G R L (x)}{\partial x} = - \lambda I, \tag {6}
$$
<sup>1</sup>The feature extractor can be any existing extractor. In this work, we verified the effectiveness of our purification network using CNN, RNN, Transformer, and Bert as feature extractors as we will see in the experiment section.
8164
Figure 2: Working of the Orthogonal Projection Layer. The example here is in a 2-dimensional space.  $f_{p}$  represents the traditional feature vector;  $f_{c}$  represents the common feature vector;  $f_{p*}$  is the projected feature vector;  $\widetilde{f}_{p}$  is our final Orthogonal Projection feature vector.
where  $\lambda$  is a hyper-parameter. We process the feature vector  $f_{c}$  through GRL as  $GRL(f_{c}) = \widetilde{f}_{c}$ , which is then fed to the classifier  $C_c$ :
$$
Y _ {G R L} = \operatorname {s o f t m a x} \left(\widetilde {f} _ {c} \cdot W _ {c} + b _ {c}\right), \tag {7}
$$
$$
L o s s _ {c} = \operatorname {C r o s s E n t r o p y} (Y _ {t r u t h}, Y _ {G R L}), \tag {8}
$$
where  $W_{c}$  and  $b_{c}$  are the weights and bias of  $C_{c}$  respectively. By optimizing the objective function  $Loss_{c}$ , the feature extractor  $F_{c}$  is able to extract the common features of different classes.
P-net Module: The goal of P-net is to first extract the full semantic information from the input example and then project it into the semantic space purified for classification. In order to achieve this, we perform the projection of the feature  $f_{p}$  extracted by the feature extractor  $F_{p}$  onto the orthogonal direction of the common feature  $f_{c}$ , extracted by  $F_{c}$ . The feature space orthogonal to the common feature vector should contain features that are pure and highly effective for classification (e.g., sentiment related information in sentiment classification). Projecting the traditional feature vector  $f_{p}$  to this orthogonal feature space preserves the discriminative information and remove those common features of the classes that are unhelpful and even confusing to the classification task.
The Orthogonal Projection Layer (OPL) helps us accomplish this goal. Figure 2 illustrates the idea of OPL using a two-dimensional space example. Mathematically, we first project the tradition feature vector  $f_{p}$  onto the common feature vector  $f_{c}$ :
$$
f _ {p} * = \operatorname {P r o j} \left(f _ {p}, f _ {c}\right), \tag {9}
$$
where Proj is a projection function.
$$
\operatorname {P r o j} (x, y) = \frac {x \cdot y}{| y |} \frac {y}{| y |}, \tag {10}
$$
where  $x, y$  are vectors. We then do the projection in the orthogonal direction of the projected feature
$f_{p}$  to get the purer classification feature vector:
$$
\widetilde {f _ {p}} = \operatorname {P r o j} \left(f _ {p}, \left(f _ {p} - f _ {p} *\right)\right). \tag {11}
$$
Clearly, it is easy to show that the feature vector  $\widetilde{f_p}$  obtained by Eq. 11 is equivalent to  $f_{p} - f_{p*}$ . Using the traditional feature vector  $f_{p}$  and the projected feature vector  $f_{p*}$ , we can build a plane (in three dimensions). The intersection of this plane and the orthogonal plane of the projected feature vector  $f_{p*}$  is our pure feature vector. In other words, the projection in Eq. 9 is a constraint on the common feature vector. That is to say: the modulus of the common feature vector is limited by projecting the traditional feature vector of the input  $x_{i}$  to the common feature vector, so the semantic information of the new common feature vector (i.e., the projected feature  $f_{p*}$ ) contains only the common semantic information in  $x_{i}$ . This makes the final purified feature vector  $\widetilde{f_p}$  coming from the traditional feature vector  $f_{p}$  rather than any vector in any plane orthogonal to the common feature vector  $f_{c}$ . Finally, we use the purified feature vector  $\widetilde{f_p}$  to do the classification.
$$
Y _ {O P L} = \operatorname {s o f t m a x} \left(\widetilde {f _ {p}} \cdot W _ {p} + b _ {p}\right), \tag {12}
$$
$$
L o s s _ {p} = \text {C r o s s E n t r o p y} \left(Y _ {\text {t r u t h}}, Y _ {\text {O P L}}\right). \tag {13}
$$
Note that here  $Loss_{p}$  and  $Loss_{c}$  are trained simultaneously, and they use different optimizers.  $Loss_{p}$  uses the Adam optimizer. Since Ganin and Lempitsky (2014) used Moment SGD as the domain classifier's optimizer, our C-net loss function  $Loss_{c}$  also uses Moment SGD optimizer. Gradients are also passed back through feature  $f_{c}$  when optimizing  $Loss_{p}$ . Although the two losses are opposite to each other in terms of optimization targets of the feature extractor  $F_{c}$ , the effect of  $Loss_{p}$  on  $F_{c}$  is in the orthogonal direction of  $f_{c}$ . A balance will be found to make the extracted feature  $f_{c}$  closer to the real common features. The complete training algorithm of the proposed FP-Net is given in Algorithm 1, which is self-explanatory.
We now evaluate the proposed FP-Net  ${}^{3}$  using four text classification datasets and compare it with baselines without the purification capability. Our goal is
2We have conducted experiments using the Adam optimizer for both C-Net and P-Net. The results are about the same as using two different optimiers.
<sup>3</sup>https://github.com/Qqinmaster/FP-Net/
8165
2: for each iteration  $b = 1, 2, \dots, M$  do
3: Sample one batch  $X_{b}$  from  $D$
4: C-net part:
5: Generate common features (CFs) (Eq. 3)
6: CFs go through GRL (Eq. 5)
7: Perform classification (Eq. 7)
8: P-net part:
9: Generate traditional features (TFs) (Eq. 4)
10: TFs projection (Eq. 9)
11: Get the purified features (Eq. 11)
12: Perform classification (Eq. 12)
13: Update parameters:
14: C-net, P-net's parameters are updated together (Eq. 8 & Eq. 13)
15: end for
to verify whether the proposed feature purification is general and effective for different deep learning classification models (or more precisely, feature extractors) on diverse datasets.
4.1 Experimental Datasets
We carried out experiments on four diverse benchmark datasets:
MR: This is a movie review dataset for sentiment classification. It has two classes: positive and negative (Pang and Lee, 2005).<sup>4</sup>
SST2: This is the Stanford Sentiment Treebank dataset. Each sample is marked as negative or positive.
TREC: This is a question classification dataset, which is to classify a question into one of the six question types (Li and Roth, 2002).<sup>6</sup>