SlowGuess/ACL_2020_OCR huggingface dataset:
SNLI: This is a popular text entailment dataset. It contains 570k human annotated sentence pairs, in which the premises are drawn from the captions of the Flickr 30 corpus and hypotheses are manually annotated (Bowman et al., 2015). For this SNLI dataset, we created the following settings to suit our needs: (1) we concatenated the two sentences (in a pair) as a single sample; (2) when using
Data	c	l	Train	Test	|V|
MR	2	45	8,529	1,066	17,884
SNLI	3	40	54,936	9,824	33,944
SST2	2	35	6,920	1,821	16,789
TREC	6	15	5,000	952	8,834
Table 1: Dataset statistics.  $c$ : number of classes.  $l$ : average length of sentences, after padding and cutting. Train, Test: number of training and testing examples.  $|V|$ : vocabulary size.
Bert as a feature extractor, we reduced the number of training set samples to 25,000 to speed up the training process. For other feature extractors (see below), the complete data is used.
The dataset statistics are given in Table 1.
Since our goal is to perform feature purification so that the purified features are more conducive for classification, to verify the validity of the proposed FP-Net model, we compare the classification results with and without purification using the following popular feature extractors:
LSTM: The long short-term memory network (LSTM) (Hochreiter and Schmidhuber, 1997) for solving the gradient disappearing problem of the traditional RNN.
CNN: We use the Convolution Neural Networks in (Kim, 2014) as the feature extractor to generate representations.
Transformer: We use the encoder part of the model proposed by (Vaswani et al., 2017) as the feature extractor, followed by a classifier.
Bert: We fine-tuned on the trained Bert base (Devlin et al., 2018). Bert base includes 12-layer, 768-hidden, 12-heads and 110M parameters. In particular, we use Bert-base Uncased, where Uncased means that the text has been lower cased before WordPiece tokenization.
Note, those existing feature learning or feature enhancement approaches discussed in Section 2 are not compared as they are entirely different from our approach. They mainly relied on external data or information to improve representation learning. Our method does not use any external data or information. However, we do include Bert as a baseline as it is perhaps one of the most successful feature learning methods using external data. Our method can improve on top of Bert.
$^{4}$ http://www.cs.cornell.edu/people/pabo/movie-review-data/ $^{5}$ http://nlp.stanford.edu/sentiment/ $^{6}$ http://cogcomp.cs.illinois.edu/Data/QA/QC/
8166
Model	MR	SNLI	SST2	TREC
LSTM	77.46(±0.41)	76.98(±0.07)	80.41(±0.20)	87.19(±0.58)
FP+LSTM	78.13(±0.18)	77.92(±0.10)	81.60(±0.17)	88.83(±0.40)
CNN	76.18(±0.45)	72.92(±0.19)	80.47(±0.59)	90.86(±0.51)
FP+CNN	78.74(±0.36)	74.38(±0.14)	82.02(±0.11)	92.78(±0.26)
Trans	75.18(±0.57)	66.71(±0.58)	76.93(±0.39)	87.33(±0.23)
FP+Trans	76.83(±0.66)	73.34(±0.43)	78.42(±0.49)	89.51(±0.79)
Bert	87.45(±0.51)	80.78(±0.42)	90.38(±0.10)	96.67(±0.22)
FP+Bert	90.56(±0.35)	81.47(±0.26)	92.24(±0.29)	98.33(±0.24)
Table 2: Results of our FP-Net against baseline methods. In each block,  $\mathrm{FP + X}$  is a model obtained by our FP-Net using X as the feature extractor. Accuracy  $(\%)$  is the evaluation metric. Each result in the table is the average accuracy of five experiments with the standard deviation in parentheses.
First, all the word embeddings in our experiments are randomly initialized as 200-dimension vectors and then modified during training (except Bert). For each type of feature extractor, we have the following configuration:
1) For the RNN-based models, we use a two-layer LSTM for feature extraction and the hidden state of each layer is set to 256.
2) For the CNN-based models, in order to obtain more fine-grained features, we use filter sizes of [2,3,4,5,6] with 100 feature maps each.
3) For the Transformer-based models, we use Transformer's encoder as the feature extractor, specifically with single-head and 3 blocks.
4) For the Bert-based models, we fine-tuned the pre-trained Bert-base parameters. These settings are exactly the same in the baseline as in our FP-Net.
In the training of the C-net module, we use a stochastic gradient with 0.9 as the momentum and the following annealing learning rate (Ganin and Lempitsky, 2014).
$$
l _ {p} = \frac {l _ {0}}{(1 + \alpha \cdot p) ^ {\beta}}
$$
where  $p$  is the training progress linearly changing from 0 to 1,  $l_0 = 0.01$ ,  $\alpha = 10$  and  $\beta = 0.75$ . In GRL, the hyper-parameters  $\lambda$  swept [0.05, 0.1, 0.2, 0.4, 0.8, 1.0].
In our experiments, we adopt the classification accuracy as the evaluation metric. We summarize the experimental results in Table 2, where FP+X means that the model trained by the proposed FP-Net using X as the feature extractor. Each of the
two lines compares the experimental results of the traditional model with our proposed model on these four datasets. From Table 2, we can make the following observations.
1. Our FP-Net model consistently improves the results of the baseline feature extractors (i.e., LSTM, CNN, Transformer and Bert) using the proposed feature projection. This verifies the effectiveness of the proposed feature purification method of projecting the traditional feature vectors to the orthogonal direction of the common features.
2. Compared with the traditional CNN, the FP+CNN model increases the accuracy by  $2.56\%$  on the MR dataset and  $1.46\%$  on the SNLI dataset. The improvement of FP+LSTM is less, increased by  $0.67\%$  and  $0.94\%$  on the MR and SNLI datasets. This shows that the way that CNN extracts input features (concatenate the feature after using different sliding window sizes for extracting local features) is quite effective in extracting more complete semantic information, which leads to more irrelevant features being used. That is why the projection on the CNN features brings more improvements compared to the RNN-based model.
3. By comparing the experimental results of the attention-base model (i.e., Transformer and Bert), we can see that our FP-Net can improve the feature representation capabilities of these feature extractors. For example, in the Bert-based experiment, our FP+Bert can increase the accuracy by  $3.11\%$  on MR and  $1.66\%$  on TREC. That is to say our orthogonal projection method can make the representation of attention-based obtain a higher discriminative power for classification. Outperforming Bert is particularly significant because Bert is perhaps one of the best feature extractors, if not the best.
8167
Model	MR	SNLI	SST2	TREC
FP+CNN	78.74(±0.36)	74.38(±0.14)	82.02(±0.11)	92.78(±0.26)
FP+CNN-G	77.71(±0.44)	72.85(±0.62)	81.09(±0.17)	91.89(±0.10)
FP+CNN-O	76.64(±0.39)	73.11(±0.22)	81.25(±0.11)	90.76(±0.37)
FP+CNN-G-O(+sum)	76.38(±0.45)	73.08(±0.19)	80.67(±0.52)	90.89(±0.41)
FP+CNN-G-O(concat)	76.18(±0.51)	72.91(±0.26)	81.02(±0.18)	91.16(±0.41)
Table 3: Ablation experiments. The first block contains the results of FP+CNN with GRL (-G) or OPL (-O) removed and the results with both GRL and OPL (-G-O) removed and the features of the two modules (or subnetworks) summed. The second block contains the results with both GRL (-G) and OPL (-O) removed and the features of the two modules concatenated.
Model	MR	SNLI	SST2	TREC
CNN	76.18(±0.45)	72.92(±0.19)	80.47(±0.59)	90.86(±0.51)
CNN_Dp	76.72(±0.50)	73.49(±0.14)	80.67(±0.40)	90.91(±0.41)
FP+CNN	78.74(±0.36)	74.38(±0.14)	82.02(±0.11)	92.78(±0.26)
Trans	75.18(±0.57)	66.71(±0.58)	76.93(±0.39)	87.33(±0.23)
Trans_Dp	75.75(±0.31)	68.36(±0.25)	77.10(±0.48)	88.16(±0.34)
FP+Trans	76.83(±0.66)	73.34(±0.43)	78.42(±0.49)	89.51(±0.79)
Table 4: Experimental results with doubled parameter size on the four datasets. For example, Trans_Dp shows the increase of the number of blocks of the Transformer from 3 to 6.
4.5 Ablation Experiments and Analysis
In order to analyze the effectiveness of each component of FP-Net, we performed the following two ablation experiments.
First, in Table 3, we report the results of the ablation test of each component of FP-Net, where FP+CNN-G (or O, G-O) represents FP-Net with the GRL (or OPL, or both GRL and OPL) removed while using CNN as the feature extractor. The parameters of all the experiments compared in the first block are exactly the same. In order to keep the parameter size consistent, we performed elementwise summation of the features of FP-Net's two sub-networks  $f_{p}$  and  $f_{c}$  in the FP+CNN-G-O experiment. By comparing the experimental results of the first block, we observe the following:
1) Whether GRL or OPL is removed or both GRL and OPL are removed at the same time, the accuracy will drop significantly compared with the complete FP-Net. For example, for the MR dataset, when we remove the GRL and keep the OPL (i.e., FP+CNN-G), the accuracy decreases by  $1.03\%$ ; When we remove both GRL and OPL, and then execute  $f_{p} + f_{c}$  (i.e., FP+CNN-G-O(plus)), the accuracy decreases by  $2.36\%$ , etc. These results show that each component in FP-Net is important, and the absence of any one component will lead to decline in accuracy.
2) In the experiment of FP+CNN-O, we remove OPL and keep GRL, which means that we use  $f_{p} - f_{c}$  instead of the orthogonal projection (i.e.,  $f_{p} - f_{p*}$ ). As stated in P-Net module of Section 3, such a replacement will give up a constraint that gets the common feature  $f_{p*}$  of the current input  $x_{i}$  from the base common feature  $f_{c}$ . The results showed that the accuracy decreases by  $2.10\%$  on MR and decreases by  $1.27\%$  on SNLI, which mean that the projection operation (i.e., Eq. 9) is necessary.
3) Clearly, adding  $f_{p}$  and  $f_{c}$  of FP-Net is not the only way to connect the two sub-networks of FP+CNN-G-O. We can do  $f_{p} \oplus f_{c}$ , where  $\oplus$  is the concatenation operator. Although this method has more parameters in the P-net classifier, we can still observe that the accuracy of FP+CNN-G-O is not as good as the accuracy of FP+CNN. For example, FP+CNN-G-O reduced the accuracy by 2.36% on MR and 1.30% on SNLI, which can also prove the effectiveness of GRL and OPL in our FP-Net.
Second, we show that the improvement in accuracy by FP-Net is not due to the increase in the number of parameters. We doubled the parameters of traditional CNN and Transformer and compared with our  $\mathrm{FP + CNN}$ ,  $\mathrm{FP + Trans}$ . The results of this part of the experiments are shown in Table 4, where the index 'Dp' means the Doubled parameter size. For example, Tans_Dp increases the number
8168
of blocks of Transformer in the baseline from 3 to 6. All experimental results show that increasing the number of parameters of the baseline models will improve classification accuracy slightly, but there is still a large gap with the proposed model.
In this paper, we proposed a novel Feature Purification Network (FP-Net) to improve the representation for text classification. The method is based on feature projection. The proposed model uses two sub-networks, one for identifying common features that are not discriminative for classification, and the other for feature projection that projects the traditional features to the orthogonal direction of the common features. To the best of our knowledge, this is the first method that uses feature projection to improve text classification. Through a large number of comparative experiments, we showed the effectiveness of the proposed feature projection method.
Our current method is designed only for traditional text classification methods such as LSTM, CNN, and Transformer. In our future work, we will consider extending it to graph-based methods such as GCN for graph data, and to generation-based methods such as GAN for adversarial learning.
The project was funded by Peking University.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Yonatan Belinkov, Adam Poliak, Stuart M Shieber, Benjamin Van Durme, and Alexander M Rush. 2019. On adversarial removal of hypothesis-only bias in natural language inference. arXiv preprint arXiv:1907.04389.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. arXiv preprint arXiv:1508.05326.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa.
2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-2537.
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. 2016. Very deep convolutional networks for text classification. arXiv preprint arXiv:1606.01781.
Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079-3087.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Kawin Ethayarajh. 2018. Unsupervised random walk sentence embeddings: A strong but simple baseline. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 91-100.
Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers Volume 2, pages 171-175. Association for Computational Linguistics.
Yaroslav Ganin and Victor Lempitsky. 2014. Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi-olette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2030.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1681-1691.
8169
Samuel R. Bowman Jason Phang, Thibault Févry. 2019. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv:1811.01088 [cs.CL].
Peng Jin, Yue Zhang, Xingyuan Chen, and Yunqing Xia. 2016. Bag-of-embeddings for text classification. In *IJCAI*, volume 16, pages 2824–2830.
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics.
Christopher D. Manning Quoc V. Le Kevin Clark, Minh-Thang Luong. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2002 conference on Empirical methods in natural language processing.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294-3302.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Twenty-ninth AAAI conference on artificial intelligence.
Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1-7. Association for Computational Linguistics.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167.
Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In Thirty-Second AAAI Conference on Artificial Intelligence.
Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2015. Natural language inference by tree-based convolution and heuristic matching. arXiv preprint arXiv:1512.08422.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of
the 43rd annual meeting on association for computational linguistics, pages 115-124. Association for Computational Linguistics.
Dominik Scherer, Andreas Müller, and Sven Behnke. 2010. Evaluation of pooling operations in convolutional architectures for object recognition. In International conference on artificial neural networks, pages 92-101. Springer.
Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018. Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079.
Kai Sheng Tai, Richard Socher, and Christopher D Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075.
Y. Tamaazousti, H. Le Borgne, C. Hudelot, M. E. A. Seddik, and M. Tamaazousti. 2018. Learning more universal representations for transfer-learning. arXiv:1712.09708v5 [cs.CV].
Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1422-1432.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Baoxin Wang. 2018. Disconnected recurrent neural networks for text categorization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2311-2320.
Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-lstm and answer pointer. arXiv preprint arXiv:1608.07905.
Yiren Wang and Fei Tian. 2016. Recurrent residual learning for sequence classification. In Proceedings of the 2016 conference on empirical methods in natural language processing, pages 938-943.
Yijun Xiao and Kyunghyun Cho. 2016. Efficient character-level document classification by combining convolution and recurrent layers. arXiv preprint arXiv:1602.00367.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: human language technologies, pages 1480-1489.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. 2017. Comparative study of cnn and rnn for natural language processing. arXiv preprint arXiv:1702.01923.
Kai Zhang, Hefu Zhang, Qi Liu, Hongke Zhao, Hengshu Zhu, and Enhong Chen. 2019. Interactive attention transfer network for cross-domain sentiment classification.
8171