SlowGuess/ACL_2020_OCR huggingface dataset:
Table 3: Generalized experimental results on benchmarks and real-world e-commerce dataset, where I and II refer to two versions of partitions respectively.
Topic	Emotion	Situation	E-commerce
I	II	I	II	I	II	I	II
Word2vec	38.16	49.08	18.42	12.17	59.02	37.89	59.52	70.17
Label similarity	39.36	45.70	27.43	17.81	67.73	39.96	61.90	72.73
FC	20.93	29.29	33.76	12.98	38.47	34.15	34.10	30.57
RNN+FC	31.09	28.63	33.05	19.47	32.98	25.61	32.44	26.52
BERT	67.73	60.20	29.31	11.96	75.08	51.48	70.77	79.74
BERT+self-training	73.24	67.97	33.71	20.76	76.03	53.18	73.95	82.74
BERT+RL	74.46	66.70	37.33	20.57	77.23	53.63	75.58	83.97
Table 4: Non-generalized experimental results on benchmarks and real-world e-commerce dataset, where I and II refer to two versions of partitions respectively.
Table 3 shows the experimental results on benchmarks and real-world e-commerce dataset in generalized setting. For baseline methods, Word2vec and Label similarity are unsupervised approaches, which cannot get desirable results as the effectiveness of these methods heavily rely on the similarity of text and label. Therefore, it may not perform well on dataset like emotion detection. Label similarity performs slightly better than Word2vec, which proves that max aggregation of n-grams is better than mean aggregation in Word2vec method. As for the supervised FC and RNN+FC method, FC gets slightly better results than RNN+FC in most datasets. As the number of categories and the scale of training dataset are small, RNN+FC may overfit on seen class data and cannot generalize well on unseen class data.
For variants of our method, we can observe that the full model BERT+RL outperforms all other baselines. On average, BERT+RL achieves an improvement of  $15.4\%$  over BERT. To be specific, the base matching model BERT performs better than previous baselines, which shows good gen
eralization results benefiting from pre-training on large-scale corpus. For BERT+self-training, the integration of unlabeled data augments the base matching model and shows superior performance than BERT. Last but not least, our full model BERT+RL shows substantial improvement over BERT+self-training in most datasets. Under the condition that the number of selected instances remains the same, reinforced selection strategy can still yield better performance than the simply confidence-based strategy, which proves the effectiveness of our RL policy.
For non-generalized ZSL setting, we can get similar results as presented in Table 4. On average, BERT+RL achieves an improvement of  $5.4\%$  over BERT. However, we notice that the improvement is more significant in generalized ZSL compared to non-generalized ZSL. The reason is that model trained on seen class data tends to bias towards seen classes, resulting in poor performance in generalized setting (Song et al., 2018). Our approach, however, could relieve the bias in favour of seen classes by incorporating pseudo-labeled unseen class data.
(a) Topic
(b) Emotion
Figure 4: Performance with regards to selected instance ratio  $\epsilon$ . One can see the RL data selection strategy does not rely on manually-set ratio and can yield consistently better performance than the competitors in most cases.
(c) Situation
(d) E-commerce
Label	BERT	BERT+RL
Joy	1. Good morning joyful people. Choose happiness to have a great day today.2. I was filled with joy when I heard I had been selected to come here at Kamuzu College of Nursing.	1. And they all rejoiced, and embraced him and kissed him without stopping.2. When I got a record as a gift from a friend.
Sadness	1. I'm sick and sad , missing out on Martini Lounge tonight.2. Crossing the bridge, leaving ocean city I'm sad .	1. When I learned that two of my friends had a serious car accident.2. Oh my god! Got in a car accident! Pray for him!
Whitening	1. Mizon Good Night White Sleeping Mask.2. Intimate Skin Whitening Cream For Face.	1. VieBeauti Dark Spot Corrector Remover.2. Intimate Skin lightening Cream.
Nordic style	1.Aah Nordic modern cloth sofa size living room.2. Nordic Side Table, Modern Decoration.	1. Fabric sofa, simple and modern apartment living room.2. Modern simple style living room chandelier.
Table 5: Qualitative comparison between BERT and BERT+RL. Left: texts predicted with high confidence; Right: texts being misclassified by BERT while being correctly labeled by BERT+RL.
4.5 Impact of Selection Ratio
When selecting the same number of instances per iteration, previous experimental results show our reinforced selection strategy can yield better performance than the greedy strategy. We define  $\epsilon$  as the ratio of selected instances size to all unlabeled instances size. In this section, we vary the selection ratio  $\epsilon$  among  $\{0.2, 0.4, 0.6, 0.8, 1.0\}$  for self-training method. For each iteration, we select top  $\frac{\epsilon}{N_1} M$  instances and add them into training set. Figure 4 shows the performances with different selection ratios in generalized ZSL setting. Clearly, the performance of self-training method varies with different ratio of instances selected. The optimal ratio of selection instances also varies with different datasets. However, our reinforced data selection strategy does not rely on manually-set ratio and can yield consistently better performance than the self-training method in most cases.
In Table 5, we listed some examples to further reveal the differences between BERT and
BERT+RL method. In the left part of the table, texts predicted by BERT with highest confidence are listed. We can easily find that these texts share a simple matching pattern that label words appear in the text, which is highlighted with red color. These simple patterns are exactly class-invariant patterns we defined previously, which can be shared among classes. In the right part of the table, we select the texts which are misclassified by BERT but are predicted correctly by BERT+RL. We can observe that those texts are harder to be distinguished since these matching patterns are more class-dependent, which cannot be directly transferred from other classes. There is no doubt that model trained on other classes would fail in such cases. For our method, we first tackle the easy instances, then add these instances into training set iteratively. With the integration of instances with easy pattern, the model can learn harder pattern gradually. In this way, our method can learn to transfer between classes even with low similarity.
In this paper, we propose a reinforced self-training framework for zero-shot text classification. To realize the transferring between classes with low similarity, our method essentially turns a zero-shot learning problem into a semi-supervised learning problem. In this way, our approach could leverage unlabeled data and alleviate the domain shift between seen classes and unseen classes. Beyond that, we use reinforcement learning to learn data selection policy automatically, thus obviating the need to manual adjustment. Experimental results on both benchmarks and real-world e-commerce dataset demonstrate the effectiveness of the integration of unlabeled data and the reinforced data selection policy.
This work is funded by NSFC U19B2027/91-846204/61473260, national key research program 2018YFB1402800, and supported by Alibaba-ZJU Frontier Technology Research Center.
Chenhua Chen, Yue Zhang, and Yuze Gao. 2018. Learning how to self-learn: Enhancing self-training using neural reinforcement learning. In 2018 International Conference on Asian Language Processing (IALP), pages 25-30. IEEE.
Minmin Chen, Kilian Q Weinberger, and John Blitzer. 2011. Co-training for domain adaptation. In Advances in neural information processing systems, pages 2456-2464.
Yann N Dauphin, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2013. Zero-shot learning for semantic utterance classification. arXiv preprint arXiv:1401.0509.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Brett Drury, Luis Torgo, and Jose Joao Almeida. 2011. Guided self training for sentiment classification. In Proceedings of Workshop on Robust Unsupervised and Semisupervised Methods in Natural Language Processing, pages 9-16.
Yang Fan, Fei Tian, Tao Qin, Jiang Bian, and TieYan Liu. 2017. Learning what data to learn. arXiv preprint arXiv:1702.08635.
Meng Fang, Yuan Li, and Trevor Cohn. 2017. Learning how to active learn: A deep reinforcement learning approach. arXiv preprint arXiv:1708.02383.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. In Thirty-Second AAAI Conference on Artificial Intelligence.
Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong. 2012. Attribute learning for understanding unstructured social activity. In European Conference on Computer Vision, pages 530-543. Springer.
Yanwei Fu, Timothy M Hospedales, Tao Xiang, and Shaogang Gong. 2015. Transductive multi-view zero-shot learning. IEEE transactions on pattern analysis and machine intelligence, 37(11):2332-2345.
Zhongqiang Huang and Mary Harper. 2009. Self-training pcfg grammars with latent annotations across languages. In Proceedings of the 2009 conference on empirical methods in natural language processing: Volume 2-Volume 2, pages 832-841. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Zornitsa Kozareva, Boyan Bonev, and Andres Montoyo. 2005. Self-training and co-training applied to Spanish named entity recognition. In Mexican International conference on Artificial Intelligence, pages 770-779. Springer.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 951-958. IEEE.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2013. Attribute-based classification for zero-shot visual object categorization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(3):453-465.
Hugo Larochelle, Dumitru Erhan, and Yoshua Bengio. 2008. Zero-data learning of new tasks. In AAAI, volume 1, page 3.
Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. 2018. Analogical reasoning on chinese morphological and semantic relations. arXiv preprint arXiv:1805.06504.
Xin Li, Yuhong Guo, and Dale Schuurmans. 2015. Semi-supervised zero-shot classification with label representation learning. In Proceedings of the IEEE international conference on computer vision, pages 4211-4219.
David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of the main conference on human language technology conference of the North American Chapter of the Association of Computational Linguistics, pages 152-159. Association for Computational Linguistics.
David McClosky, Eugene Charniak, and Mark Johnson. 2008. When is self-training effective for parsing? In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, pages 561-568. Association for Computational Linguistics.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529.
Jinseok Nam, Eneldo Loza Mencía, and Johannes Fürnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In Thirtieth AAAI Conference on Artificial Intelligence.
Mohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg S Corrado, and Jeffrey Dean. 2013. Zero-shot learning by convex combination of semantic embeddings. arXiv preprint arXiv:1312.5650.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Pushpankar Kumar Pushp and Muktabh Mayank Srivastava. 2017. Train once, test anywhere: Zero-shot learning for text classification. arXiv preprint arXiv:1712.05972.
Pengda Qin, Weiran Xu, and William Yang Wang. 2018. Robust distant supervision relation extraction via deep reinforcement learning. arXiv preprint arXiv:1805.09927.
Anthony Rios and Ramakanth Kavuluru. 2018. Few-shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Language Processing, volume 2018, page 3132. NIH Public Access.
Marcus Rohrbach, Sandra Ebert, and Bernt Schiele. 2013. Transfer learning in a transductive setting. In Advances in neural information processing systems, pages 46-54.
Marcus Rohrbach, Michael Stark, and Bernt Schiele. 2011. Evaluating knowledge transfer and zero-shot learning in a large-scale setting. In CVPR 2011, pages 1641-1648. IEEE.
Kenji Sagae. 2010. Self-training without reranking for parser domain adaptation and its impact on semantic role labeling. In Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing, pages 37-44. Association for Computational Linguistics.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. 2016. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484.
Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pages 935-943.
Jie Song, Chengchao Shen, Yezhou Yang, Yang Liu, and Mingli Song. 2018. Transductive unbiased embedding for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1024-1033.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063.
Isaac Triguero, Salvador Garcia, and Francisco Herrera. 2015. Self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42(2):245-284.
Vincent Van Asch and Walter Daelemans. 2016. Predicting the effectiveness of self-training: Application to sentiment classification. arXiv preprint arXiv:1601.03288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Sappadla Prateek Veeranna, Jinseok Nam, Eneldo Loza Mencia, and Johannes Furnkranz. Using semantic similarity for multi-label zero-shot classification of text documents.
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. 2019. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):13.
Jiawei Wu, Lei Li, and William Yang Wang. 2018. Reinforced co-training. arXiv preprint arXiv:1804.06035.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. 2018. Zero-shot learning-a comprehensive evaluation of the good, the bad and the
ugly. IEEE transactions on pattern analysis and machine intelligence.
Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017. Deeppath: A reinforcement learning method for knowledge graph reasoning. arXiv preprint arXiv:1707.06690.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach. arXiv preprint arXiv:1909.00161.
Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. arXiv preprint arXiv:1903.12626.
3024