SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2 presents the performance comparison of the proposed PCM method and other baselines on different datasets. From that, we can have the following observations. (1) By using BERT, all methods achieve reasonable performance. Even the BERT fine-tune baseline achieves good performance when there are ten samples per class. However, BERT fine-tune is still inferior to the semi-supervised approaches, especially when the number of training samples becomes smaller or the classification task becomes more challenging. (2) As expected, the MixText method excels than UDA in most cases, but performs similarly when the number of labeled samples becomes large (e.g., 50 labels/class). Since the proposed method could also be incorporated into MixText, it might be able to boost its performance. (3) the proposed PCM methods achieves significant performance improvement over UDA approaches. Please note that PCM is built on top of the UDA method and this performance gain indicates the effectiveness of using the proposed progressive training process. (4) It is clear that PCM can not only always outperform other baselines and achieve state-of-the-art text classification performance on all four datasets, but also have smaller standard error and be more stable. PCM performs especially well when the number of labeled samples becomes small. A much larger performance gain is observed when only
Figure 4: Ablation study on the DCDL strategy in PCM.
three labeled samples are available.
Furthermore, we compare PCM to baselines with 10 labeled data per class and varying number of unlabeled ones on Yahoo! Answers dataset (range from 10,000 to 100,000 unlabeled samples). Fig. 3 shows that PCM continuously benefits from more unlabeled data and can be consistently superior than other methods.
PCM model consists of several components. In this section, we perform ablation studies to examine their impact. Most of these studies are performed on Yahoo! Answers dataset with one identical labeled set, unless otherwise specified.
1. The importance of using two classifiers in PCM. The proposed PCM model contains a  $K$ -way classifier (i.e.,  $p^s$ ) and a matching classifier (i.e.,  $p^m$ ), and they are jointly trained in the proposed process. We investigate the role of them by constructing a variant of PCM by only using either one of them. As the results shown in Table 4, without using the  $K$ -way classifier, the method totally fails to a random guess. In contrast, only keeping the  $K$ -way classifier can obtain reasonable results. More interestingly, this variant actually performs better than UDA on 3 and 5 label cases (see the Table 2). The difference between this variant and UDA is that the former appends CSR to the input sequence. Its good performance shows that merely appending CSR can be helpful for semi-supervised text classification. Finally, we can see that using both classifiers can lead to the best performance. This clearly validates the necessity of the proposed joint learning process.
2. If using the dual-classifier-dual-loss is the key to success? In our method, we utilize a slightly unconventional dual-classifier-dual-loss strategy
Initial	bush, car, bomb, killed, chancellor, black, moscow, inter, leftist, putin, story, presidential, texas, president, campaign, documents, ap, unearthed, caracas, ...
Final	iraq, president, iraqi, government, baghdad, military, palestinian, security, nuclear, prime, minister, country, israeli, leader, war, peace, gaze, iran, israel, troops, ...
Table 3: The class semantic-related word lists on class "world" of AG News dataset. The top row is the initial class semantic-related words obtained from fine-tuned BERT, while the bottom one is the final class semantic-related words after PCM training with upper initial words. All models are trained on the 3 labels per class case.
ps	pm	Number of Labeled Example Per Clas
3	5	10	20	50
X	✓	10.01	10.45	10.01	10.21	10.05
✓	X	49.51	65.32	65.70	67.88	68.43
✓	✓	63.52	67.09	68.34	69.21	70.28
Table 4: Ablation study on the importance of two classifiers of the proposed PCM model.
update CSR	Number of Labeled Example Per Class
3	5	10	20	50
X	39.49	66.04	66.41	67.09	68.85
✓	63.52	67.09	68.34	69.21	70.28
Table 5: Ablation study on the importance of updating the CSR during training of PCM.
(DCDL): the pseudo-labels are generated by checking the agreement of the two classifiers, and two losses, i.e., BCE and CE, are used for training those two classifiers. One may suspect that our good performance actually stems from this DCDL scheme rather than leveraging BERT's matching capability. To investigate this problem, we conduct an ablation study by modifying UDA with this strategy. Specifically, we use two classifiers, one trained from the BCE loss and the other one trained from the CE loss. The pseudo-prediction targets are generated by following the same strategy as in PCM. The result is shown in Fig. 4. As seen, simply incorporating this training strategy does not necessarily bring better classification accuracy. This result provides evidence that the PCM's good performance can not be simply attributed to the DCDL strategy.
3. The prediction quality of the  $K$ -way classifier and the matching classifier. In our PCM model, the  $K$ -way classifier is chosen for the final testing phase. We further validate the quality of the matching classifier. As the results presented in Fig. 5, the matching classifier gains comparable performance to the  $K$ -way one. This proves that the collaborative training of two classifiers bootstraps each other to have good prediction capability.
4. The impact of updating CSR. Our PCM method dynamically updates the CSR through the
Figure 5: Ablation study on classifier quality of PCM.
training process. In this part, we investigate the impact of this updating process. Table 5 compares the results obtained by updating or not updating CSR. As seen, updating CSR leads to overall better performance. The difference becomes quite significant when only three labeled samples are used. For example, PCM may fail when the class semantic representation is fixed in the 3-label case.
6 Limitations and Potential Risks
One underlying assumption about our findings is that we mainly consider BERT-style pre-trained language models for semi-supervised text classification. The utilization of inherent knowledge of other language models (e.g., GPT (Radford et al., 2018) and XLNet (Yang et al., 2019b)) are not explored in this paper and is left for future work.
PCM algorithm has been verified to be effectiveness on texts in English, whether other languages can achieve the same performance improvement is at risk and will be explored in the future.
In this paper, we proposed a semi-supervised text classification approach by leveraging the inherent topic matching capability in pre-trained language models. The method progressively updates three components, a  $K$ -way classifier, the class semantic representation, and a matching classifier that
matches input text against the class semantic representation. We show that the updating of the three components can benefit each other and achieve superior semi-supervised learning performance.
In terms of ethics, we do not see immediate concerns for the models we introduce and to the best of our knowledge no datasets were used that have known ethical issues.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. 2019a. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09785.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019b. Mixmatch: A holistic approach to semisupervised learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5050-5060.
Kasturi Bhattacharjee, Miguel Ballesteros, Rishita Anubhai, Smaranda Muresan, Jie Ma, Faisal Ladhak, and Yaser Al-Onaizan. 2020. To BERT or not to BERT: Comparing task-specific and task-agnostic semi-supervised approaches for sequence tagging. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7927-7934, Online. Association for Computational Linguistics.
Avrim Blum and Tom Mitchell. 1998. Combining labeled and unlabeled data with co-training. In Proceedings of the eleventh annual conference on Computational learning theory, pages 92-100.
Ming-Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar. 2008. Importance of semantic representation: Dataless classification. In Proceedings of the Twenty-Third AAAI Conference on Artificial Intelligence, volume 2, pages 830-835.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online. Association for Computational Linguistics.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings
of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914-1925, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, Brussels, Belgium. Association for Computational Linguistics.
Mahak Gambhir and Vishal Gupta. 2017. Recent automatic text summarization techniques: a survey. *Artificial Intelligence Review*, 47(1):1-66.
Alex Gammerman, Volodya Vovk, and Vladimir Vapnik. 2013. Learning by transduction. arXiv preprint arXiv:1301.7375.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174-6181, Online. Association for Computational Linguistics.
Nitin Jindal and Bing Liu. 2007. Review spam detection. In Proceedings of the 16th International Conference on World Wide Web, WWW 2007, Banff, Alberta, Canada, May 8-12, 2007, pages 1189-1190. ACM.
Hwiyeol Jo and Ceyda Cinarel. 2019. Delta-training: Simple semi-supervised text classification using pretrained word embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3458-3463, Hong Kong, China. Association for Computational Linguistics.
Thorsten Joachims. 2003. Transductive learning via spectral graph partitioning. In Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pages 290-297. AAAI Press.
Diederik P. Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3581-3589.
Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Soren Auer, et al. 2015. Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
Chen Liu, Mengchao Zhang, Zhibin Fu, Pan Hou, and Yu Li. 2021. *Flitext: A faster and lighter semisupervised text classification with convolution networks*. In *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, Online. Association for Computational Linguistics.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150.
Takeru Miyato, Andrew M. Dai, and Ian J. Goodfellow. 2017. Adversarial training methods for semisupervised text classification. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairoseq: A fast, extensible toolkit for sequence modeling*. In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics* (Demonstrations), pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 79-86. Association for Computational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, volume 33, pages 596-608. Curran Associates, Inc.
Jost Tobias Springenberg. 2016. Unsupervised and semi-supervised learning with categorical generative adversarial networks. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 380-385, Minneapolis, Minnesota. Association for Computational Linguistics.
Zijun Sun, Chun Fan, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020. Neural semi-supervised learning for text classification under large-scale pretraining. arXiv preprint arXiv:2011.08626.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1195-1204.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019a. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR.
Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019b. Interpolation consistency training for semi-supervised learning. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3635-3641. ijcai.org.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019a. Uda experimental tutorial on text classification. https://github.com/google-research/uda#text-classification. Accessed: 2022-01-15.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019b. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904.12848.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019a.
Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. arXiv preprint arXiv:1509.01626.
Zhi-Hua Zhou and Ming Li. 2005. Tri-training: Exploiting unlabeled data using three classifiers. IEEE Transactions on knowledge and Data Engineering, 17(11):1529-1541.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.