SlowGuess/NAACL_2021_OCR huggingface dataset:
Danilo Croce, Giuseppe Castellucci, and Roberto Basili. 2020. GAN-BERT: Generative adversarial learning for robust text classification with a bunch of labeled examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2114-2119.
Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019. Fine-grained analysis of propaganda in news article. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5640-5650.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Pascal Diethelm and Martin McKee. 2009. Denialism: what is it and how should scientists respond? The European Journal of Public Health, 19(1):2-4.
Riley E Dunlap and Robert J Brulle. 2015. Climate change and society: Sociological perspectives. Oxford University Press.
Riley E Dunlap and Peter J Jacques. 2013. Climate change denial books and conservative think tanks: Exploring the connection. American Behavioral Scientist, 57(6):699-731.
Justin Farrell. 2016. Network structure and influence of the climate change counter-movement. Nature Climate Change, 6(4):370.
Kjersti Flottum. 2014. Linguistic mediation of climate change discourse. *ASp. la revue du GERAS*, (65):7-20.
Kjersti Flottum, Trine Dahl, and Vegard Rivenes. 2016. Young Norwegians and their views on climate change and the future: findings from a climate concerned and oil-rich nation. Journal of Youth Studies, 19(8):1128-1143.
Gary Fooks, Anna Gilmore, Jeff Collin, Chris Holden, and Kelley Lee. 2013. The limits of corporate social responsibility: techniques of neutralization, stakeholder management and political CSR. *Journal of Business Ethics*, 112(2):283-299.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A Smith. 2019. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880-5894.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Annamma Joy, John F Sherry Jr, Alladi Venkatesh, Jeff Wang, and Ricky Chan. 2012. Fast fashion, sustainability, and the ethical appeal of luxury brands. *Fashion Theory*, 16(3):273-295.
Muel Kaptein and Martien Van Helvoort. 2019. A model of neutralization techniques. Deviant Behavior, 40(10):1260-1285.
Carl B J Klockars. 1974. The Professional Fence. The Free Press.
Yiwei Luo, Dallas Card, and Dan Jurafsky. 2020. DeSMOG: Detecting stance in media on global warming. In Findings of EMNLP 2020.
Shadd Maruna and Heath Copes. 2005. What have we learned from five decades of neutralization research? Crime and justice, 32:221-320.
Ruth McKie. 2018. Rebranding the Climate Change Counter Movement through a Criminological and Political Economic Lens. Ph.D. thesis, Northumbria University.
W William Minor. 1981. Techniques of neutralization: A reconceptualization and empirical examination. Journal of Research in Crime and Delinquency, 18(2):295-318.
Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 6149-6157.
Naomi Oreskes and Erik M Conway. 2010. Defeating the merchants of doubt. Nature, 465(7299):686-687.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
James Powell. 2017. Scientists reach  $100\%$  consensus on anthropogenic global warming. Bulletin of Science, Technology & Society, 37(4):183-184.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2931-2937.
Amanda M Shigihara. 2013. It's only stealing a little a lot: Techniques of neutralization for theft among restaurant workers. Deviant Behavior, 34(6):494-512.
Gresham M Sykes and David Matza. 1957. Techniques of neutralization: A theory of delinquency. American sociological review, 22(6):664-670.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2020. Unsupervised data augmentation for consistency training. In Proceedings of NeurIPS 2020.
Yun Yang, Fengtao Nan, Po Yang, Qiang Meng, Yingfu Xie, Dehai Zhang, and Khan Muhammad. 2019. GAN-based semi-supervised learning approach for clinical decision support in health-iot platform. IEEE Access, 7:8048-8057.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In International Conference on Machine Learning, pages 3881-3890.
For the supervised BERT models, we use the following fine-tuning hyper-parameters: batch size  $= 10$ , epoch  $= 3$ , learning rate  $= 0.0005$ , number of epochs  $= 3$  and use BERT-base-uncased as the base model. We tune our decision boundary threshold to classify the presence of a label based on development set and are 0.2 for DOR, 0.2 for DOI1, 0.2 for DOI2, 0.3 for DOV, 0.3 for COC, 0.3 for AHL, 0.2 for JBC, and 0.2 for NONE.
For semi-supervised MTEXT based models, we use following hyper-parameters: labelled batch size=2, unlabelled batch size=5, sharpening temperature=0.6, epoch =3, the beta distribution parameter  $= 0.2$ , learning rate=0.00005,  $w_{1} = 1$ ,  $w_{2} = 1$ ,  $w_{3} = 0.8$  in  $w_{1}\mathcal{L}_{s} + w_{2}\mathcal{L}_{cl} + w_{3}\mathcal{L}_{em}$ ,  $\alpha$  for auxiliary objective  $\alpha \mathcal{L}_{frame} = 0.8$ . mixing layers as 7,9,12 and use BERT-base-uncased. We tune our decision boundary threshold to classify the presence of a label based on development set and are 0.75 for DOR, 0.70 for DOI1, 0.70 for DOI2, 0.80 for DOV, 0.85 for COC, 0.80 for AHL, 0.70 for JBC, and 0.60 for NONE. We use 2 augmentations (based on back translation) with Russian and German as the intermediate language.
2 Other Details
- Computing Infrastructure: We use RTX 2080 Ti and GTX 1080. In MTEXT based models we use 2 gpus when trained with RTX 2080 ti and 3 gpus when trained with GTX 1080. BERT based models are trained on a singlegpu.
- Average run time. BERT based models are quite quick and a minute per epoch to fine tune whereas MTEXT based models take around 20 minutes for each epoch.
- As all the models are based on BERT-Base-uncased the number of parameters are around  $110\mathrm{M}^2$
- Validation performance of the various models are given in Table 1
we use a small value here to ensure the generated data in the model is similar to labelled data with small noise regularization
${}^{2}$  strictly speaking number of parameters in  ${\mathrm{{MTEXT}}}_{\text{multi }}$  will be slightly more due to auxiliary objective but is insignificant in overall picture
Model	F
BERT	0.59
BERT*	0.61
MTEXT	0.62
MTEXT*	0.64
MTEXTmulti	0.66
MTEXTMulti	0.65
Table 1: NT multi-label classification performance on validation data. “F” denote micro-F1 respectively.
- Hyperparameter tuning was done using manual search and the criteria used was micro-F1 on validation set.
- Parameters used for final set of experiments are given in the above section of Training Details