SlowGuess/NAACL_2021_OCR huggingface dataset:
We use the pre-trained uncased BERTweet model for fine-tuning and augmentation under the PyTorch framework. When fine-tuning, the batch
Method	CI_ESRX	AET_HUM	CVS_AET	ANTM_CI	avgF1
PGCNN	71.34	77.43	73.73	71.70	73.55
+SR	71.96	77.25	73.54	71.78	73.63
+EDA	70.97	77.28	73.85	71.90	73.50
+BT	71.57	77.59	74.17	71.56	73.72
+CBERT	71.57	77.31	73.53	71.59	73.50
+TR	73.51	77.85	75.42*	72.57*	74.84
+CBERT-ASDA	73.02‡	78.65*‡	74.30*‡	72.19	74.54
+ASDA-base	71.61	77.97	73.77	72.14	73.87
+ASDA	74.25*†	78.36*	74.63†	72.63*	74.97
TAN	68.39	76.06	69.83	68.72	70.75
+SR	67.88	75.46	70.37	69.02	70.68
+EDA	68.02	75.40	69.86	69.06	70.59
+BT	67.69	75.19	70.57	67.99	70.36
+CBERT	68.55	75.75	70.89	68.88	71.02
+TR	68.02	75.85	69.66	69.10	70.66
+CBERT-ASDA	70.40*‡	76.35	71.50	69.87*‡	72.03
+ASDA-base	67.19	76.29	70.83	69.32	70.91
+ASDA	70.13*†	77.53*	71.73*†	70.18*	72.39
BERT	71.12	78.47	75.28	74.11	74.75
+SR	73.24	78.57	75.65	73.80	75.32
+EDA	73.44	78.51	75.85	73.98	75.45
+BT	72.30	77.47	75.97	73.80	74.89
+CBERT	72.83	77.99	75.49	73.66	74.99
+TR	74.17	78.80	76.30	74.24	75.88
+CBERT-ASDA	74.58‡	78.95‡	76.46‡	74.46‡	76.11
+ASDA-base	72.49	78.76	76.09	74.01	75.34
+ASDA	75.45*†	78.99	76.41*	74.48*	76.33
Table 6: Performance comparisons of applying different augmentation methods to the base model on the WT-WT stance dataset.  $\ast$ : the proposed methods improve the best baseline at  $p < 0.05$  with paired t-test.  $\dagger$ : ASDA improves the ASDA-base at  $p < 0.05$  with paired t-test.  $\ddagger$ : CBERT-ASDA improves the CBERT at  $p < 0.05$  with paired t-test. avg $F_1$  is the average of all target pairs.
size is 32, maximum sequence length is 128, learning rate is 2e-5 and proportion of sentence to mask is  $15\%$ . For classification, we train our PGCNN and TAN models using a mini-batch of 128 and the learning rate of Adam optimizer (Kingma and Ba, 2015) is 1e-3. Maximum sequence length is 50 and word vectors are initialized using fastText embeddings (Bojanowski et al., 2017) with dimension 300. For BERT classifier, we fine-tune the pretrained BERT to predict the stance by appending a linear classification layer to the hidden representation of the [CLS] token. The maximum sequence length is set to 128 and the learning rate is 2e-5.
We generate one augmented sentence for each training data, doubling the original train set in size for fair comparison. Experimental results on SemEval2016, WT-WT and Multi-Target datasets are shown in Tables 5, 6 and 7, respectively. Bold scores are best two results for each classifier. Each result is the average of ten runs with different initializations. Since CBERT and TR cannot be applied to the Multi-Target and SemEval-2016 datasets, respectively, we didn't report the results of these methods.
First, we can observe that our proposed ASDA performs the best in  $\mathrm{avgF_1}$  on almost all datasets. Moreover, ASDA has better performance than ASDA-base on all targets, demonstrating the effectiveness of adding reference sentences. Second, CBERT can be only used in single-target stance detection tasks due to the segmentation embeddings. In contrast, ASDA-base that achieves similar performance with CBERT can be applied to all datasets, which indicates that constructing auxiliary sentence contributes to the C-MLM task. Third, Tables 5 and 6 show that constructing the auxiliary sentence can not only perform well on the BERTweet model, but also help improve the baseline CBERT, indicating that our proposed method is not tailored to specific masked language model. Fourth, TR achieves promising improvements on WT-WT and Multi-Target datasets, outperforming the EDA in the average of  $\mathrm{avgF_1}$  on three classifiers by  $0.61\%$  and  $1.54\%$ , respectively. Further comparison between TR and Random Swap of EDA is discussed later in this section. At last, we can observe that improvements brought by the baselines are limited on three datasets, verifying that target-based stance detection tasks are more challenging.
Method	Tr-Cl	Tr-Cr	Cl-Sa	avgF1
PGCNN	57.38	52.73	50.00	53.37
+SR	57.95	53.42	50.97	54.11
+EDA	57.57	51.99	50.56	53.37
+BT	57.94	52.33	51.29	53.85
+CBERT	-	-	-	-
+TR	59.86*	56.06*	50.32	55.41
+ASDA-base	57.97	53.65	50.62	54.08
+ASDA	59.98*†	54.67*	52.66*†	55.77
TAN	56.26	53.25	50.13	53.21
+SR	56.62	53.73	50.17	53.51
+EDA	55.55	54.12	50.11	53.26
+BT	56.31	52.61	50.83	53.25
+CBERT	-	-	-	-
+TR	58.07*	56.57*	48.38	54.34
+ASDA-base	57.45*	55.45	50.99	54.63
+ASDA	57.76*	56.44*†	52.92*†	55.71
BERT	54.87	54.32	53.49	54.23
+SR	53.99	52.97	52.72	53.23
+EDA	55.46	52.69	53.21	53.79
+BT	54.81	53.95	53.30	54.02
+CBERT	-	-	-	-
+TR	61.83*	55.23	48.85	55.30
+ASDA-base	52.98	53.27	51.80	52.68
+ASDA	56.31†	55.48	53.72†	55.17
We further explore the effect of the auxiliary sentence by comparing the proposed ASDA with other Prepending based Data Augmentation (PDA) (Schick and Schütze, 2020; Kumar et al., 2020) in which no description sentence is constructed and the complete sentence is: [Label] [Target]  $E_{1}$ . Moreover, we consider the reference sample  $E_{2}$  as mentioned in Section 3.2.1 for PDA and the complete sentence is [Label] [Target]  $E_{2}$ $E_{1}$ . Comparison results on SemEval-2016 dataset are shown in Table 8. We can observe that both ASDA and PDA-ASDA show better performance over their base models, which indicates that the reference sentence contributes to the performance improvement and our proposed method is not tailored to specific auxiliary sentence.
We compare the proposed methods with other augmentation methods in Table 9. We can observe that both ASDA and TR consider the target information during augmentation. However, TR cannot be applied to SemEval-2016 dataset because unlike WT-WT dataset that corresponds to the merger of two target companies, only single target is available in SemEval-2016 dataset.
Random Swap is an augmentation method that randomly chooses two words in the sentence and
Table 7: Performance comparisons of applying different augmentation methods to the base model on the Multi-Target stance dataset.  $\ast$ : the proposed methods improve the best baseline at  $p < 0.05$  with paired t-test.  $\dagger$ : ASDA improves the ASDA-base at  $p < 0.05$  with paired t-test.  $avgF_{1}$  is the average of all target pairs.
Method	Atheism	Feminist	Hillary	Abortion
PGCNN	68.08	55.31	61.51	67.26
+PDA	65.82	56.36	61.66	65.80
+PDA-ASDA	66.77	57.34	65.50	66.36
+ASDA-base	65.61	58.11	62.10	66.75
+ASDA	69.78	59.03	63.62	68.44
TAN	59.27	56.45	56.58	59.29
+PDA	65.16	57.09	58.76	58.21
+PDA-ASDA	67.36	58.85	60.98	60.80
+ASDA-base	63.40	57.55	56.03	60.96
+ASDA	68.47	58.73	60.09	63.66
BERT	70.69	54.57	66.06	57.80
+PDA	70.35	55.27	66.50	61.11
+PDA-ASDA	72.93	56.04	66.54	59.37
+ASDA-base	70.67	54.18	64.67	61.34
+ASDA	72.30	55.97	68.09	63.02
Table 8: Performance comparisons of applying augmentation methods with different auxiliary sentences to the base model on the SemEval-2016 dataset.
Method	Target-aware	All datasets	Require FT
SR	X	✓	X
EDA	X	✓	X
BT	X	✓	X
CBERT	X	X	✓
TR	✓	X	X
ASDA	✓	✓	✓
Table 9: Overall method comparisons on the stance detection. "Target aware" means the method is aware of target information during augmentation. "All datasets" means the augmentation method can be applied to all three stance detection datasets. "Require FT" means the method requires fine-tuning before augmentation.
swaps their positions. However, Random Swap can potentially generate augmented sentences that contain contradictory content with the labels. Since TR shares similar features with Random Swap by swapping the target mentions in some cases, we compare our proposed TR with Random Swap on WT-WT and Multi-Target datasets in Table 10. The results show that TR achieves better performance on 6, 5 and 4 targets for PGCNN, TAN and BERT, respectively, demonstrating the effectiveness of this method. Note that TR does not perform well on the target pair Clinton-Sanders; one possible reason is that there is more target-related information in this target pair. Since only target words (e.g., "Hillary Clinton") are swapped in TR, target-related words like "feminism" and "Benghazi" still appear in the same position in the generated sentence, which may lead to the inconsistency of target information.
In this section, we present several augmented examples in Table 11 to show the effectiveness of our
Method	Tr-Cl	Tr-Cr	Cl-Sa	avgF1	CI_ESRX	AET_HUM	CVS_AET	ANTM_CI	avgF1
PGCNN
+RS	57.98	51.32	51.45	53.58	71.05	77.37	74.33	71.12	73.47
+TR	59.86	56.06	50.32	55.41	73.51	77.85	75.42	72.57	74.84
TAN
+RS	56.45	52.89	49.30	52.88	67.19	74.20	70.87	68.22	70.12
+TR	58.07	56.57	48.38	54.34	68.02	75.85	69.66	69.10	70.66
BERT
+RS	55.14	53.57	53.46	54.06	73.07	78.80	76.45	73.98	75.58
+TR	61.83	55.23	48.85	55.30	74.17	78.80	76.30	74.24	75.88
Table 10: Performance comparison between EDA Random Swap (RS) and the proposed Target Replacement (TR) on the Multi-Target stance dataset and WT-WT stance dataset. avgF1 is the average of all target pairs.
Target:	Feminist Movement.
Source:	What do feminists want: all humans, male and female, should have equal political, economic and social rights.
EDA:	What do libber want: all humans, male and female, should have equal political, economic and social rights.
BT:	What the feminists want: all humans, men and women should have the same political, economic and social.
CBERT:	What do feminists want: all humans, male and female, will have had what. economic and social..
ASDA:	What real feminists want: all humans, male and female, to have equal political rights and equal social rights.
Target:	Cigna and Express Scripts.
Source:	Cigna stockholders greenlight merger with Express Scripts.
EDA:	Cigna stockholders greenlight merger with Express Scripts.
BT:	Cigna merger to shareholders with GreenLight Express Scripts.
CBERT:	Cigna stockholders relight merger with Express Scripts.
TR:	Express Scripts stockholders greenlight merger with Cigna.
ASDA:	Cigna stockholders vote for merger with Express Scripts.
Target:	Donald Trump and Ted Cruz.
Source:	Make america great again!! No socialist/liberals. Principals that made this country great can make it great again! Trump Cruz
EDA:	Make america great again!! No Cruz/liberals. Principals that made this country great can make it great again! Trump socialist
BT:	Do it again!! Not great america liberal socialist/. Managers who have made this great country can do much more! Trump Cruz
TR:	Make america great again!! No socialist/liberals. Principals that made this country great can make it great again! Cruz Trump
ASDA:	Make america great again!! No democrats/liberals. The people who made this country great can make america great again! Trump Cruz
Table 11: Examples generated by the augmentation methods. Texts in bold represent generated words.
proposed methods. Synonym Replacement, Random Deletion and Random Swap of EDA are applied to the targets "Feminist Movement", "Cigna and Express Scripts" and "Donald Trump and Ted Cruz", respectively. We can observe that the generated words of ASDA and TR are more consistent with the target and label information. In contrast, the augmented words of baseline methods especially EDA could be incompatible with the labels of the original sentences.
In this paper, we presented two data augmentation methods, called ASDA and TR, for stance detection. Different from the existing augmentation methods that are either unaware of target information or hard to be applied to different stance detection tasks, ASDA performs better in generating target-relevant and label-compatible sentences and can be easily applied to various tasks. Results
show that ASDA can not only achieve best performance on BERTweet model but also help improve the existing augmentation method such as CBERT. Unlike other rule-based word replacement methods that may produce undesirable correlation between a target and its contexts, TR replaces a target mention with the other, generating qualified sentences with meaningful lexical patterns. In addition, both ASDA and TR will be applicable if we need to detect the stances toward more than two targets simultaneously in the future.
Future work includes extending the proposed methods to various directions, e.g., argument mining, aspect-based sentiment analysis and hate-speech detection, and generating more diversified samples through conditional generation.
This work is partially supported by the NSF Grants IIS-1912887 and IIS-1903963. We thank our reviewers for their insightful comments.
Abeer AlDayel and Walid Magdy. 2020. Stance detection on social media: State of the art and trends. arXiv preprint arXiv:2006.03644.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, pages 7383-7390.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Costanza Conforti, Jakob Berndt, Mohammad Taher Pilehvar, Chryssi Giannitsarou, Flavio Toxvaerd, and Nigel Collier. 2020. Will-they-won't-they: A very large dataset for stance detection on Twitter. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1715-1724.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, page 933-941.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 3988-3994.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, page 2672-2680.
Binxuan Huang and Kathleen Carley. 2018. Parameterized convolutional neural networks for aspect level sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1091-1096.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations.
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457.
Dilek Kucuk and Fazli Can. 2020. Stance detection: A survey. ACM Comput. Surv., 53(1):1-37.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained transformer models. In Proceedings of the 2nd Workshop on Life-long Learning for Spoken Language Systems, pages 18-26.
Kun Li, Chengbo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7056-7066.
Yingjie Li and Cornelia Caragea. 2019. Multi-task stance detection with sentiment and stance lexicons. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6298-6304.
Junjie Lin, Qingchao Kong, Wenji Mao, and Lei Wang. 2019. A topic enhanced approach to detecting multiple standpoints in web texts. Information Sciences, 501:483-494.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems - Volume 2, page 3111-3119.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. 2016. A dataset for detecting stance in tweets. In LREC.
Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: A pre-trained language model for English tweets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 9-14.
Timo Schick and Hinrich Schütze. 2020. Exploiting cloze questions for few-shot text classification and natural language inference. Computing Research Repository, arXiv:2001.07676.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Umme Aymun Siddiqua, Abu Nowshed Chy, and Masaki Aono. 2019. Tweet stance detection using an attention based neural ensemble model. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1868-1873.
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2017. A dataset for multi-target stance detection. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 551-557.
Parinaz Sobhani, Diana Inkpen, and Xiaodan Zhu. 2019. Exploring deep neural networks for multi-target stance detection. Computational Intelligence, 35(1):82-97.
Chi Sun, Luyao Huang, and Xipeng Qiu. 2019. Utilizing BERT for aspect-based sentiment analysis via constructing auxiliary sentence. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 380-385.
Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierarchical attention network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2399-2409.
William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using #petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557-2563.
Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382-6388.
Penghui Wei, Junjie Lin, and Wenji Mao. 2018a. Multi-target stance detection via a dynamic memory-augmented network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1229-1232.
Penghui Wei, Wenji Mao, and Daniel Zeng. 2018b. A target-guided neural memory model for stance detection in twitter. In 2018 International Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13, 2018, pages 1-8.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional bert contextual augmentation. In International Conference on Computational Science, pages 84-95.
Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2019. Recognising agreement and disagreement between stances with reason comparing networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4665-4671.
Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2514-2523.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, page 649-657.