SlowGuess/NAACL_2021_OCR huggingface dataset:
Yingjie Li
yli300@uic.edu
Cornelia Caragea
The goal of stance detection is to identify whether the author of a text is in favor of, neutral or against a specific target. Despite substantial progress on this task, one of the remaining challenges is the scarcity of annotations. Data augmentation is commonly used to address annotation scarcity by generating more training samples. However, the augmented sentences that are generated by existing methods are either less diversified or inconsistent with the given target and stance label. In this paper, we formulate the data augmentation of stance detection as a conditional masked language modeling task and augment the dataset by predicting the masked word conditioned on both its context and the auxiliary sentence that contains target and label information. Moreover, we propose another simple yet effective method that generates target-aware sentence by replacing a target mention with the other. Experimental results show that our proposed methods significantly outperforms previous augmentation methods on 11 targets.
Nowadays, people often take to social media to express their stances toward specific targets (e.g., political figures or abortion). These stances in an aggregate can provide valuable information for obtaining insight into some important events such as presidential elections. The goal of the stance detection task is to determine from a piece of text whether the author of the text is in favor of, neutral or against toward a specific target (Mohammad et al., 2016; Lin et al., 2019), which indicates that all elements, the sentence, the target, and the label, are used to train a stance detection model. We can further classify the task as single-target stance detection and multi-target stance detection (Küçük and Can, 2020; AlDayel and Magdy, 2020) where we need to detect the stances toward two different targets simultaneously.
Orig.	We all have a duty to protect the sanctity of life.
G1	We all have a life to protect the sanctity of duty.	✕
G2	We all have a duty to protect the sanctitude of life.	✕
G3	We all have a responsibility to protect the unborn lives.	✓
Table 1: Examples of data augmentation on the target "Legalization of Abortion".
One of the biggest challenges for stance detection tasks is the scarcity of annotated data. Data augmentation (DA) is an effective strategy for handling scarce data situations. However, we face three main obstacles when applying the existing augmentation methods to the stance detection tasks. First, existing augmentation methods do not generalize well, which means some methods are tailored to specific tasks and models, and thus difficult to be extended to the stance detection tasks. Second, consider an original sample that is against to the target "Legalization of Abortion" in Table 1. Using previous augmentation methods we may end up with the first generation example  $(G_{1})$  that deviates from its original meaning due to the unawareness of target and label information during augmentation. Third, previous augmentation methods could generate the sentence  $(G_{2})$  with less diversified patterns. To address these issues, we propose an augmentation method that can generate more diversified sentence  $(G_{3})$  that is consistent with target and label information. Moreover, we expect the proposed method to generalize well to other tasks.
A common data augmentation strategy is based on word replacement. Zhang et al. (2015) augmented a sentence by substituting the replaceable words with synonyms from WordNet (Miller, 1995). However, synonym replacement can only generate limited diversified patterns. Wu et al. (2019) formulated the text data augmentation as a Conditional Masked Language Modeling (CMLM) task and proposed a Conditional BERT
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1850-1860 June 6-11, 2021. ©2021 Association for Computational Linguistics
(CBERT) where segmentation embeddings of BERT (Devlin et al., 2019) are replaced with the annotated label during augmentation. This method seems to be able to generate label-compatible sentences, yet it does not consider the target information for stance detection. Moreover, CBERT can hardly be extended to other pre-trained language models that do not use segmentation embeddings in inputs, and cannot be applied to the multi-target stance detection due to the inability to encode two stance labels in segmentation embeddings. Wei and Zou (2019) proposed a simple effective method that uses operations such as random deletion or swap to help train more robust model. However, similar to the above methods, it fails to take target information into considerations. Another commonly used strategy for augmentation is back-translation (Yu et al., 2018), however, it is less controllable and may change the target information unpredictably.
Inspired by the recent advances of applying auxiliary sentence to aspect-based sentiment analysis (Sun et al., 2019) and the task of recognising agreement and disagreement between stances (Xuet al., 2019), in this paper, we propose an Auxiliary Sentence based Data Augmentation (ASDA) method that generates target-relevant and label-consistent data samples based on the C-MLM task. Specifically, we fine-tune a pre-trained BERTweet (Nguyen et al., 2020) model through C-MLM task in which the masked word is conditioned on both its context and the prepended auxiliary sentence that contains target and label information. The same task is also adopted in the augmentation stage to generate data samples. Besides, we propose a simple Target Replacement (TR) method that generates target-aware sentence by replacing a target mention in a sentence with the other.
Our contributions include the following: 1) In this paper, we propose a novel data augmentation method called ASDA. As far as we know, this is the first attempt to explore the conditional data augmentation of stance detection. Our proposed ASDA significantly outperforms strong baselines on three different stance detection datasets with 11 targets in total, demonstrating its effectiveness. Experimental results show that prepending auxiliary sentence contributes to the performance gain; 2) We further propose a simple yet effective method called Target Replacement (TR) that achieves highly competitive performance even without fine-tuning during the augmentation; 3) Our proposed ASDA can be
also employed on other baseline to help improve the performance, which indicates that ASDA is not tailored to specific model.
Most previous studies on stance detection focused on the detection of stance from text that contains expressions of stance towards one single target, i.e., single-target stance detection. Mohammad et al. (2016) presented the SemEval-2016 dataset that contains 5 independent targets, e.g., Legalization of Abortion and Hillary Clinton. Conforti et al. (2020) constructed WT-WT, a financial dataset on which the task is to detect whether two companies (e.g., Cigna and Express Scripts) will merge or not. Inspired by the attention mechanism (Bahdanau et al., 2015), various target-specific attention-based approaches (Du et al., 2017; Sun et al., 2018; Wei et al., 2018b; Li and Caragea, 2019; Siddiqua et al., 2019; Sobhani et al., 2019) were proposed to connect the target with the sentence representation. Moreover, gated mechanism (Dauphin et al., 2017) and BERT (Devlin et al., 2019) have drawn a lot attention these years and achieved promising performance on aspect-based sentiment analysis (Xue and Li, 2018; Huang and Carley, 2018). We used the models from Du et al. (2017), Huang and Carley (2018) and Devlin et al. (2019) as strong base classifiers for our evaluation.
Sobhani et al. (2017) introduced the multi-target stance detection task and presented the Multi-Target stance dataset. The task is to detect the stances toward two presidential candidates (e.g., Donald Trump and Ted Cruz) simultaneously. They also proposed an attention-based encoder-decoder (Seq2Seq) model that predicts stance labels by focusing on different parts of a tweet. Wei et al. (2018a) proposed a dynamic memory network for detecting stance. We used the above three datasets (Mohammad et al., 2016; Sobhani et al., 2017; Conforti et al., 2020) for our evaluation.
One of the main challenges for stance detection tasks is the scarcity of annotated training data, which is costly to obtain. Therefore, data augmentation becomes appealing, particularly when the training models become increasingly large. Generative models are commonly used for data augmentation in previous studies, including variational
autoencoders (VAE) (Kingma and Welling, 2014), generative adversarial networks (GAN) (Goodfellow et al., 2014) and pre-trained language generation models (Anaby-Tavor et al., 2020; Li et al., 2020; Kumar et al., 2020). Besides, Sennrich et al. (2016) and Yu et al. (2018) generated the data by using back-translation, which first translates the English sentence into another language (e.g., French) and then translates it back to English.
Another commonly used way for data augmentation is to substitute local words. Zhang et al. (2015) and Wang and Yang (2015) substituted the replaceable words with synonyms from WordNet (Miller, 1995) and Word2Vec (Mikolov et al., 2013), respectively. Kobayashi (2018) proposed a contextual data augmentation method. A bidirectional language model is used to predict the word given the context surrounding the original word. Wu et al. (2019) formulated the text data augmentation as a C-MLM task, retrofitting BERT (Devlin et al., 2019) to predict the masked word based on its context and annotated label. Wei and Zou (2019) boosted the performance on text classification by using simple operations such as random deletion or insertion, and received substantial attention from the research community recently.
However, the augmentation methods mentioned above mostly focus on the sentence-level natural language processing tasks and the resulting augmented sentence can either change the stance toward the given target unexpectedly or generate only limited diverse patterns for stance detection tasks.
Suppose a given training dataset of size  $n$  is  $D_{train} = \{(x_i, t_i, y_i)\}_{i=1}^n$  where  $x_i = [x_i^1, x_i^2, \dots, x_i^l]$  is a sequence of  $l$  words,  $t_i$  is the corresponding target and  $y_i \in \{1, \dots, c\}$  is the label. The objective of our data augmentation task is to generate an augmented sentence  $\hat{x}_i$  that is consistent with the target  $t_i$  and label  $y_i$ . Note that  $t_i = [t_i^1, t_i^2]$  and  $y_i = [y_i^1, y_i^2]$  for multi-target stance detection, which makes the augmentation task more challenging.
3.2 Auxiliary Sentence based Data Augmentation
Previous conditional data augmentation methods such as (Wu et al., 2019) could generate target-unaware data samples and cannot be applied to the multi-target stance detection task. In this pa
per, we propose an Auxiliary Sentence based Data Augmentation (ASDA) method that can generate target-relevant and label-consistent data samples based on the C-MLM task.
3.2.1 Construction of the Auxiliary Sentence
ASDA generates augmented sentence by predicting the masked word that is conditioned on both its context and the auxiliary sentence. We propose the following method to construct the auxiliary sentence.
ASDA: Given a training sample  $E_{1}$ , we pretend both another training sample  $E_{2}$  with the same target and label as  $E_{1}$  and the description sentence that contains target and label information to  $E_{1}$ . The complete sentence is: The authors of the following tweets are both [Label] [Target]. The first tweet is:  $E_{2}$ . The second tweet is:  $E_{1}$ .
The sentences before  $E_{1}$  are the auxiliary sentences we construct. "Target" and "Label" are the target name and stance label with regard to the given training sample.  $E_{2}$  that contains the same target and stance label with  $E_{1}$  is sampled from the training dataset. Specifically, suppose we are given a training example in the SemEval-2016 dataset: We all have a duty to protect the sanctity of life. Target: Legalization of Abortion; Label: Against. We can have the following masked words and auxiliary sentences in fine-tuning or augmentation stage: The authors of the following tweets are both against to legalization of abortion. The first tweet is: Every human life is worth the same, and worth saving. The second tweet is: We all have a [MASK] to protect the [MASK] of life. Target: Legalization of Abortion; Label: Against. With the auxiliary sentence, the masked word is not only conditioned on its context in the second tweet, but also conditioned on the first tweet of same target "Legalization of Abortion" and label "Against".
We expect the agreement between stances to benefit the data augmentation by adding a reference sentence  $E_{2}$ . The introduction of the  $E_{2}$  not only generates more diversified samples for fine-tuning the pre-trained language model, but also provides a strong guideline to help generate target-relevant and label-compatible sentences in the augmentation stage. Moreover, ASDA is not tailored to specific model because it does not rely on the model architecture, and thus can be easily extended to different language models.
3.2.2 Conditional DA using BERTweet
BERTweet (Nguyen et al., 2020) is a large-scale language model pre-trained on 850M English tweets. BERTweet follows the training procedure of RoBERTa (Liu et al., 2019) and uses the same model configuration with BERT-base (Devlin et al., 2019). We fine-tune the pre-trained BERTweet via C-MLM on stance detection tasks. The fine-tuning step is summarized in Algorithm 1. Note that words of auxiliary sentence  $A$  are never masked (see Algorithm 1 lines 4-6) because we want to preserve all target and label information.
After fine-tuning the BERTweet on the training dataset for a few epochs, we use the well-trained model for augmentation. Similar to the fine-tuning procedure as shown in Algorithm 1, for a training sentence  $s$  from  $D_{train}$ , we randomly mask words in  $s$  and prepend the corresponding auxiliary sentence  $A$  to obtain the masked sentence  $\hat{s}$ . Then, the BERTweet model is used to predict the masked words and we repeat these steps over all training data to get  $\hat{D}_{train}$ . The above steps can be implemented multiple times with different masked positions, and hence, different augmented samples can be generated from the original training dataset. Finally, we merge the  $D_{train}$  with  $\hat{D}_{train}$  and per
form classification task on this combined dataset.
3.3 Target Replacement Method
Besides ASDA, we propose a Target Replacement (TR) method to increase the size of the training set by replacing a target mention in a sentence with the other, which improves model robustness so that meaningful lexical patterns are learned by the model instead of learning undesirable correlation between a target and its contexts. In case a target is mentioned more than once, we continue to replace the target until all targets are replaced. Hash-tags and mentions that contain target information (e.g., #Cigna) are also considered for replacement. Consider the following example in single-target stance detection: #CI Shareholders vote to approve merger Cigna and Express Scripts. Target: Cigna and Express Scripts; Label: Support. After applying TR, we have: #ESRX Shareholders vote to approve merger Express Scripts and Cigna. Target: Cigna and Express Scripts; Label: Support. CI and ESRX represent Cigna and Express Scripts, respectively.
TR can be also applied to the multi-target stance detection with minor changes. Consider the following example: #Cruz supporters want people to think his words alone are good enough. #DonaldTrump has created jobs and businesses we need in this country. Target1: Donald Trump; Target2: Ted Cruz; Label1: Favor; Label2: Against. TR could potentially generate contradictory content with the labels if we only replace the target mentions since the task is to detect the stances toward two different targets simultaneously. Therefore, we replace the target mentions and swap the stance labels for multi-target stance detection. Consider the same example as above after applying the target replacement and label swap: #DonaldTrump supporters want people to think his words alone are good enough. #Cruz has created jobs and businesses we need in this country. Target1: Donald Trump; Target2: Ted Cruz; Label1: Against; Label2: Favor.
In this section, we first describe three stance detection datasets used for evaluation and several baseline methods of data augmentation and stance detection. Then, we introduce the evaluation metrics and report the experimental results.
Target	#Train	%Favor	%Against	%None	#Test	%Favor	%Against	%None
Atheism	513	17.93	59.26	22.81	220	14.54	72.73	12.73
Climate	395	53.67	3.80	42.53	169	72.78	6.51	20.71
Feminism	664	31.63	49.40	18.97	285	20.35	64.21	15.44
Hillary	689	17.13	57.04	25.83	295	15.25	58.31	26.44
Abortion	653	18.53	54.36	27.11	280	16.43	67.50	16.07
Total	2,914	25.84	47.87	26.29	1,249	24.34	57.25	18.41
Table 2: Data distribution of SemEval-2016 dataset (Mohammad et al., 2016).
Target	#Total	%Refute	%Comment	%Support	%Unrelated
Cigna and Express Scripts	2,527	10.01	37.47	30.58	21.92
Aetna and Humana	7,897	14.00	35.50	13.14	37.34
CVS Health and Aetna	11,622	4.45	47.49	21.24	26.80
Anthem and Cigna	11,044	17.82	28.05	8.78	45.33
Total	33,090	11.62	37.38	15.87	35.13
Table 3: Data distribution of WT-WT dataset (Conforti et al., 2020).
Target Pair	#Total	#Train	#Dev	#Test
Trump-Clinton	1,722	1,240	177	355
Trump-Cruz	1,317	922	132	263
Clinton-Sanders	1,366	957	137	272
Total	4,455	3,119	446	890
Table 4: Distribution of instances in Multi-Target stance dataset (Sobhani et al., 2017).
Three stance detection datasets, the SemEval-2016 dataset (Mohammad et al., 2016), the WT-WT financial dataset (Conforti et al., 2020) and the Multi-Target election dataset (Sobhani et al., 2017), are used to evaluate the performance of augmentation methods. The SemEval-2016 dataset and WT-WT dataset are both single-target stance datasets and the third dataset is a multi-target stance dataset, which contains stances toward two targets in each tweet. Summary statistics of three datasets are shown in Tables 2, 3, 4, respectively.
SemEval-2016 SemEval-2016 is a benchmark dataset containing five different targets: "Atheism", "Climate Change is a Real Concern", "Feminist Movement", "Hillary Clinton" and "Legalization of Abortion". The dataset is annotated for detecting whether the author is against to, neutral or in favor of a given target. We split the train set in a 5:1 ratio into train and validation sets and removed the target "Climate Change" because of the limited and highly skewed data. The test set of each target is the same as provided by the authors.
WT-WT WT-WT is a financial dataset and the task aims at detecting the stance toward mergers and acquisition operations between companies. This dataset consists of four target pairs in the healthcare domain and each data is annotated with four labels (refute, comment, support and unre
lated). We split the dataset in a 10:2:3 ratio into train, validation and test sets.
Multi-Target Multi-Target stance dataset consists of three sets of tweets corresponding to target pairs: Donald Trump and Hillary Clinton, Donald Trump and Ted Cruz, Hillary Clinton and Bernie Sanders. The task aims at detecting the stances (against, none or favor) toward two targets for each data. We used the train, validation and test sets as provided by the authors.
We compare the proposed augmentation methods with the following baselines:
- Synonym Replacement (SR): A data augmentation method that randomly replaces words with their synonyms from WordNet.
- EDA (Wei and Zou, 2019): A simple data augmentation method that consists of four operations: synonym replacement, random deletion, random swap and random insertion.
- BT (Yu et al., 2018): A back-translation method that first translates the English sentence into French and then translates back to English.
- CBERT (Wu et al., 2019): A C-MLM method that generates label-compatible words by replacing the segmentation embeddings of BERT with label embeddings.
Three base classifiers are used to evaluate the performance of different augmentation methods:
- PGCNN (Huang and Carley, 2018): A parameterized convolutional neural network that uses target-sensitive filters and gated mechanism to incorporate the target information.
- TAN (Du et al., 2017): An attention-based LSTM model that extracts target specific features.
Method	Atheism	Feminist	Hillary	Abortion	\(avg_{1}\)
PGCNN	68.08	55.31	61.51	67.26	63.04
+SR	66.67	56.07	61.12	67.14	62.75
+EDA	66.43	55.26	62.06	66.07	62.46
+BT	67.52	56.01	61.70	65.18	62.60
+CBERT	66.36	57.24	61.97	66.19	62.94
+TR	-	-	-	-	-
+CBERT-ASDA	\(68.70^{\ddagger}\)	56.74	\(64.97^{\ddagger}\)	\(68.77^{* \ddagger}\)	64.80
+ASDA-base	65.61	58.11	62.10	66.75	63.14
+ASDA	\(69.78^{* \dagger}\)	59.03*	63.62	\(68.44^{* \dagger}\)	65.22
TAN	59.27	56.45	56.58	59.29	57.90
+SR	63.49	55.76	56.61	59.52	58.85
+EDA	64.86	57.23	55.35	59.69	59.28
+BT	65.36	57.37	58.95	58.45	60.03
+CBERT	65.43	57.27	59.39	60.18	60.57
+TR	-	-	-	-	-
+CBERT-ASDA	66.60	\(58.59^{\ddagger}\)	61.02	\(62.72^{* \ddagger}\)	62.23
+ASDA-base	63.40	57.55	56.03	60.96	59.49
+ASDA	\(68.47^{* \dagger}\)	\(58.73^{\dagger}\)	\(60.09^{\dagger}\)	\(63.66^{* \dagger}\)	62.74
BERT	70.69	54.57	66.06	57.80	62.28
+SR	72.78	54.85	66.20	59.42	63.31
+EDA	71.69	53.82	65.51	60.18	62.80
+BT	72.49	54.98	66.74	61.91	64.03
+CBERT	71.88	55.79	64.29	62.36	63.58
+TR	-	-	-	-	-
+CBERT-ASDA	\(74.93^{* \ddagger}\)	56.43	\(67.01^{\ddagger}\)	61.66	65.01
+ASDA-base	70.67	54.18	64.67	61.34	62.72
+ASDA	\(72.30^{\dagger}\)	\(55.97^{\dagger}\)	\(68.09^{\dagger}\)	\(63.02^{\dagger}\)	64.85
Table 5: Performance comparisons of applying different augmentation methods to the base model on the SemEval2016 stance dataset. \*: the proposed methods improve the best baseline at  $p < 0.05$  with paired t-test. †: ASDA improves the ASDA-base at  $p < 0.05$  with paired t-test. ‡: CBERT-ASDA improves the CBERT at  $p < 0.05$  with paired t-test. avg $F_1$  is the average of all target pairs.
- BERT (Devlin et al., 2019): A pre-trained language model that predicts the stance by appending a linear classification layer to the hidden representation of  $[CLS]$  token. We fine-tune the BERT-base on various stance detection tasks.
The proposed methods are listed as follows:
- Target Replacement (TR): A method that replaces target words with the other.
CBERT-ASDA: The CBERT that uses our proposed auxiliary sentences during fine-tuning and augmentation.
- ASDA-base: A variation of ASDA that only prepends the description sentence to the given training sample. The complete sentence is: The author of the following tweet is [Label] [Target].  $E_{1}$ .
- ASDA: The full method that uses both description and reference sentences as auxiliary sentences during fine-tuning and augmentation.
4.3 Evaluation Metric and Hyperparameters
$F_{avg}$  is adopted to evaluate the performance of the proposed model. First, the F1-score of label "Favor"
and "Against" is calculated as follows:
$$
F _ {f a v o r} = \frac {2 P _ {f a v o r} R _ {f a v o r}}{P _ {f a v o r} + R _ {f a v o r}} \tag {1}
$$
$$
F _ {\text {a g a i n s t}} = \frac {2 P _ {\text {a g a i n s t}} R _ {\text {a g a i n s t}}}{P _ {\text {a g a i n s t}} + R _ {\text {a g a i n s t}}} \tag {2}
$$
where  $\mathrm{P}$  and  $\mathbf{R}$  are precision and recall respectively. After that, the  $F_{avg}$  is calculated as:
$$
F _ {a v g} = \frac {F _ {f a v o r} + F _ {a g a i n s t}}{2} \tag {3}
$$
We calculate the  $F_{avg}$  for each target. The same evaluation metric was used in SemEval-2016 dataset and Multi-Target stance datasets. To be consistent with the previous work, we evaluate the performance of augmentation methods on WT-WT dataset by using the same evaluation metric  $F_{avg}$ , which is calculated by averaging the F1-scores of label "Support" and "Refute". Moreover, we get  $avgF_{1}$  by calculating the average of  $F_{avg}$  across all targets for each dataset.