PeerDA: Data Augmentation via Modeling Peer Relation for Span Identification Tasks∗
Year: 2023

Abstract:
Span identification aims at identifying specific text spans from text input and classifying them into pre-defined categories. Different from previous works that merely leverage the Subordinate (SUB) relation (i.e. if a span is an instance of a certain category) to train models, this paper for the first time explores the Peer (PR) relation, which indicates that two spans are instances of the same category and share similar features. Specifically, a novel Peer Data Augmentation (PeerDA) approach is proposed which employs span pairs with the PR relation as the augmentation data for training. PeerDA has two unique advantages: (1) There are a large number of PR span pairs for augmenting the training data. (2) The augmented data can prevent the trained model from over-fitting the superficial span-category mapping by pushing the model to leverage the span semantics. Experimental results on ten datasets over four diverse tasks across seven domains demonstrate the effectiveness of PeerDA. Notably, PeerDA achieves state-of-the-art results on six of them.1

heading: 1 Introduction
text: Span Identification (SpanID) is a family of Natural Language Processing (NLP) tasks with the goal of detecting specific spans from the input text and further classifying them into pre-defined categories (Papay et al., 2020). It serves as the initial step for complex text analysis by narrowing down the search scopes of important spans, which holds a pivotal position in the field of NLP (Ding et al., 2021; Xu et al., 2021). Recently, different
∗ This work was supported by Alibaba Group through Alibaba Research Intern Program. It was also partially supported by a grant from the Research Grant Council of the Hong Kong Special Administrative Region, China (Project Code: 14200620). † This work was done when Weiwen Xu was an intern at Alibaba DAMO Academy. ‡ Xin Li is the corresponding author.
1Our code and data are available at https://github. com/DAMO-NLP-SG/PeerDA
domain-specific SpanID tasks, such as social media Named Entity Recognition (NER) (Derczynski et al., 2017), Aspect-Based Sentiment Analysis (ABSA) (Liu, 2012), Contract Clause Extraction (CCE) (Chalkidis et al., 2017), Span Based Propaganda Detection (SBPD) (Da San Martino et al., 2019) and Argument Extraction (Cheng et al., 2020), have emerged for various NLP applications.
Precisely, as shown in Figure 1 (a), the process of SpanID can be reinterpreted as extracting spancategory Subordinate (SUB) relation — if a span in the input text is an instance of a certain category. Early works (Chiu and Nichols, 2016) typically tackle SpanID tasks as a sequence tagging problem, where the SUB relation is recognized via predicting the category for each input token under certain context. Recently, to better utilize category semantics, many efforts have been made on reformulating SpanID tasks as a Machine Reading Comprehension (MRC) problem (Liu et al., 2020; Yang et al., 2021). As shown by the example in Figure 1 (b), such formulation first creates a SUB query for each category and then recognizes the SUB relation by
8681
detecting relevant spans in the input text (i.e., context) as answers to the category query.
However, only leveraging the SUB relation in the training data to build SpanID models may suffer from two limitations: 1) Over-fitting: With only SUB relation, SpanID models tend to capture the superficial span-category correlations. Such correlations may misguide the models to ignore the semantics of the given span but make predictions based on the memorized span-category patterns, which hurts the generalization capability of the models. 2) Data Scarcity: For low-resource scenarios or long-tailed categories, the number of span-category pairs with SUB relation (SUB pairs) could be very limited and insufficient to learn a reliable SpanID model.
In this paper, we explore the span-span Peer (PR) relation to alleviate the above limitations. Specifically, the PR relation indicates that two spans are two different instances of the same category. The major difference between PR relation and SUB relation is that the former one intends to correlate two spans without giving the categories they belong to. For example, in Figure 1 (a), "Hawaii" and "London" are connected with the PR relation because they are instances of the same category. By jointly recognizing SUB relation and PR relation in the input text, the model is enforced to favor the usage of span semantics instead of span-category patterns for prediction, reducing the risk of over-fitting. In addition, the number of spanspan pairs with the PR relation (PR pairs) grows quadratically over the number of SUB pairs. Therefore, we can still construct a reasonable number of training data with PR pairs for categories having insufficient examples.
In this paper, with the aim of leveraging the PR relation to enhance SpanID models, we propose a Peer Data Augmentation (PeerDA) approach that treats PR pairs as a kind of augmented training data. To achieve this, as depicted in Figure 1 (b), we extend the usage of the original training data into two views. The first view is the SUB-based training data. It is used to directly solve the SpanID tasks by extracting the SUB relation, which is the typical formulation of MRC-based approaches. The second view is the PR-based training data. It is our augmentation to enrich the semantics of spans by extracting the PR relation in the original training data, where one span is used to identify its peer from the input context. Note that our PR-based
training data can be easily formulated into the MRC paradigm. Therefore, the knowledge learned from such augmentation data can be directly transferred to enhance the model’s capability to capture SUB relation (i.e., the SpanID tasks).
To better accommodate the MRC-style SUB and PR data, we develop a stronger and more memoryefficient MRC model. Compared to the designs in Li et al. (2020b), our model introduces a bilinear component to calculate the span scores and consistently achieves better performance with a 4 times smaller memory consumption. Besides, we propose a margin-based contrastive learning strategy to additionally model the negative spans to the query (e.g., when querying the context in Figure 1 for “ORG” entities, “London” becomes a negative span) so that the spans from different categories are separated more apart in the semantic space.
We evaluate the effectiveness of PeerDA on ten datasets across seven domains, from four different SpanID tasks, namely, NER, ABSA, CCE, and SBPD. Experimental results show that extracting PR relation benefits the learning of semantics and encourages models to identify more possible spans. As a result, PeerDA is a new state-of-the-art (SOTA) method on six SpanID datasets. Our analyses further demonstrate the capability of PeerDA to alleviate scarcity and over-fitting issues.
Our contributions are summarized as follows:
• We propose a novel PeerDA approach to tackle SpanID tasks via augmenting training data with PR relation. • We conduct extensive experiments on ten datasets, including four different SpanID tasks across seven domains, and achieve SOTA performance on six SpanID datasets. • PeerDA is more effective in low-resource scenarios or long-tailed categories and thus, it alleviates the scarcity issue. Meanwhile, jointly recognizing the SUB and PR relations makes the MRC model rely less on memorizing the SUB patterns in the training set for inferring the span label, which prevents over-fitting.

heading: 2 Related Work
text: DA for SpanID: DA, which increases the diversity of training data at a low cost, is a widelyadopted solution to address data scarcity (Feng et al., 2021). In the scope of SpanID, existing DA approaches aim to introduce more span-category patterns, including: (1) Word Replacement either
replaces or paraphrases some context tokens using simple rules (Wei and Zou, 2019; Dai and Adel, 2020; Xiang et al., 2021) and strong language models (Kobayashi, 2018; Wu et al., 2019; Li et al., 2020a; Yoo et al., 2021), or applies synonym dictionaries or masked language models to replace the labeled tokens with other tokens of the same type (Wei and Zou, 2019; Zhou et al., 2022b). (2) Fine-grained Augmentation Data Generation first trains an auto-regressive language model, and then leverages the model to generate new sentences with entity tags as a special kind of tokens (Ding et al., 2020; Liu et al., 2021b). (3) Self-training is to continually train the model on its predicted data (Xie et al., 2019, 2020; Wang et al., 2020; Zhou et al., 2023; Tan et al., 2023), while consistency training also leverages unlabeled data by imposing regularization on the predictions (Zhou et al., 2022a) (4) Distantly Supervised Training focuses on leveraging external knowledge to roughly label spans in the target tasks (Bing et al., 2013, 2015; Xu et al., 2023a). Huang et al. (2021) leverage Wikipedia to create distant labels for NER. Chen et al. (2021) transfer data from high-resource to low-resource domains. Jain et al. (2019); Li et al. (2020c) tackle cross-lingual NER by projecting labels from highresource to low-resource languages, which is particularly common in real applications (Kruengkrai et al., 2020). Differently, the motivation of PeerDA is to leverage the augmented data to enhance models’ capability on semantic understanding by minimizing(maximizing) the distances between semantically similar(distant) spans.
MRC: MRC is to extract an answer span from a relevant context conditioned on a given query. It is initially designed to solve question answering tasks (Hermann et al., 2015), while recent trends have shown great advantages in formulating NLP tasks as MRC problems. In the context of SpanID, Li et al. (2020b); Xu et al. (2022b, 2023b) address the nested NER issues by decomposing nested entities under multiple queries. Mao et al. (2021); Zhang et al. (2021a) tackle ABSA in a unified MRC framework. Hendrycks et al. (2021) tackle CCE with MRC to deal with the extraction of long clauses. Moreover, other tasks such as relation extraction (Li et al., 2019), event detection (Liu et al., 2020, 2021a), and summarization (McCann et al., 2018) are also reported to benefit from the MRC paradigm.

heading: 3 PeerDA
text: Overview of SpanID: Given the input text X = {x1, ..., xn}, SpanID is to detect all appropriate spans {xk}Kk=1 and classify them with proper labels {yk}Kk=1, where each span xk = {xsk , xsk+1, ..., xek−1, xek} is a subsequence of X satisfying sk ≤ ek and the label comes from a predefined category set Y (e.g. "Person" in NER).

heading: 3.1 Training Data Construction
text: The training data D consists of two parts: (1) The SUB-based training data DSUB, where the query is about a category and the MRC context is the input text. (2) The PR-based training data DPR is constructed with PR pairs, where one span is used to create the query and the input text containing the second span serves as the MRC context.

heading: 3.1.1 SUB-based Training Data
text: First, we need to transform the original training examples into (query, context, answers) triples following the paradigm of MRC (Li et al., 2020b). To extract the SUB relation between categories and relevant spans, a natural language query QSUBy is constructed to reflect the semantics of each category y. Following Hendrycks et al. (2021), we include both category mention [Men]y and its definition [Def]y from the annotation guideline (or Wikipedia if the guideline is not accessible) in the query to introduce more comprehensive semantics:
QSUBy = Highlight the parts (if any)
related to [Men]y. Details : [Def]y. (1)
Given the input text X as the context, the answers to QSUBy are the spans belonging to category y. Then we can obtain one MRC example denoted as (QSUBy , X , {xk | xk ∈ X, yk = y}Kk=1). To guarantee the identification of all possible spans, we create |Y | training examples by querying the input text with each pre-defined category.

heading: 3.1.2 PR-based training data
text: To construct augmented data that derived from the PR relation, we first create a category-wise span set Sy that includes all training spans with category y:
Sy = {xk | (xk, yk) ∈ DSUB, yk = y} (2)
Obviously, any two different spans in Sy have the same category and shall hold the PR relation. Therefore, we pair every two different spans in Sy to create a peer set Py:
Py = {(xq,xa) | xq,xa ∈ Sy,xq ̸= xa} (3)
For each PR pair (xq,xa) in Py, we can construct one training example by constructing the query with the first span xq:
QPRy = Highlight the parts (if any)
similar to x q . (4)
Then we treat the text Xa containing the second span xa as the MRC context to be queried and xa as the answer to QPRy . Note that there may exist more than one span in Xa satisfying PR relation with xq, we set all of them as the valid answers to QPRy , yielding one training example (Q PR y , X
a, {xak | xak ∈ Xa, yak = y}Kk=1) of our PeerDA.
Theoretically, given the span set Sy, there are only |Sy| SUB pairs in the training data but we can obtain |Sy| × (|Sy| − 1) PR pairs to construct DPR. Such a large number of augmented data shall hold great potential to enrich spans’ semantics. However, putting all PR-based examples into training would exacerbate the skewed data distribution issue since the long-tailed categories get fewer PR pairs for augmentation and also increase the training cost. Therefore, as the first step for DA with the PR relation, we propose three augmentation strategies to control the size and distribution of augmented data.
PeerDA-Size: This is to increase the size of augmented data while keeping the data distribution unchanged. Specifically, for each category y, we randomly sample λ|Sy| PR pairs from Py. Then we collect all sampled PR pairs to construct DPR, where λ is the DA rate to control the size of DPR. PeerDA-Categ: Categories are not evenly distributed in the training data, and in general SpanID models perform poorly on long-tailed categories. To tackle this, we propose PeerDA-Categ to augment more training data for long-tailed categories. Specifically, let y∗ denote the category having the largest span set of size |Sy∗ |. We sample up to |Sy∗ | − |Sy| PR pairs from Py for each category y and construct a category-balanced training set DPR using all sampled pairs. Except for the extreme cases where |Sy| is smaller than √ |Sy∗ |, we would get the same size of the training data for each category after the augmentation, which significantly increases the exposure for spans from the long-tailed categories.
PeerDA-Both (The final version of PeerDA): To take advantage of the above two strategies, we further propose PeerDA-Both to maintain the data distribution while effectively increasing the size of
training data. In PeerDA-Both, we randomly sample max(λ|Sy∗ |+ (|Sy∗ | − |Sy|), 0) PR pairs from Py for each category y to construct DPR, where λ|Sy∗ | determines the size of the augmented data, and |Sy∗ | − |Sy| controls the data distribution.

heading: 3.1.3 Data Balance
text: We combine the DSUB and the DPR created above as the final training data. Since an input text usually mentions spans from a few categories, when converting the text into the MRC paradigm, many of the |Y | examples are unanswerable. If a SpanID model is trained on this unbalanced data, then the model may favor the majority of the training examples and output an empty span. To balance answerable and unanswerable examples, we follow Hendrycks et al. (2021) to randomly remove some unanswerable examples from the training data.

heading: 3.2 Model Architecture
text: As shown in Figure 2, to achieve the detection of multiple spans for the given query, we follow Li et al. (2020b) to build the MRC model. Compared to the original designs, we further optimize the computation of span scores following a general way of Luong et al. (2015); Xu et al. (2022b).
Specifically, the base model consists of three components: an encoder, a span predictor, and a start-end selector. First, given the concatenation of the query Q and the context X as the MRC input X = {[CLS],Q, [SEP],X, [SEP]}, where [CLS], [SEP] are special tokens, the encoder would encode the input text into hidden states H:
H = ENCODER(X) (5)
Second, the span predictor consists of two binary classifiers, one to predict whether each context token is the start index of the answer, and the other to predict whether the token is the end index:
Pstart = HW s Pend = HW e (6)
where W s,W e ∈ Rd×2 are the weights of two classifiers and d is the dimension of hidden states. The span predictor would output multiple start and end indexes for the given query and context.
Third, the start-end selector matches each start index to each end index and selects the most possible spans from all combinations as the outputs.
Different from the concat way that would create a large R|X|×|X|×2d-shape tensor (Li et al., 2020b), we leverage a general way following Luong et al. (2015); Xu et al. (2022b) to compute the span score, consuming fewer resources for better training efficiency:
Ps,e = FFN(Hs)THe (7)
where FFN is the feed-forward network (Vaswani et al., 2017), Ps,e denotes the likelihood of Xs:e to form a possible answer.

heading: 3.3 Training Objective
text: The standard objective is to minimize the crossentropy loss (CE) between above three predictions and their corresponding ground-truth labels, i.e., Ystart, Yend, Ys,e (Li et al., 2020b):
Lmrc = CE(σ(Pstart), Ystart) + CE(σ(Pend), Yend) + CE(σ(Ps,e), Ys,e) (8)
where σ is the sigmoid function. However, these objectives only capture the semantic similarity between the query and positive spans (i.e., the span instances of the query category). In this paper, we propose to explicitly separate the query and its negative spans (i.e., the span instances of other categories) apart with a marginbased contrastive learning strategy, for better distinguishing the spans from different categories.
Specifically, given the MRC input X with query of category y, there may be multiple positive spans X+ = {xk ∈ X, yk = y} and negative spans X− = {xk′ ∈ X, yk′ ̸= y}. We leverage the following margin-based contrastive loss to penalize negative spans (Chechik et al., 2010):
Lct = max xk∈X+
x k′∈X
−
max(0,M − (σ(Psk,ek )− σ(Psk′ ,ek′ )))
(9)
where M is the margin term, max(·, ·) is to select the larger one from two candidates, and the span score Psk,ek can be regarded as the semantic similarity between the query and the target span xk. Note that our contrastive loss maximizes the
similarity difference between the query and the most confusing positive and negative span pairs (Max-Min), which we demonstrate to be effective in Sec. 5.3.
Finally, the overall training objective is:
L = Lmrc + αLct (10)
where α is the balance rate.

heading: 4 Experimental Setup

heading: 4.1 Tasks
text: Note that PeerDA is a method for augmenting training data, it is not applied to test sets during evaluation. Therefore, we only use the SUB-based test data to evaluate the models’ capability to extract spans according to the category. We conduct experiments on four SpanID tasks from diverse domains, including NER, ABSA, Contract Clause Extraction (CCE), and Span-Based Propaganda Detection (SBPD). The dataset statistics are summarized in Table 1. The detailed task description can be found in Appendix A.1.
NER: It is to detect named entities (i.e. spans) and classify them into entity types (i.e. categories). We evaluate five datasets, including four English datasets: OntoNotes52 (Pradhan et al., 2013), WNUT17 (Derczynski et al., 2017), Movie (Liu et al., 2013b), and Restaurant (Liu et al., 2013a) and a Chinese dataset Weibo (Peng and Dredze, 2015). We use span-level micro-averaged Precision, Recall, and F1 as evaluation metrics.
ABSA: We explore two ABSA sub-tasks: Aspect Term Extraction (ATE) to only extract aspect terms, and Unified Aspect Based Sentiment Analysis (UABSA) to jointly identify aspect terms and their sentiment polarities. We evaluate the two sub-tasks on two datasets, including the laptop domain Lap14 and restaurant domain Rest14. We use micro-averaged F1 as the evaluation metric.
2In order to conduct robustness experiments in Sec. A.3, we use the datasets from Lin et al. (2021) with 11 entity types.
SBPD: It aims to detect both the text fragment where a persuasion technique is used (i.e. spans) and its technique type (i.e. categories). We use News20 and Social21 from SemEval shared tasks (Da San Martino et al., 2020; Dimitrov et al., 2021). For News20, we report the results on its dev set since the test set is not publicly available. We use micro-averaged Precision, Recall, and F1 as evaluation metrics.
CCE: It is a legal task to detect and classify contract clauses (i.e. spans) into relevant clause types (i.e. categories), such as "Governing Law". We conduct CCE experiments using CUAD (Hendrycks et al., 2021). We follow Hendrycks et al. (2021) to use Area Under the Precision-Recall Curve (AUPR) and Precision at 80% Recall (P@0.8R) as the evaluation metrics.

heading: 4.2 Implementations
text: Since legal SpanID tasks have a lower tolerance for missing important spans, we do not include start-end selector (i.e. CE(Ps,e, Ys,e) and αLct in Eq. (10)) in the CCE models but follow Hendrycks et al. (2021) to output top 20 spans from span predictor for each input example in order to extract spans as much as possible. While for NER, ABSA, and SBPD, we use our optimized architecture and objective.
For a fair comparison with existing works, our models utilize BERT (Devlin et al., 2019) as the text encoder for ABSA and RoBERTa (Liu et al., 2019) for NER, CCE, and SBPD. Detailed configurations can be found in Appendix A.2.

heading: 4.3 Baselines
text: Note that our main contribution is to provide a new perspective to treat the PR relation as a kind of training data for augmentation. Therefore, we
compare with models built on the same encoderonly PLMs (Devlin et al., 2019; Liu et al., 2019). We are not focusing on pushing the SOTA results to new heights though some of the baselines already achieved SOTA performance.
NER: We compare with Tagging (Liu et al., 2019) and MRC (Li et al., 2020b) baselines. We also report the previous best approaches for each dataset, including RB-CRF+RM (Lin et al., 2021), CL-KL (Wang et al., 2021), T-NER (Ushio and Camacho-Collados, 2021) KaNa (Nie et al., 2021), and RoBERTa+BS (Zhu and Li, 2022).
ABSA: In addition to the MRC baseline, we also compare with previous approaches on top of BERT. These are SPAN-BERT (Hu et al., 2019), IMNBERT (He et al., 2019), RACL (Chen and Qian, 2020) and Dual-MRC (Mao et al., 2021).
SBPD: For News20 we only compare with MRC baseline due to the lack of related work. For Social21, we compare with top three approaches on its leaderboard, namely, Volta (Gupta et al., 2021), HOMADOS (Kaczyński and Przybyła, 2021), and TeamFPAI (Hou et al., 2021).
CCE: We compare with (1) MRC basline, (2) stronger text encoders, including ALBERT (Lan et al., 2019) and DeBERTa (He et al., 2020), (3) the model continually pretrained on contracts: RoBERTa + CP (Hendrycks et al., 2021) and (4) the model leveraged the contract structure: ConReader (Xu et al., 2022a).

heading: 5 Results

heading: 5.1 Comparison Results
text: NER: Table 2 shows the performance on five NER datasets. Our PeerDA significantly out-
performs the Tagging and MRC baselines. Precisely, compared to RoBERTabase MRC, PeerDA obtains 0.3, 6.0, 3.2, 1.5, and 2.9 F1 gains on five datasets respectively. When implemented on RoBERTalarge, our PeerDA can further boost the performance and establishes new SOTA on three datasets, namely, OntoNotes5, Movie, and Restaurant. Note that the major improvement of PeerDA over MRC comes from higher Recall. It implies that PeerDA encourages models to give more span predictions.
ABSA: Table 3 depicts the results on ABSA. Compared to previous approaches, PeerDA mostly achieves better results on two subtasks, where it outperforms vanilla MRC by 2.7 and 1.0 F1 on UABSA for two domains respectively.
SBPD: The results of two SBPD tasks are presented in Table 4. PeerDA outperforms MRC by 8.2 and 9.2 F1 and achieves SOTA performance on News20 and Social21 respectively.
CCE: The results of CCE are shown in Table 5. PeerDA surpasses MRC by 8.7 AUPR and 13.3 P@0.8R and even surpasses the previous best model of larger size (ConReaderlarge) by 3.2 AUPR, reaching SOTA performance on CUAD.

heading: 5.2 Analysis on Augmentation Strategies
text: To explore how the size and category distribution of the augmented data affect the SpanID tasks, we conduct ablation study on the three augmen-
tation strategies mentioned in Sec. 3.1.2, depicted in Table 6. Overall, all of the PeerDA variants are clearly superior to the MRC baseline and the PeerDA-both considering both data size and distribution issues performs the best. Another interesting finding is that PeerDA-Categ significantly outperforms PeerDA-Size on SBPD and CCE. We attribute the phenomenon to the fact that SBPD and CCE have a larger number of categories and consequently, the MRC model is more prone to the issue of skewed data distribution. Under this circumstance, PeerDA-Categ, the variant designed for compensating the long-tailed categories, can bring larger performance gains over MRC model. On the other hand, if the skewed data distribution is not severe (e.g. NER), or the category shows a weak correlation with the spans (i.e. UABSA), PeerDASize is more appropriate than PeerDA-Categ.

heading: 5.3 Analysis on Model Designs
text: Calculation of Ps,e (Top part of Table 7) Under the same experimental setup (RoBERTabase, batch size=32, sequence length=192, fp16), using our general method (Eq. (7)) to compute span score Ps,e greatly reduces the memory footprint by more than 4 times with no performance drop, compared to the original concat method. Therefore, our general method allows a larger batch size for accelerating the training.
Contrastive Loss (Bottom part of Table 7) After we have settled on the general scoring function, we further investigate different methods to compute
contrastive loss. We find that the Average method, which averages similarity differences between the query and all pairs of positive and negative spans, would affect SpanID performance when the task has more long-tailed categories (i.e. SBPD). While our Max-Min (strategy in Eq.(9)) is a relaxed regularization, which empirically is more suitable for SpanID tasks and consistently performs better than the Average method.

heading: 6 Further Discussions
text: In this section, we make further discussions to bring valuable insights of our PeerDA approach.
Out-of-domain Evaluation: We conduct out-ofdomain evaluation on four English NER datasets, where the model is trained on OntoNotes5, the
largest dataset among them, and evaluated on the test part of another three datasets. Since these four datasets are from different domains and differ substantially in their categories, this setting largely eliminates the impact of superficial span-category patterns and thus it can faithfully reflect how well the MRC model exploits span semantics for prediction. The results are presented in Table 8. PeerDA can significantly exceed MRC on all three transfer pairs. On average, PeerDA achieves 2.9 and 4.1 F1 gains over base-size MRC and large-size MRC respectively. These results verify our postulation that modeling the PR relation allows models to weigh more on the semantics for making predictions, and thus mitigates the over-fitting issue.
Semantic Distance: To gain a deeper understanding of the way in which PeerDA enhances model performance, we consider the span score (Eq. 7) as a measure of semantic similarity between a query and a span. In this context, we can create queries for all categories and visualize the similarity distribution between the categories and their corresponding positive and negative spans on Ontonote5 test set. As shown in Figure 4, we can observe that
the use of PeerDA leads to an increased semantic similarity between spans and their corresponding categories, resulting in higher confidence in the prediction of correct spans. Furthermore, PeerDA has been shown to also create a larger similarity gap between positive and negative spans, facilitating their distinction.
Low-resource Evaluation: We simulate lowresource scenarios by randomly selecting 10%, 30%, 50%, and 100% of the training data for training SpanID models and show the comparison results between PeerDA and MRC on four SpanID tasks in Figure 3. As can be seen, our PeerDA further enhances the MRC model in all sizes of training data and the overall trends are consistent across the above four tasks. When training PeerDA with 50% of the training data, it can reach or even exceed the performance of MRC trained on the full training set. These results demonstrate the effectiveness of our PeerDA in low-resource scenarios.
Error Analysis: In order to know the typical failure of PeerDA, we randomly sample 100 error cases from Ontonotes5 test set for analysis. As shown in Table 9, there are four major groups:
• Multiple Labels: PeerDA would assign multiple labels to the same detected span. And in most cases (35/41), this error occurs among similar categories, such as LOC, GPE, and ORG. • Incorrect Label: Although spans are correctly detected, PeerDA assigns them the wrong categories. Note that MRC even cannot detect many of those spans (23/37). As a result, PeerDA sig-
nificantly improves the model’s capability to detect spans, but still faces challenges in category classification. • Missing Prediction: Compared to MRC, PeerDA tends to predict more spans. Therefore it alleviates the missing prediction issue that MRC mostly suffers. • Other Errors: There are several other errors, such as the incorrect span boundary caused by articles or nested entities.

heading: 7 Conclusions
text: In this paper, we propose a novel PeerDA approach for SpanID tasks to augment training data from the perspective of capturing the PR relation. PeerDA has two unique advantages: (1) It is capable to leverage abundant but previously unused PR relation as additional training data. (2) It alleviates the over-fitting issue of MRC models by pushing the models to weigh more on semantics. We conduct extensive experiments to verify the effectiveness of PeerDA. Further in-depth analyses demonstrate that the improvement of PeerDA comes from a better semantic understanding capability.
Limitations
In this section, we discuss the limitations of this work as follows:
• PeerDA leverages labeled spans in the existing training set to conduct data augmentation. This means that PeerDA improves the semantics learning of existing labeled spans, but is ineffective
to classify other spans outside the training set. Therefore, it would be beneficial to engage outer source knowledge (e.g. Wikipedia), where a variety of important entities and text spans can also be better learned with our PeerDA approach. • Since PeerDA is designed in the MRC formulation on top of the encoder-only Pre-trained Language Models (PLMs) (Devlin et al., 2019; Liu et al., 2019), it is not comparable with other methods built on encoder-decoder PLMs (Yan et al., 2021b; Chen et al., 2022; Zhang et al., 2021b; Yan et al., 2021a). It would be of great value to try PeerDA on encoder-decoder PLMs such as BART (Lewis et al., 2020) and T5 (Raffel et al., 2020), to see whether PeerDA is a general approach regardless of model architecture. • As shown in Table 9, although PeerDA can significantly alleviate the Missing Predictions, the most prevailing error in the MRC model, PeerDA also introduces some new errors, i.e. Multiple labels and Incorrect Label. It should be noted that those problematic spans are usually observed in different span sets, where they would learn different category semantics from their peers. Therefore, we speculate that those spans tend to leverage the learned category semantics more than their context information to determine their categories. We hope such finding can shed light on future research to further improve PeerDA.

heading: A Appendix
text: A.1 Task Overview We conduct experiments on four SpanID tasks with diverse domains, including Named Entity Recognition (NER), Aspect Based Sentiment Analysis (ABSA), Contract Clause Extraction (CCE) and Span Based Propaganda Detection (SBPD), to show the overall effectiveness of our PeerDA. The dataset statistics are summarized in Table 1.
NER: It is a traditional SpanID task, where spans denote the named entities in the input text and category labels denote their associated entity types. We evaluate five datasets from four domains:
• OntoNotes5 (Pradhan et al., 2013) is a largescale mixed domain NER dataset covering News, Blog and Dialogue. To make a fair comparison in the robustness experiments in Sec. A.3, we use the datasets from Lin et al. (2021), which only add adversarial attack to the 11 entity types, while leaving out 7 numerical types.
• WNUT17 (Derczynski et al., 2017) is a benchmark NER dataset in social media domain. For fair comparison, we follow the data preprocessing protocols in Nie et al. (2020).
• Movie (Liu et al., 2013b) is a movie domain dataset containing movie queries, where long spans are annotated such as a movie’s origin or plot. We use the defaulted data split strategy into train, test sets.
• Restaurant (Liu et al., 2013a) contains queries in restaurant domain. Similar to Movie, we use the defaulted data split strategy.
• Weibo (Peng and Dredze, 2015) is a Chinese benchmark NER dataset in social media domain. We exactly follow the official data split strategy into train, dev and test sets.
ABSA: It is a fine-grained sentiment analysis task centering on aspect terms (Zhang et al., 2022). We explore two ABSA sub-tasks:
• Aspect Term Extraction (ATE) is to extract aspect terms, where there is only one query asking if there are any aspect terms in the input text.
• Unified Aspect Based Sentiment Analysis (UABSA) is to jointly extract aspect terms and predict their sentiment polarities. We formulate it
as a SpanID task by treating the sentiment polarities, namely, positive, negative, and neutral, as three category labels, and aspect terms as spans.
We evaluate the two sub-tasks on two datasets, including the laptop domain dataset Lap14 and restaurant domain dataset Rest14 from SemEval Shared tasks (Pontiki et al., 2014). We use the processed data from Zhang et al. (2021b).
CCE: It is a legal NLP task to detect and classify contract clauses into relevant clause types, such as "Governing Law" and "Uncapped Liability". The goal of CCE is to reduce the labor of legal professionals in reviewing contracts of dozens or hundreds of pages long. CCE is also a kind of SpanID task where spans are those contract clauses that warrant review or analysis and labels are predefined clause types. We conduct experiments on CCE using CUAD (Hendrycks et al., 2021), where they annotate contracts from Electronic Data Gathering, Analysis and Retrieval (EDGAR) with 41 clause types. We follow Hendrycks et al. (2021) to split the contracts into segments within the length limitation of pretrained language models and treat each individual segment as one example. We also follow their data split strategy.
SBPD: It is a typical SpanID task that aims to detect both the text fragment (i.e. spans) where a persuasion technique is being used as well as its technique type (i.e. category labels). We use the News20 and Social21 from two SemEval shared tasks (Da San Martino et al., 2020; Dimitrov et al., 2021) and follow the official data split strategy. Note that News20 does not provide the golden label for the test set. Therefore, we evaluate News20 on the dev set.
A.2 Implementations
We use Huggingface’s implementations of BERT and RoBERTa (Wolf et al., 2020) 3. The hyperparameters can be found in Table 10. We use Tesla V100 GPU cards for conducting all the experiments. We follow the default learning rate schedule and dropout settings used in BERT. We use AdamW (Loshchilov and Hutter, 2019) as our optimizer. The margin term M is set to 0 for NER and ABSA, and 1 for SBPD. The balance rate α is set to 0.1.
3Chinese RoBERTa is from https://github.com/ymcui/ Chinese-BERT-wwm.
A.3 Robustness
To verify the advantage of PeerDA against the adversarial attack, we conduct robustness experiments using the adversarial dev set of OntoNotes5 (Lin et al., 2021) on NER and adversarial test set of Lap14 (Xing et al., 2020) on UABSA. Table 11 shows the performance on the original and the adversarial sets. On OntoNotes5 full adversarial set, PeerDA improves the robustness of the model compared to MRC but slightly degrades compared to Tagging. To investigate why this happens, we evaluate each type of adversarial attack independently, including entity attack that replaces entities to other entities not presented in the training set and context attack that replaces the context of entities. It shows that PeerDA does not work well on entity attack because we only use entities in the training set to conduct data augmentation, which
is intrinsically ineffective to this adversarial attack. This motivates us to engage outer source knowledge (e.g. Wikipedia) into our PeerDA approach in future work. On Lap14, PeerDA significantly improves Tagging and MRC by 5.6 and 3.2 F1 on the adversarial set respectively.
A.4 Peer-driven DA
We compare PeerDA with Mention Replacement (MenReplace) (Dai and Adel, 2020), another Peerdriven DA approach randomly replaces a span mention in the context with another mention of the same category in the training set. The results of four SpanID tasks are presented in Table 12. PeerDA exhibits better performance than MenReplace on all four tasks. In addition, MenReplace would easily break the text coherence as a result of putting span mentions into the incompatible context, while PeerDA can do a more natural augmentation without harming the context.
A.5 Effect of DA Rate We vary the DA rate λ to investigate how the volume of PR-based training data affect the SpanID models performance.
Figure 5 shows the effect of different λ in four SpanID tasks. PeerDA mostly improves the MRC in all different trials of λ and we suggest that some parameter tuning for λ is beneficial to obtain optimal results.
Another observation is that too large λ would do harm to the performance. Especially on CCE, due to the skewed distribution and a large number of categories, PeerDA can produce a huge size of PRbased training data. We speculate that too much PR-based training data would affect the learning of BL-based training data and thus affect the model’s ability to solve a SpanID task, causing the optimal λ to be a negative value. In addition, too much PRbased training data would also increase the training cost. As a result, we should maintain an appropriate ratio of BL-based and PR-based training data to keep a reasonable performance on SpanID tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract, Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4, Appendix A.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4, Appendix A.2, Appendix A.5

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4, Section A.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope such finding can shed light on future research to further improve PeerDA