SynRE: Two-stage Self-training with Synthetic data for Low-resource Relation Extraction
Year: 2023

Abstract:
Current relation extraction methods suffer from the inadequacy of large-scale annotated data. While distant supervision alleviates the problem of data quantities, there still exists domain disparity in data qualities due to its reliance on domain-restrained knowledge bases. In this work, we propose SynRE, a framework of two-stage Self-training with Synthetic data for Relation Extraction. We first leverage the capability of large language models to adapt to the target domain and automatically synthesize large quantities of coherent, realistic training data. We then propose an accompanied two-stage self-training algorithm that iteratively and alternately learns from synthetic and golden data together. We conduct comprehensive experiments and detailed ablations on popular relation extraction datasets to demonstrate the effectiveness of the proposed framework. Code is available at https: //github.com/BenfengXu/S2ynRE.

heading: 1 Introduction
text: Relation extraction systems aim at discovering relational knowledge between entities by reading from unrestricted texts (Cardie, 1997). Although neural methods, especially pre-trained language models, have greatly advanced the state-of-the-art relation extraction capability (Zeng et al., 2014; Wu and He, 2019), they still require large quantities of training data (Han et al., 2020). However, high-quality annotations are usually very expensive to obtain, making low-resource relation extraction a very practical challenge in many real-world scenarios.
Distant supervision (Mintz et al., 2009), which automatically annotates relational statements by aligning entities with an existing knowledge bases (Bollacker et al., 2008; Vrandečić and Krötzsch, 2014), has been widely explored as an effective way to construct large scale relational
∗Joint work of USTC and Baidu Inc. †Corresponding author: Zhendong Mao
dataset. Many recent works exploit such data in a pretraining stage to learn relational representations (Baldini Soares et al., 2019; Peng et al., 2020; Qin et al., 2021). Although this line of methods have seen certain improvements, they still inevitably raise the concern that the distantly annotated data can vary considerably from downstream tasks both in target schema and in context distributions, thus may not be able to offer optimal transferability. For instance, due to the reliance on existing knowledge bases, current works mostly resort to Wikidata as the source of relational triples and Wikipedia (Vrandečić and Krötzsch, 2014) as the corpus for distant supervision. This circumscribes distant data to only factual knowledge between world entities, while downstream tasks may be of other special interests involving various domains, ranging from semantic relation between nominals (Hendrickx et al., 2009) to chemicalprotein interactions (Kringelum et al., 2016).
Meanwhile, recent advances in large-scale pretrained language models (LLM) (Radford et al., 2019; Brown et al., 2020; Raffel et al., 2020) have demonstrated their great potential in generating realistic texts of various domains (Radford et al., 2019). Accordingly, several very recent works have explored the possibility to exploit LLM as an alternative training data pool (Schick and Schütze, 2021; Vu et al., 2021). However, these attempts are confined to NLI task, while still not effectively explored in the area of relation extraction.
In this paper, we study the construction of synthetic data for relation extraction tasks to simultaneously address both training data scarcity in low resource scenarios and domain disparity in distant supervision. We employ LLM to estimate and adapt to the target domain distribution with only a few training instances, and synthesize a large amount of ones accordingly. The procedure is overall very simple but also carefully designed with two critical choices: 1) we linearize relational state-
8186
ments into natural language sequences where entity pairs are indicated by special marker tokens; 2) we resort to unconditional generation instead of label-conditioned ones, which relaxes the requirements for strict label-semantic correspondence but increases sample availability and diversity.
We experiment with both GPT2 and the recent very large LLMs like GPT-3.5. For standard size generative LMs like GPT2-Large, we first finetune it to adapt to the target domain, while for the capable GPT-3.5 model, we directly apply In-Context Learning (Brown et al., 2020; Xu et al., 2023). We empirically found finetuned GPT2-Large produces synthetic data of equivalent quality to prompted GPT-3.5. In general, it is observed that with only a few accessible samples, we are able to successfully synthesize a large amount of domain-customized training data with satisfactory quality.
To effectively learn from such synthetic data, we novelly advocate a two-stage self-training algorithm. The approach in general follows the selftraining framework (Yarowsky, 1995; Xie et al., 2020), which is widely employed to exploit unlabeled data. Typically, such methods iteratively annotate and learn pseudo labels for unlabeled data to bootstrap the model’s performance. Distinctively, we make a two-stage adaptation where in each of the iterations, the model is firstly trained on synthetic instances, then on golden ones. Such sequential training procedure favors golden data with more importance since they are introduced in the latter stage of the training curriculum.
We refer to our method as S2ynRE, a framework of two-stage Self-training with Synthetic data for Relation Extraction. The contributions of this paper is three-fold:
• Conceptual Contribution We exploit LLM to generate large quantities of domain adaptive synthetic data for low-resource relation extraction, and challenge the long-prevailing distant supervised methods restricted by KB domain coverages. The proposed solution novelly mitigates the problems of both data scarcity and domain disparity.
• Technical Contribution We propose a novel two-stage self-training algorithm to effectively learn from unlabeled synthetic data and golden data together. We demonstrate that this is a non-trivial adaptation that significantly outperforms standard self-training widely employed in semi-supervised learning.
• Experimental Contribution We conduct comprehensive experiments on 6 popular relation extraction datasets to investigate, analyze the propose method and make comparisons. We achieve new state-of-the-art for low-resource relation extraction. Compared to standard finetuning baseline, we obtain up to 17.18% absolute improvements, and 11.09% on average across all datasets.

heading: 2 Related Works
text: Relation Extraction Relation extraction is one of the fundamental tasks in natural language processing (Cardie, 1997), where lots of research efforts have been made to advance the state-of-theart methods (Zeng et al., 2014; Zhou et al., 2016; Zhang et al., 2018; Baldini Soares et al., 2019), as well as the low-resource scenario (Han et al., 2018; Sainz et al., 2021; Dong et al., 2021; Chen et al., 2022). One of the most prominent methods is distant supervision (Mintz et al., 2009), which automatically constructs annotated relational data by aligning corpus with existing knowledge base. Many recent works investigate how to learn effectively with such distant data (Baldini Soares et al., 2019; Peng et al., 2020; Ding et al., 2021; Qin et al., 2021). Generally, they propose various pretext tasks that pre-train a model to learn relational representation. We will further explain some of these works for comparison in Section 5.3.
Learning from Synthetic Data Built upon massive corpora, pre-trained language models are promising at producing texts of eligible quality, resulting in a surge of research interests in its usage for data augmentation (Feng et al., 2021). One straightforward way is to introduce mask corruptions in the way language models are pretrained, then collect predictions as augmented data (Kobayashi, 2018; Ng et al., 2020). Later works further developed such technique into conditional augmentation (Wu et al., 2019; Kumar et al., 2020). Nevertheless, these methods are mostly editing existing instances, which limits the diversity and scale of augmented data.
With increasingly powerful LLMs, recent works turn to direct synthesis of new instances (Schick and Schütze, 2021; Wang et al., 2021; Meng et al., 2022; Ye et al., 2022). Different from this work, most of them focus on zero-shot language understanding where no labeled data is available (Schick and Schütze, 2021; Wang et al., 2021; Meng et al.,
2022; Ye et al., 2022). They investigate ways to generate label-conditioned data by prompting LLMs, but these methods can hardly be applied to low-resource or full data scenarios while still preserving effectiveness.
With the existence of labeled data, synthetic data needs to be of higher quality to bring further utility. Several works thus propose to finetune the generator (Anaby-Tavor et al., 2020; Vu et al., 2021; He et al., 2021). There are also explorations for learning from synthetic and golden data together, including threshold-based confidence filtering (AnabyTavor et al., 2020), classical semi-supervised learning (He et al., 2021) or restricting the usage of synthetic data within a supplemental intermediate task (Vu et al., 2021).
For structured learning tasks, Ding et al. (2020) similarly formulates NER task data as sequential language. Specifically for relational data synthesis, Papanikolaou and Pierleoni (2020) explore the biomedical domain and Chia et al. (2022) focus on zero-shot setting of triplet extraction. By contrast, Syn2RE distinguishes not only in applied scenario and synthesis strategy, but also in the two-stage learning framework, which is specially designed for improved synthetic data adaptation.

heading: 3 Preliminary
text: This section formulates the task of relation extraction and the baseline models used throughout all experiments.
Task Formulation A typical relation extraction task is defined by a corpus of relational statements and a set of relations, i.e., schema S. Assume the training dataset Dtr = {(xi, si, oi)}Ni=1 and its corresponding labels Y tr = {yi}Ni=1, where xi is a sequence of words {wil}Ll=1, yi ∈ S, si = [wsstart : wsend ] and oi = [wostart : woend ] are subject and object entities within the context. The target is to learn a function fθ(xi, si, oi) that predicts the correct relation label yi.
Baseline Model As S2ynRE is a data-centric framework, we keep the model architecture simple but competitive, which is the vanilla finetuning of pre-trained language models. Instead of autoregressive LMs, we use auto-encoding networks like BERT as they usually perform better on language understanding downstream tasks. Following Baldini Soares et al.’s (2019) comprehensive study of building relation extractors, we inject spe-
cial marker tokens to the input word sequence:
xmarked = (..., [Sub], s, [\Sub], ...
..., [Obj], o, [\Obj], ...) (1)
After the encoding process of transformer, the representation h in corresponding positions will be concatenated for classification:
ŷ = softmax(W|S|[h[Sub];h[Obj]]) (2)
where W |S| is a feedforward network and the predicted categorical distribution ŷ will be trained against y using cross-entropy loss.

heading: 4 Methodology

heading: 4.1 Relational Data Synthesis
text: Training instances of relation extraction task is of specific structure (xi, si, oi), i.e., the relational statement is expected to be a sentence containing exact two entities as subject and object. Inspired by Paolini et al. (2021), we linearize relational data into marked natural language sequence as in Eq 1. The synthesizer can be built upon any existing LLMs. In this paper, we explore both GPT2 and the even larger GPT-3.5 as two representative LLMs and respectively employ finetuning or incontext learning treatment.

heading: 4.1.1 Finetuning for GPT2
text: The finetuning process is performed in the same autoregressive way as how it is pre-trained:
L = − L+4∑
l=1
logP (wl|w0, ..., wl−1;LLM) (3)
where {wl} = xmarked, and a <bos> token is prepended as w0. Note that we ignore relation labels y in training data and approach it as unconditional generation. This eliminates the noise caused by label-semantic inconsistency, and leaves it to model itself to learn from unlabeled synthetic data.
After the finetuning is completed, we simply prepend the <bos> token to prompt the generation, and repeatedly perform inference using multinomial sampling until we obtain the expected scale of synthetic data Dsyn. We show in appendix G that these synthetic data are coherent, realistic, and most importantly, customized to the target domain.
We elaborate on the framework of S2ynRE (see Fig. 2) in this section, including the construction of an LLM-based synthesizer, and the two-stage self-training algorithm.

heading: 4.1.2 In-Context Learning for GPT-3.5
text: Even larger LLMs naturally exhibit few-shot learning capabilities, and can be elicited through very few in-context demonstrations (Brown et al., 2020; Xu et al., 2023). We directly prepend 5-shot exemplars and provide a specific instruction asking the LLM to generate more examples. The process is illustrated in Figure 1.

heading: 4.2 Two Stage Self-training
text: Self-training is a widely adopted learning algorithm for semi-supervised learning. Typically, to jointly learn from an unlabeled dataset and a labeled dataset, it iteratively samples from the unlabeled set, assigns them with pseudo labels, merges them with the labeled dataset, and re-trains the model. In this paper, we argue that this design of naive merging is built upon a strong assumption that the unlabeled dataset must be in the exact distribution with the labeled ones, for which the synthetic data does not strictly satisfy.
In S2ynRE, differently, we make a two-stage adaptation: where synthetic data and golden data are trained sequentially (Figure 2). We start from a base model initialized using any auto-encoding language models, e.g., BERT (Devlin et al., 2019), and train it on Dtr to produce a teacher model η, as introduced in Section 3. We first use η to annotate the unlabeled synthetic data Dsyn:
ŷsyni = η(x syn i , si, oi) (4)
and we keep Ŷsyn = {ŷsyni } as soft pseudo labels of Dsyn, note that here theˆdenotes soft as we keep the categorical distribution intact instead of keeping its argmax. Inspired by Li and Qian (2021), to further eliminate fluctuations in pseudo labels, we train multiple teachers using different random seeds, and the pseudo labels annotated by k-th teacher is referred to as Ŷsynk .
We then re-initialize a new student model θ, and apply a two-stage training strategy. In stage-one training, student θ is trained on synthetic data using soft pseudo labels:
θ′ ← LKD(θ,Dsyn, {Ŷsynk }Kk=1) (5) This can be seen as a distillation procedure that transfers knowledge from η to θ based on synthetic data DSyn. And LKD is calculated as:
LKD = 1
K
K∑
k=1
DKL(ŷ syn i ‖ θ(x syn i , si, oi))
(6) where DKL is the Kullback-Leibler divergence. Then in stage-two training, we take from θ′, and train it on labeled training dataset:
θ′′ ← LCE(θ′,Dtr,Y tr) (7) where LCE is the standard cross-entropy loss, and θ′′ is the resulting model in this iteration. We then use θ′′ as the teacher model η for the next iteration to re-annotate Dsyn, and this procedure is
repeated T times. Following the standard practice of self-training, in each iteration, we incrementally sample 1/T more synthetic data from Dsyn until in iteration T, where Dsyn will be running out of new instances. The entire two-stage self-training process can be formulated as Algorithm 1.
Algorithm 1: Two-stage Self-training. Input: Golden training dataset Dtr, Y tr,
synthetic dataset Dsyn /* ===== Iteration 1 ===== */ t = 1; Dsyn1 = ∅; Initialize θ from auto-encoding LM; θ1 ← Train(θ,Dtr,Y tr) ; // Eq.7 θTea1 ← θ1; // assign teacher model /* ===== Iteration 2∼T ===== */ repeat
t = t+ 1; Dsynt = Dsynt−1 ∪Dsyn[ t−1T : tT ]; Ŷsynt ← Annotate(θTeat−1 ,Dsynt ); // Eq.4 Re-initialize θ from auto-encoding LM; /* stage-one training */ θ′t ← Train(θ,Dsynt , Ŷsynt ); // Eq.5 /* stage-two training */ θ′′t ← Train(θ′t,Dtr,Y tr) ; // Eq.7 θTeat ← θ′′t ; // update teacher model
until performance converges or t reaches maximum iteration limit T; Output: Final model θ′′t

heading: 5 Experiments

heading: 5.1 Experimental Settings
text: We evaluate S2ynRE on popular datasets including SemEval 2010 Task 8 (Hendrickx et al., 2009), TACRED (Zhang et al., 2017), TACREDRevisited (Alt et al., 2020), Re-TACRED (Stoica et al., 2021), ChemProt (Kringelum et al., 2016) and Wiki80 (Han et al., 2019). Their statistics are given in Table 1 and we refer to detailed introduction in Appendix A.
For each dataset, we set three different prerequisites of resource availability. Respectively, FULL for 100% training data, LIMITED for 10% training data and FEW for 1% training data. To provide robust and convincing conclusions, we run all experiments (including ablation studies) with 5 different random seeds and report their average. With each random seed, we employ grid search to select the best model as well as the teacher model in each iteration. We use only development set for such selection, and report the corresponding test set score as the final results.
For data synthesis, we use GPT2-Large and GPT3.5 as the aforementioned LLMs. Specifically, for ChemProt, we use an adapted version of GPT-2 (Pa-
panikolaou and Pierleoni, 2020), which is further trained on 500k PubMed abstracts. When generating, we restrict sequence length to 128, and perform necessary filtering by removing instances that do not conform with the relational structure, i.e., there must exist 4 exact special markers and each start position marker shall appear before its end position marker. The synthesis efficiency is 24.05 instances per second before any filtering. In total, we collect 10,000 samples for FEW setting, and 100,000 synthetic samples for LIMITED and FULL settings. We leave other hyper-parameters to Appendix B.

heading: 5.2 Capability of LLM
text: We first validate the capability of LLMs as data synthesizer and their respective treatment. Considering both affordability and model capability, we use the recently released gpt-3.5-turbo-0301 API2 as even larger LLMs in comparison with GPT2-Large. For in-context learning, we repeatedly sample 5- shot random examples from the golden training set as demonstrations, followed by an instruction that asks LLMs to generate more, with domain, format and diversity constraints. We sent 1,321 queries in total to collect 10,000 synthetic data (the same as our previous experimental settings using GPT-2). Each query produces different synthetic results due to LLM sampling strategy and varied selection and permutation of in-context demonstrations.
The results are shown in Table 2. We empirically found finetuned GPT2-Large produces synthetic data of equivalent quality to prompted GPT-3.5, while both are effective training data synthesizers that significantly outperforms baselines. In the following experiments, we use GPT2-Large to further verify the proposed S2ynRE framework.

heading: 5.3 Main Results
text: We choose competitive baselines and reproduce them under comparable settings to provide more reliable conclusions. These baseline methods are: BERT We finetune BERT model (Devlin et al.,
2https://openai.com/blog/introducing-chatgptand-whisper-apis
2019) in a straightforward way for relation extraction as explained in Section 3 and implemented in many existing works. This serves as our reimplemented Finetune Baseline and will be referred to in the following figures.
MTB (Baldini Soares et al., 2019) pre-trains a relational encoder using matching the blanks task, which is built on the hypothesis that two relational statements containing the same entity pair should express similar relational representations. Note that this is a weaker reliance than distant supervision as it only aligns entities, and does not need relations.
CP (Peng et al., 2020) proposes a contrastive learning pretext task that encourages sentence representations with the same relation to be similar and different ones to be disparate.
ERICA (Qin et al., 2021) further extends distant supervision to document-level corpus, and design similar pretext task that discriminates relational representations across sentences.
We provide an overview of these works regarding various resource usage and requirements in Table 4. The main results are shown in Table 3. On Wiki80, we directly use distant data as they are available in the general wiki domain, we analysis the effects later in Table 5. Under all three settings across five datasets, S2ynRE outperforms the BERT finetune baseline. Specifically for the FEW setting, improvements are much more significant, respectively +17.18, +15.47, +16.86, +8.07, +5.59, and +3.34, resulting an average improvements of +11.09 across all 6 datasets. We further employ CP as a stronger base model to initialize the students, and the performances are even better. This implies that the improvements of S2ynRE are mostly orthogonal with those of the distantly pre-trained methods. In general, S2ynRECP achieves new stateof-the-art for low resource relation extraction tasks.
2We obtain MTB and CP checkpoints from https://gith ub.com/thunlp/RE-Context-or-Names and ERICA checkpoint from https://github.com/thunlp/ERICA

heading: 5.4 Ablation Study
text: We investigate the advantages of S2ynRE via comprehensive ablations. In accordance with the main claim, all experiments are conducted under the lowresource (FEW) setting unless otherwise stated.
Synthetic Data Instead of Distant Data Distant supervision has long been the prevailing solution to automatically construct relational data. We make its comparison against the proposed synthetic data in Table 5. We keep the two-stage self-training algorithm intact, only replace the synthetic data with distant data3. On 5 investigated datasets, distant data can provide appreciable improvements ranging from +2.06 to +13.25, however, synthetic data brings much more significant improvements ranging from +5.59 to +17.18, which clearly demon-
3The distant data is produced and released by Peng et al. (2020), we randomly sample 100,000 instances out of it
strates the superiority of being domain-customized for target tasks. However, on Wiki80, which very closely follows identical distribution of distant data as both are constructed using distant supervision on wikipedia and wikidata, result shows that synthetic data provides competitive improvements but no longer outperforms distant ones. This verifies the importance and advantage of domain-customized data from an opposite perspective. Nevertheless, real-world scenarios mostly involve distribution beyond the scope of wikipedia, and only the proposed synthetic approach can offer such advantage. We also provide qualitative comparisons for synthetic and distant data in Appendix G to better illustrate the discussed domain disparity.
Two Stage Self-training Typical self-training algorithms merge the pseudo-labeled data into existing labeled data in each iteration, and minimize the model’s empirical loss on a mixture of both. We refer to such classical implementation as Mixed Self-training as opposed to the proposed Twostage Self-training. Fig. 3 compares these two approaches. In each iteration ( transparent blue bar), there will be one evaluation for mixed self-training (blue curve), but two evaluation for Two-stage Selftraining (teal for stage one, Red for stage two). We observe that in stage-one training, the performance might drop a few compare to its previous iteration, however, it effectively provides a better initialization where the model can further learn from the golden data. Overall, the model can continually bootstrap its performance by learning from synthetic and golden data iteratively and alternately. While in mixed self-training, the golden data are treated equally as synthetic ones, and the model is overwhelmed by large amounts of the latter. Therefore, the improvement quickly saturates to a limited plateau. We also provide illustrations of the bootstrapping performance over iterations on other datasets in Appendix C.
Comparison Under Semi-supervised Setting Standard semi-supervised setting also investigates low-resource relation extraction by joint learning from both labeled data and unlabled data. However, they make a strong assumption of identical distribution between unlabeled data and labeled ones, and most existing works actually directly sample from the golden training data and remove the labels to
construct the unlabled set. We provide comparison with state-of-the-art methods of semi-supervised learning in Table 6 (under the LIMITED setting). Results show that 1) the proposed two-stage selftraining outperforms other semi-supervised learning algorithms, and 2) synthetic data demonstrates better or comparable performance compared to unlabled set constructed from golden training data. We attribute the latter to its domain-customized quality and unlimited large-scale quantity.
Unconditional Generation Although a lot of previous works intuitively resort to conditional synthesis, we show that this is not the optimal choice for relation extraction task. We finetune the synthesizer by prepending label-specific prompts: "write a sentence describing relation V (r): ", where V (r) is the verbalizer for each relation r and we directly use corresponding label strings, e.g., ComponentWhole(e2,e1). We synthesize each relation class proportional to its original distribution in golden dataset. As conditional generation provides already labeled data, we can directly finetune the student model instead of self-training. We still train synthetic and golden data sequentially as we empirically found it a better choice. The results show that conditional generation only brings minimum or no benefits. We attribute this to the difficulty of preserving required label semantics for highly abstractive tasks like relation extraction. As a consequence, while these extra amounts of data can still provide certain usability, they also most likely
cause considerable distractions.
Scale of Synthetic Samples Figure 4 investigates the scale of synthetic samples. The improvements are approximately increasing in log scale w.r.t. the number of synthetic samples. The best performance is reached at 10,000, after which if we keep adding more samples, the performance saturates. As the synthesis of data is a repeatedly sampling process, we think exploiting too much data will deteriorate the diversity at the same time. We verify this by evaluating its diversity using type-token ratio (Roemmele et al., 2017; Kumar et al., 2020), which is defined as the ratio of unique n-grams out of all n-grams (see Table 8). We can see that the diversity gap between synthetic and golden data is enlarged when increasing the data scale.
We also report supervised results using additional golden training data to measure the utility of synthetic data. We can achieve two conclusions: 1) the advantage of golden training data are more significant when it is scaled up (104). However, this also takes substantially expensive costs. 2) S2ynRE approximately achieves the utility of 1,000 additional annotated golden data (103), and it only costs several hours of GPU computation to produce according synthetic data (104) as needed.

heading: 6 Discussion
text: Distant supervision is the most prevalent solution for low-resource relation extraction, and also the main investigated and compared baseline in this paper. Both distant data and the proposed synthetic data can essentially be recognized as ways of data augmentation to produce sufficient number of additional data. The critical difference which determines the effectiveness lies in their consistency with golden training data, i.e., domain affinity. And in this paper, the superiority of synthetic data is both experimentally proved (Table 5) and qualitatively explained (Appendix G). In conclusion, leveraging LLM to adapt to target domain and generate synthetic data of high utility is in general an performant solution and we hope this novel perspective can further inspire future insights in many related areas that have been greatly impacted by the idea of distant supervision.

heading: 7 Conclusion
text: In this paper, we present S2ynRE, a framework of two-stage self-training with synthetic data for relation extraction. We show that synthetic data generated using LLMs can resolve data scarcity in low-resource scenarios and mitigate domain disparity compared to distant supervision. To enable effective learning from such synthetic data, we then propose a novel two-stage self-training algorithm that continually bootstraps model performance by iteratively and alternately training the synthetic and golden data together. The proposed framework brings substantial improvements and achieves new state-of-the-art for low-resource relation extraction. In the future, we expect new possibilities brought by LLMs and will further explore accompanied techniques to exploit their potential.
Ethical Considerations
Synthetic data generated by language models may involve potential ethical risks regarding fairness and bias (Bommasani et al., 2021; Blodgett et al., 2020), which results in further consideration when they are employed in downstream NLP tasks. Although the scope of this paper remains how to produce and leverage such synthetic data to improve relation extraction system, it is worth further investigation to investigate in conjunction with wellestablished methods that can measure (Nadeem et al., 2021) and mitigate (Nadeem et al., 2021; Gupta et al., 2022) such ethical risks.

heading: Acknowledgements
text: This work is supported by the National Key Research and Development Program of China No.2020AAA0109400, the National Natural Science Foundation of China under Grant 62222212, U19A0527, 62121002 and 61876223.

heading: A Datasets
text: SemEval 2010 Task 8 (Hendrickx et al., 2009) is a widely used testbed for relation extraction, the schema targets at semantic relations between pairs of nominals, which requires certain level of abstractive capabilities. TACRED (Zhang et al., 2017) is a large-scale dataset annotated using Amazon Mechanical Turk crowdsourcing. It was initially created for the TAC knowledge base population and mainly covers common relations between people, organizations, and locations based on the TAC KBP scheme. TACRED-Revisited (Alt et al., 2020) is a label-corrected version of the TACRED dataset, which motivates from the unresolved challenging cases in original TACRED dataset. ReTACRED (Stoica et al., 2021) further conducted a more comprehensive analysis and re-annotated the entire dataset. Besides, it made alternations to the schema to make it more clear and intuitive, which greatly improved the dataset quality. ChemProt (Kringelum et al., 2016) is a biodomain dataset that extracts 13 kinds of chemicalprotein interactions. It is widely used for evaluating domain-specific model capabilities (Lee et al., 2019; Beltagy et al., 2019).

heading: B Experimental Settings
text: S2ynRE involves three different training processes, respectively the finetuning of LLM, stage-one training, and stage-two training. Except for training steps or epochs, we do not exhaust further search for other hyper-parameters and set them empirically.
For the finetuning of LLM as synthesizer, we set batch size to 64, learning rate to 3e-5. We found that the quality of generated samples is sensitive to the finetuning steps. Considering that the scale of training samples varies from 73 (SemEval 1%) to 68,124 (TACRED 100%) w.r.t. different datasets and different settings, we search steps within different ranges accordingly. The final choices are listed in Table 9.
For stage-one training, we set batch size to 64, learning rate to 3e-5, and fix the training steps as 1500. We save the checkpoint from 500, 1000, and 1500 steps respectively and select the best one. For stage-two training, we set batch size to 16, learning rate to 3e-5, and the epochs are set as Table 10. These epoch settings are empirically chosen in our pilot study to obtain a competitive baseline performance. We set the number of teacher models
K in each iteration to 5 without further searching. We use bert-base-uncased to initialize the student model. All experiments are conducted on 40GB A100 machines.

heading: C Performance Over Self-training Iterations
text: We provide the performance curve w.r.t. iterations in Figure 5. It shows that the iterative training procedure following the classical self-training method is indeed effective. We simply set iteration to 10 as most of the self-training methods did and find it already a robust choice across different datasets.

heading: D Scale of Synthesizer Model
text: We test S2ynRE with a different scale LLM, i.e., GPT-Small with 117M parameters. The results in Table 11 show that even with such a small size LM, S2ynRE can still bring significant improvements. But in general, larger model unsurprisingly performs better. With the emergence and applicability of increasingly stronger LLMs, we can look forward to further advancement of relation extraction task.

heading: E Ablation on Multi-teacher Distillation
text: We provide ablation on multi-teacher distillation in Table 12, and demonstrate that the primary improvements come from the utilization of synthetic data and the proposed two-stage self-training. The choices of ablations are:
+Multi-teacher (Niave Ensemble) We train k models on the golden training set and ensemble their predicted logits. This only provides marginal benefits (+3.62, +3.13).
+Synthetic Data We incorporate synthetic data by performing knowledge distillation from multiteacher to students. This provides rather significant benefits (+6.01, +7,83).
+Two-stage Self-training We further introduce the proposed two-stage self-training method, which brings about the most remarkable improvements (+16.72, +14.55).

heading: F Effects of Domain-Augmented LLM
text: In the main results (Table 3) we have specifically used a domain-augmented version of GPT2 (GPT2PubMed) for biomedical task ChemProt. This is our initial choice of design and intuitively should
bring better performance. Here we further analysis the effects of such LLM choices in detail. Table 13 provides comparison between GPT2-PubMed and vanilla GPT2, both LLMs can effectively produce synthetic data and bring expected improvements. Nonetheless, we empirically find that vanilla GPT2 would need a bit more finetuning steps to adapt to the target domain (256 steps compared to 32 steps using GPT2-PubMed). In general, the proposed method is rather robust to choices of LLM.

heading: G Case Study
text: We provide randomly sampled case studies of synthetic data for SemEval, TACRED, and ChemProt in Table 14, 15, and 16 respectively as well as distant data in Table 17. These cases show that LLMs are capable of synthesizing coherent, realistic sentences with relational structure. Most importantly, such synthetic data are customized to target domains with various topics and styles.
Nevertheless, we also notice several limitations, especially in low-resource scenarios where it’s still challenging to get a good estimation of the target dataset distribution:
• Lack of diversity. For example, instances 2.1, 2.2, 2.3 all start with "the marmalade".
• Fragmentary structure. For example, instances 2.4 and 2.8 contain atypically lengthy object.
For pseudo labels, most of the time teacher model confidently assigns one specific label with very high probabilities (> 0.95), but for some other cases, it goes for more than one possible label, such as 1.8, 2.8, 4.1, etc. We attribute this to two possible reasons: 1) the limited capability of the teacher model to accurately recognize all relations, and 2) the imperfections of certain synthetic data, i.e., some synthetic instances do not well align with pre-defined schema and are difficult to be assigned exact relation labels. In these cases, forcing the student to learn from hard labels assigned using argmax might introduce severe noise, while the proposed knowledge distillation process using soft labels in S2ynRE can properly put these imperfect data still into usage.
For distant data, as these instances are produced from wikipedia texts, we can clearly identify that they are quite different from other downstream task data ether in content, or in relation schema. This further verifies the superiority of the proposed synthetic in-domain data qualitatively.

heading: H Potential Limitations
text: We empirically conclude two limitations for S2ynRE in the hope of inspiring more future research. On one hand, its advantages are less significant when a large amount of annotated data is available. For example, TACRED training set has 68,142 annotated instances. Under this setting, even if we add another 100,000 synthetic samples, the improvement is only +0.98 compared to +22.02 under 1% training set. This means that the quality of synthetic data, although superior to distant ones, is still not as good as golden ones. Thus they can hardly provide identical utility the same as 100,000 golden data. Nevertheless, with the development of LLMs and their powerful generation ability, we look forward to accessing higher-quality synthetic data.
On the other hand, when training data are limited to a few samples (for example, 1% setting for SemEval only includes 73 training instances), even strong LLMs like GPT-2 can not perfectly fit the structure of relational statements within a few steps of finetuning (See Appendix G for illustration of cases). Therefore, many generated sentences may not contain correct subject or object entity markers as requested and have to be discarded. In general, although the formation of marked natural language sequence proposed in this work made such structured synthesis feasible, we look forward to further improving the synthesis efficacy in future works.
4https://www.wikidata.org/wiki/Property_talk: P609
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: A1. Did you describe the limitations of your work?
Left blank.
A2. Did you discuss any potential risks of your work? Left blank.
A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
A4. Have you used AI writing assistants when working on this paper? Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.
B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Left blank.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Left blank.

heading: Future Work
text: We empirically conclude two limitations for S2ynRE in the hope of inspiring more future research. On one hand, its advantages are less significant when a large amount of annotated data is available. For example, TACRED training set has 68,142 annotated instances. Under this setting, even if we add another 100,000 synthetic samples, the improvement is only +0. 98 compared to +22. 02 under 1% training set. This means that the quality of synthetic data, although superior to distant ones, is still not as good as golden ones. Thus they can hardly provide identical utility the same as 100,000 golden data. Nevertheless, with the development of LLMs and their powerful generation ability, we look forward to accessing higher-quality synthetic data. On the other hand, when training data are limited to a few samples (for example, 1% setting for SemEval only includes 73 training instances), even strong LLMs like GPT-2 can not perfectly fit the structure of relational statements within a few steps of finetuning (See Appendix G for illustration of cases). Therefore, many generated sentences may not contain correct subject or object entity markers as requested and have to be discarded. In general, although the formation of marked natural language sequence proposed in this work made such structured synthesis feasible, we look forward to further improving the synthesis efficacy in future works. 4https://www. wikidata. org/wiki/Property_talk: P609
ACL 2023 Responsible NLP Checklist