Generating Labeled Data for Relation Extraction: A Meta Learning Approach with Joint GPT-2 Training
Year: 2023

Abstract:
Relation Extraction (RE) is the task of identifying semantic relation between real-world entities mentioned in text. Despite significant progress in RE research, a remaining challenge for RE concerns the lack of training data for data-hungry deep learning models. Cost of annotation and difficulty of the task are among hindrance to collect a large-scale RE dataset in different domains. To address this limitation, we propose a novel framework to automatically generate labeled data for RE. Our framework presents the pre-trained language model GPT-2 for data generation. In addition, to optimize the generated samples for an RE model, we introduce a meta learning approach to allow the GPT-2 model to be updated during the training process for RE. In particular, to leverage the feedback from the RE model to improve the data generation from GPT-2, we propose a novel reward function to update the GPT-2 model with REINFORCE, seeking to promote the similarity of the RE loss function’s gradients computed for generated data and a meta development set. We conduct extensive experiments on two benchmark datasets to produce state-of-the-art performance for RE.

heading: 1 Introduction
text: One of the fundamental tasks in Information Extraction (IE) involves Relation Extraction (RE) that aims to identify semantic relations between two entities mentioned in textual data. For instance, in the sentence “After XZY’s decision to move to Europe, they selected Paris as the final location for their headquarters.”, the semantic relation PARTWhole between two entity mentions “Europe” and “Paris” should be detected. An RE system can be employed to populate a knowledge base with relations among entities, provide information for question answering systems, and present facts for text summerization tools.
∗ Work done at Raytheon BBN Technologies (prior to joining AWS AI).
Due to the importance of RE, in recent years various methods and models have been proposed for this task. These models can be categorized into feature-based (Zelenko et al., 2003; Zhou et al., 2005; Bunescu and Mooney, 2005; Sun et al., 2011; Chan and Roth, 2010; Nguyen and Grishman, 2014; Nguyen et al., 2015c) and deep learning (Zeng et al., 2014; Nguyen and Grishman, 2015a; dos Santos et al., 2015; Wang et al., 2016; Nguyen and Grishman, 2016; Zhou et al., 2016; Zhang et al., 2017; Nguyen et al., 2019a) models. The existing models provide solutions for RE in various settings including monolingual (Zhang et al., 2018), cross-lingual (Ni et al., 2020), cross-domain (Pouran Ben Veyseh et al., 2020), and joint models (Nguyen et al., 2021, 2022). Despite those progress, one limitation that hinders on-going research for RE is labeled data scarcity. Annotating a large-scale RE dataset is challenging, due to the expensive nature of annotation task and the high requirement for expertise in specific domains. As such, prior methods have resorted to distantly supervised setting (Mintz et al., 2009; Zeng et al., 2015; Ji et al., 2017) or pseudo labeling techniques (Hu et al., 2021b,a) that leverage vast amounts of unlabeled data to address the labeled data scarcity issue for RE. Although these methods are helpful to substantially increase the size of RE datasets, they also introduce massive noisy samples which might hurt the training of an RE model. Consequently, creating cost-efficient large-scale labeled datasets for specific domains remains highly challenging for RE.
To achieve large-scale labeled datasets, in this work we introduce a novel data augmentation method to automatically generate labeled data for RE. In particular, instead of using unlabeled data, we propose to employ the pre-trained language model GPT-2 (Radford et al., 2019) to generate synthetic labeled data for RE. In our method, the GPT-2 model is first fine-tuned on available manually labeled RE datasets. Concretely, the language
11466
model is trained on the label-augmented sentences in which positive and negative RE samples are marked with special tags surrounding two input entity mentions. Next, the fine-tuned GPT-2 model is employed to generate new label-augmented indomain sentences that can be map back to produce new labeled data for RE. The new labeled data is then combined with the original manually labeled data to train an RE model. However, an issue with this approach involves the separation between the fine-tuning process of GPT-2 and the target RE model that might cause a mismatch between the generated data from GPT-2 and the data expected by the RE model (e.g., the generated data can be noisy or redundant for RE). As such, to improve the effectiveness of the generated data for an RE model, we propose to further optimize GPT-2 parameters during the training of the RE model, thus enabling the interactions between the GPT-2 and RE models to generate optimal/customized data for RE. In particular, we propose a meta learning framework to treat the parameters of the GPT-2 model as meta-parameters for the RE model that will be fine-tuned based on the performance of the RE model on a separate meta development set.
To leverage the performance on meta development set to optimize GPT-2 parameters, one solution is to employ reinforcement learning where the rewards for the generated sentences can be directly based on some performance metric (e.g., F1 score). However, due to the small size of the available data, this reward can lead to unstable training with high variance. To remedy this issue, in this work we propose a novel reward function that instead relies on gradients of the RE model’s loss to produce more robust training signals. In particular, our intuition is that a generated sample should have a higher reward if the direction in which the RE model should be updated to perform well on the sample and the development data are similar. To fulfil this objective, in the proposed training procedure, after one iteration of training, we first compute the average gradient of the RE model’s loss function over the meta development set. Next, the gradient of the loss of the RE model over a generated sample is computed. Finally, the reward for the generated sample is obtained via the cosine similarity between the gradients from the development set and the generated sample. While this reward is backed up with intuitive objectives, we also provide mathematical derivation of the reward based on bi-level optimiza-
tion to further demonstrate the advantages of our method. Finally, we evaluate the effectiveness of the proposed method on two benchmark datasets for RE. The experiments show the superiority of the proposed model compared to strong baselines.

heading: 2 Model
text: Task Definition: We study the problem of sentencelevel relation extraction. In this setting, the objective is to identify semantic relation between two input entity mentions in a sentence. Formally, the input to our model involve a sentence T = [w1, w2, . . . , wn] and two indices s and o (1 ≤ s, o ≤ n) to indicate the positions of the subject and object in the relation1. Our goal is to predict label y representing semantic relation between the entity mentions ws and wo from a predefined relation label set R (y ∈ R). Note that if the two entity mentions are not involved in a relation the special label None is employed. Also, for convenience, let Otrain be the set of available training data for our RE problem (i.e., T ∈ Otrain).
Model Overview: In this work we propose a meta-learning framework to train a deep learning model for relation extraction and a generative language model, i.e., GPT-2, to automatically generating training data for the deep learning RE model. In particular, our approach consists of a base model Mθ to be trained on the combination of original manually labeled RE data and automatically generated data. This base model is finally employed at inference time. Also, our approach involves a pretrained language modelMψ that will first be trained on the manually labeled data for RE to prepare it for in-domain synthetic data generation. Afterward, the language model will be jointly optimized with the RE model Mθ to leverage the feedback to each other from the models to improve the effectiveness of the generated data for RE. To realize the second objective, we present a reinforcement learning procedure that employs performance of the RE model Mθ as the reward to update the parameters of the generative model Mψ. More specifically, a reward function based on agreement of the gradients from a development set and generated data is introduced. In the rest of this section, we first describe the details of the proposed approach. We will then present the derivation of the proposed reward function.
1Note that semantic relation between two entity mentions can be directed.

heading: 2.1 Base Model
text: In this work we employ a BERT-based model (Devlin et al., 2019) to implement the base model Mθ for RE (θ involves the learnable parameters for the RE model). Concretely, the input sentence T is provided to BERTbase in the form of [[CLS], w1, w2, . . . , wn, [SEP]]. For each word wi ∈ T , the corresponding hidden vector ei in the final layer of the BERT model is employed to represent wi, leading to the sequence of vectors E = [e[CLS], e1, e2, . . . , en, e[SEP]] for T . Note that if wi contains multiple word-pieces, we utilize the hidden vector for its first word-piece for ei. Next, to create an overall representation vector h for the input sentence T with input entity mentions ws and wo, we employ the Dynamic Pooling mechanism (Chen et al., 2015): h = [e[CLS] : f(e1, . . . , es−1) : es : f(es+1, . . . , eo−1) : eo : f(eo+1, . . . , en)], where “:” indicates vector concatenation and “f(·)” is the Max Pooling operation over a set of vectors. Finally, the feature vector h is fed into a network architecture to produce a label distribution P (y′|T, s, o) = σ(FFC(h)), where σ is the softmax function and FFC is a two-layer feed-forward network. To train the base model Mθ, we employ the negative log-likelihood function: LC(T, y; θ) = − logP (y|T, s, o).

heading: 2.2 Generating Labeled Data
text: This section describes our approach to employ the pre-trained language model GPT-2, i.e., Mψ, to generate synthetic labeled data for RE (ψ contains the learnable parameters for GPT-2). The training of GPT-2 for this purpose is divided into two stages: (1) Pre-training to generate in-domain labeled data for RE and (2) Fine-tuning to improve the effectiveness of the generated data for the RE model. Pre-Training: To generate additional labeled data in the same domain as existing manually labeled data, we first train the GPT-2 model on the available RE training samples Otrain. In particular, we augment each training sentence T ∈ Otrain with special tags surrounding the input entity mentions to imply the existence of a relation. Formally, the label-augmented sentence T ′ for T is prepared as T ′ = [w1, w2, . . . , <SUB-l>ws</SUBl>, . . . , <OBJ-l>wo</OBJ-l>, . . . , wn], where l is p for positive samples (i.e., the subject and object entity mentions are in relation); and n otherwise. To train the GPT-2 model Mψ on the label-augmented
sentences T ′, denoted by T ′ = [w′1, w ′ 2, . . . , w ′ m] with m tokens for convenience, we employ autoregressive training. In particular, the model Mψ is trained to predict the the next token w′i using the left context [w′1, . . . , w ′ i−1]. Formally, the following loss function is employed to train Mψ: LG = − ∑m i=1 logP (w ′ i|w′1, . . . , w′i−1).
Once pre-trained, the GPT-2 model Mψ can be used to generate new label-augmented sentences that can be decoded to obtain new sentences along with markers for entity mention positions and relation labels. This newly generated labeled data can then be combined with the original training data Otrain to train the base RE model Mθ. It is noteworthy that our label-augmented sentences T ′ do not encode actual relation labels (i.e., only the information about the positive or negative examples is included) to simplify the generation task for GPT-2. As such, the new synthetic labeled data can only provide a binary label to indicate the existence of relation. Consequently, to employ the generated data to train the RE model Mθ, we integrate a classification head into the RE base model Mθ in which the overall representation vector h is fed into another feed-forward network with one output to serve as a binary classifier to predict positive/negative examples for the synthetic data. Accordingly, the cross-entropy loss for the binary classifier is computed over generated data for training Mθ (i.e., multi-task learning): LB(T, yb; θ) = −[yb ∗ log(δ(FFB(h))) + (1 − yb) log(1 − δ(FFB(h)))] where δ is the sigmoid function, and yb is 1 for positive samples and 0 otherwise.
Fine-Tuning: The pre-training of GPT-2 model is helpful to generate in-domain labeled data for RE. However, as this pre-training step is done separately from the RE model Mθ, the generated data from GPT-2 might not be optimal for the RE model. For instance, due to the lack of consultancy with Mθ, the generated data can introduce redundant/noisy information to hinder the training of the RE model. As such, it is necessary to allow the RE model to provide feedback for the training of the GPT-2 model so that the generated data from GPT-2 can be directly optimized/customized for our RE model to improve the model performance. To this end, we propose to further fine-tune the GPT-2 model during the training process of the RE model (i.e., joint training) that facilities the exploitation of training guidance from the RE model
Algorithm 1 Training of the ED model and finetuning of the GPT-2 model Input: Otrain,Dmeta Output: Optimal Models Mψ and Mθ Initialize θ0 and ψ0 For t = 1 to num_train_steps do
Sample |BO| data points from Otrain Generate |BG | data points (Tg, yg) using GPT-
2 with T ′g as the label-augmented texts BC ← BO ∪ BG ▷ Optimize θ gθ ← 1|BC |Σ(T,y)∈BC∇θLbase(T, y; θt−1) θt ← GradientUpdate(θt−1, gθ) ▷ Evaluate Mθ on Dmeta dθ ← 1|Dmeta|Σ(T,y)∈Dmeta∇θLbase(T, y; θt) ▷ Optimize ψ rg ← d⊤θ · ∇θLbase(Tg, yg; θt−1) gψ ← 1|BG |Σ |BG | g=1rg · ∇ψ logP (T ′g;ψt−1)
ψt ← GradientUpdate(ψt−1, gψ) end
to improve the data generation process in GPT-2. In particular, we present a meta-learning framework for joint training of the GPT-2 and RE model. At each training iteration t, a batch of training examples Btrain is sampled from the original training data Otrain. The GPT-2 model Mψt−1 at the current iteration is then employed to generate a batch of synthetic data BG . The combination of the original and generated data batches BC = Btrain∪BG is next leveraged to update the current base RE model Mθt−1 using the loss functions LC and LB . For convenience, we use Lbase to refer to both LC and LB . We can decide which loss to use depending on the type of data, i.e., LC for original humanlabeled data and LB for generated labeled data. Afterward, the current GPT-2 model Mψt−1 is updated using the feedback of the base RE model over the effectiveness of the generated samples BG (i.e., leading to Mψt . In this way, the GPT-2 model will be adapted along the training process to be generate effective data for the next training iteration of RE model.
To measure the effectiveness of the generated data batch BG for the RE model for GPT-2 updating, one straightforward solution is to employ the performance (e.g., F1 score) of the updated RE model Mθt over a separate meta development set Dmeta as a reward to update the GPT-2 model Mψt−1 with the REINFORCE algorithm (Williams, 1992) (i.e., to account for the discreteness of gen-
erated data). However, as we might not have sufficient labeled data to offer a large meta development set, this approach can have high variance for the reward, thus causing unreliable estimation and limiting the effectiveness of generated data for RE (Du et al., 2018). To address this issue, we propose a novel reward to avoid direct reliance on performance metrics and improve the robustness for the meta learning process. Accordingly, we devise the reward function based on the gradient of the training loss Lbase for Mθt over the meta development setDmeta, which captures the the direction to cause largest reduction for the loss function (i.e., the steepest direction). Intuitively, a generated sample Tg is helpful for the RE model Mθt if the gradient ofLbase with this sample aligns with the steepest direction with the development data (i.e., similar gradients from Tg and Dmeta). Formally, our reward to train GPT-2 is obtained via the dot product: rg = d⊤θ · ∇θLbase(Tg, yg; θt−1), where the dθ is the average of the gradients of the loss function Lbase for the RE model on the development set Dmeta, i.e., dθ =
1 |Dmeta|Σ(T,y)∈Dmeta∇θLbase(T, y; θt). We
use θt for dθ to inform the GPT-2 model with the latest RE model to generate better data in the next iteration. Finally, the parameters of the generative model Mψ is also updated using REINFORCE algorithm in our framework. The details of the proposed procedure are presented in Algorithm 1.

heading: 2.3 Derivation of Gradient-based Reward
text: This section aims to justify the proposed gradientbased reward with a mathematical foundation to better reveal its effectiveness for updating GPT-2 in our framework for RE. For simplicity, we assume that only one example (Tg, yg) is generated in an iteration, i.e., |BG | = 1. Using the reward rg for (Tg, yg), we leverage the REINFORCE algorithm to update ψt in the last GradientUpdate(ψt−1, gψ) step of Algorithm 1, leading to the update rule:
ψt ← ψt−1 + γrg · ∇ψ logP (T ′g;ψt−1) (1)
where γ is the learning rate. As such, to justify this update rule, we consider a bi-level optimization problem that starts with (Tg, yg) sampled from P (T ′g;ψt−1), which is the distribution induced by the GPT-2 model Mψt−1 . Next, our first level of optimization aims to optimize the loss function Lbase for the RE model using (Tg, yg), leading to the following update rule with gradient descent: θt = θt−1 − γ∇θLbase(Tg, yg; θt−1).
Here, θt can be seen as a function of ψ due to the dependence on (Tg, yg), which is in turn computed over ψt−1 (i.e., θt(ψ)). For convenience, we also compute the expectation over generated samples for ψt, i.e., θ̄t = ET ′g∼P (T ′g ;ψt−1)[θt] = θt−1 − γET ′g∼P (T ′g ;ψt−1)[∇θLbase(Tg, yg; θt−1)].
Afterward, we estimate the loss function Lbase of the new RE model θt over the meta development set Dmeta: J(θt(ψ),Dmeta) =
1 |Dmeta| ∑ (T,y)∈Dmeta Lbase(T, y; θt), serving as a measure for the effectiveness of the generated sample (Tg, yg) to provide feedback/training signals for the GPT-2 model. To this end, our second level of optimization is to optimize J(θt(ψ),Dmeta) with respect to ψ to update the GPT-2 model for the next iteration. Using gradient descent, our optimization procedure thus needs to compute the gradient ∇ψJ(θt(ψ),D) that can be computed via the chain rule:
∇ψJ(θt(ψ),D) = ∇θ̄tJ(θt(ψ),D)
⊤ · ∇ψ θ̄t(ψ) ≈ ∇θJ(θt(ψ),D)⊤ · ∇ψ θ̄t(ψ) (substitute the formula for θ̄t above) = ∇θJ(θt(ψ),D)⊤ · ∇ψ(θt−1− γET ′g∼P (T ′g ;ψt−1)[∇θLbase(Tg, yg; θt−1)]) (assume ∇ψθt−1 ≈ 0 with Markov assumption) ≈ −γ∇θJ(θt(ψ),D)⊤· ∇ψET ′g∼P (T ′g ;ψt−1)[∇θLbase(Tg, yg; θt−1)] (using the log-gradient trick)
=− γET ′g∼P (T ′g ;ψt−1) [( ∇θJ(θt(ψ),D)⊤
· ∇θLbase(Tg, yg; θt−1) ) · ∇ψ logP (T ′g;ψt−1)
]
To this end, using one roll-out sample and gradient descent, we can eventually derive the update rule for the GPT-2 parameters ψ in Equation 2.3, thus justifying our gradient-based reward function rg for REINFORCE to highlight its advantage for labeled data generation for RE.

heading: 3 Experiments

heading: 3.1 Dataset & Hyper-Parameters
text: To evaluate the effectiveness of the proposed model, i.e., called Data Generation for Relation Extraction (DGRE), we employ two English benchmark datasets for RE, i.e., ACE 2005 (Walker et al., 2006) and SPOUSE (Hancock et al., 2018). For ACE 2005, similar to previous work (Nguyen and Grishman, 2016; Shi et al., 2018; Pouran Ben Veyseh et al., 2020), we use the dataset split and prepro-
cessed by (Yu et al., 2015) for compatible comparison. There are 6 different domains in this dataset setting, i.e., (bc, bn, cts, nw, un, and wl), covering text from news, conversations and web blogs. As such, the union of the domains bn and nw (called news) is used as training data; a half of the documents in bc is reserved for the development data, and the remainder (cts, wl and the other half of bc) serve as the test data. In this way, our data organization presents different domains for the training and test data to focus on cross-domain generalization evaluation of the models (Pouran Ben Veyseh et al., 2020).
In addition, we employ the standard data split for the SPOUSE dataset, involving 22,195 sentences for training data, 2,796 sentences for development data, and 2,697 sentences for test data as done in (Hancock et al., 2018; Pouran Ben Veyseh et al., 2020). Each sentence in SPOUSE2 contains two marked person names (i.e., the entity mentions) and the goal is to predict whether the two people in the sentence are spouses. For both datasets, we sample 10% of the training data portions to serve as meta development data for our model.
We utilize the development set of ACE 2005 dataset to fine-tune the hyper-parameters for our model. Based on the F1 score on the development set, the following hyper-parameters are selected: 8 for the mini-batch size; 2 layers for the feedforward networks with 250 hidden dimensions; and 1e-2 for the learning rate for the GradientUpdate steps in our meta learning framework. Moreover, we use the default hyper-parameter values provided by Huggingface3 for the pre-training step for the GPT-2 model. Finally, the num_train_steps in Algorithm 1 is set to the number of training batches in each dataset.

heading: 3.2 Baselines
text: For experiments on ACE 2005, we compare DGRE with prior models reported on this dataset and also the related data augmentation methods. In particular, we consider the following baselines:
RE Models: (i) Feature based models: These models hand-design linguistic features for RE, i.e., FCM, Hybrid FCM, and LRFCM (Yu et al., 2015; Hendrickx et al., 2010). (ii) Deep learning models: These models employ deep learning architectures for RE, i.e., CNN, Bi-GRU (Nguyen
2extracted from news articles 3https://huggingface.co/ (Apache License 2.0.)
and Grishman, 2016), CNN+DANN (Fu et al., 2017), GSN (Shi et al., 2018), AGGCN (Attention Guided GCN) (Guo et al., 2019), SACNN (Segment-level Attention-based CNN) (Tran et al., 2019), DRPC (Dependency Relation Prediction and Control model) (Veyseh et al., 2019), EABERT (Wang et al., 2019), CEON-LSTM (Pouran Ben Veyseh et al., 2020), MapRE (Dong et al., 2021), and A-GCN (Qin et al., 2021). Note that CEON-LSTM and A-GCN have the best reported performance with different settings over ACE 2005 and SPOUSE.
Data Augmentation Models: These methods employ data augmentation (DA) techniques to address labeled data scarcity for RE or related tasks. In particular, we compare with GradLRE (Hu et al., 2021b) that proposes a Gradient Imitation Reinforcement Learning method to encourage pseudo labeled data to imitate the gradient on labeled data, and MetaSRE (Hu et al., 2021a) that employs pseudo label generation in a self-training procedure. Both methods use existing unlabeled data. In addition, we explore DA methods for IE tasks that exploit GPT-2 for data generation, including Filter-GPT (Anaby-Tavor et al., 2020) that filters the generated data based on confidence scores of a pre-trained RE model before combining them with original data; and Novelty-GPT (Yang et al., 2020a) that computes novelty scores for generated data, in comparison to original training data, to weight the samples in the combined dataset for training.

heading: 3.3 Results
text: The performance for the models on the test set of ACE 2005 is presented at Table 1. This table shows that the proposed method significantly outperforms all the baselines with p < 0.01 (except for A-GCN over cts)). Specifically, compared to the baselines that employ richer information from the input (e.g., syntactic structures in CEON-LSTM or label semantics in MapRE), the improvement obtained by DRGE is important as it requires only the surface form of the input text. This advantage is helpful in domains and settings that suffer from the lack of rich resources and data. Moreover, compared to the models that employ data augmentation (DA) to address data scarcity, the proposed method achieves significantly better results on all three domains. In particular, compared to “Filter-GPT” and “NoveltyGPT”, which are the most relevant approaches to DRGE, our method can substantially improve the
performance by up to 2.6% on the average F1 score. We attribute this improvement to the fact that other DA methods do not interact with the target RE model to guide the labeled data creation for optimal performance. In contrast, our method DRGE embeds the data generation process into the training process for RE to allow direct communication between GPT-2 and the RE model to produce more effective labeled data for the RE models.
In addition, Table 2 reports the performance of the model on test data of the SPOUSE dataset. The table corroborates our findings for the advantages of our labeled data generation method
for RE over competitive baselines. Specifically, DRGE is significantly better than all the baselines (p < 0.01); the performance improvement over GPT-based baselines is at least 2%, thus suggesting the ability to extend to different datasets and domains for RE of our method.

heading: 3.4 Ablation Study
text: To provide more insight into the performance of DGRE, this section studies the contribution of different components of the model to its final performance. Specifically, we examine the following variants of DGRE: (1) No GPT-2 Data: For this variant, we entirely remove the GPT-2 model so that the base RE model is only on original labeled data Otrain; (2) Separate Fine-Tuning: In this baseline, the GPT-2 model is separately fine-tuned on the training set Otrain to generate new labeled data, i.e., no information from the RE base model is employed to optimize GPT-2; (3) Dev Perf. Reward: To study the importance of the proposed gradient-based reward, we report the performance of the model that replaces the proposed reward in DGRE with direct F1 scores of the RE model on the meta development set (i.e., performance-based reward); and (4) No Pre-training: This variant is intended to show the benefit of the initial pretraining step of the GPT-2 model using the original training data Otrain.
Table 3 shows the performance of the models on the ACE 2005 development data. This table shows that all stages and components in the proposed method are necessary to achieve the best performance for DGRE. In particular, removing GPT-2 hurts the performance the most, demonstrating the importance of augmenting RE models with diverse samples generated by GPT-2. Moreover, replacing the proposed reward with the performance on the meta-development set in REINFORCE algorithm reduces the performance significantly, clearly confirming the advantages of the proposed reward
with gradient agreement to train our meta learning framework. Finally, we observe worse performance when the GPT-2 model is optimized separately from the RE model, thus testifying to our proposal of joint training to leverage the interaction between the two models for RE.

heading: 3.5 Analysis
text: Error Analysis: To better understand the effectiveness of the proposed reward to update the parameters of the GPT-2 model for RE, we analyze a sample of generated labeled data from GPT-2. A key insight from our analysis is that the proposed gradient-based reward is able to reduce noises in the generated data from GPT-2, thus better supporting the training of the base model for RE. In particular, we compare the frequencies of errors in the generated samples in two scenarios: (1) GPT-2 is fine-tuned by the proposed reward (i.e., DGRE), and (2) No fine-tuning is applied to the pre-trained GPT-2 (i.e., the GPT-2 is only pre-trained separately from the RE model as discussed in Section 2.2). 100 generated examples are reviewed for each scenario in our study. To this end, we consider the following categories of noises in the generated samples by GPT-2 for the RE model: (1) Missing Entity: In the generated texts, there is no tags for entity mentions, or only the subject or the object mention exists; (2) Wrong Entity: The special tokens “<SUB-l>”, “</SUB-l>”, “<OBJ-l>”, or “<OBJ-l>” do not match or surround correct entity mentions in the generated text; (3) Incorrect Relation: GPT-2 generates samples with correct tags for entity mention spans; however, the relation labels are incorrect (e.g., using the negative tags <SUB-n> and <OBJ-n> for samples with relation and vice versa); (4) Semantics: The semantics of the generated text is not sound (e.g., inconsistent topics, repeated words, etc.).
Table 4 shows the frequency of each noise category in the study. As can be seen, fine-tuning the GPT-2 model using the proposed gradient-based
reward for RE significantly reduces error rates in all categories. Interestingly, the RE-related errors, i.e., Wrong Entity, Missing Entity and Incorrect Relation, enjoy larger error reduction. This fact corroborates the necessity of integrating the finetuning process of the GPT-2 model with the training for the RE model. Moreover, the table shows that among all error categories, Wrong Entity is the major source of noises in the generated samples from GPT-2. Future work can thus explore approaches to integrate entity knowledge into the GPT-2 model to address this major for RE.
Case Study: Finally, to shed more light on the quality of the generated text, we present three positive and three negative samples produced by the GPT-2 model fine-tuned in the final epoch of the proposed training procedure for RE on ACE 2005. The sentences are shown in Table 5, highlighting the diverse nature of the generated samples (e.g, different distances and orders between the subject and object mentions) from GPT-2 for RE models.

heading: 4 Related Work
text: Relation Extraction is one of the fundamental tasks in Information Extraction. Due to its importance, various methods have been proposed for RE, ranging from feature-based and kernel-based techniques (Zelenko et al., 2003; Zhou et al., 2005; Bunescu and Mooney, 2005; Sun et al., 2011; Chan and Roth, 2010; Nguyen and Grishman, 2014; Nguyen et al., 2015c) to recent advanced deep learning models (Zeng et al., 2014; dos Santos et al., 2015; Zhou et al., 2016; Verga et al., 2018; Veyseh et al., 2019). The typical neural architectures for RE include Convolutional Neural Networks (Zeng et al., 2014; Nguyen and Grishman, 2015a; dos Santos et al., 2015; Wang et al., 2016), Recurrent Neural Networks (Nguyen and Grishman, 2016; Zhou
et al., 2016; Zhang et al., 2017), and self-attentions in Transformer (Verga et al., 2018).
To address the key challenge of data scarcity for RE, prior work has resorted to distantly supervised methods (Mintz et al., 2009; Zeng et al., 2015; Ji et al., 2017; Chen et al., 2021) or pseudo labeling techniques (Hu et al., 2021b,a). However, such methods suffer from low quality of obtained training data, thus hindering performance for RE. Also, we note that data augmentation based on GPT-2 has also been explored for other tasks, such as event extraction (Pouran Ben Veyseh et al., 2021; Papanikolaou and Pierleoni, 2020; Zhang et al., 2020; Yang et al., 2020b; Madaan et al., 2020). Compared to such prior work, our work features a new meta learning framework to jointly train GPT-2 with the downstream RE model, leveraging gradient agreement-based reward to improve the quality of generated labeled data.

heading: 5 Conclusion
text: We present a novel data augmentation method for RE using the pre-trained language model GPT2. The language model is fine-tuned over labelaugmented texts to generate in-domain and labeled samples for RE. To improve the quality of generated data for RE, the GPT-2 model is further optimized along the training process of a RE model in a novel meta learning framework (i.e., joint training to promote model interaction). Agreement scores between gradients of the RE loss function over generated data and a meta development set are proposed as the reward to update the GPT-2 model. We conduct extensive experiments on two benchmark datasets to demonstrate the benefits of the proposed method for RE. In the future, we will explore the application of the proposed methods to other related tasks in Information Extraction.
Limitations & Risks
Limitations: In this work we present a novel method to address data scarcity issue for Relation Extraction (RE). Although our experiments demonstrate the effectiveness of the proposed method, there are still some limitations that can be improved in future work. First, similar to previous work (dos Santos et al., 2015; Veyseh et al., 2019), the current method assumes golden entity mentions to perform RE that might not be the case in different applications. It is thus helpful to explore the method in a more realistic setting where entity mentions are predicted, e.g., using joint inference models to simultaneously extract entity mentions and relations in an end-to-end fashion. Second, our method is currently evaluated only for sentence-level RE (i.e., entity mentions are in the same sentences). Future work can further explore our method for documentlevel RE to allow entity mentions to appear in different sentences to better demonstrate its advantage. Finally, our method requires the generative GPT-2 model for data generation. To perform well, GPT-2 needs to be trained on large unlabeled datasets that might not be readily available for low-resource languages. As such, it is important to further evaluate our method on low-resource languages to better reveal its effectiveness. Risks: In this work, we employ GPT-2 to generate new training samples for the task of RE. Although GPT-2 is publicly available and the datasets employed in this work to fine-tune GPT-2 for RE are also publicly available, a generative language model might produce biased sentences, insulting texts or reveal private information. As such, it is necessary to take further measures before publicly releasing the automatically generated labeled data. To this end, we inspect the data employed for finetuning to exclude any offensive text and identity information. The generated data will also be inspected for purpose before publicly releasing the data.

heading: Acknowledgement
text: This research has been supported by the Army Research Office (ARO) grant W911NF-21-1-0112, the NSF grant CNS-1747798 to the IUCRC Center for Big Learning, and the NSF grant # 2239570. This research is also supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract 2022-
22072200003. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Limitations & Risks
3 A2. Did you discuss any potential risks of your work? Limitations & Risks
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 2.2
3 B1. Did you cite the creators of artifacts you used? Introduction
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The license information is publicly available
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Limitations and Risks
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Limitations and Risks
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? The information is publicly available.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Experiments
C 3 Did you run computational experiments? Experiments
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Experiments
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we will explore the application of the proposed methods to other related tasks in Information Extraction. Limitations & Risks
Limitations: In this work we present a novel method to address data scarcity issue for Relation Extraction (RE). Although our experiments demonstrate the effectiveness of the proposed method, there are still some limitations that can be improved in future work. First, similar to previous work (dos Santos et al. , 2015; Veyseh et al. , 2019), the current method assumes golden entity mentions to perform RE that might not be the case in different applications. It is thus helpful to explore the method in a more realistic setting where entity mentions are predicted, e. g. , using joint inference models to simultaneously extract entity mentions and relations in an end-to-end fashion. Second, our method is currently evaluated only for sentence-level RE (i. e. , entity mentions are in the same sentences). Future work can further explore our method for documentlevel RE to allow entity mentions to appear in different sentences to better demonstrate its advantage. Finally, our method requires the generative GPT-2 model for data generation. To perform well, GPT-2 needs to be trained on large unlabeled datasets that might not be readily available for low-resource languages. As such, it is important to further evaluate our method on low-resource languages to better reveal its effectiveness. Risks: In this work, we employ GPT-2 to generate new training samples for the task of RE. Although GPT-2 is publicly available and the datasets employed in this work to fine-tune GPT-2 for RE are also publicly available, a generative language model might produce biased sentences, insulting texts or reveal private information. As such, it is necessary to take further measures before publicly releasing the automatically generated labeled data. To this end, we inspect the data employed for finetuning to exclude any offensive text and identity information. The generated data will also be inspected for purpose before publicly releasing the data