A Two-Agent Game for Zero-shot Relation Triplet Extraction (2024)

Abstract: Relation triplet extraction is a fundamental task in natural language processing that aims to identify semantic relationships between entities in text. It is particularly challenging in the zeroshot setting, i.e., zero-shot relation triplet extraction (ZeroRTE), where the relation sets between training and test are disjoint. Existing methods deal with this task by integrating relations into prompts, which may lack sufficient understanding of the unseen relations. To address these limitations, this paper presents a novel Two-Agent Game (TAG) approach to deliberate and debate the semantics of unseen relations. TAG consists of two agents, a generator and an extractor. They iteratively interact in three key steps: attempting, criticizing, and rectifying. This enables the agents to fully debate and understand the unseen relations. Experimental results demonstrate consistent improvement over ALBERT-Large, BART, and GPT3.5*, without incurring additional inference costs in all cases. Remarkably, our method outperforms strong baselines by a significant margin, achieving an impressive 6%-16% increase in F1 scores, particularly when dealing with FewRel with five unseen relations†.

heading: 1 Introduction
text: Relation triplet extraction (RTE; Miwa and Bansal, 2016) is a pivotal task in information extraction (Yang et al., 2022), which aims to extract the relation triplets in the form of (head entity, tail entity, relation label) within a given sentence. RTE boosts a broad spectrum of downstream applications across domains, such as machine reading comprehension (Qiu et al., 2019) and machine translation (Zhao et al., 2020). Existing methods (Zheng
*Work done when Ting Xu was interned at IDEA. †Corresponding authors. *In the following section, we use the term GPT-3.5 to refer to the gpt3.5-turbo model. https://platform.openai.com/ docs/guides/gpt/chat-completions-api
†Our code will be publicly available at https://github. com/Mizar77/TAG_ZeroRTE.
" Rise Up Singing " was published by " Sing Out ! "', 'head': 'Rise Up Singing', 'tail': 'Sing Out !’
Sibling
'He was the father of actresses Catherine Deneuve and Françoise Dorléac .', 'head': 'Catherine Deneuve', 'tail': 'Françoise Dorléac’
Mother
'Cynidr was the son of St. Gwladys , T.', 'head': 'Cynidr', 'tail': 'Gwladys'}
'The theme so is " Tightrope " by Stephanie McIntosh .', 'head': 'Tightrope', 'tail': 'Stephanie McIntosh’
Architect
'The Mells W r Memorial i al o by Edwin Lutyens was unveiled in 1921 .', 'head': 'Mells War Memorial', 'tail': 'Edwin Lutyens’
Publisher
" Rise Up Singing " was published by " Sing Out ! "', 'head': 'Rise Up Singing', 'tail': 'Sing Out !’
Sibling
'He was the father of actresses Catherine Deneuve and Françoise Dorléac .', 'head': 'Catherine Deneuve', 'tail': 'Françoise Dorléac’
Mother
'Cynidr was the son of St. Gwladys , T.', 'head': 'Cynidr', 'tail': 'Gwladys'}
et al., 2017; Wang and Lu, 2020) rely on large amounts of labeled data, limiting the scalability and applicability of these methods.
To reduce the overreliance on labeled data, Chia et al. (2022) introduce a challenging task, zero-shot relation triplet extraction (ZeroRTE), where relation sets during the training and test stages are disjoint. For example, in Fig. 1, training samples may belong to the seen relation set {sibling, performer}, while test samples may belong to the unseen relation set {mother, writer}. Generalizing knowledge from the training set to the test set is critical.
Existing methods deal with ZeroRTE by formulating it into prompts to leverage the power of language models. The main idea is to fine-tune the model on seen relations from the training dataset. Next, the model adapts to unseen relations by integrating the semantics of those unseen relations into the prompt templates (Chia et al., 2022; Lv et al., 2023; Lan et al., 2022; Kim et al., 2022). Recent research based on large language models (LLMs) removes the fine-tuning stage and integrates the semantics of relations via chain-of-thought fewshot prompting (Wadhwa et al., 2023) or multi-turn question answering (Wei et al., 2023b).
Despite achieving favorable results, simply integrating relations into prompts is superficial and may lead to an insufficient understanding of the un-
7510
seen relations. For example, consider the two relations, "located on terrain feature" and "contains administrative territorial entity". Both deal with location but a subtle difference exists between the two relations. Merely merging them into the prompt without distinguishing their nuances may cause confusion between similar relations, ultimately resulting in a misunderstanding of the relations.
To tackle the above issues, we construct a twoagent game (TAG) to facilitate a process where both agents can engage in deliberation and debate to grasp the semantics of unseen relations. TAG aims to comprehensively understand the nuances of unseen relations to improve the performance in the subsequent test phase. Specifically, TAG consists of two agents: an extractor responsible for extracting relation triplets from sentences and a generator for generating sentences and triplets based on the given relation. They deliberate and debate through three key steps: attempting, criticizing, and rectifying. First, the generator attempts to understand the unseen relation by generating sentences and triplets associated with the relation. These initial attempts will serve as the target of debate in the later stages of the game. Subsequently, the extractor criticizes the generator on the quality of the synthetic data via evaluating the semantic matching degree between sentences and triplets, with higher log-likelihood values indicating better quality. Finally, we use the results of attempting as data and the results of criticizing as rewards to rectify both agents through reinforcement learning. In this way, both agents can learn the preference between synthetic data and tend to produce highquality data. Through this deliberation and debate process, two agents gradually develop a comprehensive understanding of unseen relations. Experiments demonstrate that TAG achieves consistent improvement across different model scales from ALBERT (Lan et al., 2020) with 18M parameters to large language models like GPT-3.5.
Our contributions are three-fold: (1) We introduce a novel two-agent game (TAG) method for ZeroRTE that facilitates deliberation and debate on unseen relations. This approach enables the generator and extractor agents to enhance their understanding of these relations collaboratively. (2) TAG is a versatile approach that can be applied to different backbone models, i.e., ALBERT, BART, and GPT3.5. Moreover, during deployment, the generator agent can be omitted without incurring additional computation and storage costs for the original ex-
tractor agent. (3) Experimental results demonstrate that TAG consistently improves the performance across different backbone models. This highlights the effectiveness and scalability of our approach.

heading: 2 Preliminaries
text: Before delving into the details, we introduce the task definitions of RTE and ZeroRTE. And then introduce the definitions of the two agents in our approach, the generator and extractor. To make the notations consistent throughout the paper, we define the important ones in Table 5 in the appendix.

heading: 2.1 Task Definitions
text: RTE Given a dataset D = {(si, ti)}|D|i=1, where si ∈ S represents the i-th input sentence and ti ∈ T represents the corresponding output triplet, Relation Triplet Extraction (RTE) aims to extract relation triplet t ∈ T from a sentence s ∈ S, following the form t = (ehead, etail, r). Here, the head entity ehead and the tail entity etail are represented as token spans or word sequences referring to real-world entities. The relation r belongs to the setR, encompassing a predefined collection of relations between the head and tail entities.
ZeroRTE The objective of ZeroRTE (Chia et al., 2022) is to leverage the knowledge from the seen dataset Ds and generalize to the unseen dataset Du. Let Ds and Du represent the training and test datasets, respectively, derived from the original full datasetD. The relation sets during training and test are denoted as Rs = {rs1, rs2, . . . , rsn} and Ru = {ru1 , ru2 , . . . , rum}, where n = |Rs| and m = |Ru| indicate their respective sizes. Importantly, it is worth noting that ZeroRTE does have training data Ds; zero-shot refers to the fact that the relation sets for training and test are disjoint, i.e.,Rs ∩Ru = ∅.

heading: 2.2 Agent Definitions
text: Generator The generator G aims to generate a relation-specific sentence s and triplet t = (ehead, etail, rs) given the relation rs. To adapt to language models, we define two transformations Ein(rs) and Eout(s, ehead, etail), mapping the input and output into natural language space. They transform structured information into text sequences using the template defined by Ein and Eout. In this work, we use a simple transformation function: Ein(rs) = "Relation: rs" and Eout(s, e head, etail) = "Head Entity: ehead, Tail
Entity: etail, Context: s". The two transformations linearize each element into natural languages.
We optimize the generator through autoregressive objective on the training set Ds:
LG(s, t) =− logP (out|Ein(rs);G) (1)
where out = Eout(s, ehead, etail) and t = (ehead, etail, rs), (s, t) ∈ Ds.
Extractor The extractor E aims to extract the structured triplet t of the form (ehead, etail, rs) given the sentence s. To ensure the generality of the method, we do not impose any restrictions on the model structure of the extractor. It can be an encoder-decoder model like BART (Lewis et al., 2020) or any encoder-only model like ALBERT (Lan et al., 2020). The objective of the extractor is to minimize the negative log-likelihood on the training dataset Ds:
LE =− logP (t|s; E) (2)
During inference, the extractor extracts triplet of the highest probability:
t̂ = argmax t∈T
P (t|s; E) (3)
Our method works with extractors of different structures, so we’re not going into the extraction steps for each one in detail here. You can find a full description of the extraction process we use during inference in Appendix A.1.

heading: 3 Methodology
text: The above section illustrates how the generator and extractor function. Our challenge is to make the two agents work collaboratively and improve the overall extraction performance. To achive this, we propose a two-agent game and collaborate through the following three key steps: (1) attempting: the generator attempts to express its understanding of the unseen relation by generating sentences and triplets for the relation; (2) criticizing: the extractor criticizes the generator on its synthetic data; (3) rectifying: both agents rectify on their extraction task and generation task individually. The cycle of attempting, criticizing, and rectifying goes iteratively to refine the two agents’ abilities. The overall procedure is described in Fig. 2.

heading: 3.1 Attempting for Understanding
text: The game begins with an unseen relation, and the generator expresses its understanding of the relation. This understanding is reflected in constructing sentences and triplets related to the relation.
To generate relation-specific sentences and triplets for an unseen relation rui ∈ Ru, we follow these steps:
• Map the relation to natural language space using the Ein(rui ) transformation.
• Randomly sample K times from the generator to obtain text sequences textik ∼ P (textik |Ein(rui );G), k = 1, . . . ,K.
• Extract the sentences, head entities, and tail entities from the generated text sequences using E−1out, the inverse function of Eout, sik , e head ik , etailik = E −1 out(textik).
• Combine the entities and the input relation to form the triplet tik = (e head ik , etailik , r u i ).
We aggregate the extracted sentences and triplets to create the synthetic dataset Dsyni = {(si1 , ti1), . . . , (siK , tiK )}. The overall synthetic dataset is the union of data on all relations: Dsyn = Dsyn1 ∪ . . .∪Dsynm , with |Dsyn| = mK, which represents the generator’s initial understanding of the relation and serves as the target of debate in the later stages of the game.

heading: 3.2 Criticizing for Quality Assessment
text: After the generator expresses its understanding of the relations, we use an extractor to evaluate whether the generator’s comprehension is correct. The motivation is: the generator’s synthetic data may contain noise. We introduce the extractor to differentiate the quality of synthetic data.
Intuitively, the probability P (tj |sj ; E) can be interpreted as the likelihood of triplet tj given sentence sj . This can be regarded as a semantic matching degree between tj and sj , indicating the quality of (sj , tj). Given the synthetic data (sj , tj) ∈ Dsyn, we compute the log-likelihood of tj extracted from sj :
llj = logP (tj |sj ; E) (4)
Using this formula, we can assign a higher loglikelihood to high-quality text-triplet pairs and a lower log-likelihood to low-quality text-triplet
pairs, thereby distinguishing the quality of different synthetic data.
Since all log-likelihood values are negative, we normalize‡ them using the following formula to obtain the quality value αj for (sj , tj) and distinguish between good and bad data quality based on the sign of the αj :
αj = llj − µ
δ (5)
where µ and δ are the mean and standard deviation of llj on all text-triplet pairs over the dataset Dsyn.

heading: 3.3 Rectifying via Reinforcement Learning
text: Previous research (Ouyang et al., 2022; Rafailov et al., 2023) demonstrate that reinforcement learning can effectively learn preferences among data. Building on this, we apply a reinforcement learning approach to rectify the model. By providing higher rewards to better-quality text-triplet pairs, the two agents can better understand the semantics of unseen relations through data.
More specifically, we consider the quality value αj , as defined in Eq. (5), as a reward and aim to maximize the expected reward E[α] on Dsyn through gradient ascent by optimizing the parameters of E and G to yield high-quality data:
Ei+1 ← Ei + γ1∇EE[α] (6) Gi+1 ← Gi + γ2∇GE[α] (7)
‡"Normalizing reward" is a common operation in RL (van Hasselt et al., 2016) that can make the distribution of rewards more stable, making it easier for the algorithm to converge.
where γ1 and γ2 are the learning rates for the extractor and generator, respectively. The gradients for the extractor ∇EE[α] and generator ∇GE[α] are computed as follows:
∇EE[α] = E(αj∇E logP (tj |sj ; E)) (8)
∇GE[α] = E(αj∇G logP (outj |inj ;G)) (9) where the expectation is taken over αj on Dsyn, inj = Ein(r u j ), outj = Eout(sj , e head j , e tail j ).
During the iterative process of attempting, criticizing, and rectifying, the two agents engage in detailed semantic debates, enabling them to discern subtle semantic distinctions, and ultimately refine their extraction and generation tasks.

heading: 4 Experiments
text: To validate the effectiveness of TAG, we conduct experiments on both small and large language models to address the following questions: (1) Can TAG be applied to different extractor model structures? (2) Is TAG effective at different scales of model parameters?

heading: 4.1 Experimental Settings
text: Datasets We evaluate TAG on FewRel (Han et al., 2018), Wiki-ZSL (Chen and Li, 2021), and TACRED (Zhang et al., 2017). The data statistics are shown in Table 3. We follow the same process as Chia et al. (2022) to partition the data into seen and unseen label sets. For each dataset, we set the
unseen label size to m = {5, 10, 15} and randomly select m relation labels for testing and treat the remaining labels as seen labels during training in the experiments. To reduce the effect of experimental noise, the label selection process is repeated for five random seeds to produce different folds.
Evaluation Metrics Following the work of Chia et al. (2022), we evaluate the results of triplet extraction separately for sentences containing single triplets and multiple triplets. We use the standard micro precision (P.), recall (R.), and F1 metrics commonly used in structured prediction tasks for multiple triplet extraction. On the other hand, evaluating single triplet extraction involves only one possible triplet for each sentence. Hence, the metric of Accuracy (Acc.) is employed. We report the average results across five data folds.

heading: 4.2 Experiments on Small Language Models
text: Baselines We compare our proposed TAG, with competitive baselines in ZeroRTE.
• TableSequence (Wang and Lu, 2020) casts the ZeroRTE task as a table-filling problem and uses ALBERT-Large (Lan et al., 2020) to encode the textual information.
• RelationPrompt (Chia et al., 2022) uses GPT2 (Radford et al., 2019) to generate synthetic data for unseen relations and then trains the extractor model BART (Lewis et al., 2020) on the synthetic data from GPT-2.
We choose not to compare with DSP (Lv et al., 2023) because this approach heavily relies on the
initial parameters of soft prompts, making it hard to reproduce the results. We set the hyperparameters as those reported in Appendix B because they can attain good performance in our experiments.
Results We report experimental results in Table 1. We omit the multi-triplets for TACRED, which exclusively contains single triplets. From the table, we can observe that TAG consistently improves over TableSequence (encoder-only) and RelationPrompt (encoder-decoder). Compared with RelationPrompt, TAG achieves an absolute F1 improvement of 8.23% and 16.47% on Wiki-ZSL and FewRel in multi-triplet with m = 5. Such improvement indicates the effectiveness of TAG across different model architectures. Moreover, TAG can achieve a more balanced precision-recall ratio, leading to better overall F1 results. When comparing TAG + TableSequence to TAG + RelationPrompt on TACRED, we notice a significant performance advantage for the former. This discrepancy is because many of the relations labels in TACRED start with the same token, RelationPrompt solely considers the probability of the first token during relation decoding, so it struggles to differentiate between relations with identical initial tokens.

heading: 4.3 Experiments on Large Language Models
text: Given the strong performance of LLMs across various downstream tasks (Zan et al., 2023), we investigate the performance of GPT-3.5, a highly influential LLM, on the ZeroRTE task. We remove the fine-tuning data Ds to compare with previous methods based on LLMs and evaluate the perfor-
mance using the test sets with m = 5. Detailed experiment settings can be seen in Appendix B.
Baselines We compare our proposed TAG with GPT-3.5 under different prompting methods.
• ICL is an in-context-learning method that directly prompts LLMs, we follow the prompting method in Wei et al. (2023b).
• ChatIE (Wei et al., 2023b) transforms ZeroRTE task into a multi-turn question answering problem with a two-stage framework.
• RelationPrompt is the same as described in the previous section with the generation model replaced from GPT-2 to GPT-3.5.
Results We report experimental results in Table 2. The results reveal the following key observations: (1) Based on the results of TAG + RelationPrompt, ICL, and ChatIE, we can conclude that small extractors trained on synthetic data of LLMs can achieve comparable results while reducing the inference time from 78% to 91%. (2) TAG + RelationPrompt’s performance on FewRel is lower than ICL’s. This is because TAG + RelationPrompt uses BART (140M) as an extractor, which has a much smaller number of model parameters than ICL’s GPT-3.5. However, TAG’s inference speed is 78% faster than ICL. (3) TAG can still improve the performance of the RelationPrompt by 1.93% and 2.81% in Wiki-ZSL and FewRel, respectively. This underscores the effectiveness of TAG under large language model scales. Moreover, it does not increase the inference cost.

heading: 5 Analysis

heading: 5.1 Analysis on Attempting
text: To assess whether the generator can understand new relations and produce relation-specific sentences and triplets during the attempting step, we randomly sample 50 outputs from GPT-2 and GPT3.5 and manually evaluate the quality of synthetic data. Specifically, we categorize the generated data into five categories: correct (CORRECT), incorrect sentences (INC_SENT), incorrect entities (INC_ENT), incorrect relations (INC_REL), and cases of incorrect relations and entities (INC_REL&ENT). Detailed settings and generated examples are shown in Appendix B.2. Fig. 3a and Fig. 7 display the analysis of generated samples from GPT-2 and GPT3.5, showing that the generator accurately produces nearly half of the data. The results suggest that the generator can capture the semantic information of relation and generalize to unseen relations to some extent. Consequently, this validates our approach of employing language models as generators for data generation.

heading: 5.2 Analysis on Criticizing
text: To assess the extractor’s ability to differentiate between varying data quality, we use text-triplet pairs from the evaluation set as matching examples and construct mismatching examples by randomly replacing relations, head entities, and tail entities. We calculate the log-likelihood for each pair. Fig. 3b and Fig. 8 show the results from two seminal studies, RelationPrompt (Chia et al., 2022) and TableSequence (Wang and Lu, 2020), respectively. The figure shows a noticeable difference between matching and mismatching examples. The model tends to assign higher log-likelihood values to matching examples and lower values to mismatching examples.
In addition to the manually constructed mismatching data, we also conduct qualitative analysis
on the reward for synthetic data in appendix B.4. We can conclude that the extractor serves as an effective semantic matching evaluator for unseen relation types and, thus has the potential to detect errors from the generator.

heading: 5.3 Stability of Reinforcement Learning
text: We assess the reinforcement learning stability by analyzing the extractor and generator’s training loss dynamics, as well as the extractor’s F1 score on the validation set. As shown in Fig. 4a, the loss of the extractor and generator steadily decreases during the training process, indicating that our algorithm can learn a stable policy. Meanwhile, the sharper decrease in the generator’s loss is attributed to handling a more complex task than the extractor. Extractor’s F1 scores on FewRel validation set (with unseen labels 5-15) are shown in Fig. 4b. TAG exhibits progressive improvement up to 4 iterations,
with minimal decreases beyond that point. This suggests a delicate balance between capability enhancement and potential overfitting. In conclusion, reinforcement learning in TAG gradually improves both the extractor and generator but may lead to an over-fitting with excessive training. Here, we set overall interaction times to 5 based on Fig. 4.
In Appendix B.5, we compare the results of Reinforcement Learning (RL) and Maximum Likelihood Estimation (MLE) and discover that RL significantly outperforms MLE, which further demonstrates the effectiveness of our approach.

heading: 5.4 Case Study
text: As presented in Table 4 in the appendix, we compare the outcomes of RelationPrompt and TAG + RelationPrompt in the ZeroRTE task. RelationPrompt tends to confuse the meanings of similar relations like "located on terrain features" and "con-
tains administrative territorial entity", "publisher" and "distributed by", whereas TAG distinguishes between them effectively. This showcases TAG’s superior ability to understand the nuances of unseen relations through its deliberation and debate process. However, in the fourth example, both methods incorrectly swap the head and tail entities. This suggests that TAG’s entity distinction capability is somewhat weaker, which could be a research direction for the future.
We further analyze the effectiveness of reward value, reinforcement learning, and generated data size, and put them in Appendix B.4, Appendix B.5 and B.6 due to space limitation.

heading: 6 Related Work
text: Zero-shot Relation Triplet Extraction Zeroshot relation triplet extraction (ZeroRTE) is a challenging task that aims to extract relation triplets from unstructured text, where the relation sets between training and test are disjoint. There are many different approaches to ZeroRTE. Conventional methods tackle ZeroRTE by formulating ZeroRTE into prompts to leverage the power of language models. For example, Chia et al. (2022), the seminal work in ZeroRTE, first prompt a generative model to generate synthetic training data for the unseen relations, then they use the synthetic data to further train the extraction model. Lv et al. (2023); Lan et al. (2022); Kim et al. (2022) directly incorporate the semantics of relations through soft prompt§.
Recent research has shown that large language models (LLMs) can achieve strong performance on downstream tasks (Brown et al., 2020; Kojima et al., 2022; Zan et al., 2023; Zhang et al., 2024) without tuning the parameters. For the RTE task, Wadhwa et al. (2023) explore chain-ofthought prompting under few-shot settings, Wei et al. (2023b) transforms RTE into a multi-turn question answering task and extract the relation triplets through chatting with LLMs.
However, these methods simply integrate relations into prompts, which is superficial and may lead to an incomplete understanding of the unknown relations.
Multi-agent Game Multi-agent game studies the behavior of multiple language models through de-
§Though KnowPrompt (Chen et al., 2022) has been proposed in the literature for relation classification, it differs from our work in both the task and experiment settings.
bate or cooperation (Talebirad and Nadiri, 2023; Li et al., 2023; Fu et al., 2023; Dasgupta et al., 2023; Wei et al., 2023a). The core idea is the agents can improve each other through debate and cooperation. For example, Fu et al. (2023) improve the negotiation ability by two agents bargaining and one agent criticizing. Talebirad and Nadiri (2023) propose a collaborative multi-agent framework for handling complex tasks more efficiently and effectively. In this paper, we borrow the idea of multi-agent game and construct a two-agent game for ZeroRTE to deliberate and debate the semantics of the relation.
Reinforcement Learning Reinforcement learning is a machine learning paradigm that focuses on how intelligent agents can make decisions in an environment to optimize a given notion of cumulative rewards (François-Lavet et al., 2018). Unlike supervised learning, reinforcement learning does not require labeled data to be presented. Instead, it interacts with the environment to collect information. Reinforcement learning has found applications in various domains such as game playing (Mnih et al., 2015) and recommendation systems (Afsar et al., 2023). In this paper, we regard ZeroRTE as the interaction game between the extractor and generator and employ reinforcement learning to mutually improve the two agents.

heading: 7 Conclusion
text: In this paper, we propose TAG, a two-agent game for ZeroRTE, by introducing a new generator agent to communicate with the original extractor agent. Through iterative processes of attempting, criticizing, and rectifying, the generator and extractor agents engage in a deliberative and collaborative exploration of the semantics of unseen relations, facilitating a comprehensive understanding of these relations. Experimental results demonstrate that TAG consistently enhances performance across different model architectures and scales without incurring additional inference costs. Remarkably, our method outperforms strong baselines by a significant margin, achieving an impressive 6%-16% increase in F1 scores, particularly when dealing with FewRel with five unseen relations. We believe that TAG is a promising approach for ZeroRTE and holds potential for applications in other natural language processing tasks.

heading: Limitations
text: The proposed method has some limitations with LLMs. Specifically, the criticizing step requires probability values from the extractor, which are difficult to obtain for LLMs that can only be accessed via API. Additionally, the rectifying step necessitates gradient updates to the generator and extractor, which is impractical for LLMs.
In order to address these challenges, future work could focus on the following directions: (1) Replacing probability values with textual feedback, which can be readily provided by LLMs; (2) Exploring alternative approaches to rectify LLMs without relying on gradient updates.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments. This work was fully supported by the IDEA Information and Super Computing Centre (ISCC) and NSFC (No. 62206126, 62376120, 61936012, 62201576).

heading: A Preliminaries

heading: A.1 Extractor
text: We base our experiments on two extraction models, RelationPrompt (Chia et al., 2022) and TableSequence (Wang and Lu, 2020). The detailed extraction process is outlined as follows:
• RelationPrompt (Chia et al., 2022) is a sequence-to-sequence model that generates
text sequences from “Head Entity: h, Tail Entity: t, Relation: r”. Then the generated text sequences are decoded into triplets of the form (h, t, r) through regular expressions. It proposes a Triplet Search Decoding method to extract multiple triplets in a sentence. The core concept is enumerating multiple output sequences during generation by considering multiple candidates for the head entity, tail entity, and relation.
• TableSequence (Wang and Lu, 2020) consists of two distinct encoders: a sequence encoder to extract entities and a table encoder to extract relations for entity pairs. The proposed model is capable of adaptively discovering multiple triplets simultaneously in a sentence via the classification results of the table encoder. It formalizes the RTE task as a tablefilling task and decodes the triplet from the table.
It is important to note that RelationPrompt and TabelSequence represent two distinct extraction models, while TAG serves as a framework that enhances the performance of extractors by incorporating both extractors and generators, making it suitable for a variety of extraction models.

heading: B Experiments and Analysis

heading: Training Details for Small Language Models
text: We use the pretrained model provided by HuggingFace¶ and run all the experiments on NVIDIA A100 GPU with pytorch. For the hyper-parameters of the baseline models, we follow the original settings in their paper (Chia et al., 2022; Wang and Lu, 2020). We use GPT-2 as the generator in small language models. We set the learning rates for the extractor and generator of TAG + TableSequence as γ1 = 3 × 10−3, γ2 = 3 × 10−5, respectively. We set the learning rates for the extractor and generator of TAG + RelationPrompt as γ1 = 3 × 10−5, γ2 = 3 × 10−5, respectively. The only hyper-parameter in TAG is the number of generated data size K. We search K in {100, 300, 500, 700}. For each value, we conduct experiments with five random data splits and set K = 500 by the F1 score on the validation set. The parameter search costs about 50 GPU hours.
¶https://huggingface.co/
Algorithm 1 The two-agent game framework Input: Unseen relation set Ru with |Ru| = m, extractor E and generator G after supervised training on Ds, learning rates
γ1, γ2 for the extractor and generator, respectively. Output: Trained extractor and generator which can generalize to the unseen relation set Ru 1: repeat 2: ▷ Attempting for Understanding 3: for i = 1, 2, . . . ,m do 4: Select relation rui from Ru 5: Sample K text sequences texti1 , . . . textiK from P (textik |Ein(rui );G) 6: Decode sentences and triplets (si1 , ti1), . . . (siK , siK ) from text sequences 7: Constructing synthetic data Dsyni = {(si1 , ti1), . . . (siK , siK )} 8: end for 9: Union synthetic data for all relations to construct Dsyn = Dsyn1 ∪ . . . ∪ Dsynm 10: ▷ Criticizing for Quality Assessment 11: for (sj , tj) ∈ Dsyn do 12: Measure the quality of the jth data using Eq. (4) and Eq. (5). 13: end for 14: ▷ Rectifying via Reinforcement Learning 15: Compute the stochastic gradient of E and G using Eq. (8) and Eq. (9). 16: Update the model parameters using Eq. (6) and Eq. (7). 17: until convergence

heading: Training Details for Large Language Models
text: Due to the model’s large size, updating its parameters is challenging. Previous research has shown that prompts have a similar effect to gradient updates (Von Oswald et al., 2023; Dai et al., 2023).
Therefore, we have modified the reinforcement learning part of the large model to use data with high rewards as demonstration inputs. For the extractor, we fine-tune the model using the generated output from the generator and the scoring results from the extractor. We show the detailed prompts for ICL, ChatIE, and TAG in Fig. 9, Fig. 10, and Fig. 11, respectively.

heading: B.1 Fine-tuning Large Language models
text: To show the effectiveness of our proposed framework and reduce the effect of the small backbone model, we replace BART with LLaMa-7b and conduct experiments on FewRel. We conduct inference on a 32GB V100 using beam search with num_beams=16, which results in longer inference times. The input and output definitions for LLaMa remain consistent with ICL. The experimental results are shown in Table 7. It shows that our approach TAG with LLaMa-7b significantly outper-
forms the GPT-3.5 using ICL by a margin of 2.16%.

heading: B.2 Analysis on Attempting
text: We evaluate the zero-shot generation ability on two models: GPT-2 and GPT-3.5. For GPT-2, we first fine-tune it on the training set Ds, then evaluate its ability on unseen relation set Ru. We randomly select 50 generated examples and manually assess the quality. We categorized the generated data into five categories:
• CORRECT: The sentence and the triplet are both correct, and the sentence implies the meaning of the triplet.
• INC_SENT: The sentence is grammatically or semantically incorrect, making it impossible to understand.
• INC_ENT: The sentence and the relation are correct, but the corresponding head or tail entity is incorrect.
• INC_REL: The sentence does not preserve the semantics of the corresponding relation.
• INC_REL&ENT: The combination of INC_REL and INC_ENT.
Analysis statistics for GPT-2 and GPT-3.5 are shown in Fig. 3a and Fig. 7. To show the results more specifically, we randomly select 10 examples generated by GPT-2 and GPT-3.5, and present them in Table 8 and Table 9, respectively.

heading: B.3 Analysis on Criticizing
text: We first train the two models on Ds of seen relations. Next, we calculate the log-likelihood of matching and mismatching data. The results for RelationPrompt and TableSequence are reported in Fig. 3b and Fig. 8, respectively.

heading: B.4 Qualitative Analysis of the Reward
text: In our study, we employ a qualitative analysis to assess the efficacy of the reward generated by the extractor. Specifically, we scrutinize the reward values assigned to various synthetic data, as depicted in Fig. 5. Notably, the initial instance, boasting the highest reward value, exhibits consistent sentences and triplets. Conversely, the second instance accurately extracts head and tail entities, but the relation is not implied in the sentence, resulting in
.
a moderate reward value. Finally, the last instance extracts incorrect and duplicate head and tail entities, warranting the lowest reward value. This analysis leads us to the conclusion that the reward from the extractor effectively evaluates the quality of synthetic data.

heading: B.5 Effect of Reinforcement Learning
text: We compare reinforcement learning with two methods to demonstrate its effectiveness, i.e., Maximum-Likelihood Estimation (MLE) and Maximum Marginal Likelihood (MML) (Guu et al., 2017). MLE maximizes the likelihood over the whole synthetic dataset, which replaces the update in Eq. (6) and Eq. (7) into:
Ei+1 = Ei + γ1∇E ∑
j
logP (tj |sj ; E) (10)
Gi+1 = Gi + γ2∇G ∑
j
logP (outj |inj ;G) (11)
MLE maximizes the marginal likelihood over the whole synthetic dataset with a self-normalized gra-
dient estimation technique. It follows to the following gradient update :
αj∇E log ∑
j
P (tj |sj ; E) = E[ eαj
ZE × pEj ] (12)
αj∇G log ∑
j
P (outj |inj ;G) = E[ eαj
ZG × pj ]
(13) where pEj = P (tj |sj ; E), pGj = P (outj |inj ;G).
ZE = ∑
j′ p E j′ , Z
G = ∑
j′ p G j′ .
The results are shown in Table 6, we can conclude that our proposed TAG outperforms MLE and MML in most of the experimental settings. We believe this is because TAG can provide different rewards for data of varying quality, allowing the model to favor high-quality data.

heading: B.6 Effect of Generated Data Size
text: We explore the impact of generated data size on ZeroRTE performance. F1 scores of FewRel validation set with m = 10 and varying generated data sizes (100-700) are shown in Fig. 6. Our findings indicate that TAG’s performance improves with increasing generated data up to 500. However, further increasing the data size to 700 does not significantly enhance the final performance.
Prompt for relations The given sentence is César Gaviria Trujillo Airport is an airport serving the town of Inírida in the Guainía Department of Colombia . List of given relations: ['field of work', 'contains administrative territorial entity', 'located on terrain feature', 'distributed by', 'spouse'] What relations in the given list might be included in this given sentence? If not present, answer: none. Respond as a tuple, e.g. (relation 1, relation 2, ......) Response (contains administrative territorial entity, located on terrain feature)
Prompt for entities According to the given sentence, the relation between them is contains administrative territorial entity, find the head and tail entities and list them all by group if there are groups. If not present, answer: none. Respond in the form of (head entity1, tail entity1), (head entity2, tail

heading: Future Work
text: In order to address these challenges, future work could focus on the following directions: (1) Replacing probability values with textual feedback, which can be readily provided by LLMs; (2) Exploring alternative approaches to rectify LLMs without relying on gradient updates