Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction
Authors: Martin Josifoski
Marija Sakota
Maxime Peyrard
Robert West
Venue: EMNLP 2023 Main (2023)
Keywords: LLM
Synthetic Data Generation
Information Extraction
Closed Information Extraction
Parsing
Structured Output
TL;DR: Using cIE as a representative task, we show that exploiting asymmetries in synthetic data creation is a highly effective and versatile method for generating high-quality data, even for tasks beyond the direct problem-solving capabilities of the LLM.

Abstract: Large language models (LLMs) have great potential for synthetic data generation. This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in task difficulty makes it possible to produce large-scale, high-quality data for complex tasks. We demonstrate the effectiveness of this approach on closed information extraction, where collecting ground-truth data is challenging, and no satisfactory dataset exists to date. We synthetically generate a dataset of 1.8M data points, establish its superior quality compared to existing datasets in a human evaluation, and use it to finetune small models (220M and 770M parameters), termed SynthIE, that outperform the prior state of the art (with equal model size) by a substantial margin of 57 absolute points in micro-F1 and 79 points in macro-F1. Code, data, and models are available at anonymous.

Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction
Martin Josifoski, Marija Sakota, Maxime Peyrard, Robert West
EPFL
marttin.jsiffoki, marija.sakota, peyrard robert. west) @epfl.ch
Abstract
P(Y)
Task of interestt hard
Large language models (LLMs) have great po- tential for synthetic data generation. This work shows that useful data can be synthetically gen- erated even for tasks that cannot be solved di- rectly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to per- form the task the reverse direction, by gen- erating plausible input text for a target output structure. Leveraging this asymmetry in task difffculty makes it possible produce large- scale. high-quality data for complex tasks. We demonstrate the effectiveness of this approach closed information extraction, where col- lecting ground-truth data is challenging, and no satisfactory dataset exists date. We synthet- ically generate a dataset of 1.8M data points, establish its superior quality compared to ex- isting datasets human evaluation, and it to finetune small models (220M and 770M parameters), termed SynthLE, that outperform the prior state of the art (with equal model size) by a substantial margin of 57 absolute points micro-F1 and 79 points macro- F1. Code, data, and models are available https://gi thub. thub.
X
Inverse task,
easy for LLM
X1 +
Y1
X2 +
Y/2
Xn +
LLM
Yn
Figure 1: Exploiting asymmetry for SDG. For hard tasks of interest with input X and output the reverse task (from Y to X) may be much easier for an LLM. If so, we can generate high-quality training pairs (X,Y) by prompting L.M generate plausible inputs X from outputs Y. This often holds true for tasks with structured as in closed information extraction, where would be the input text and Y would be the list of (subject, relation, object) triplets expressed in the input text. Furthermore, this ensures full control over the sampling distribution P(Y), and thus balanced datasets.
we propose to alleviate this issue by generating syn- thetic data in the reverse direction by first sampling an output structure y and then prompting the LLM to generate corresponding input text x (see Fig. 1), We postulate that for many tasks, given appropri- ate in-context information, an LLM can generate meaningful corresponding when the original task cannot be solved directly by the LLM. Exploiting this asymmetry, then, will allow us to synthetically generate high-quality data even for hard tasks. Furthermore, the flexibility to choose the distribution over output structures y gives us fine-grained control over the data.
www.ttttww......aaeeeeete cww/epfl -epppllllbbbbia http://pf1--lla/Sytttte Sb/hhth....
1 Introduction
Large language models (LLMs) have demonstrated the ability to generate highly fluent and coherent textual data. One promising application of this capability is the generation of large amounts of high-quality synthetic data for training and evalu- ating smaller models. This becomes particularly useful for tasks where high-quality datasets are not readily available or access to real data is limited or expensive. However, in many complex natural lan- guage processing (NLP) tasks, the textual input is mapped to structured (rather than free-text) out- puty: y. and in such cases, LLMs may perform poorly as synthetie-data generators, since pretraining did not gear them to produce the specific required out- put format (even with in-context learning). Here
structure y, even
A good example of such task, which we focus in this work, is closed information extraction (cIE). In cIEè, model must extract set of disam- biguated triplets (i.e., facts) from a given text x. These triplets are of the format (subject, relation, object), with the subject and object being entities

---

knowledge base (e.g., Wikidata) and the relation being specified in the knowledge base schema.
(iii) We introduce SyuthE, a series of Flan-T5 models (Chung et al., 2022) finetuned on our syn- thetic data. In contrast to previous models, which perform well only for the few most frequently oc- curring relations, SynthlE: achieves high precision and recall across all relations. On REBEL Clean, a manually annotated subset of REBEL, SynthlEEs macro-F1 score even exceeds that of the original REBEL data's gold annotations. On Wiki-clE Text, our highest-quality test set, SynthIE outperforms the previous state of the art (with equal model size) by a substantial margin of 57 and 79 absolute points in micro-F1 and macro-F1, respectively.
Collecting datasets for this task is time-consum- ing and expensive, it requires annotators to know the entire entity and relation catalogs and reason about all possible facts expressed in the text x, As a result, only small or noisy datasets cxist to date. The largest dataset available, REBEL (Huguet Cabot and Navigli, 2021), suffers from several problems (Josifoski et al., 2022): (i) Noise: is collected with a mix of heuristics, and for many data points, the target output y does not contain all the facts expressed in the input is (partially) incorrect. (ii) Skewness: most relations appear only rarely the dataset, which results in mod- els that ignore most of the information when the data is used for training and in poor performance estimates when it is used for evaluation.
Overall, we demonstrate that by exploiting asym- metry between structured outputs y and textual in- puts x, LLMs can generate high-quality synthetic data to train smaller, specialized models. This way, task that was previously not solvable by LLMs now feasible for small #20M-parameter model. Our code, models, and datasets will be released for future researchers to reuse and extend.
Importantly, cIE is also not solvable directly by LLMs such as GPT3.5, as such models are unaware of the entity and relation identifiers of interest (for examples of failures, see Appendix A). We show that, although the LLM cannot be used to produce output structures directly, we can use it to generate high-quality input texts by reversing the task.
Background and Related Work
2.1 Synthetic Data Generation
Several approaches to data augmentation rely on pretrained language models (PLMs). Early efforts include works that require the pre-existence of a da- taset, which is then used to finetune the pretrained generator network (Anaby- Tavor et al., 2020; Pa- panikolaou and Pierleoni, 2020; Yang et al., 2020; Mohapatra et al., 2020; Kumar al., 2020).
Contributions. (i) We propose strategy for de- signing effective synthetic data generation (SDG) pipelines and apply it to cIE. Concretely, we start by sampling triplet sets from the Wikidata knowl- edge graph (KG) such that each triplet set is co- herent (i.e., can be expressed by short text) and the selected triplet sets cover all relations more uniformly than REBEL does. For each triplet set y, we then prompt an LLM to generate text x that expresses these, and only these, triplets. This re- sults in a high-quality synthetic dataset (1.8M data points) that we use to replace the noisy REBEL dataset. The process is illustrated in Fig. 2.
Recent work focuses on methods not requiring supervision beyond a few demonstrations. Wang et al. (2021) generate synthetic labels by provid- ing unlabeled samples as examples to the LLM. Ye et al. (2022) and Gao et al. (2022) use PLMs to generate data with carefully designed prompts. They evaluate text classification, question an- swering, and natural language inference. There are similar procedures for GLUE (Wang al., 2018) tasks (Meng et al., 2022a), intent classifi- cation (Sahu et al., 2022), and question answer- ing (Li et al., 2022). Alternatively, Meng et al. (2022c) tune a PLM on few demonstrations and use it as synthetic data generator; and Smith et al. (2022) incorporate prompted PLM for weak-super- vision; while Shao et al. (2023) generate synthetic demonstrations improve the propmting of LLMs; and Honovich et (2022) generate synthetic in- structions for instruction tuning of LLMs. Con- trary to these works, we do not generate labels but
(ii) human evaluation, we show that our syn- thetically generated data is of significantly higher quality than the existing REBEL dataset, has a more uniform relation-frequency distribution, and can be generated cheaply at scale. The evaluation reveals that, with 70% of the information from the text not present in the target set and 45% of the target triplets not actually expressed in the text, REBEL'S test cannot be used for an accurate estimation of performance. In contrast, for our qualtst-quality test set, the corresponding numbers are 15% and 22%, respectively.

---

prompt an LLM to perform the reverse task when the reverse task is easy for the LLM, which re- sults in high-quality data points. To the best of our knowledge, only Meng et al. (2022b) and Gao et al. (2021) also perform the reverse task by prompting an LLM to generate comments x given a sentiment y. However, their direct task is simple and can al- ready be solved easily by an LLM, and their is not structured.
Hiler subset Wikidata Graph
Sample in (1.MM) tripltt
For cach generale teat (X)
"Armtn" file Croatia dirrcctdd
Aratn S4793E0S
Creath 10024 File 011474
Suiliese.
Figure 2: Synthetic data generation flow. We start by filtering the Wikidata knowledge graph to a subset of relations and entities comparable to REBEL. Then, we sample coherent triplet sets, encouraging uniform cov- erage of relations. Finally, we prompt OpenAl LLMs to generate text for each triplet set.
2.2 Closed Information Extraction
In this work, we exemplify our synthetic data gener- ation approach with elosed information extraction (cIE), which is the task of extracting the exhaustive set of facts from natural language, expressible un- der the relation and entity constraints defined by a knowledge base (KB). More formally, assuming a KB concerning collection of entities & and col- lection of relations R, the goal is to extract the haustive set of fact triplets yse r,) (s) E R x E expressed in given textual input Contrary to previous pipeline approaches (Galár- raga et al., 2014; Angeli et al., 2015; Chaganty Josifoski et al., Josffsssi et al. (2022) proposed GenlE, a model approaching cIE in an autoregressive end- to-end formulation. Their results suggest that GenlE is more sample-efficient and achieves state- of- of-the-art art performance, with substantial improve- ment over previous work. However, despite be- ing 3x higher than the strongest baseline, GenlE's macro-F1 score is less than 35%. They identify the heavily skewed data distribution, with most re- lations appearing only a few times in the training data, as the main reason for the low macro score.
access more annotated data covering the large number of rare relations which cannot be easily obtained using human annotations or distant super- vision heuristics (Huguet Cabot and Navigli, 2021). Instead, our procedure allows us to synthetically generate large amount of data with good cover- age of all relations. This data scarcity problem is not unique cIE, and our procedure can bene- fit other structured NLP tasks such as entity link- ing, olE, or abstract meaning representation pars- ing (Banarescu et al., 2013), as well as other tasks such as writing assistants (Schick et al., 2022).
E x
3 Exploiting Asymmetry for Synthetic Data Generation
In this section, we demonstrate how to exploit asymmetry for synthetic data generation with cIE as an example task. Our pipeline, depicted in Fig. 2, comprises three primary components: (i) construc- tion of a knowledge graph (KG) containing the entities and relations of interest; (ii) sampling of coherent triplet sets from the KG with compre- hensive coverage of the entities and relations, and (ii) generation of tigh-quality text, expressing the triplets without any additional triplets. Next, we describe these three components in turn.
Whereas cIE requires the constituent elements of the output triplets to be entities and relations from the KB, the output triplets in open information traction (olE) are free-form text. Therefore, cIE is a fundamentally harder task than oTE, closely related to full spectrum of tasks that are central to the field of information extraction (e.g., entity linking, relation classification, slot filling, etc.). We focus on the task of cIE as use case for studying the benefits SDG could have for information extraction tasks more generally.
3.1 Knowledge Graph Construction
We start from the Wikidata KG (Vrandevčié, 2012). To remain comparable to previous work (Josifoski et al., 2022), we filter the KG to the subset of 2.7M entities E and 888 relations R appearing least once in REBEL'S training set. Each entity in the KG is associated with a unique English Wikipedia page title, and each relation is linked to a unique Wikidata label, which we use as their textual iden- tifiers (see Appendix B. for details).
2.3
Data as a Core Limitation
As Josifoski al. (2022) noticed, cIE datasets nat- urally present large imbalances in relation frequen- cies. Training on such data results models that perform well only for the few frequent relations and poorly on the rest. Addressing this issue requires

---

3.2 Sampling Triplet Sets
sampled. We denote the two distributions as B and D R. Appendix B.2 details the choice of K.
In the Wikidata KG, nodes represent entities, and edges represent relations between two entities. Therefore, a triplet can be seen as an edge together with its endpoint nodes. For the synthetic data to be valuable, we design the triplet set sampling with two objectives in mind. First, it is crucial that the triplet sets be coherent: the triplets in each set must be able to conceivably co-occur in murttewritten text. Second, the dataset should have (approxi- mately) uniform coverage of entities and relations.
Ensuring coverage. Rare entities and relations do not often appear in other entities' local neigh- borhoods. Even if they are associated with a high probability of being sampled when encountered, they are encountered rarely and, therefore, sampled rarely. One way to ensure that rare entities and relations are selected is to explicitly choose them as a starting point of random walk (i.e., triplet sets). We describe two strategies for achieving this: Entity-centrie strategy: sample the start- ing entities of each triplet set according to the reweighted entity sampling distribution D
Encouraging coherence. We observed that uni- form edge sampling from the KG does not pro- duce coherent triplet sets. Instead, coherent sets of triplets tend to focus on a small number of entities, with one or two entities being repeated in most of the triplets. These entities serve as the "protag- onists", ", or anchors, of the sentence. To capture this property, we propose sampling procedure based on a random walk with backtracking applied to the KG. Concretely, given starting point-a node (i.c., an entity (ub)) or an edge (i.c., a triplet to - (e (0) eoh)) from the KG-we maintain a set of already sampled triplets T, and until the de- sired number of triplets is reached, iteratively sam- ple: subject (77))
(ii) Relation-centric strategy: sample relation according to the reweighted relation sampling distribution D Then, among the triplets corre- sponding relation sample one according to the probability assigned to the subject entities by the reweighted entity sampling distribution Dy renor- malized to the ones available.
In both cases, reweighting favors the rarest enti- ties and relations as the next starting points.
For comprchensive coverage of both entities and relations, employ mixed strategy, which switches between the entity-based and relation- based strategy on every K samples-at at the same time when the empirical relation and entity distri- butions are recomputed.
& starting new triplct; or (ii) an object cbbj E (e(((T))) where Me) corre- sponds the of entities adjacent e, forming triplet (7) = (1 ,,(())) eobj (((71)))00))) to be added to T. The entity sampling is biased towards entities al- ready appearing as subject an object of triplet in by parameter controlling the strength of the bias. The desired number of triplets per set sam- pled from Poisson distribution. Appendix B.2 details the choice of parameters.
sub
(TT)
3.3 Triplet-Set-to-Text Generation
In this section, we denote by query the triplet set for which we want to generate text. Also, we refer to the in-context examples demonstrations. The demonstrations consist of triplet sets and sentences selected from REBEL's training set. When choos- ing the triplet-set-to-text generation setup, we made the following considerations.
Encouraging coverage. When sampling triplet sets from the graph uniformly, some entities and relations are so central that they appear most local neighborhoods. These end up being over- represented, heavily skewing the distribution.
LLM choice. We consider two models from Ope- nAI's GPT series: t. text.-davinci--nn -003 (details in Appendix B.3).
and
To alleviate this issue, we implement an aggres- sive reweighting of the entity and relation distri- bution. After every K sampled sets, we craft new relation and entity distributions, where the prob- ability of sampling specific entity relation is inversely proportional to its frequency in the set of already sampled triplet sets S. As a consequence, after each reweighting, the rarest entities and re- lations are given the highest probability of being
Prompting strategy. We evaluated both models in a zero-shot and few-shot setting. In the zero- shot setting, we experimented with different in- structions. In the few-shot setting, varied the instruction, the number of demonstrations, and the formatting of the demonstrations.
Generation parameters. We experimented with different values for temperature and top-p.

---

min
1st quartile
median
Aed quartlle
max
3.5 Human Evaluation
Wiki <lE Code
65
934 42
1380
3629
429.200
Experimental setup. We randomly selected 50 data points from REBEL's test and synthetically generated the text for the corresponding triplet sets following the procedure outlined Sec. 3.3, with the generation parameters and prompts used to gen- erate the Wiki-cIE Code and Wiki-cIE Text datasets. As result, two more versions of the (small) data- set, differing only in the textual sequences, were created- following the Wiki-clE Code and an- other following the Wiki-clE Text text generation procedure. We evaluate the match between the triplet-set-to-text pairs for each dataset by scoring data points in terms of standard precision, recall, and FI (see Appendix D) for definitions). Con- cretely, we extract the triplets actually expressed in each of the three versions of the text by human annotation and compare them against the corre- sponding target set (i.e., REBEL's gold triplet set) as ground truth. Appendix B.5 details this process. In this setting, precision corresponds to the fraction of triplets expressed in the text that are present the target set, and recall to the proportion of triplets in the target set that were expressed in the text.
Wiki-ClE
Teat
4
62
136
14,323
Table Relation occurrence count statistics.
Micro Recall
Macre Recall
Precision
REBRI.
29.35 +077
56.05
39.87 +740
24.20
Wiki-ellE Code
57.40
70.38 anss
65.08
273
50.70 2330
Wiki-clB Test
84.78 23.0
78:45
28.00
82.97 2339
22.14
Table 2: SDG quality (human evaluation) results.
We ran the inference on selected set of 12 triplet sets from REBEL's validation set and manu- ally evaluated the quality of the generated outputs in terms of precision, recall, and fluency. The best- performing prompt setup and optimal set of genera- tion parameters for both models given in Fig. 5 and Table 4, respectively. In Table 5, we showcase example generations for a few data points. In this setup, we generated two datasets:
Wiki-elE Code consists of around 1.8M training, 10K validation, and 50K test samples generated with code dieeeedaviiciii -02.
Results. The results are summarized in Table 2. First, the results indicate that the synthetically gen- erated text has substantially higher precision and re- call than REBEL'S original text, with Wiki-clE Text reaching 84.8% precision and 78.5% and 72.1% in micro- and macro-recall, respectively. Second, REBEL texts score low in precision (29.4%), sug- gesting that over 70% of the information in REBEL text is absent from the target set. On the other hand, a micro-rectt score of 56,, implies that REBEL'S gold annotations are actually wrong 44% of the time. Finally, our datasets' micro and macro scores are much closer than REBEL's, indicating that our datasets have more consistent quality across rela- tions.
Wiki-clEE Text consists of 10K validation and 50K test samples generated with text-davinci- 003 using the same triplet sets as in Wiki-cIE Code. Appendix B.3 contains the inference costs details.
3.4 Distributional Properties of Data
One important problem we aimed to address with our synthetic data generation is the imbalance in relation frequencies. Table 1 reports the basic statistics of the relation frequency distribution in REBEL, Wiki-cll Code and Wiki-clE Text (see Ap- pendix B.4 for the cumulative distribution plots). While REBEL very skewed towards few relations appearing most of the time, Wiki-cIE Code has a particular, the rarest relations in Wiki-clE Code appears more often than the median relation in REBEL. Josi- foski et al. (2022) show that supervised models perform poorly on rare relations. Therefore, we expect Wiki-clE Code to help supervised models perform well on much larger subset of relations, which necessary for exhaustive extraction. In terms of entity coverage, the training split of our Wiki-cIE Code contains 1,805,504 unique entities, compared to the 1,715,922 in REBEL's.
much more balanced distribution.
Synthetic Data in Action
In this section, evaluate the benefits of training on synthetically generated data.
4.1 Modeling and Inference
Model. Given textual input x, our proposed model, SynthlE, autoregressively generates the linearized sequence representation y of the exhaustive set of facts Yset expressed in x. The conditional probabil- ity (parametrized by f) assigned to the target set

---

b pa xer,)), Ysel is computed as: pa(y) x) -I:- The training consists maximizing the target se- quence's 8 conditional log-likelihood with teacher forcing (Sutskever et al., 2011, 2014), using the cross-entropy loss, and dropout (Srivastava et al., 2014) and label smoothing for regulariza- tion (Szegedy et al., 2016). We use the same task formulation and training as GenlE (Josifoski et al.. 2022), but unlike GenlE, which employs the BART architecture (Lewis et al., 2020), SynthlE is based on FLAN-T5 (Chung et al., 2022), a choice mainly motivated by the availability of pre-trained check- points with different parameter counts.
leverage the datasets generated by the SDG proce- dure described in Sec. 3. Coneretely, we use the larger Wiki-elE Code for training and testing and the smaller Wiki-clE Text for testing purposes only. Following the main evaluation setup from Josifoski et al. (2022), we use their version of REBEL for training and testing in the non-synthetic regime (see Appendix C for details).
Baselines. To isolate the effect of training on syn- thetic data, we keep the same architecture and vary the training (and validation) data. We use the GenlE identifier to refer to models trained in the non-synthetic and SynthIE to refer to models trained in the synthetic regime, See Appendix for details on the apper-parametrrs and the compute time.
Output linearization. To represent the set of facts Yset as sequence of symbols compatible with requence-t-ssquenne architectures, we in- troduce two mappings: (i) fully expanded (FE) and (ii) subject-collapsed (SC), which used by Huguet Cabot and Navigli (2021). While FE concatenates the textual representation of each triplet in the set to obtain the sequence, SC groups the triplets based on the subject entity and con- catenates their grouped representations. For more details and examples, see Appendix F.
Evaluation metrics. We evaluate the performance in terms of micro and macro precision, recall, and Fl, and report point estimate with a 95% confi- dence interval constructed from 50 bootstrap sam- ples. Appendix D formally describes the metrics.
4.3 Results
4.3.1 Human Evaluation on REBEL
Inference. At inference time, it would be pro- hibitively expensive to score every set of triplets in the output space. Instead, we search for the top-k eligible options by using constrained beam search (Sutskever et al., 2014; Josifoski et al., 2022) paired with a strategy for dynamically generating the valid prefixes. More concretely, we enforce a bi- level constraint where (i) the high-level structural constraint asserts that the prefix follows a specific linearization schema, and (ii) lower-level validity constraints (via a pre-computed entity and relation trie) ensure only valid entity or relation identifiers (depending on the given element) are generated.
The human evaluation Sec. uncovers funda- mental flaws in REBEL'S annotations. Approxi- mately 70% of the information from the text is not included in the "gold" set of triplets, and 45% of the triplets are not expressed in the input text (cf. Table 2). As result, evaluation on REBEL would provide highly inaccurate picture of the models' performance. Specifically, triplets missing from the "gold" set Icad to an underestimation of truc precision, while incorrect triplets the "gold" set can result in an overestimation of precision and an underestimation of recall. These findings raise serious doubts about the validity of REBEL an evaluation dataset for cIE. Both of these problems (missing and extra triplets) addressed by our proposed evaluation datasets (cf. Table 2).
4.2 Experimental Setup
Knowledge base constraints. We maintain our world of concern to all the entities and relations from the KG described in Sec. 3.1 corresponding to labels that can be fully tokenized by the model's s tokenizer (i.e., tokenized labels do not contain un- known tokens), which rules out around 5% of the entities the KG. Our final entity catalog contains around 2.6M, and the relation catalog 888 items.
To understand the implications of these limita- tions, we manually annotate 360 randomly selected samples from REBEL. This process results in a new dataset that we refer to as REBEL Clean. We provide detailed information about the annotation procedure in Appendix G.I.
We first evaluate REBEL'S gold triplet sets against the hand-annotated triplet sets, by treat- ing the original ones if they were the output of a model (referred to as REBEL Gold in Table 3). The original annotations achieve an Fl score of 73.8
Datasets. We differentiate between two data regimes: (i) synthetic and (ii) non-synthetic (dis- tantly supervised). For the synthetic regime, we

---

Distant Supervision
Synthetically Generated
REBEL
Wiki-elE Recall
Wiki-elE Code
Precision
Recall
F1
Precision
Precision
Recall
Mícro
REBEL
60.68
73.76
GenlE TS-base
76.06
51.81
+344
62.17
+301
49.10
26.69 =0.17
34.58 +000
41.56 an
23.94 .024
30.38 =0.30
SynthlE TS-hase
53.02
+30
48.05 +341
92.08 2017
90.75 =0.1
91.41 =01
79.99 -02
70.47 +030
74.93 -m
SynthlE T5-bax-SSC SC
59,97 =434
40.76 2014
2157
92.79 2012
90.50
=0.00
91.63
81,58 20.5
69.48 2029
75,05 20.19
SynthIE Tunge
68.25 4401
54.37 1308
61.26 3300
93.38 2011
92.69
20.19
93.04 2013
82.60
73.15 1.29
77.59 20.4
Macro
REBEL Gold
51.21 250
41.02 14.9
43.76 4522
GenlE 15-hase
39.36 Lans
31.46
1424
33.33 4.0
29.82 Lo0
11.14 Lo.5
13.94
25.78
9.81
12.12 co12
SynthIE TS-hase
35.57 1483
34.05 1440
33.13
1.44
94.10 2015
92.42 L0177
93.05 2011
83.76 auw
74.05 104s
77.91 1..2
SynthIF: TS-hase-SC
20.07 1.76
12.82 2005
14.65 +2700
94.35 -0
92.39
-0.30
93.15 -011
84.32 uum
73.57 40a
77.88 -0.1
SynthlE T5-luge
54.11 =5225
52.01 2464
51.04
4477
95.27 #022
94.95
20.13
94.99 20122
86.43 2025
78.78 +027
81,95 e.22
Table 3: Main results. Performance of our model SynthlE, the baseline GenlE, and REBEL's target annotations, evaluated on the hand-annotated but biased REBEL Clean, Wiki-clB Code, and the highest-quality Wiki-clE Text.
micro and 43.76 macro, which is unsatisfactory for dataset intended for estimating model perfor- mance. Furthermore, the significant gap between miero and macro scores confirms that the quality of original annotations varies greatly across relations. Next, we evaluate the predictions from our mod- els and the baseline, and observe that, in terms macro performance, (i) SynthIE T5-Iarge outper- forms REBEL Gold; and (ii) SynthlE T5-base is on par with GenlE T5-base. Crucially, the first obser- vation suggests that the predictions of SynthlE T5- large- model trained on synthetic data generated with our proposed methodology-exhiee quality than REBEL's original gold triplet sets. On one hand, this highlights the quality of SynthTE, and on the other hand, it further undermines the credibility of an evaluation on REBEL.
and two-thirds macro Fl on Wiki-clE Text and Wiki-clE Code. This due to several reasons. Cru- cially, the annotations per relation in GenlE's train- ing dataset, REBEL, are not uniform in terms of representation and quality. As result, the model performs well on a few relations and badly on the rest. This is exposed by the synthetic datasets, which (i) contain triplets expressing every relation seen at least once in REBEL's training set as op- posed to REBEL's test set, which does not cover 300 relations (around a third); and (ii) express all of the relations as uniformly as possible.
higher
Additionally, while GenlE's precision also de- creases, Fl performance is particularly affected by the strong decrease in recall. For instance, on Wiki- cIE Text, in comparison to REBEL Clean, the preci- sion drops by 17 (micro) and 10 (macro) absolute points while the recall drops by 25 (miero) and 20 (macro) absolute points. The drop in recall is more pronounced as consequence of (i) training on an imbalanced dataset (REBEL), with non-exhaustive annotations missing 70% of the information; and (ii) evaluation on balanced, diverse dataset in which almost 85% of the information from the text is present in the target triplet set (see Sec. 3.5).
It is worth noting that the REBEL Clean dataset inherits some important problems from REBEL: (i) high imbalance in terms of the relation occur- rence counts, which can be exploited by models like GenlE (trained on REBEL) to achieve strong (micro) performance despite only performing well for few relations, (ii) text often containing informa- tion for entities that cannot be resolved.
These findings emphasize the importance of the proposed Wiki-clE Text as a reliable evaluation da- taset for the cIE task. By design, Wiki-clE Texi does not suffer from these issues.
On the other hand, SynthIE T5-base, which is trained on data synthetically generated by the pro- posed methodology, and differs from GenlE T5-base only in terms of the training data, achieves 91.4 micro, and an even higher 93.1 macro-Fl score on Wiki-clE Text. Going to the larger SynthIE Ta-le model, the performance on both datasets increases from 2 to absolute points.
4.3.2 Performance Evaluation
On Table 3, we start by noticing that GenlE T5-base achieves an Fl score 62.2 micro and 33.33 macro on REBEL Clean. However, the model's perfor- mance decreases by almost half terms of micro
Finally, the subject-collapsed (SC) lincarization decreases the target sequence length (see Fig. 8)

---

5 Discussion
Syrhhliy
budon roladons
0.8-
Implications for clE. The lack of large, balanced, high-quality dataset has been a significant obsta- cle for cIE. The synthetic data generated by the methodology proposed in this work satisfies all of the desiderata in terms of size, coverage, and qual- ity (cf. Sec. 3.4 and Sec. 3.5). As an alternative to REBEL which greatly suffers from both false positives and negatives, Wiki-clE Text enables a substantially more accurate evaluation.
Nummeer of midiitr
0.6
0.4
0.2
20
0.0
2
Relatio Eequanicy
REBEL
Isiiiing
Figure 3: Impact of the relation frequency. Relations are bucketed based on their frequency: bucket 2' con- tains relations occurring between 2' and 2 times. The histogram shows the number of relations per bucket. The line plots depict the per bucket Fl scores for GenlE and SynthIE evaluated on Wiki-cIE Text, with confi- dence intervals constructed by bootstrapping.
Similarly, Wiki-clE Code enables the training of models performing well across all relations: (i) in terms of macro performance, SynthlE's s predic- tions are of higher quality than REBEL's gold an- notations; (ii) on the highest quality test set (Wiki- cIE Text), SynthIE pushes the macro-Fl score of 14%, for GenlE, to 93% (cf. Table and Fig. 3).
without any performance costs on Wiki-clE Text and Wiki-elE Code. However, the autoregressive na- ture of the model, paired with the often ill-defined nature of the task in REBEL, renders SC an inap- propriate choice for REBEL Clean that comes with a substantial performance cost. This result highlights that the choice of the output format should not be neglected. We further discuss this in Sec. 5.
While SynthlE's performance on REBEL Clean is better than the original gold annotations, it is still lower than the performance on Wiki-clE Text. Analyzing the errors committed by our models and looking at the REBEL data allowed us to uncover an interesting property of cIE that, to the best of our knowledge, has not been identified by prior work. Assume that a model was trained on exhaustively annotated data (the holy grail which is now within reach) and presented with text containing central in- formation that cannot be linked to the KB, (i) either in theory (e.g.. "He plays the guitar") or (ii) under the output constraints assumed by the model. This would place the model in an undesired space of the likelihood distribution where it attempts to ex- press information it cannot express, and is therefore bound to make mistakes. Fortunately, mistakes of this type can be avoided by (i) generating training data covering such scenarios and/or (ii) expanding the expressivity of models by extending the cata- logs or developing novel output formats beyond subject, relation, object triplets. The latter goes towards exhaustive IE, whereas the former relaxes this desideratum. Overall, we believe that SDG could bring us closer to practical cIE systems.
4.3.3 Performance by Relation Frequency
There is a natural imbalance in relation frequencies in text. In existing datasets, most of the triplets correspond to only few relations (Josifoski et al., 2022). Models trained on such data are good at extracting information concerning a few relations and ignore the rest, which is a major obstacle to exhaustive CIE. For this reason, we bucket relations based on their number of occurrences in REBEL'S training set and compute the per-bucket (micro) Fl performance. The results are reported in Fig. 3. For 46% of the relations that have less than 2 32 oc- currences, GenlE's performance is close 0. The model's s performance slowly starts to rise for rela- tions with at least 2 = 32 occurrences, reaching a maximum of around 58% F1 for the few most fre- quent relations. Overall, the performance is higher than 50% for buckets that cover only 1.5% of the re- lations in the dataset. In contrast, training on Wiki- cIE Code, which has a uniform quality and coverage of annotations across relations, makes SynthlE per- form well across all buckets. This translates to an Fl performance corresponding to a straight line at the top of the plot, at around 93%.
Implications for SDG. This study highlights the efficacy of leveraging asymmetries in difficuly for SDG by developing specific pipeline for cIE. However, the idea can be readily applied to dif- ferent pipelines for cIEE, or any other IE or pars- ing task, such as entity linking, oIE, or abstract meaning representation parsing. With cIE, which necessitates the most information external to the

---

input (i.e., knowledge of entities, relations, and their IDs), arguably being the hardest task, we are confident that applying our approach to these tasks would translate to similar results.
ation; Debjit Paul and Veniamin Veselovsky for their helpful feedback on a draft version of the paper; and Akhil Arora for providing us with a mapping of Wikipedia page titles to Wikidata iden- tifiers. West's lab is partly supported by grants from Swiss National Science Foundation (200021_- 185043), Swiss Data Science Center (P22_08). H2020 (952215), Microsoft Swiss Joint Research Center, and Google, and by generous gifts from Facebook, Google, and Microsoft.
Finally, the proposed methodology is not limited to tasks with structured ouiput spaces. Rather, any problem with an inverse formulation that can be addressed more effectively by the LLM can benefit from our approach.
Limitations
Exhaustive cIE. We focus on exhaustive cIE, a for- mulation of the IE task where the model is asked to exhaustively annotate all the information expressed in the text. This focus is reflected in the synthetic data generation procedure, where we generate text faithful to the target set. Notably, obtaining large dataset of comparably faithful text-triplets pairs was not feasible prior to this work. Our results (see Sec. 4.3) suggest that SynthlE, the line of models trained on such high-quality data, can solve the task of exhaustive cIE remarkably well whenever all the information in the text can be linked to the KB. However, when this is not possible either due to missing information in the text limited model expressivity the models are placed in an undesired space of the likelihood distribution and, consequently, fail to achieve the same performance. As discussed in Sec. 5, we can address this by modifying the data generation procedure or by ex- panding the expressivity of our models. However, future work would need to decide how important exhaustiveness is for cIE.
References
Ateret Anaby-Tavor, Boaz Carmeli, Fsster Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue!. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pages 7383-7390.
Gabor Angeli, Victor Zhong, Danqi Chen, Arun Te- jasvi Chaganty, Jason Bolton, Melvin Jose Johnson Premkumar, Panupong Pasupat, Sonal Gupta, and Christopher D. Manning. 2015. Bootstrapped self training for knowledge base population. Proceed- ings of the 2015 Text Analysis Conference.
Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob. Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings the 7th linguis- annotation workshop and interoperability with discourse, pages 178-186.
Arun Chaganty, Ashwin Paranjape, Percy Liang, and Christopher D. Manning. 2017. Importance sam- pling for unbiased on-demand evaluation of knowl- edge base population. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1038- 1988-004, Copenhagen, Denmark. Association for Computational Linguis- tics.
Model bias. Synthetic data generation is an inte- gral part of this work. Naturally, relying on an LLM to generate the training data opens the door for the LLM's biases to be introduced into the newly gen- erated synthetic data well.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gauray Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instuuetionfinetuueed language models. CoRR, abs/2210. 11416.
LLM availability. As described in Sec. 3.3, for the SDG we consider two models from Ope- nAl's GPT 3.5 series: code davinci-OO0 and text-ddavinci--00: While the API for us- ing text-
publicly available, code-davinci-o0n is not. However, getting access to it is still possible through OpenAl's Re- searcher Access Program.
Hady Elsahar, Pavlos Vougiouklis. Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Lafor- est, and Elena Simperl. 2018. T-REx: A large scale alignment of natural language with knowledge base triples. Proceedings of the Eleventh International Conference on Language Resources and Evaluation
Acknowledgements
We would like to thank Jiheng Wei, Yifei Li, and Saibo Geng for their help with the human evalu-

---

(LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022b. Generating training data with language models: Towards zero-shot language understanding. arXiv preprint arXi::202.044: 04538.
Luis Galárraga, Geremy Heitz, Kevin Murphy, and Fabian M. Suchanek. 2014. Canonicalizing open knowledge bases. In Proceedings of the 23rd ACM International Conference on Conference on Infor- mation and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pages 1679- 1688. ACM.
Yu Meng, Martin Michalski, Jiaxin Huang. Yu Zhang, Tarek Abdelzaher, and Jiawei Han. 2022c. Tun- ing language models training data generators for augmentation-ennanced few-shot learning.
Biswesh Mohapatra, Gaurav Pandey, Danish Contrac- tor, and Sachindra Joshi. 2020. Simulated chats for building dialog systems: Learning to gener- ate conversations from instructions. arXiv preprint 1.Xi::0010.102/::
Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2022. Zerogen": Self-guided high- quality data generation in efficient zero-shot learning.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 38 3816-3830, Online. Association for Computa- tional Linguistics.
Yannis Papanikolaou and Andrea Pierleoni. 2020. Dare: Data augmented relation extraction with gpt-2. arXiv preprint arXiv:2004. 13845.
Gaurav Sahu, Pau Rodriguez, Issam H. Laradji. Parmida Atighehchian, David Vazquez, and Dzmitry Bah- danau. 2022. Data augmentation for intent classi- fication with off-the- shelf large language models.
Or Honovich. Thomas Seialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. arXiv preprint arXiv:2212.09689.
Timo Schick, Jane Dwivedi- Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebas- tian Riedel. 2022. Peer: A collaborative language model. arXiv preprint 12X.::2208..166:
Pere-Lluis Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2370- 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zhihong Shao, Yeyun Gong, Yelong Shen, Min- lie Huang, Nan Duan, and Weizhu Chen. 2023, Synthetic prompting: Generating chain-of-thouge demonstrations for large language models.
Ryan Smith, Jason A. Fries, Braden Hancock, and Stephen H. Bach. 2022. Language models in the loop: Incorporating prompting into weak supervi- sion.
Martin Josifoski, Nicola De Cao, Maxime Peyrard, Fabio Petroni, and Robert West. 2022. GenlE: Gen- erative information extraction. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4626-4643, Seattle, United States. Association for Computational Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Re- search, 15956):12991195.
Varun Kumar, Ashutosh Choudhary, and Eunah Cho. 2020. Data augmentation using pre-trained trans- former models. arXiv preprini arXiv:2003.02245.
Ilya Sutskever. James Martens, and Geoffrey E. Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference Machine Learning, ICML 2011, Bellevue, Wash- ington, USA, June 28 July 2, 2011, pages 1017- 1024. Omnipress.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. pre-training
BART: Denoising gqueecc-to-sequenee for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. 2014. Sequence sequence learning with neural networks. In Advances in Neural information Processing Sys- tems 27: Annual Conference on Neural Information Processing Systems 2014, December 8- 8-13 2014, Mon- treal, Quebec, Canada, pages 3104-3112.
Junlong Li, Zhuosheng Zhang, and Hai Zhao. 2022. Self-prompting large language models for open- domain qa.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zhigniew Wojna. 2016. Re- thinking the inception architecture for computer vi- sion. In 2016 IEEE Conference on Computer Vision and Paltern Recognition, CVPR 2016, Las Vegas,
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022a. Generating training data with language mod- els: Towards zero-shot language understanding.

---

USA, June 27-30, 2016, pages 2816.886. IEEE Computer Society.
Denny Vrandevčić. 2012. Wikidata: A new platform for collaborative data collection. In Proceedings of the 21st international conference on world wide web. pages 1063-1064.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task task benchmark and analysis platform for natural language understanding. arXiv preprint vrXiv:1804.0744
Zirui Wang, Adams Wei Yu, Orhan Firat, and Yuan Cao. 2021. Towards zero-label language learning. CoRR, abs/2109.091
Yiben Yang. Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta. Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, Doug Downey. 2020. Generative data augmentation for common- sense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1008-1025, Online. Association for Computational Linguistics.
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiang- tao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. Zerogen: Efficient zero-shot learning via data- set generation.

---

A LLM cLE failure cases
parameter
code-davinci-002
text-davinci-003
In Fig. 4 we showcase examples of LLMs not being able to solve the problem of cIE effectively. While text--daas nci-003 can often identify the core information in the text, it is not able to map the names to the Wikidata entities relations.
max_tokenn
100
50
temperature
0.7
0.7
top-p
1
frequency_penatty
0.2
0.2
presence penaltteeeettyyyyee
0
0
stop
"In"
B Synthetic Data Generation
best_of
5
In this section, we give details on the sampling and the synthetic data generation process. Addi- tionally, we provide examples of prompts used to generate the data in Fig. 5, as well as generated sentences with text davinci-000 and code- davinci-002 in Table
Table Optimal generation parameters the LLMs used in the SDG.
but allows for more diversity, whereas 10 always focuses on a single anchor entity. Therefore, we fix the bias factor 7.
B.1 Details about the Knowledge Graph
For sampling the starting point, we opt for mixed strategy described in Sec. 3.2. The parameters for this procedure are the dampening factor d of the entity and relation distributions as well as K: how often do we recompute the entity and relation distri- bution from the empirical observation in the current sample (and switch strategy). To choose these pa- rameters, we sample 120K data points with varying ([0.01, 0.05, 0.1, 0.5, 1]) and K (I2K, 10K, 20K, 120KJ). A dampening factor of means damp- ening, and K N the number of samples means no computation of empirical distributions and only using the relation based strategy. We then look at the skewness, entropy and median number of appearance of relations among the 120K sampled triplet sets, as well as the number of entities cov- ered. We found that good compromise is given by d = 0.01, K 20K.
We first select entities and relations which appear in the REBEL dataset as described in Sec. 3.1 and relations that do not take literal arguments. We filter out all the entities whose names cannot be associated with Wikipedia page. This subset com- prises 2,715,483 entities and 888 relations. We also exclude entities with a degree of 0, as they do not contribute to any facts the graph. As a result, our filtered graph includes 2,715,483 nodes and 17,655,864 edges. It is worth noting that our synthetic data generation approach can easily be applied to larger subsets of the Wikidata KG or even the full KG. However, we subsample the KG remain comparable to previous research.
B.2 Triplet Sampling
To sample a coherent triplet set, we follow the iter- ative procedure explained in Sec. 3.2. The param- eters involved in this procedure: (i) the number of triplets per triplet sets and (ii) the bias factor. We sample the number of triplets from Poisson distri- bution of mean 3. Assuming a triplet set compris- ing N different entities, the next (subject or object) entity is sampled from a probability distribution where entities: (i) not appearing in the triplet set are sampled with probability proportional to 1; (ii) entities already appearing in the triplet set are as- signed probability proportional to
B.3 Triplet Set to Text
Choice of LLM. The code-davinci-002 model has been trained on mix of language and code with further instruction finetuning. text- davinci-00s was further finetuned with rein- forcement learning from human feedback making more effective at zero-shot learning with instruc- tions but less capable of in-context learning. We query the models through the OpenAT API.
where bf is the bias factor, and r is the rank of the current triplet. To choose the bias factor, fix random seed for starting nodes and apply the sampling procedure described above for varying choices of bias factor ([1, 3, 7, 10]). We manually inspect the result triplet set and judge their coher- ence. We find that 7 and 10 yield good coherence
Inference costs. At the time of writing, code- davinci- 002 was free with limitation of 20 requests and 150k tokens per minute. Therefore, the cost for constructing the Wiki-clE Code data- set was 0. text-davvnnci-ee at $ 0.02 per 1K tokens, and the total cost of con- structing the Wiki-clE Text was $ 223.55.
was priced

---

Sentence
ser Rerteroa in fimm Souleck, reaten cleect) fest, more
Songe Fred originlly Warner fimes
Edhasshelly excress the lnfordiio monion
ingrass Trap sia should cover all intormataon from
Folk So For Out Folk is an dlcun by Frod ofgifiilly
Triplets:
Sungs Out
albu by Fred
frok Sonpx, for taa, Out frake
orighn released Wamer Bross
perforerr Fied_ Kazz jooliett
(Fred Fort 3tum
Foa Bonge Our Foli
Folk Songa, Far: Out Folk, tratanic
Songe For Folk, altut
Fer: lsbal, Varmer Iime
Fied Katz
38um
Folk Songo Far Our PoRd
Ful Songs For Out Folk, by Free Kata
Bros)
Fndd Kalz 4bam
Falk Bonie Far Our Folk
code-dadees-222
Ediaustivey excresa the infommtiio from sentence a form (sudject, reledon, object) Tippess oower information
Extoust
otjedij Whkkls Trppets shousd cower al infommtion from lest. but more
informatio
form relation
Aathus strport serves the city hartus.
Aathes aipoat survica the Autua.
Trowes: I ('uegeon Arrtur Alport, "wetetoo" place served Imanspont huc, "otject"
('sujjoot haruus Arport "ndition: place served by trnsport hub, "ooject" Aartus]
recourtng afvertiess Luphn
Wisbin Lupin, Gentlemon Burgla first collection
Maurice
rocouniing odvemarcs Arsène Lupii.
Lupan Burpler oolection stories by Maurica Lettonc
Trewss
"hiNeet: Amdeoo Lupin Conntemann Burgle "nelation: author "ooject" satennff
Mouric__ Lebanee t'sujjor Anstnn Luph, "hlttoon" creator "oejern Mourice Lebbane)
Gondoman Eurola: "rodation "obreot"
wnd)
Pork Songs for For Out Folk buu by Fred Kata originally re relosedd Warner Bros.
That:
Folk Songe for For Out Folk album by Fied Katz originall released Wamer Bros,
Triskta
Poulioor Fork Songs, for Far Out Fol: "reatorn ortat "ootect" Kaed ('sutjoot: lajeon Wanter lime)
Khemer time; (hrnjesf FRl, Srnnge mo. Far_Crr, rook "reastonn anten jeyy Kild)
Songs For Out weett abum receaee "ob
Figure 4: Examples failure cases of LLMs attempts solve cIE task. In some cases, models are not able to recognize all the facts present in the sentence. Even when this is possible, they not able to map subjects, relations, and objects to Wikidata concepts.
Exhaustivly information from tripiets qughiqualty, flunt, natural The leod should contein all the infommaiinn from the prpleis, but no more.
Triplets:
Demonszation
toxt davinci 000
( ''uunjec""A Aathus Airport "re relation:: place served by transport hub, "object": Aartue)
information
Text:
hiph qualtyy fuent netural toxt. The toxt should contain The metion the hnprets, bui more.
sirport serves the city of Aarhua.
Trilets:
"subject" CoriiottAArrnnrr Lupin,
Arsere "reation":
Burgiar, "relation author, "abicet" Loblanc) Maurice Lebland)
"uuject The Sacaalls. "ralation" publicatoon date. objac"" 1979) Tre screento for "ajjc" Louds L'Amou)) ('uubeeff Louis L'Amour, Telaloo" gante "ojject Westem (genre))
("Suh beet Logend the Golden "relation":
Triplete:
date "oosedt
Arsèno Lupin, Gondeman Burbior the first collection stories by Maurice Lebiono recouniing nigg the adventuree oosoo Aroien Lupi.
Triplets
1979) "suryes"" The Sacketts, the publicsion "otject" 1879) Csubject"
"relation": publication
Legand Golden Gun The Socivotts were both 1879. Louie L'Amour screerçlay for the fimm Shadow Riders and & renowhed author Wester genre.
Text:
Shadow Ridars (flm), "elelco" acreewwller, "objer"" Lous L'Amouu)) ('uljje"" Louis
L'Amour, "elatoo" genre, "object" Westem igenrel)
Logand Goldon L'Amour, who whole westerne, erote the screen play for The Shadow Riders (fim).
Sacketts both publisned 1978, Louis
Figure 5:
Best performing prompts. We present the berf-perf prompts both models, Lext-davinei- 003 and code davvddvinnc 0002. code code-davinci. Dlavinci-00 1eci-0000 -002 makes use of demonstrations, Text highlighted in green corresponds to the output of the model.

---

dicted triplets and allows us compute the micro- and macro-recall.
100.00 0+
Percentage of relations
75.0-
Number of triplets expressed in the text. For this analysis, we aim to exhaustively annotate all the information in the text, even that which not contained in the triplet set. However, doing this properly requires a non-trivial understanding the closed information extraction task. Because of that, we opted to estimate this number with three of the authors acting as annotators. Appendix B.5.4 contains the details of the annotation procedure. Given the number of total triplets in the text, we can calculate the micro-precision using the standard definition. This estimate of the precision is, fact, (very) tight upper bound on the true precision for each model, as there may exist triplets that were accidentally missed by the annotators, Finally, we note that to enable exhaustive annotation, we relaxed the catalog constraints on the relations in the annotated triplets, which prevented from computing the macro-precision.
50.0
Wiki-clE Tee Wiki-clE REBEL
25.0
0.01
10
10'
10'
Number of occurences
Figure 6: Cumulative distribution function (CDF) plot of the relation frequencies each dataset. The relation frequencies in Wiki-cIE Code and Wiki- cIE Text have a similar CDF graph follow similar the difference
distribution - shifted on the x-axis in the dataset size. In contrast, the relation frequency distribution REBEL heavily skewed, with most relations having few occurrences. Despite being larger than Wiki-eIE Code, more than half of the relations in REBEL have fewer occurrences than the least frequent relation in Wiki-cIE Code.
B.4 Distributional Properties of the Data
In Fig. 6, we showcase the cumulative distribu- tion function of the relation frequencies in REBEL, Wiki-clE Code and Wiki-elE Text, providing a more complete view of how REBEL differs from the synthetically generated datasets on this important dimension.
B.5.2 Human Annotation Task
For each annotation task, presented the MTuurk workers with single text. Along with the text, a list of potential triplets was provided, and the workers were instructed to indicate which of the triplets were present in the text. In the instructions, we included an example text, set of triplets paired with explanations why each should or should not be marked, depicted in Fig. 7.
B.5
SDG Evaluation
B.5.1 Computing Performance Metries
To compute the precision and recall (cf. Ap- pendix D), we need the number of: (i) target triplets; (ii) correctly predicted triplets: (iii) triplets expressed in the text (i.e.. the total number of pre- dicted triplets).
B.5.3 Quality of annotations
To ensure high-quality annotations, took the following measures:
Crowdworkers criteria. In an attempt to get reliable ratings, we recruited workers that have previously completed at least 1000 tasks with 99% acceptance rate. Workers were stricted by location to US, UK and Canada. We targeted pay rate of $ 8-10 per hour, guided by US minimum wage.
Number of target triplets. This number can be trivially calculated by simply counting the num- ber of triplets in the target set, which comes from REBEL.
Number of correctly predicted triplets. We esti- mate this quantity by hiring Amazon Mechanical Turk (MTurk) workers to annotate the data. They are presented with sentence and a set of triplets and asked to select those actually expressed in the sentence. A more detailed description of the task is provided in Appendix B.5.2. Appendix B.5.3 details the quality checks and the inter-annotator agreement for the task. In conclusion, this proce- dure leaves us with the number of correctly pre-
Honeypots. Each MTurk task consisted of 10 rating tasks. Among the 10 rating tasks, there were 1-2 honeypots, each constructed by randomly selecting
different REBEL samples and pairing the sentence from the first sample with the target set from the second. this case, workers were expected not to mark any of the given triplets, as none of them were

---

Multiple annotators. Each task was done by three different workers. The final set of Which theee
serience? 1110))
Santence The 2004 Monacs Grand Pro the Giand Pirx Monasii Formuls
bncilded bads reymoadd Fooeaal
Ciruut lested
2004 Formuls
memppeene somes de
Cipass Monacc Grand Pite locasion: Chicut de Monaco) i0004 Monaez Grand Fiig ever dirowoee 77)
Facts net:
cor ol cles man en de ourser pun our move
CD004 Monazo fiand Frik san ot 2034 Formllo One World Cha mpiooneti
22000 Formala Worki Champienllpp point lame, 2004)
C0004 Permula One World Champorat
comportoon Porrala Orai
Scnade crtncc 'Dden comean bors hemennen' rind""
beok boon pessent aate staes you sbouc
Lie fiee mrams Evemphunee- tee mpliety in
Figure 7: Mturk setting, Workers presented with the sentence and a list of triplets. Their task is to decide which triplets are present in the presented sentence, They are also presented with detailed instructions and examples.
expressed in the sentence. This allowed to filter out unreliable workers.
alignment between sentences, Wikipedia hy- perlinks, and their corresponding Wikidata enti- ties and relations. REBEL proposed an alignment expanding on Elsahar et al. (2018), pipeline mention detection, coreference resolution, entity disambiguation, and then mapping triplcts to each sentence. Huguet Cabot and Navigli (2021) fur- ther filtered false positives using natural language inference model to check if the relation entailed by the text. We use the processed version of the dataset for training and testing. Sec. 3.4 contains additional statisties for the dataset.
triplets expressed in the sentence was con-
structed by considering the majority vote for
each of the triplet in the set. The Fleiss' kappa was 0.2239.
truly
B.5.4 Precision Annotation Procedure
The annotation performed in two steps. In the first step, one of the annotators annotated every sen- tence with the triplets expressed the text external to the original triplet set. the second step, two other annotators independently decided whether each annotated triplet was indeed expressed in the generated text. Conflicts were resolved via dis- cussion. This was done without knowing whether the text-triplets pair came from REBEL synthetically generated datasets. The estimated prediction was constructed as the union of the original triplet set and the missing triplets that the annotators provided. data points were excluded from the final analysis due to problems with the original data.
D Performance Metrics
We measure standard precision, recall, and FI for all settings. A fact is regarded as correct if the relation and the two corresponding entities are all correct. More precisely, we denote the set of all predicted triplets of document E D as Pa, and the set of gold triplets as Gg- Then:
of
micro-preeistoe and
Σ Pan deD
Σ deD
Pal (1)
micro-recall
Σ Pannaal
Σ
Gal
(2)
C Datasets
dED
deD
REBEL Following the processing in Josifoski et al. (2022), we adapt REBEL (Huguet Cabot and Navigli, 2021) for close information extraction by linking entities their corresponding Wikipedia page and filtering out those that do have an associated Wikipedia page (i.e., entities not asso- ciated unique name). The original dataset is created from Wikipedia abstracts. consists of
Micro scores are useful for measuring the overall performance of model, but they are less infor- mative for imbalanced datasets (c.g., when some entities or relations are disproportionaaely more present in both training and test sets). Indeed, mi- стo scores assign equal weight to every sample, while macro scores assign equal weight to every class. For this reason, we also measure and report

---

triplets
rebel text
text-davinci-003
code-davinci-M02
(Samsung Galaxy Xcover
Samsung Galaxy Xcover Android smartphone produced by Samsung Electronies and released July 2015.
Sumsung Guluxy Xeover smartphone manufactured by Samsung Electronics that the Android operating system.
The Samsung Galaxy
operating system; Android
smartphone manufactured by Samsung Electronics that
(operating system)), (Sam-
sung Galaxy Xcover 3; man-
Audroid operating system.
ufacturer, Samsung Electron- ies)
(Cleveland Cavaliers; league;
During his freshman year he helped lend them to West Coust Ath- letic Conference championship and into the NCAA where Pepperdine would lose the first round. Muidlehrooks was named the WCAC Freshman of the Year. The following scason, the Waves once again won the WCAC, qualified for the 1986 NCAA Tour- nament, and lost This would Middlebrooks' last experience with the NCAA Tourna- ment the Waves did make it during his junior season and only qualified for the 1988 National lavi- lation Tournument in his senior year. Pix Brook a stream that flows through Letebworth Garden City in in Bediori- shire, and meets River Hiz north of Arlesey.
The Cleveland Cavaliers are a professional baskethall team thut plays the National Basketball Association (NBA).
The Cleveland Cavaliers are a pro- fessionul haskethall team hased in Cleveland, Ohio. They compete the National Basketball Associ- ation (NBA) member of the league's Eastern Conference Central Division.
National Baskethall Assacia-
tion)
round.
(Pix Brook; located in the ad- ministrative territorial eatity; Hertfondshire), (Pix Brook; mouth the watercourse;
Pix Brook is located in adminis- trative teriitoiial entity of Hertford- shire, and tercourse River Hiz, which trib- utary of Pix Brook. Hertfordshire shures border with Redfordshire, and likewise Bedfordshire shares border with Hertfordshire.
Pix Brook a tributary of the River Hiz and is located in Hert- fordshire, which burders Bed- fordshire.
Hertfondhhrre,
mouth
wa-
River Hiz), (Hertfordshine: shares border with; Bedford- shire), (Redfordshire; shares border with; Hertfordshire). (River Hiz; tributary: Pix Brook)
(1999 Nunavut general elec- tion: point in time; 1999)
lquurialu July 2, 1954)
general tion took place 1999.
The Nunavut general elec- tion held 1999.
former territorial politician from Clyde River, Northwest Terri- tories (now Nunavut), Canada, He served as member of the Nunavut Legislature from 1999 until 2004.
(Poltava Governorate: coun- try: Russian Empire)
Andriy Bobyr was bor in the vil- lage of Nychyparivka, in the Pollava Governorate Russion Empire.
The Poltava Governorate was part the Russian Empire.
Poltava Governorate was part of Russian Empire.
(United State;; contains
This genus comtains six species that distributed
contains ad- ministrative territoiial entity of Wis- consin, which is located within the United States.
United
Wisconsin state in the United States.
ministrative territorial entity:
the Eastern continental United States far
Wisconsin), (Wisconsin; lo-
cated in the administraiive ter-
Oklahema and Wisconsin.
ritorial entity; United States)
(Two Weeks with the Queen;
Two Weeks with the Queen a 1990 novel by Australion author Morris Gleitzman.
Two Weeks with the Queen, novel by Morris Gleitzman, was first published in 1990.
Morris Gleitzman's 's book "Two Weeks with the Queen" was pub- lished in 1990.
publication date; 1990), (Two
Weeks with the Queen; nu-
thor; Morris Gleitzman)
(Ciudud Bste; country; Paraguay)
Sarubhi", foot- ball stadium in the city Ciudad Este, Paraguny.
Antonio
Ciudad Este located Paraguay.
Este city in Paraguay.
Ciudad
Table 5: Text comparison for REBEL triplet sets This table contains the original REBEL data, as well as text generated using two OpenAl models, Synthetic data samples are hetter in terms of recall, especially precision, and remain fluent. Samples that are overall better in terms precision, recall, and fluency are bolded.
and text-davinei-00f for the same triplet sets.

---

Dataset
# Data Points
# Triplets
# Entities
Relations
val
train
val
train
REBEL
2,813,210
155,926
156,449
7,187,915
397,326
398,252
2,038,741
205 0800
205,549
1071
691
690
REBEL"
2,069,780
114,448
114,953
4,642,624
256,327
257,129
1,537,472
151,617
151,997
863
595
580
Wiki-clE Code
1,815,378
10,000
50,286
6,055,911
34,262
172,991
1,806, 126
27,553
105,176
888
883
888
Wiki-cIE Code"
1,669.708
9,222
46,210
5,482,658
31,073
156,350
1,668,198
25,295
96,337
876
869
876
Wiki-elE Text
10,000
50,286
34,262
172.991
27.553
105.176
883
888
Wiki-clE Text"
9,230
46,295
31,117
156,805
25,323
96,483
869
876
Table 6: Statistics of the datasets. "The filtered version of the dataset used in this work. We filter out data points that have: (i) triplets in their respective target set corresponding to entities and relations outside of the pre-defined knowledge base constrained (cf. Sec. 4. )); (ii) imput longer than 256 tokens; (iii) linearized output longer than 256 tokens (always according to the longer fully expanded linearization schema in order keep the same data points across runs).
performance in terms of macro scores. If we de- note p(r) and G as the predicted and gold set only containing the relation E R of document d, then macro-precision is defined as:
better-optimized set of hyper-parameters gives an advantage to the baseline models, which we expect to be insignificant. The complete param- eter configuration and the code to reproduce the experiments is available in the GitHub repository provided in the abstract.
Σ Pa
n
Σ DED
(3)
R
F
DED
Inference. Following Josifoski et al. (2022), we use Constrained Beam Search with 10 beams. We normalize the log probabilities by sequence length and allow for any number of n-gram repetitions. Additionally, we experimented
and macro-recall as:
Σ
pl (nok!
Σ dED
(4)
R
the validation set with the value of the length penalty parameter. The value of 0.8 was optimal for the models with fully- eully-expanded linearization and 0.6 for the mod- els with subject collapsed linearization. The other parameters are kept to their default values.
ER
dED
E Experiment Implementation Details
Data.. The train, test, and validation splits are inherited for REBEL (Huguet Cabot and Navigli, 2021) and sampled at random for the newly re- leased datasets. We restrict the data points to those with input and target sequences with at most 256 tokens. To facilitate reproducibility, release the exact splits used in our experiments.
Infrastructure and compute time. For training all of the models except for the SynthlE T5-large, used single machine with 24 Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz processor cores and 441 GB of RAM, equipped with Tesla V100- PCIE-16GB GPUs. Each training run took 40- 45 wall-clock hours (160-180 GPU hours), and each inference run on REBEL took around 16 wall- clock hours (64 GPU hours) and 6 wall-clock hours (24 GPU hours) on the Wiki-clE Code and Wiki- cIE Text datasets.
The models were trained using the Adam optimizer with a learning rate of 3e-4, 0.1 gradi- ent clipping the Euclidean norm, and weight decay of 0.05. We trained the models for 8000 steps, with a batch size of 2568, and a polynomial learning rate scheduler with 1000 warm-up steps and a final learning rate of 3e-05. Following the hyper-parameter optimization for GenlE (Josifoski et al., 2022), we used the default values for most of the parameters and tuned: the number of training and warm-up steps, the batch size, and the weight decay. Importantly, due to the high training costs, the parameters were tuned on the REBEL dataset (the GenlE models), and the best performnnnnnng set parameters was reused for training on the syn- thetic datasets (the SynthIE models). Having a
Training.
The SynthlE T5-large model was trained on a ma- chine with 96 Intel(R) Xeon(R) CPU @ 2.20GHz processor cores and 680 GB of RAM, equipped with 8 Tesla A100-PCIE-40GB GPUs. A train- ing run, in this case, took around 30 wall-clock hours (240 GPU hours), and an inference run took around 11 and 4 wall-clock hours (88 and 24 GPU hours) on REBEL, and the Wiki-clE Code or Wiki- cIE Text datasets, respectively.

---

F
Output Linearization
Triplet Set
[Mount Latning: assunce: Mpuntann, (Mount Laming: mountain.
Sentuet Rangel, (Newesmer Glacler,
Sentictt Range!
Fully Expanded
Mo] Lanning instanse Mo] [e][a] Moust Lan-
sing [r] mountain mane [o] Senvinel, Ringe [c] [s] Nowoomer.
Glacie [r] mouttois raige [e] Seatine
[e]
Subject Collapeed
[5] Mount Lamning [2] instance [oj Moumain [e] mountoin
Perconage datapcints
Text (PE)
ragg [o] Sertinel Ringe [e] [4] Newoomer Obuier [i] mounoin
(Si)
range [o] Senvinel Range [e]
1.0
Table 7: Example for the different linearization meth-
ods. the first row, we showcase the original triplet set. The following two rows show the linearization of the original triplet set according to the two methods we consider: fully expanded and subject collapsed.
0.5
0.0
100
200
300
400
Number of tokens
the datasets considered in this work. This effect is particularly pronounced for Wiki-elE Code and Wiki-cIE Text where the triplet sets are more exhaus- tive, leading to repetitions. For REBEL, which is missing many triplets from the target set the differ- ence in output length between the two linearization methods is not substantial.
Figure 8: Histogram of the number of output tokens according to different linearization strategies. The subject-collapsed (SC) linearization results in shorter sequences an effect which is particularly pronounced for Wiki-elIE Code and Wiki-elE Text where the triplet sets are more exhaustive.
In Sec. 4,1, we introduced two output lineariza- tion methods: (i) fully expanded (FE) and (ii) sub- ject collapsed (SC). This section expands on the two methods. Table 7 presents example outputs.
While the sequence representation has an intrin- sic notion of order, the output set of triplets does not. To mitigate the effects of this discrepancy, we enforce consistent ordering of the target triplets during training by considering first the triplets cor- responding to subjects appearing earlier in the sen- tence. Ties are resolved by the appearance position of the object entity. Whenever the triplets' entities are not linked to entity mentions in the textual in- put,
Fully expanded lincarization (FE). FE mapping, used by GenlE, starts by linearizing each (subject, relation, object) triplet by using the delimiters [s]. [r). [o] to demarcate the start of the subject entity, the relation type, and the object entity, respectively. The end of each triplet is demarcated with (e). The final representation y is constructed by concatenat- ing the textual representations of all of the triplets in the set Yser- The advantage of this method lies in its simplicity. However, with most textual se- quences introducing several facts per entity (most concerning the same subject), many entities are repeated. To alleviate this, we consider the SC mapping.
heuristic that links cach entity to the largest sequence of words from the textual input appearing in the entity name (or to the beginning of the sentence in case of no overlap).
G SynthIE in Action
G.1 Human Evaluation of REBEL
G.1.1 Constructing REBEL Clean
Subject collapsed lincarization (SC). SC map- ping, used by Huguet Cabot and Navigli (2021), starts by grouping all of the triplets based on the subject and then linearizing them separately by expressing the group's 's subject once and then list- ing the relation and object for each of the triplets in alternating fashion, demarcating the start of each part with the previously introduced delim- iters. The end of each triplet group is demarcated by the same delimiter [c], and the final represen- tation is constructed by concatenating the textual representations of all of the triplet groups. The SC linearization results in shorter target sequences at the cost of more complex subpart dependencies in the output. Figure Fig. 8 captures this effect for
Choice of data points. We started by randomly selecting 1000 data points from REBEL'S test set, ordering them according to their corresponding numeric identifier (ID), and printing their ID and input text (crucially, without looking at the target triplets). By manually inspecting them in order, we selected data points until we reached 360. We used two simple criteria for choosing the data points,
Criterion 1: The text should have substantial "extractable" information.
One of the samples that were discarded due to this criterion is "In addition to saxophone, he plays
1350, and 10 as backup in case of annotation issues, but we did not encounter any issues and kept all of them.

---

clarinet, bass clarinet, French horn, flute, and cornet.".
We start from the assumption that any candidate target set should be super-set of REBEL's orig- inal target set, which we know is not exhaustive. Therefore, to increase the coverage, we enlarge the target set in REBEL. with the predictions of our best model SynthIE T5-large, which is trained to provide exhaustive annotations. The human annotators will then filter out any incorrectly annotated triplets in the candidate target set.
Rationale: Without extractable information, we would be spending our "sample budget" on examples that cannot be resolved even theory and are, therefore, misleading and not informative. Criterion 2: The central information in the text should not be of a literal type.
One of the samples that were discarded due to this criterion is: "Incumbent Republican Alfred Driscoll defeated Democratic nominee Elmer H. Wene with 51.54% the vote."
This design of the human- aanootation procedure ensures that (i) our estimate of the true precision will be correct (up to human error) and (ii) the mea- sured recall will be an upper bound with a constant multiplicative gap to the true recall for all the mod- els of interest (equal to the portion of triplets that are potentially missing from the curated target set). We discuss these consequences in Sec. G.1.2 and Sec. G.1.3
Rationale: GenIE (Josifoski et al., 2022) does not support literals. While extending GenlE (and therefore SynthlE) to literals is possible, that is not the focus of our work. To isolate the effect of synthetic data, we choose to keep the same setting as Josifoski et al. (2022) and limit our SDG relations that do not rely on literal arguments. Therefore, following the same reasoning as in Criterion 1, data points for which the central information involves literals would be misleading and not informative.
Human annotation task. This annotation task followed the same format as the task described in Appendix B.5.2. However, to ensure the highest possible quality, for this task, instead of relying on MTurk workers, we hired two Ph.D. and two MSc students, who were not familiar with our work (to avoid potential bias). For their time, they were compensated 25 CHF per hour.
Construction of target-triplet candidate sets. is relatively straightforward and accurate to estab- lish which of the triplets in given set are expressed in specific text by considering whether each of them is expressed separately (see Appendix B.5). However, providing workers with our entity catalog containing around 2.6M and the relation catalog of 888 items and asking them to annotate the sen- tence exhaustively is a difficult task that is bound to result in inaccurate annotations. To circumvent this issue, instead of asking annotators to annotate a sentence exhaustively, we cast the task of "al- most" exhaustive annotation in the same format as the task of estimating precision by providing the annotators with larger "target set" of candidate triplets (with high recall) and ask them to establish which of triplets are indeed expressed in the text. This process will be as exhaustive as the recall of the candidate set, i.e., we will correctly detect the triplets within the candidate set that are expressed in the text and overlook any triplets outside of the candidate set.
We conducted the annotation in two stages. In the first stage, of the Ph.D. students annotated all of the data points, while each MSc student an- notated half of the data points such that every data point was annotated twice. In the second stage, the second Ph.D. student annotated each data point where the two annotations (from the first Ph.D. stu- dent and from one of the MSc students) did not match, resolving the conflicts.
G.1.2 Precision on REBEL Clean
To compute the precision (see Appendix D), we need the number of: (i) predicted triplets; and (ii) correctly predicted triplets.
The total number of predicted triplets across all documents d DD, by the model m, is given as:
Pm
Σ Pa JED
(5)
where Pmd is the set of triplets predicted by model for document d. This number can be trivially calculated for any model and dataset.
In the construction of REBEL Clean, for each datapoint, we construct the candidate target set as the union of the triplets in the corresponding target set in REBEL and the set of predicted triplcts by SynthIE 15-large. The rationale here is as follows.
Let G5 be the gold set of triplets in the
REBEL Clean dataset and G = U the tree set of target triplets where U corresponds to the

---

set of triplets in the that are missing from Ga- With this notation, the total number of correctly predicted triplets across all documents d E D, by the model m, is defined as:
A similar argument can be made for the macro- recall (sce Appendix D for the formal definition). Let clrl calrl G G denote their respective quantities before but computed the subset of triplets corresponding relation E R. Then, for model m, we have:
C
Σ
P A nGal
dCD
(m(r)
Since G2 is unknown in practice, to estimate preci- sion, we rely on
macro- R,
[R
(
VE
Gm ElPam dED
Pa nGa
(7)
R
PER
dr)
ELED /
However, in the construction of the gold of triplets in REBEL Clean, for document d, the human annotators considered the triplets Aa = Pa.ReBELous U Sy.ShIE 75 large where PREBELOWS corresponds to the gold (target) set of triplets in REBEL (as the predictions of a model REBELGod and PsynilE 75 large to the predictions from SynthlE T5-large. As consequence, for REBELGold and SynthlE T3-large the estimated and the true total number of correctly predicted triplets, and therefore the estimated and the true micro- and macro-precision precision will be equivalent (up to human error). More generally, for any model m, if:
R
EdeD|Ga Gal
EdED G
Σ E G
R
PR EdeD Gdl
dED |G'
macro-R, micro- RREBEL Clean
Rlemn
However, the equivalence, in this case, relies on an additional assumption used in the penultimate equality: the recall of the ground truth annotations in REBEL Clean should be approximately the same across all relations.
macro-
Σ Pmnd O Gill =o IED
Eipma IED
Pmd DAg (8)
the true precision will be estimated well by the stan- dard precision on the REBEL Clean. Considering that GenIE was trained on REBEL, and SynthIE T5- base is a smaller version of SynthlE TS-lare, this criterion should satisfied, and therefore, we ex- pect to have good estimates of micro- and macro- precision for these models as well.
G.1.3 Recall on REBEL Clean
The previous section argues that, for all of the mod- els of interest, Cm C As consequence, using the same notation as the previous section, for the computation of the micro-recall (see Appendix D) of model m, we have:
C₁
micro-R. R
EdeDlog G
Cm
EdeD G7
Cm
E EdED Gal
EdED Gal
ELeD G
micro-Re micro-R- REBEL Clean


Reviews:
[{"review_id": "WzKLRwTUvK", "reviewer": "Reviewer_qw9q", "summary": "", "questions": "In Figure 2, it is illustrated that multiple triplets (more than 2 entities and 1 relation) are sampled to generate texts but actually only one triplet is sampled to generate a text each time, right?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased to read that the reviewer agrees regarding the effectiveness of the proposed approach and appreciates the systematicity and overall soundness of the analysis and paper.\n\n**Question: Are the generated data points expressing only a single triplet or multiple triplets as suggested by Figure 2?**\\\nThe triplet set sampling procedure starts by sampling the number of triplets for a given data point from a Poisson distribution. We then iteratively sample the triplets following the process outlined in the Section 3.2., which ensures coherent triplet sets until we reach the desired number of data points. In conclusion, we sample triplet *sets* and Figure 2 is consistent with the process. However, there was a typo in the caption, suggesting that the generated text is conditioned on a *triplet* rather than a *triplet set*, which we assume is the original source of confusion. The caption was updated; apologies for the inconvenience this typo has caused.\n\n**Weakness 2: More detailed analysis of the performance of SynthIE models on REBEL.**\\\nWe actually performed such an analysis, as reported in the first subsection of the Discussion.\\\nIn a nutshell, the performance drop is caused by flaws in the *text* associated with REBEL’s data points. More concretely, the text is not appropriate for *exhaustive* closed information (cIE). The GenIE models were trained on REBEL text and annotations, so they learn the systematic anomalies of the data – which, in fact, results in poor performance (14% macro-F1) of the baseline on higher quality data – but the SynthIE models, which were optimized to perform exhaustive cIE, are thrown off by them. This observation surfaces an important implication of the exhaustiveness assumption made by cIE more broadly (we discuss this in greater detail in the limitations section). In the discussion section, we outline a solution to this issue: applying the proposed synthetic data generation pipeline with different assumptions about the output distribution. We would like to note that this important implication has remained undetected thus far precisely because exhaustive annotation was out of reach before this work.\\\nAccording to the human evaluation, despite the data anomalies, the same-sized SynthIE model is on par with GenIE in terms of macro scores (the large imbalances make the micro scores on REBEL misleading), while the predictions generated by the larger SynthIE model are of superior quality even compared to the gold annotations associated with the REBEL dataset on which GenIE was trained.\n\n**Weakness 1: The task, data, and baselines are limited, so the audience may be narrow.**\\\nWe would like to emphasize that, as illustrated by Figure 1, the paper proposes a general strategy for synthetic data generation and comprehensively evaluates it on the task of cIE as a purposefully chosen *case study* (a central IE task for which non-synthetic data collection is prohibitively expensive and which cannot be reliably solved by today’s LLMs directly). Overall, the proposed methodology and findings are valuable and apply to any IE or parsing task. Furthermore, any task with an inverse formulation that can be addressed more effectively by LLMs would benefit from the proposed approach as well. For more details on the last two points, see the second subsection in the Discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and well-organized. \n2. Through human evaluation and model performance, the paper shows that model-generated training data can be of higher quality than human-curated data in the field of closed information extraction (cIE). This can be regarded as prior evidence of the population of model-generated datasets.", "reasons_to_reject": "1. The task, data, and baselines are limited so the audience may be narrow.\n2. According to Table 3, the model trained on the synthetic data under-performed baselines on the existing dataset, REBEL. I know there exist some flaws in REBEL's annotation, which are uncovered by the human evaluation, but this does not fully convince me the significant performance loss is acceptable. It would be better to take a closer look at the samples in REBEL that SynthIE failed but GenIE succeed and show that a majority of such samples are contributed by the ill-annotation of REBEL or some other acceptable reasons.", "questions_to_authors": "In Figure 2, it is illustrated that multiple triplets (more than 2 entities and 1 relation) are sampled to generate texts but actually only one triplet is sampled to generate a text each time, right?", "paper_topic_and_main_contributions": "This paper proposed an approach to construct synthetic training data for information extraction using OpenAI's LLM APIs. \n\nMain contributions include:\n1. A synthetic data generation approach for information extraction, i.e., first properly sample a triplet from a knowledge graph and then generate a desired text expressing the triplet.\n2. Human evaluation of the synthetic data and its counterpart, a human-curated dataset called REBEL. The evaluation demonstrates the high quality of the synthetic data and the important flaws of REBEL.\n3. An information extraction model trained on the constructed synthetic data that achieves strong performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "XwrMdhJvFt", "reviewer": "Reviewer_CoFT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful the reviewer highlights the effectiveness of the proposed approach as well as the systematicity of the analysis and the overall soundness of the paper.\n\nIn our response, we would like to provide clarifications (reacting to potential misunderstandings) that we believe may further emphasize the benefits of our approach. \n\n**Knowledge distillation and using an LLM *only***\\\nIn the context of closed information extraction (cIE), as illustrated in Figure 4, the LLM cannot solve the task directly (X → Y) because it has no knowledge of the entity and relation catalogs. However, thanks to the proposed synthetic data generation strategy that exploits an asymmetry in difficulty when going in the reverse direction Y → X, and leverages external information in the form of a knowledge graph in the triplet set sampling, the LLM can provide substantial value, although the direct task is out of reach. The resulting data generation process uses the LLM as a tool and goes way beyond vanilla knowledge distillation.\n\n**The value for LLMs**\\\nThe paper proposes a general method, which is by no means limited to cIE and can provide value to any IE or parsing task, as well as any task with an inverse formulation that can be addressed more effectively by LLMs. For more details on this, please see our response to R1. \n\nAlthough we focus on fine-tuning a “small” LM (FLAN-T5) in this paper, our method may be used to further improve large LMs (LLMs) – especially since, as emphasized above and as argued in the paper, there are still many tasks (including cIE, our focus task) that LLMs cannot solve out of the box, but that they can learn to solve based on data as generated by our method. Additionally, our proposed approach could be used for augmenting data used in LLM fine-tuning. Augmentations that could provide value include pairing answers with (i) instructions; (ii) questions and tool calls; or (iii) reasoning traces in domains where the model struggles. These could be useful to improve the model’s performance more generally or to introduce the model to novel tools or/and reasoning patterns. \n\nOutside of LLMs, the proposed method can result in models (LMs, classifiers, etc.) that outperform the original (non-finetuned) LLM at a fraction of the compute cost (e.g., cIE), thereby enabling new use cases or simply cost-efficient solutions. Closed information extraction and tasks suffering from imbalances or a general lack of diversity, if addressed by a general-purpose LLM, would greatly benefit from the proposed methodology (e.g., sarcasm categorization, fine-grained sentiment analysis, etc.).\n\nIn conclusion, the proposed method creates many opportunities in the context of LLMs and smaller models alike.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose an effective method to generate large-scale dataset for closed IE tasks.\n2. The authors propose many methods for different tasks to generate the dataset.\n3. The authors conduct so many experiments and analyze the methods, the paper is solid. ", "reasons_to_reject": "I think generate the dataset only depending on LLMs just like a knowledge distillation. Although the method is effective for smaller model, I think it is useless for LLMs. I think this kind of work does not make me exciting.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper demonstrates the effectiveness of this synthetic data generation by LLMs on closed information extraction, where collecting ground-truth data is challenging, and no satisfactory dataset exists to date. Authors synthetically generate a dataset of 1.8M data points,\nestablish its superior quality compared to existing datasets in a human evaluation.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZaqLsNVojw", "reviewer": "Reviewer_kAnw", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive and encouraging review.\n\nIndeed, we acknowledged in the limitations section that as a method for generating synthetic data that leverages an LLM, its effectiveness will be affected by the capabilities as well as biases of the specific LLM used in the process. However, considering the capabilities of today’s LLMs and the effectiveness of the proposed method on the task of closed information extraction, we believe that a carefully designed data-generation procedure, which could potentially be augmented by external knowledge or tools, will readily provide value for many tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper uses relation extraction as a case study and synthesizes data based on the REBEL dataset. In the experimental section, the author conducts a thorough comparative analysis between the generated data and the original dataset, thus validating the effectiveness of the method. The paper provides comprehensive and detailed data, ensuring a valid analysis.", "reasons_to_reject": "The generated data completely relies on appropriate templates and the generation capabilities of LLMs, which may have problems (like noise) in some week models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The author proposes utilizing a large model to generate data for training, thereby circumventing the issue of structured output in information extraction. The paper takes relation extraction as an example and conducts a comparative analysis of the quality of synthetic data and human-generated data. In conclusion, it is demonstrated that large models are capable of generating high-quality data suitable for training purposes.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work shows that LLMs can be used to synthesize data even when tasks cannot be solved directly by LLMs, particularly for problems with structured outputs. In this case, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in the task difficulty makes it possible to produce large-scale, high-quality data for complex tasks.  This leads to improved capabilities of smaller LLMs at structured prediction tasks, particularly IE. This is a simple but useful idea - and it has been properly investigated in the paper. The reviewers seem to like the paper as well.
Decision: Accept-Main