ASPIRO: Any-shot Structured Parsing-error-Induced ReprOmpting for Consistent Data-to-Text Generation
Authors: Martin Vejvar
Yasutaka Fujimoto
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Data-to-Text
data disambiguation
structured data verbalisation
few-shot learning
multi-shot re-prompting
TL;DR: ASPIRO leverages LLMs and rule-based parsing to generate concise verbalisation templates from single-triple data entries, reducing parsing errors by 66% against 0-shot setting and performing competitively on standard automatic metrics.

Abstract: We present ASPIRO, an approach for structured data verbalisation into short template sentences in zero to few-shot settings. Unlike previous methods, our approach prompts Large Language Models (LLMs) to directly produce entity-agnostic templates, rather than relying on LLMs to faithfully copy the given example entities, or validating/crafting the templates manually. We incorporate LLM re-prompting, triggered by algorithmic parsing checks, as well as the PARENT metric induced consistency validation to identify and rectify template generation problems in real-time. ASPIRO, compared to direct LLM output, averages 66% parsing error rate reduction in generated verbalisations of RDF triples on the DART dataset. Our best 5-shot text-davinci-003 setup, scoring BLEU of 50.62, METEOR of 45.16, BLEURT of 0.82, NUBIA of 0.87, and PARENT of 0.8962 on the Rel2Text dataset, competes effectively with recent fine-tuned pretrained language models.

ASPIRO: Any-shot Structured Parsing-error-Induuee ReprOmpting for Consistent Data-to-Texx Generation
Martin Vejvar
Yasutaka Fujimoto
Graduate School of Engineering Science
Faculty of Engineering
Yokohama National University
Yokohama National University
vejvar-mart wataaaaartinknnnnnnnemm
fujimoto@ynu.ac.
Abstract
knowledge graph. Xiang al. (2022) nullify the requirements for human labelling entirely by utilis- ing GPT3-davinci (Brown et al., 2020), large lan- guage model (LLM) with broad general knowledge, to disambiguate RDF triples into short sentences and automatically parse them into reusable sen- tence templates as an alternative to human-crafted templates. In this paper introduce ASPIRO, bust N-shot variant of the data disambiguation step presented by Xiang et al. (2022) and promising alternative to fine-tuning PLMs for crafting RDF verbalisations (Kasner et al., 2023). At its core, AS- PIRO uses simple rules to algorithmically flag er- rors in the templates (such as missing subject, mul- tiple objects, etc.) and re-prompt the LLM until all errors are alleviated or maximum (N) retries have been reached. We evaluate changes in automated metrics and reduction of parsing errors in different configurations of ASPIRO on DART (Nan et al., 2021) and Rel2Text (Kasner et al., 2023) and com- pare the original RDF verbalisation prompt used by Xiang et al. (2022) with our prompt focused on enforcing structured json output with intermediate fields as guidelines.
We present ASPIRO, approach for struc- tured data verbalisation into short template sen- tences in zero to few-shot settings. Unlike previous methods, our approach prompts large language models (LLMs) to directly produce antity-agnostic templates, rather than relying on LLMs to faithfully copy the given exam- ple entities, or validating/crafting manually. We incorporate LLM re- prompting, triggered by algorithmic parsing checks, as well the PARENT metric induced consistency val- idation to identify and rectify template genera- tion problems in real-time. ASPIRO, compared direct LLM output, averages 66% parsing error rate reduction in generated verbalisations RDF triples on the DART dataset. Our best 5-shot text-davinci-003 setup, scoring BLEU of 50.62, METEOR of 46.16, BLEURT of 0.82, NUBIA 0.87, and PARENT 0.8962 on the Rel2Text dataset, competes effectively with recent fine-tuned pre-trained language models.
templates
1 Introduction
Data-to-text task (Reiter, 1996) aims to build faithful natural language interpretation of struc- tured data such as relational tables or Resource De- scription Framework (RDF) triples (Miller, 2001). However, without proper context, the given struc- tured data may not sufficiently represent the rela- tionships between entities, leading to ambiguity (Dušek et al., 2019). To battle this, some works rely on fine-tuning pre- pre-trained language models (PLMs) on task-specific datasets in supervised or semi-supervised ways (Ke al., 2021; Agarwal et al., 2021), but the domain of the resulting sys- tem is limited and requires well-labelled training data (Keymanesh et al., 2022). In contrast to fine- tuning, Kasner and Dusck (2022) prove that zero- shot neural systems are possible solution, where in-domain data is introduced via simple human- crafted templates for each unique relation in the
2 Related Work
Single triple verbalisation: Mainly leveraged for reducing ambiguity in structured data before specific D2T task (Laha et al., 2019; Dušek and Kasner, 2020; Xiang al., 2022) as well trans- forming inputs to better suited for existing NLG models (Gupta ct al., 2020; Kasner and Dusek, 2022; Xiang et al., 2022), verbalisation templates fall into three main categories:
human-crafted (Kale and Rastogi. 2020: Kas-
ner and Dusek, 2022)
2) rule-based (Laha et al., 2019; Gupta et al.,
2020)
3) neural model-based (Xiang et al., 2022; Kas- ner al., 2023)
code available at github.com//iyyam/AAPPP....
ASPIRO combines aspects of both 2) and 3).

---

Delexicalization: Einolghozati et al. (2020) and Heidari al. (2021) find that without delexical- ization, generative models can produce incomplete representations of the entities and concepts in the structured data verbalisations, leading to misinter- pretation and failures in production. Our JSON structured prompt ($G.2) enforces the LLM to di- rectly produce named-entity agnostic templates.
3.1 N-shot Generator
N-shot Generator further fractures into an LLM stack and a Rule-hased parser. The LLM Stack is tasked with generating verbalisation attempts based on given initial prompt ($G.1). It does so with the help of the Rule-based parser. This parser checks the generated completions for structural accuracy, ensuring they adhere expected patterns.
3.1.1 LLM Stack
O-shot to N-shot: Our work is heavily inspired and builds upon the disambiguation step from Xi- ang et al. (2022), which equivalent to O-shot setting for our N-shot Generator. We also use their prompt ($G.1) as baseline against our JSON prompt ($G.2).
The LLM stack is a sequence of N + LLMs, in- dexed from 0 to N. Lo is responsible for the initial completion and each further retry shot, initiated by the Rule-based parser ($3.1.2), increments the index by 1. Each L, is instantiated separately and does not have to be the same model. Equation (1) shows the single completion for structured input sample æ shot
Refining LLM outputs: Madaan et al. (2023) and Shinn al. (2023) show that iterative prompt- ing and chain-of-thought reasoning can signifi- cantly improve the outputs of I.LMs. We lean on their findings in designing our ASPIRO pipeline. However, back and forth prompting of LLMs can be expensive, which we counterweight by using our Rule-based parser ($3.1) and the PARENT (Dhin- gra et al., 2019) Fl score ($3.2) cost-efficient gateways decide if additional prompting is nec- essary.
yn = (T())))
(1)
where T is given prompt and can be either Tr (initial) or TR (retry).
3.1.2 Rule-based parser
A purely algorithmic module, which validates yo against set of conditions {C} one by one. If yo does pass the condition C,, a respective parsing error is logged into set &,, The aggregated rules for each given completion are formally given below (see ŠA for detailed Python implementation).
3 Methods
Co has exactly one '<subject> substring.
C1 has exactly one '<object> substring.
The proposed method (ASPIRO) revolves around the conversion of structured data samples into ver- balisation templates using a two-stage pipeline: N- shot Generator ($3.1) and Consistency Validator ($3.2). The pipeline processes structured data sam- ples, wherein each sample comprises of one or more RDF triples which share the same relation. ASPIRO (see Figure 1) starts with initial prompt verbally articulate the structured data. This is equivalent to prompting a single LLM directly. the zeroth attempt isn't accurate, it will retry a max- imum of N times, refining the previous completion based on parsing errors 1$3.1.2). Subsequently, the outputs are validated for consistency, ensuring faithful and reliable verbalisations. We explain the individual stages and their sub-modules in the sec- tions below. Refer to Figure for full pipeline and terminology on general input. Step-by-step flow of the pipeline and example on specific input are provided in section $3.3 and Figure 2 respectively.
C2 has no other '<..>' substrings.
If the parser identifies any error in the structure, the next LLM in the LLM stack is re-prompted with Retry prompt ($G.3) to generate new completion.
3.2 Consistency Validator
Even if the outputs from the N-shot Generator adhere to the structural patterns, they might still contain inaccuracies, such as hallucinated content. This module assesses the quality of the verbalisa- tions, using the PARENT statistical metric (Dhin- gra et al., 2019). IF PARENT F1 score is too low, the module will utilise an LLM with specialised Consistency prompt (SG.4) to improve the tence.
3.2.1 PARENT threshold To gauge the quality the completion yv from N- shot Generator, we set a minimal threshold (μi) for the PARENT score of yo- The score is calculated

---

Inital
N-Shot Generator
Consistency Validator
Data Enery
Fi(e")
V
Rule-based Parner
FARENTI
Argmnaa Figy))
Find Completion
ColTolr mi))
Coles
Toln s4
Co
Ratry Prompt
Conslatency Prompt
Figure I: ASPIRO pipeline for general input sample x' E X.
using eq. (3) against artificially constructed table and reference.
pen (ONaT
Ailldll (p 0.7
no riptac lrfeed and dipil.
syriee cratty
MOTE SELOMEC
First, we construct the respective hypothesis, ta- ble and reference entries:
B.OT (.)))
Commitency
h Vn replacc((s,o)),))
gan (ibeet
"Jrned pooctsay and
Marry Pronef
t (c, sppltt" "),e) p
(2)
Soithey im
Porept
where "<subject>" and "<object>" are replaced with "<entity>" to prevent penalising order dis- crepancy between hypothesis and table.
Figure 2: Example flow of ASPIRO pipeline with input
We then calculate the PARENT F1 score using equation (3).
sample x" [(Mario, creator, Shigeru Miyamoto)]
F1(y.) = PARENT (h, p,t)
(3)
Step 2
Use $3.1.2 to validate yn against all con- ditions C. If errors (E,) are found, run equation (5) and return to Step 1. Otherwise go to Step 3.
3.2.2 Consistency LLM
If the calculated PARENT score from $3.2.1 is not sufficient, we call another LLM with prompt To as in eq. (4).
T = TR(x", Vhe 8 n 12 +
(5)
Step 3
Use $3.2.1 and calculate F1(v") via eq. (3). If the calculated F1 score is lower than our chosen threshold 0 < μ S 1, continue to Step 4. Otherwise, output current yo as the final comple- tion y".
yc = Cc(Torr yn))
(4)
The prompt To designed to guide Lo to iden- tify problems with the given completion, provide advice how to fix it and subsequently produce fixed completion in structured json output. See $G.4 for full version of the prompt.
Step Use $3.2.2 to get revised completion se-
Step 5 Compute F1 scores of y" and ve using eq. (3) and take the completion with higher score via eq. (6) to produce the final completion y".
3.3 Stepwise Pipeline Formulation
Given dataset of structured data samples {x") (e"), where x" {af ..,, and æt is a single RDF triple x² = (5), of) with relation r E R, the pipeline for one x² is as follows:
y" argmax FF(
(6)
vE(vA.
Experiments
Step 0
Set n and To = (x").
The following sections show results on several se- tups of ASPIRO. In section $4.1 we compare auto-
Step 1
Calculate V/ using eq. (1).

---

Table Average values from 5 independent runs of 4 ASPIRO confiuraatios on Rel2Text test set using automatic metrics (desc. $C), compared with Kasner et al. (2023)'s BART-BASE models, fine-tuned on full (full-) only X (fewshot-X) examples from Rel2Text training set. See Table 2 for models and Table for standard deviations.
test samples (616)
BLEU
METEOR
BLEURT
NB
SS
(( 4
(
(()
PARENTp1
(Kasner el nl., 2023) full-relStext
52.54
44,86
0.54
0.88
4.72
3.50
4,65
91.85
(Kasner el al., 2023) fewshot-25
31.13
35.52
-0.02
0.65
3.94
8.35
27.26
64.39
(Kasner et al., 2023) fewshot-200
48.67
43.34
0.44
0.83
4.58
5.40
9.03
85,57
(A) G3.5
51.22
44.73
0.82
0.87
4.71
4,88
7.34
87.77
0.8883
(A) SxG3.5 (G3.5)
51.40
44.94
0.82
0.87
4.72
3.65
7.98
88.38
0.8895
(J) G3.3
50.63
45.13
0.82
0.87
4.79
1.60
7.36
91.04
0.8963
(J) 5xG3.5 (G3.5)
50.62
45,16
0.82
0.87
4.79
1.54
7.25
91.21
0.8962
Table 2: LLM variants used in our experiments.
competitive to the full-training-set-ffn----ne full- rel2text model with ca 1-2 point reduction in BLEU, but 28 % points increase in BLEURT. This implies higher semantic similarity, however Semantic Similarity sub-score of NUBIA only shows small increments. Despite the overall NB score being same for all ASPIRO setups, the sub- metrics of NUBIA show steady improvement be- tween our models. Most noticeable change is in the Contradiction percentage, which the 5-shot setting improves by ca 1.2 % points and further 2 % points by introducing JSON prompt, suggesting higher capacity to disambiguate the correct direction of relation between subject and object entities in the input triples. PARENT FI score slightly favours the JSON prompted setups ASPIRO, but only by ca 0.6 % points.
label
foll
model family
original publication
G3
davinci
GPT3
(Winata al., 2021)
G3.5
text eax-iavinni-O03
InstrurCiPP
(Oyang 2022)
C3.ST
gpt-3 13t-b---urbo-000
CharGPT
(0eenA1, 2022)
G4
spt-4-0314
GPT4
(OpenAl, 2023)
matic metrics on Rel2Text test set ($D.3) with Kas- ner et al. (2023)'s fine-tuned BART-BASE models. In section $4.2 we report on the number of parsing errors tagged by Rule-based parser ($3.1) both DART (SD.1) and Rel2Text ($D.3) datasets. In $4.3 we also provide brief ablation study of CV.
Setup: For N-shot generator ($3.1), Lo marks initial model choice and NxC, max N retry shots using model L. We limit experiments to Ln being same for all N shots. For Consistency Valida- tor ($3.2), we set μ 0.7 and only use it in some setups (marked by Lo in brackets). For reference on LLMs used as C in ASPIRO setups, see Tab. 2.
Additional experiments: For metric results and discussion on DART, see appendix $E.1. For full experiment results with fine-tuned pre-trained lan- guage models refer to (Kasner et al., 2023).
While Retry prompt TR ($G.3) and Consistency prompt To ($G.4) constant across all our experiments, we compare two variants of the Initial prompt Tr:
Prompts:
Table 3: DART dataset counts of templates with parsing errors. RR %: error Rate Reduction percentage of best N-shot setup (bold) vs Oshot model.
(A) ASDOT: proposed by Xiang et al. (2022) in their Data Disambiguation step to produce short sentence representation of given triple. (full form SG.1)
Initial run
N-shot Gen Setup LNxC, (+Cc))
SxG3.5T
Co
6x
IxG3,5T
1xG3.5
1xG4
RR %
(+G3.5T)
239
66.76
(A)
719
467
490
351
(227)
(68.43)
(A)
G3.5T
996
727
835
784
497
50.10
(J) JSON:
proposed prompt, which enforces json-like output with auxiliary fields to guide the creation of named-entity agnostic tem- plates directly. (full form $G.2)
156
G3.5
94
75
59
24
74.47
Table 4: Rel2Text counts of templates with errors.
4.1 Automatic Metries
Initial
N-shot Gen Sctup [NxC., (+Cc))
5xG3.5T
5xG3.5
We evaluate Automatic metrics Rel2Text test set ($D.3) with 4 ASPIRO setups (see Table 1 for 5 run averages; Table 7 for standard deviations). ASPIRO outperforms Kasner et al. (2023)'s srushot-fine-uumee PLMs on all metrics and is
1xG3.ST
1xG3.5
1xG4
(+G3.5T)
(+G3.5)
(A)
G3.5
21
20
21
19
(9)
(14)
2
())
G3.5
2
(1)
(2)

---

Table 5; Automatic metrics without and with consistency validation (CV) for Rel2Texx test set.
Rel2Text test samples (616)
BLEU
METEOR
BLEURT
NB
SS
((%)
N (%+)
(
PARENTpi 1
5xG3,5T
50.88
44.97
0.8215
0.8725
4.73
3.36
8.04
88.59
0.8910
5xG3.5T (CV G3:5T)
50.89
44.95
0.8213
0.8718
4.73
3.02
8.34
88. 64
0.8908
5xG3
50.96
44.69
0.8200
0,8680
4.71
4.92
7.59
87.48
0.8879
5xG3 (CV G3)
51.32
44.77
0.8200
0,8702
4.72
3.35
7.70
88.94
0.8895
4.2
Parsing Errors
ments with less informational value. Conversely, parsing errors (Table 6) are reduced notably by CV,, with counts decreasing from 12 to 10 and 23 16.
Parsing eπror analysis does not require specific references from the dataset. After ASPIRO pro- duces the verbalisation templates (y"), we run them through our Rule-based parser ($3.1) to flag and count the number of errors. As source data (X), similar to (Xiang et al., 2022), we collect at most 2 triple examples for each unique relation in the dataset and use them to prompt our pipeline.
Table 6: Individual error counts (EE) without and with CV on Rel2Text test set with (AJSDOT prompt.
NALEM
Total [8]
Templttos w/
Missing SURJ.
Mixsing OR).
SUGAST (GLST)
10
5uG3
23
Parsing error counts: For DART (Table 3) we use the full dataset (SD.1), producing 4299 unique template sentences in each experiment run. In Rel2Text (Table 4) we only the test split (SD.3) with 226 unique relations and G3.5 (T2) base model with either (A)SDOT or (J)SON prompts and different N-shot Generator setups. For Rel2Text, we don't provide RR % as the reduc- tion is evident from counts.
Conclusion
We proposed and evaluated ASPIRO, a general domain-agnostic pipeline for verbalisation of sin- gle triple data entries to short template sentences, utilising rule-based re re-prompting of LLMs. The pipeline comprises of N-shot Generator ($3.1) and Consistency Validator ($3.2). We show that AS- PIRO compares fine-tuned pre-trained language models' automatic scores the Rel2Text test set ($4.1) and significantly reduces the parsing error count in O-shot outputs of LLMs ($4.2). The ab- lation study ($4.3) revealed that Consistency Val- idator ASPIRO further reduces error counts, but does not significantly affect automatic metrics.
Introducing N-shot Generator ($3.1) shows significant reduction in parsing error counts (Tables 3 and 4) even with N 1. In the retry shot setting, GPT4 (G4) is most effective reduc- ing parsing errors. However, we introduce up to retry shots, we can see that gpt-3.5-turbo (G3.5T) reduces parsing errors further. The exception is (J)SON prompt on DART where G4 keeps the lead. Interestingly, while text-davinci-003 (G3.5) per- forms well as O-shot model, it generally performs worse than G3.5T in N-shot settings, contrasted again DART by prompt. It is also evident that J prompt provides more robust O-shot base- line compared to (A)SDOT prompt. The values in parentheses reveal that including Consistency Vali- dation yields only slight reduction error count.
Discussion:
4.3 Ablation of Consistency Validator
To investigate the efficacy of Consistency Valida- tor, conduct a brief ablation study on Rel2Text test set ($D.3). For statistical metrics (Table 5), CV provides only marginal gains. This effect may be attributed to the improvement of Contradiction score and degradation Neutrality score, implying that CV moves the templates closer to general state-

---

Limitations
on maximum likelihood estimation, which inher- ently leaves room for potential variation errors in our experimental results.
Operational costs: When contrasted with O-shot setting, ASPIRO significantly escalates the opera- tional costs (see appendix $F) due to the repeated calls of the N-shot Generator and the lengthy Con- sistency prompt ($G.4) associated with the Consis- tency Validator ($3.2). Following the brief ablation study of CV ($4.3) and the cost analysis, it remains debatable whether the performance of the Consis- tency Validator reported in this paper justifies the additional expense incurred in prompting the LLM for the flagged examples.
Ethics Statement
In the course of this research, we have employed various Generative Pre-trained Transformer mod- els, including GPT3 davinci, InstructGPT text- davinci-003 and spt-3.5-turbo-0301 and gpt-4- 0314, each demonstrating inherent biases as out- lined in their respective publications, which are listed in Table 2. These biases often favour popular opinions and can lead to a distortion in the model's outputs. This reflects the models' training on large- scale internet text, which is not entirely neutral and contains biased or skewed perspectives. We ac- knowledge this limitation and highlight that despite implementing pipeline designed to minimise the inclusion of unnecessary and irrelevant informa- tion, the potential for biased outcomes cannot be entirely eliminated.
Isolated triples: Generating verbalisations from single isolated triples doesn't account for situations where context from other triples is necessary to fully interpret the final natural language verbalisa- tion. As exemplified by the DART dataset, con- textual integration significant and should be ex- plored further.
Backup template: In instances where the pars- ing of the <subject> and <object> within the gener- ated completion of the LLM proved unsuccessful, Xiang al. (2022) introduced a general backup template as fallback. In our research, we did not use any backup templates and did not investigate their potential impact on automated metric scores. Nonetheless, it's important to acknowledge that within a production environment, the incorporation of backup template is a fundamental necessity, warranting further assessment of its effects.
References
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhance model pre-training. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3554-3565, Online. As- sociation for Computational Linguisties.
language
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Direction of relation:
The capacity to accurately discern the correct direction of the relation between subject and object is a notable feature of Data-to- text systems. In our experiments, we report on contradiction statistic (C %), which can roughly translate to measure this ability. Although ASPIRO generally shows to improve this statistic, there are no specific guardrails to validate the ambiguity other than the general knowledge of the LLM itself.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Variance of experiment runs: Due to the sub- stantial expenses associated with prompting large language models (LL.Ms) and the considerable size of the DART dataset, each experiment on DART was conducted only once. The same is true for Re12Text parsing error analysis in Table 4. It should be noted that, although the temperature parameter was uniformly set to 0 for all the employed LLMs, the underlying generative process remains reliant
Thiago Castro Ferreira, Diego Moussallem, Emiel Krah- mer, and Sander Wubben. 2018. Enriching the WebNLG corpus. In Proceedings of the 11th Interna- tional Conference on Natural Language Generation,

---

pages 171-176, Tilburg University, The Netherlands. Association for Computational Linguistics.
Hassan Kane, Muhammed Yusuf Kocyigit, Ali Abdalla, Pelkins Ajanoh, and Mohamed Coulibali. 2020. NU- BIA: NeUral based interchangeability assessor for text generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 28-37, On- line (Dublin, Ireland). Association for Computational Linguisties.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884 4895, Florence, Italy, Asso- ciation for Computational Linguistics.
7denčk Kasner and Ondrej Dusek, 2022. Neural pipeline for zero-shot data-to-text generation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Lingussiics (Volume 1: Long Papers), pages 3914-3932, Dublin, Ireland. As- sociation for Computational Linguistics.
Ondřej Dušek, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural lan- guage generation. In Proceedings of the 12th Interna- tional Conference on Natural Language Generation, pages 421-426, Tokyo, Japan. Association for Com- putational Linguisties.
Zdenčk Kasner, loannis Konstas, and Ondrej Dusek. 2023. Mind the labels: Describing relations in knowl- edge graphs with pretrained models. In Proceed- ings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pages 2398-2415, Dubrovnik, Croatia. Association for Computational Linguistics.
Ondfej Dušek and Zdeněk Kasner. 2020. Evaluating semantic accuracy of data-to-text generation with nat- ural language inference. In Proceedings of the 13th International Conference on Natural Language Gen- eration, pages 131-137, Dublin, Ireland. Association for Computational I Linguistics.
Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Lin- feng Song, Xiaoyan Zhu, and Minlie Huang. 2021. JointGT: Graph-text joint representation learning for text generation from knowledge graphs. In Find- ings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021. pages
Arash Finolghozati, Anchit Gupta, Keith Diedrick, and Sonal Gupta. 2020. Sound natural: Content rephras- ing in dialog systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP). pages 5101-5108, On- line. Association for Computational Linguistics.
Online. Association for Computational Linguistics.
Moniba Keymanesh, Adrian Benton, and Mark Dredze. 2022. What makes data-to-text generation hard for pretrained language models?
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The WebNLG challenge: Generating text from RDF data. In Pro- ceedings of the 10th International Conference on Natural Language Generation, pages 124-133, San- tiago de Compostela, Spain. Association for Compu- tational Linguistics.
Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 539-554, Ahu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Anirban Laha, Parag Jain, Abhijit Mishra, and Karthik Sankaranarayanan. 2019. Scalable micro- micro-planned generation of discourse from structured data. Com- putational Linguistics, 45(4):737-763.
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference tables as remi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 2309-233 2224, Online. Association for Computational Linguistics.
Aman Madaan, Niket Tandon. Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self- feedbeee..kk.
Peyman Heidari, Arash Einolghozati. Shashank Jain, Soumya Batra, Lee Callender, Ankit Arun, Shawn Mei, Sonal Gupta. Pinar Donmez, Vikas Bhardwaj. Anuj Kumar, and Michael White. 2021. Getting to production with few-shot natural language genera- tion models. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 66-76, Singapore and Online. Association for Computational Linguistics.
Eric J. Miller. 2001. An introduction the resource description framework. Journal of Library Adminis- tration, 34(3-4):245-22 255.
Linyong Nan, Dragomir Radev. Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xi- angru Tang, Aadit Vyas, Neha Verma, Pranav Kr- ishna, Yangxiaokang Liu, Nadia Irwanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Mutethia Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, Richard Socher, and Nazneen Fatema Rajani. 2021. DART: Open- domain structured data record to text generation. In
Mihir Kale and Abhinav Rastogi. 2020. Template guided text generation for task-oriented dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6505 65200,6500000 Online, Association for Computa- tional Linguistics.

---

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 432-447, Online. Association for Computa- tional Linguistics.
OpenAl. 2022. Introducing chatgpt.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Ehud Reiter: 1996. Building natural-language genera- tion systems.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. In Proceedings of the 1st Workshop on Multilingual Representation Learning, pages 1-15, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jiannan Xiang, Zhengzhong Liu, Yucheng Zhou, Eric Xing. and Zhiting Hu. 2022. ASDOT: Any-shot data- to-text generation with pretrained language models. In Findings the Association for Computational Linguistics: EMNLP 2022, pages 1886-1899, Abu Dhabi, United Arab Emirates. Association for Com putational Linguistics.

---

Table Standard deviations of results from Table 1, The values for pre-trained language models in first three rows are copied for reference from Kasner al. (2023)'s Table 6.
all RellTTxxt test samples (616)
BLEU
METEOR
BLEURT
NB
1
SS
((%)
(
E(%)
PARENT
(Kasuer et al., 2023) full-rel2text
0.30
0.01
0.01
0.02
0.41
0.38
(Kasner al.,, fewshot-25
(Kasner al., fewshot-200
(A) G3.5
0.02
0.00
0.00
0.00
0,0002
(A) 5xG3.5 (G3.5)
0.63
0.11
0.00
0.00
0.01
0.16
0.12
0.14
0,0005
(J) G3.5
0.13
0.06
0.00
0.00
0.00
0.08
0.08
0.13
0,0004
(J) 5xG35 (G3.5)
0.18
0.07
0.00
0.00
0.00
0.03
0.25
0.25
0,0002
A Rule-based Parser
B Tools and Repositories
Below a code snippet of all rules checked by the Rule-based parser. For full implementation refer to our GitHub"
We used Python 3.9 and 3.10 to run our experi- ments with LangChain and OpenAl API' for ef- ficient work with large language model pipelines. Metrics were calculated using the following exist- ing implementations:
SUBJECT "<subject>"
OBJECT "<object>" errors []
BLEUMETTER GEM-metrics benchmark
# SUBJECT entity errors:
if SUBJECT not in text:
PARENT: multiprocessing variant from Clé- ment Rehuffe
aprors. weteeeeaeerrrrrr......I.I elif coxnt (SUBJEC) > 1:
errors. wwrssappeed(Teeelateerrroo
MULTIPLE_SUBJEE
BLEURT:
code from google-research' with default BLEURT-20 checkpoint
# OBJECT entity errors:
if OBJECT not in text:
elif text. (08JECT))
NUBLA: original NUBIA repository"
errors. penerssappeedTTTeeaaaeeerro Www.EEErrrrr....LPLE_LOOOOOA MULTIPLE_OORJCCTS
# PLACEHOLDER mismatch errors
# e.g. "<author>" instead of "<subject>//*
object>
if contains_ii sillesssslllllllaleeeel paa.egalpplcchooleerrccc..
errors. (Templat terrorrs..rrrr....
AALLGGAL_LACEHHOA
ppthon. langchain.comm
"platform. wwwprr/m.ppeaa..oom www//ooc/appireeeeeeeooooeoooeee refference
"githuu..com com/GEM benchmarl/GEM --metiiss
http:/ttt...cm/eeeaaa/////////dd @F86ead6218001em0
"githu..com www.///oooooeeeeeeeacbbbbbeeee

---

C Metrics
D Datasets
For comparability of our automatic metric evalu- ations ($4.1), we leverage most of the lexical and semantic similarity metrics used by Kasner et al. (2023), Below is brief explanation of their signif- icance.
D.1 DART
DART dataset introduced by Nan et al. (2021) is large <triple-set, sentence> pair dataset aggregated from WikiSQL, WikiTableQues tions, WebNLG2017 and CleanedE2E with 22.659/6,980/12.555 samples in the train/dev/test sets respectively. All splits combined have total 4299 unique relations. Each sample contains a set of up to RDF triples with different relations, and human-crafter sentences as natural verbalisations of the given structured data.
BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005) metries that quan- tify the lexical similarity between model-gen generrtee outputs and (typically human- pronnn-roddued) references by utilising n- n-gram overlap.
PARENT (Dhingra et al., 2019) additionally as- sesses n-gram overlap of generated outputs with source structured data (i.e., table), which acts as auxiliary reference beyond the reference text. This metric rewards instances where the hypothesis encupsulates all the information derived from the table, even if some elements are absent from the reference. Conversely, the PARENT score discrim- inates situations where the reference text contains supplementary information, which is absent in the structured data, implying the integration of external knowledge embellishments in the reference.
D.2 DART-SingleRDF Dataset
As DART (SD.1) combines multiple relations within one sample, we opted extract only samples with single triple in the input set from all DART splits and combine them into DART- SingleRDF subset. DART-SingleRDF contains a total 2,947 <single-triple, sentence> entries with 1,439 unique relations.
D.3 Rel2Text
Rel2Text is specifically designed by Kasner et al. (2023) for the verbalisation task of single-triples focused out-of-domain and previously unseen relations. It contains a total of 4,097 <relation, description, single-triple, verbalisation> samples. In $4.1, we use the test split Re12Tex! with 616 total samples and 226 unique relations, unseen in the training set.
BLEURT (Sellam al., 2020) is trained met- rie that complements the above lexical similarity metrics by capturing semantic similarity.
NUBIA (Kane et al., 2020) also a trained met- ric that combines multiple sub-metrics to assess the interchangeability or equivalence of two texts. On the surface, this metric generates single score (NB), that ranges between 0 and 1. Similar to Kas- al. (2023), also report the sub-metrics which are used for the total NB value:
D.4 WebNLG
WebNLG is commonly used dataset by Gardent et al. (2017) where entries contain up to 7 triples extracted from DBpedia categories and labelled by human-crafted sentences. We specifically the enrichment of WebNLG version 1.4 from (Cas- tro Ferreira et al., 2018), which contains 354 unique relations in total.
SS Semantic Similarity operates on scale of 0-5, where higher values suggest higher semantic similarity
C% Contradiction percentage increases the out- put and reference contradict each other in their meaning.
E
Additional Experiments
N % Neutral percentage (also referred to as "chance of irrelevancy") increases if the out- put contains new information information which is irrelevant to the reference.
E.1
Automatic metrics for DART-SingleRDF
We used DART-SingleRDF ($D.2) as test set for automatic metric evaluation of ASPIRO. The Full results are reported Table and Reduced results in Table 9.
E% Entailment percentage increases as the infor- mation in reference is entailed by the model output.

---

Table 8: Evaluation of ASPIRO with T, =A/SDOTKOSNN and C. G33.5/G3.51 on DART DART--SngleRRDF dataset using automatic metrics. See Table 2 for for model descriptions, C for metric descriptions.
all samples (2947)
bleu
meteor
BLEURT
SS
(%%)
N(%)
E (%)
PARENTF)
(A) G3.5= Lo
34.41
34,48
0.69
0.67
4.00
16.97
14.50
68.53
0,8257
33.60
34.19
0.67
4.01
15.56
15.20
0.8286
(A) L0+IxG3.5
34.11
34.38
0.69
0.67
4.00
17.04
14.50
68.47
0.8250
(A) Lo+1xG4
33.23
34.39
0.69
0.67
4.01
16.82
14.60
68.58
0,8261
(A) 50+5xG3.5T
33.69
34.59
0.69
0.68
4.07
14.02
15.36
0.8348
all samples (2947)
bleu
meteor
BLEURT
NB
SS
C. (%)
N (%)
E (%)
PARENTp1
G3.5T= La
33.76
35.25
0.71
0.71
4.12
14.53
14.32
0.8290
(A) L0+1x
34.23
35.58
0.71
0.72
4.19
11.38
14.49
74.13
0.8347
Lo+1x
34.03
35.88
0.71
0.73
4.20
11.16
14.52
74.32
0.8342
(A) Lo+1xG4
33.48
35.67
0.71
0.73
4.20
11.29
14.75
73.96
0.8341
(A) 5xG55T
34,25
35,91
0.71
0.73
4.24
9.84
14.97
75.18
0.8406
all samples (2947)
bleu
meteor
BLEURT
SS
((%)
(%))
E (%)
PARENTF:
(J) G3.5= Lo
35.49
36.96
0.72
0.76
4.42
4.68
12.05
83.27
0.8534
Lo+IxG3.5T
35.16
36.64
0.72
0.75
4.40
4.86
12.13
83.01
0,8517
(J) Cy+1xG3.5
35.48
36.95
0.72
0.76
4.42
4.67
11.98
83.35
0,8534
(J) Lo+1xG4
35.49
36,97
0.72
0.76
4.43
4.62
11.95
83.43
0,8535
C.+5xG3.ST
35.44
36,90
0.72
0.76
4.42
4.66
11.94
83.41
0,8532
all samples (2947)
bleu
meteor
BLEURT
NB
SS
(%)
(%)
E (%)
PARENTF:
(J) G3.5T= Co
32.76
36.34
0.70
0.74
4.36
5.56
16.03
78.41
0.8408
(J Co+IxG3.ST
31.13
35.82
0.70
0.73
4.33
5.72
15.91
78.38
0,8375
20+1xG3.5
32.76
36.35
0.70
0.74
4.36
5.55
15.94
78.51
0,8408
Lo+1xG4
32.89
36.40
0.70
0.74
4.37
5.47
15.67
0,8408
(J) 50555G3
32.13
36.15
0.70
0.73
4.35
5.63
16.01
78.36
0.8400
Table 9: (Subsets of only generated templates with ('((o))
0.7)
Evaluation of ASPIRO with
Ti =(A)SDOTKJJSON and Lo -G3.5/G3.57 on DART-SingleRDF dataset using automatic metrics. See Ta- ble for model descriptions, C metric descriptions.
Pl(Ca) 0.7 only (491)
bleu
meteor
BLEURI
NB
SS
(())
((6)
E (%)
PARENTPI 1
(A) G3.55
16.76
24.20
0.57
0.49
3.19
27.74
23.56
48.70
0.5324
(A) L.tIxG3
17.09
24.59
0.56
0.49
3.27
24.12
24.60
51.27
(A) L,+1xG3.5
16.09
24.01
0.56
0.48
3.17
28.09
23.37
48.55
0,5331
(A) La+1xG4
15.66
24.49
0.57
0.49
3.25
27.43
22.47
50.10
0.5397
(A) 555553.T
18.10
25.76
0.58
0.52
3.46
20.70
23.60
55.69
0.5706
F1(Cg) only (473)
bleu f
BLEURT 1
NB 1
SS f
N (%)
PARENTFi
(A) G3.5T= Lo
17.24
25.13
0.60
0.55
3.45
23.09
22.08
54.83
0,5300
(A) 5.355T
18.16
26.12
0.61
0.58
3.58
17.24
22.56
60.21
0,5513
(A) Cot1x33.5
17.71
26.70
0.62
0.58
3.60
18.28
22.90
58.81
0.5453
Lot1xG4
16:74
26.31
0.62
0.58
3.61
18.29
22.90
58.82
0.5499
(A) 5xG5STT
19.42
27.24
0.62
0.59
3.73
14.16
62.63
0.5691
(1(( 0.7 only (334)
bleu f
meteor ?
BLEURT f
NB
SS
((%)
N (%)
(%)
PARENTE: 1
(J) G3.5 Lo
19.23
28.26
0.64
0.64
4.01
7.95
17.79
74.26
0.5317
(J) Lo+IxG3.5T
19.34
28.19
0.63
0.64
4.00
7.99
18.14
73.87
0.5310
(J) LotIxG3.5
28.21
0.64
4.01
74.08
0.5316
(J) CotIxG4
19.27
28.34
0.64
0.64
4.02
7.89
17.62
74.49
0.5322
(J) L0+5xG3.5T
19.42
28.08
0.63
0.64
4.00
7.91
17.80
74.29
0.5304
Fl(Cp) 0.7 only (387)
bleu 1
metcor
BLEURT
NB
SS
((%)
N (%)
E (%)
PARENTET
(J) G3.5T= Lo
16.70
27.25
0.61
0.61
3.91
8.45
21.34
70.21
0.5313
(J) Lo+IxG3.ST
14.31
26.01
0.60
3.82
8.56
21.81
69.63
0.5215
(J) Lo+IxG3.5
16.44
27.27
0.61
0.61
3.91
8.42
21.23
70.35
0,5314
(J) LotIxG4
16.73
27.43
0.62
0.62
3,93
8.08
21.31
70.61
0.5325
(J) Lot5xG3.5T
16.04
26.75
0.61
0.60
3.87
8.35
22.18
69.47
0.5296

---

Full: Results in Table 8 show slight no discrep- ancies in the metrics between all the experiments, which could attributed variational error. Con- sidering the high API model costs (SF), we did not run DART experiments multiple times to pro- vide deviations. Instead, we reduce the data to only problematic samples by taking a subset of v/ Lo(x") generated templates which satisfy PARENT F1 (v6) 0.7. In other words, take a subset of samples, for which oulputs of O-shot model in the respective sub-table were flagged as inconsistent by the Consistency Validator ($3.2) us- ing μ 0.7. We report the same metric evaluation process in Table 9.
vide significant improvement over O-shot method on the WebNLG dataset.
Run time and cost of ASPIRO
$191 US is the overall expenditure for OpenAl API calls during our experiments. However, it is important to note that made many redundant API calls in the development of ASPIRO so the necessary costs should be lower. The main bulk of the costs amounts to GPT3-davinci and text- davinci-003 calls.
Run time analysis
Table 12 presents the average run time in seconds across five experimental runs using the WebNLG dataset, which comprises 354 unique relations. This translates to total of 354 calls required for the Ox model, zero-shot call the initial model (GPT3.5-turbo), Subsequent retry shots only need as many calls as there are templates with parsing errors.
Discussion: For the Reduced evaluation in Table 9, we found that ASPIRO shows significant im- provement only when (A)SDOT is initial prompt and only with 5-shot gpt-3. ept-3.5-turrb setting, A point of interest is also the Neutral (irrelevance) score (N %). which the 5-shot setting generally increases, suggesting the N-shot setting reducing the rele- vance of generated verbalisations to the references. For JSON prompts 1-shot gpt-4 setting has slight, albeit marginal lead over other settings.
Cumulative mean time: Given the nature of our experiments, where subsequent runs leverage sults from preceding run utilises results from the 1-shot run and only re-prompts those with parsing errors), we intro- duce Cumulative mean time to illustrate the total time necessary to execute all shots of the respective experiment.
(for instance, the 2-shot
E.2 Performance on WebNLG
We additionally evaluated performance on WebNLG ($D.4), following a similar approach with Rel2Text in the main experiments ($4). GPT-3.5-turbo-O33 is used as LLM instances of all calls to both N-shot generator LLM stack and Consistency Validator.
F.2 Estimated API call costs
For a cearst-case-scenarii cost estimate of AS- PIRO (all templates are tagged for retry shot), we made calculations for the GPT3.5-turbo model, which charges $0.002 per 1000 tokens (as of the time of our experiments). Table 13 provides cost estimations for the experiments conducted on the Rel2Text, WebNLG, and DART datasets using GPT3.5-turbo. To derive the costs associated with the GPT3-davinei or text-davinci-003 mod- els (charged at $0.02 per 1000 tokens), multiply the presented Table 13 values by factor of 10.
Parsing errors: We observed (Table 10) that for WebNLG, ASPIRO is generally not able fix any errors and CV conversely increases the number total errors, making of the templates more "flawed" than without CV.
We compare the templates generated by ASPIRO to the manually crafted tem- plates from Kasner and Dusck (2022) to evaluate lexical similarity using PARENT, BLEU and ME- TEOR. The results, seen in Table 11, are marginal best and can only observe improvement in BLEU score, while PARENT Fl and METEOR are highest for zero-shot setting. Due to time re- straints, we did not include BLEURT and NUBIA cvaluations.
Automatic metrics:
Conclusion: Contrary to our original belief, we can conclude that ASPIRO pipeline does not pro-

---

Table 10: Number of errors tagged in generated templates by our Rule-based parser ($3.1.2) for different experiment setups with gpt3. 1-t.b-turo--031 on the full Enriched WebNLG v1.4 dataset. All models (base, retry and CV) are instances of GPT-3.5-turbo-0301. Multiple SUBJECTs column was ommitted (0 for all experiments).
WebNLG dutaset
unique templates
Total Errors
Template Errers
Missing SUBJECT
Missing OBJECT
Multiple OBJECTs
Illegal PLACEHOLDER
(Usbod) (
30
79
24
36
0
(Ishod)
(Ishot) (
50
79
22
II
(2shot)
S1
80
23
(34hod) 0
23
(4shod) O
80
79
22
58
(5chod) (CN)
83
79
23
Table 1I: Automatic Metries on WehNLG vl.4 against human-crafted verbalisation templates from Kasner and
Dusek (2022).
Oshot model = G3.5T
PARENT
PARENTR
PARENT F1 BLEU
METEOR (%)
(retry model) (Nshot) (CV model)
O (Oshot) ()
0.7455
0,9594
0.8308
54.50
46.71
(0shot) (G3.5T)
0.7461
0.9524
0.8278
55.00
46.20
(G3.5T) (Ishot) 0
0.7454
0.9591
0.8305
54.37
46.54
(G3.5T) (2shot)
0.7455
0,9592
0.8306
54.44
46.56
(G3.5T) (3shot)
0.7456
0,9595
0,8308
54.47
46.58
(G3.5T) (4shot)
0.7456
0,9595
0.8308
54.44
46.59
(G3.5T) (5shot)
0.7456
0,9595
0,8308
54.44
46.59
(5shot) (G3.5T)
0.7461
0.9537
0,8285
$4.89
46.34
Table 12: Average run time (in seconds) of experimental runs on WebNI.G dataset with the GPT3.5-turbn model.
Table 13: Calculated API call cost (USD) estima- tions the experiments conducted on Rel2Text, WebNLG, and DART datasets using the GPT3.5-turbo (x10 for GPT3-davinci and text-davinci-0033 model with either ASDOT initial prompt (A) JSON initial prompt (J). The costs are detailed for both individual N-Shot settings and Consistency Validation, with the
Experiment
Mean time (S)
Cumulative mean time (8)
Ox
377.5
377.5
(x (CV)
173.3
550.8
1x
142.4
519.9
total cost calculated for each experiment setting.
2x
118.3
638.1
3x
159.3
797.5
dalaset
4x
111.7
909.2
(A) RelTTest 0xG3.5T
0.07
0.11
0.22
5x
108.1
1017.2
RelTess IxG35T
0.14
0.29
5x (CV)
138.8
1156.0
(A) ReElixt
( RelZTest 0xG3.5T
0.14
0.11
0.28
RelZTest IxG3.5T
0.27
0.42
(n) RetZTes 5xG3.5T
0.81
0.66
(A) WebNLG 0xG3.5T
0.11
0.23
0.3
WoNLG 1GG3
(A) WeNNGG 5xGA.T
0.68
091
()) WaNLG ISG3.5T
0.21
1123
0.44
WONLG 1GG35T
0.42
(J) WeNNLG SxG3.51
1.27
1.50
(A) DART DAGAET
0.46
1122
1.38
(A) DART 5xG3ST
2.76
0.92
3.68
DART 0cG3.5T
0.56
1.78
(J) DART SxG3.5T
5.18
0.92
6.10

---

G Prompt Templates
G.3
Retry Prompt
G.1
Initial Prompt: ASDOT
Completion
[completion]
Table: Michael birth Place USA
Text:
Michael was born in the USA.
Above, the Completion not satisfy the constraints given in Prompt.
Table:
First Clearing location On NYS 52 Mi.
Youngsville
Details: [error]"
Text:
First Clearing is located On NYS 52 Mi.
Please try again.
Youngsville.
Prompt
Table: Abilene Regional Airport city Served
(prompt)
Abilene Texas Abilene Regional Airport serves Abilene Texas.
Text:
G.4
Consistency Prompt
Table: Altred Moore Scales active Years Start Date
1875-03-04
Your task is to evaluate a [string] ] based on the following [rules] ] and output a vvaidd' flag which is either 1 ([ string ] complies with [rules D) or 0 (I string J hreaks some [rules J) and an advice" which explains in one short sentence how to fix [ [string to comply with all [ J (if valid==1, advice=""). You will also output a valid string which will follow the advice and comply to the rules (if valid=1, valid_string =| string ])
Text: Alfred Moore Scales started to be active
1875-03-04.
eexmpleetaable table_str
Text:
G.2 Initial Prompt: JSON
### example:
json
rules J:
((" intput data World Trade Center", " architect
string must contain exactly one <subjeet>>" substring.
"Minoru Yamasaki"], ["Syymour Centre", architect "Allen Cottir "Ii)
string must contain exactly one "objects>"
suhstring
string must s=nor contain any named entities specific references other than "<subject>' and '<object>" **.
json
((" subject entities Centre").
("oll Trade Center", "Seymour
string should align with the semantic meaning of the relation provided, without adding implying any information beyond it.
relation architect ".
objoct_entitics entities ": ["Minoru Yamasaki", "Allen Jack+
Cottier "I.
string must concisely and factually represent the information relation without embellishment unnecessary details string [template] relation ]: (data)
aggnsstic template ": "<ohject>> architect subject>."
! ###
(1 json
### your task: ..* json
"valid ":
"advice";
(["input_data" example_rdf pimpple_rdd_lis_ Jist }}}
valid, string
...
json ((
modification of LangChain's MNTVE_COMPLETINN _EERRR_RRRR___ SETRY_WIIH, prompt from Retry-
We
use
WithEmorrOnppuaaaaeer (langchain/output parsers/reryy.py)


Reviews:
[{"review_id": "YOnC58qgf3", "reviewer": "Reviewer_4Jj1", "summary": "", "questions": "* Question A: Have you conducted experiments on popular datasets like WebNLG and E2E NLG? If so, how does the proposed ASPIRO pipeline compare to baseline methods in terms of performance? \n* Question B: How does the ASPIRO method perform without the second step, i.e., the consistency validator? It would be valuable to include discussions on the significance of the consistency validator and its impact on the overall performance. \n* Question C: In Table, does \"(Kasner et al., 2023) fewshot-200\" mean that Kasner et al. used 200 input-output examples in their few-shot setup?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review, we have gone through your remarks and wanted to provide some clarifications below:\n\n## **ad. Reasons To Reject:**\n### 1) experiments on WebNLG and E2E NLG. \n - Please see Question A in the ad. Questions section below.\n### 2) Methods section: \n - We have rewritten the Methods section to increase readability and completeness, introducing all sub-modules in their own sections and separating the step-by-step explanation in it's own section. Please see it at the end of this rebuttal (it is automatically converted to Markdown from LaTeX, so the formatting may be off in some sections).\n\n## ad. Questions:\n### Question A: \n\"Have you conducted experiments on popular datasets like WebNLG and E2E NLG? If so, how does the proposed ASPIRO pipeline compare to baseline methods in terms of performance?\"\n#### Answer: \nWe have originally not done any experiments on WebNLG or E2E NLG. Our reasoning was that DART already contains entries from both WebNLG and E2E. Moreover the E2E dataset only has 7 unique relations (8 if you count familyFriendly yes/no as 2), meaning that there are only 8 templates to produce in each experiment, so statistical evaluation of ASPIRO effectiveness would be difficult with such small sample size. \n\nWe also believed that the performance on DART would also be representative of expected performance on WebNLG and E2E, but after running the additional analysis below, it is clear that our assumtion was wrong.\n#### Results on WebNLG\nTo analyze performance on WebNLG, we followed a similar approach as with Rel2Text in the paper. We observed both the Parsing Errors and the Automatic Metrics. Please see the parsing errors table below, which \n##### Parsing Errors\nIn the below results, *gpt-3.5-turbo* is used as LLM instances of all calls to both N-shot generator LLM stack and Consistency Validator. \nThe table shows number of errors tagged in generated templates by our Rule-based parser for different experiment setups (a1 to f2).\n\n| **Parsing errors, WebNLG dataset (354 total templates)** | **Total Errors** | **Template Errors** | **Missing SUBJECT** | **Multiple SUBJECTs** | **Missing OBJECT** | **Multiple OBJECTs** | **Illegal PLACEHOLDER** |\n|----------------------------------------------------------|:----------------:|:-------------------:|:-------------------:|:---------------------:|:------------------:|:--------------------:|:-----------------------:|\n| **a1_GPT3.5T(NONE)(0shot)(NONE)**                        |        80        |          79         |          24         |           0           |         56         |           0          |            0            |\n| **a2_(NONE)(0shot)(GPT3.5T)**                            |        83        |          79         |          23         |           0           |         57         |           2          |            1            |\n| **b_(GPT3.5T)(1shot)(NONE)**                             |        80        |          79         |          22         |           0           |         58         |           0          |            0            |\n| **c_(GPT3.5T)(2shot)(NONE)**                             |        81        |          80         |          23         |           0           |         58         |           0          |            0            |\n| **d_(GPT3.5T)(3shot)(NONE)**                             |        80        |          79         |          23         |           0           |         57         |           0          |            0            |\n| **e_(GPT3.5T)(4shot)(NONE)**                             |        80        |          79         |          22         |           0           |         58         |           0          |            0            |\n| **f1_(GPT3.5T)(5shot)(NONE)**                            |        80        |          79         |          22         |           0           |         58         |           0          |            0            |\n| **f2_(NONE)(5shot)(GPT3.5T)**                            |        83        |          79         |          23         |           0           |         57         |           2          |            1            |\n\nThe table for parsing errors shows that on WebNLG, ASPIRO is generally not able to fix any errors and Consistency Validation actually increases the number of total errors, making 3 of the templates more flawed than without CV. \n##### Automatic Metrics\nWe also compare the templates generated by ASPIRO to the manually crafted templates from Kasner and Dusek, (2022)'s zero-shot Neural pipeline paper to evaluate lexical similarity using PARENT, BLEU and METEOR. \n\n| **Automatic Metrics vs human-crafted templates** | **PARENT_P** | **PARENT_R** | **PARENT_F1** |  **BLEU**  | **METEOR (%)** |\n|--------------------------------------------------|:------------:|:------------:|:-------------:|:----------:|:--------------:|\n| **a1_GPT3.5T(NONE)(0shot)(NONE)**                |    0.7455    |    0.9594    |  **0.8308 **  |   54.50    |   **46.71 **   |\n| **a2_(NONE)(0shot)(GPT3.5T)**                    |    0.7461    |    0.9524    |   _0.8278 _   | **55.00 ** |    _46.20 _    |\n| **b_(GPT3.5T)(1shot)(NONE)**                     |    0.7454    |    0.9591    |    0.8305     |   54.37    |     46.54      |\n| **c_(GPT3.5T)(2shot)(NONE)**                     |    0.7455    |    0.9592    |    0.8306     |   54.44    |     46.56      |\n| **d_(GPT3.5T)(3shot)(NONE)**                     |    0.7456    |    0.9595    |    0.8308     |   54.47    |     46.58      |\n| **e_(GPT3.5T)(4shot)(NONE)**                     |    0.7456    |    0.9595    |    0.8308     |   54.44    |     46.59      |\n| **f1_(GPT3.5T)(5shot)(NONE)**                    |    0.7456    |    0.9595    |    0.8308     |   54.44    |     46.59      |\n| **f2_(NONE)(5shot)(GPT3.5T)**                    |    0.7461    |    0.9537    |   _0.8285 _   | **54.89 ** |    _46.34 _    |\n\nThe results are marginal at best and we can only see increase in BLEU score, while PARENT_F1 and METEOR are highest for zero-shot setting.\n(We are in process of calculating the semantic scores \\[BLEURT and NUBIA\\], but it requires more time.)\n#####  Conclusion\nContrary to our original belief, we can conclude that ASPIRO pipeline does not perform well on WebNLG and it is important to state that in the final paper. Thank you for pointing this out. We will provide these additional results in the appendix of the final paper.\n\n### Question B:\n\"How does the ASPIRO method perform without the second step, i.e., the consistency validator? It would be valuable to include discussions on the significance of the consistency validator and its impact on the overall performance.\"\n#### Answer: \nAs we mention in the **Operational costs** paragraph of the **Limitations** section, we believe that the consistency validator is not significantly beneficial to ASPIRO performance. We have conducted an ablation study on Rel2Text dataset, but due to the monthly cost hard-cap on OpenAI API it was not very thorough and we did not give it enough focus. Below is the result of our experiments. First 2 rows present GPT3.5Turbo as the base model for all parts of ASPIRO, while rows 3 and 4 use GPT3-davinci as the base model.\n\nNumber of parsing error counts without and with (**CV** ) consistency validation:\n\n| **error counts**                     | **Total Errors** | **Template Errors** | **Missing SUBJECT** | **Missing OBJECT** | **Illegal PLACEHOLDER** |\n|--------------------------------------|:----------------:|:-------------------:|:-------------------:|:------------------:|:-----------------------:|\n| **5xGPT3.5Turbo**                    |        12        |          11         |          2          |         10         |            0            |\n| **5xGPT3.5Turbo (CV GPT3.5Turbo)**   |        10        |          9          |          2          |          8         |            0            |\n| **5xGPT3-davinci**                   |        23        |          21         |          5          |         18         |            0            |\n| **5xGPT3-davinci (CV GPT3-davinci)** |        16        |          14         |          3          |         13         |            0            |\n\nAutomatic metrics without and with (**CV** ) consistency validation:\n\n| **automatic metrics for Rel2Text test set** | **BLEU** | **METEOR (%)** | **BLEURT** | **NB** | **SS** | **C (%)** | **I (%)** | **E (%)** | **PARENT_F1** |\n|---------------------------------------------|:--------:|:--------------:|:----------:|:------:|:------:|:---------:|:---------:|:---------:|:-------------:|\n| **5xGPT3.5Turbo**                           |   50.88  |      44.97     |   0.8215   | 0.8725 |  4.73  |    3.36   |    8.04   |   88.59   |     0.8910    |\n| **5xGPT3.5Turbo (CV GPT3.5Turbo)**          |   50.89  |      44.95     |   0.8213   | 0.8718 |  4.73  |    3.02   |    8.34   |   88.64   |     0.8908    |\n| **5xGPT3-davinci**                          |   50.96  |      44.69     |   0.8200   | 0.8680 |  4.71  |    4.92   |    7.59   |   87.48   |     0.8879    |\n| **5xGPT3-davinci (CV GPT3-davinci)**        |   51.32  |      44.77     |   0.8200   | 0.8702 |  4.72  |    3.35   |    7.70   |   88.94   |     0.8895    |\n\nThe above tables show, that CV is effective at reducing the Parsing error count slightly (12 to 10 and 23 to 16), but only shows marginal gains on statistical metrics.\nWe will additionally include these ablation results and the discussion on the importance of Consistency Validation in the appendix. \n### Question C:\n\"In Table, does \"(Kasner et al., 2023) fewshot-200\" mean that Kasner et al. used 200 input-output examples in their few-shot setup?\"\n#### Answer: \nYes, the number X (in fewshot-X) refers to the number of input-output examples from Rel2Text training set, that were used to fine-tune the BART model before evaluating it on the Rel2Text test dataset. It is stated in the caption of Table 2.\n\n## Note on Reproducibility\nWe did not mention it in the paper, but we did provide both working code as well as all result data in the supplementary files during the paper submission. We did not publish the code on GitHub as we feared it would clash with the anonymity requirements. We do have working code prepared for GitHub and we will add a reference to it in the final paper version.\n\n# Methods (revised)\nThe proposed method (ASPIRO) revolves around the conversion of structured data samples into verbalisation templates using a two-stage pipeline: **$N$-shot Generator** (§subsec:n-shot-generator) and **Consistency Validator** (§subsec:consistency-validator). The pipeline processes structured data samples, wherein each sample comprises of one or more RDF triples which share the same relation. ASPIRO (see Figure `fig:ASPIRO-schema`) starts with an initial prompt to verbally articulate the structured data. This is equivalent to prompting a single LLM directly. If the zeroth attempt isn't accurate, it will retry a maximum of $N$ times, refining the previous completion based on parsing errors (§subsec:rule-based-parser). Subsequently, the outputs are validated for consistency, ensuring faithful and reliable verbalisations. We explain the individual stages and their sub-modules in the sections below. Refer to Figure `fig:ASPIRO-schema` for full pipeline and terminology and Figure `fig:ASPIRO-example` for a more illustrative example of ASPIRO pipeline on a single input.\n\n## $N$-shot Generator\n§subsec:n-shot-generator\n\n$N$-shot Generator further fractures into an LLM stack and a Rule-based parser. The LLM Stack is tasked with generating verbalisation attempts based on given initial prompt (see §subsec:app-prompt-asdot). It does so with the help of the Rule-based parser. This parser checks the generated completions for structural accuracy, ensuring they adhere to expected patterns.\n\n### LLM Stack\n§subsec:llm-stack\n\nThe LLM stack is a set of $N+1$ LLMs, indexed from $0$ to $N$, where $\\mathcal{L}_0$ is responsible for the initial completion, whereas each further retry shot initiated by the Rule-based parser (§subsec:rule-based-parser) increments the index by $1$. Each $L_n$ is instantiated separately and does not have to be the same model. Equation (1) shows the single completion for structured input sample $x$ at shot $n`.\n\n$$\ny_n = \\mathcal{L}_n(\\mathcal{T}(x)) \\tag{1}\n$$\n\nwhere $\\mathcal{T}$ is a given prompt and can be either $\\mathcal{T}_I$ (initial) or $\\mathcal{T}_R$ (retry).\n\n### Rule-based parser\n§subsec:rule-based-parser\n\nA purely algorithmic module, which validates $y_n$ against a set of conditions $\\{\\mathcal{C}\\}$ one by one. If $y_n$ does not pass the condition $C_i$, a respective parsing error is logged into set $\\mathcal{E}_n$. The rules for each given completion are as follows:\n\n$\\mathcal{C}_0$ ... has exactly one `<subject>` substring.\n\n$\\mathcal{C}_1$ ... has exactly one `<object>` substring.\n\n$\\mathcal{C}_2$ ... has no other `<...>` substrings.\n\nIf the parser identifies any error in the structure, the next LLM in the LLM stack is re-prompted with Retry Prompt (§subsec:app-prompt-retry) to generate new completion.\n\n## Consistency Validator\n§subsec:consistency-validator\n\nEven if the outputs from the $N$-shot Generator adhere to the structural patterns, they might still contain inaccuracies, such as hallucinated content. This module assesses the quality of the verbalisations, using the PARENT statistical metric [dhingra-etal-2019-handling]. If PARENT F1 score is too low, the module will utilise an LLM with specialised Consistency Prompt (§subsec:app-prompt-consistency) to improve the sentence.\n\n### PARENT$_{F1}$ threshold\n§subsec:parent-score\n\nTo gauge the quality of the completion $y_n$ from $N$-shot Generator, we set a minimal threshold ($\\mu$) for the PARENT score of $y_n$. The score is calculated using eq. (3) against artificially constructed table and reference.\n\nFirst, we construct the respective hypothesis, table and reference entries:\n\n$$ h = y_n\\texttt{.replace}([s, o], e) \\\\ t = \\langle e, r\\texttt{.split(\" \")}, e \\rangle \\\\ \\rho = r \\tag{2}\n$$\n\nwhere `<subject>` and `<object>` are replaced with `<entity>` to prevent penalizing order discrepancy between hypothesis and table.\n\nWe then calculate the PARENT F1 score using equation (3).\n\n$$\nF1(y_n) = \\texttt{PARENT}(h, \\rho, t) \\tag{3}\n$$\n\n### Consistency LLM\n§subsec:consistency-llm\n\nIf the calculated PARENT score from §subsec:parent-score is not sufficient, we call another LLM with prompt $\\mathcal{T}_C$ as in eq. (4).\n\n$$\ny_C = \\mathcal{L}_C(\\mathcal{T}_C(r, y_n)) \\tag{4}\n$$\n\nThe prompt $\\mathcal{T}_C$ is designed to guide $\\mathcal{L}_C$ to identify problems with the given completion, provide advice how to fix it and subsequently produce fixed completion in a structured json output. See §subsec:app-prompt-consistency for full version of the prompt.\n\n## Detailed Pipeline Formulation\n§subsec:formulation\n\nGiven a dataset of structured data samples $\\{x^r\\}_{r\\in\\mathcal{R}}$, where $x^r=\\{x^r_1, x^r_2, ..., x^r_m\\}$ and $x^r_j$ is a single RDF triple $x^r_j=\\langle s^r_j, r, o^r_j\\rangle$ with relation $r\\in \\mathcal{R}$, the pipeline for one $x^r$ is as follows:\n\n**Step 0** Set $n=0$ and $\\mathcal{T}^r_0=\\mathcal{T}_I(x^r)$.\n\n**Step 1** Calculate $y^r_n$ using eq. (1).\n\n**Step 2** Use §subsec:rule-based-parser to validate $y^r_n$ against all conditions ${\\mathcal{C}}$. If errors ($\\mathcal{E}^r_n$)  are found, run equation (5) and return to **Step 1**. Otherwise go to **Step 3**.\n\n$$\n    \\mathcal{T}^r_{n+1} = \\mathcal{T}_R(x^r, y^r_n, \\mathcal{E}^r_n) \\\\\n    n = n+1 \\tag{5}\n$$\n\n**Step 3** Use §subsec:parent-score and calculate $F1(y^r_n)$ via eq. (3). If the calculated $F1$ score is lower than our chosen threshold $0\\leq\\mu \\leq 1$, continue to **Step 4**. Otherwise, output current $y_n^r$ as the final completion $y^r$.\n\n**Step 4** Use §subsec:consistency-llm with prompt $\\mathcal{T}_C^r$ and calculate $y^r_C$ via eq. (4).\n\n**Step 5** Compute \\(F1\\) scores of \\(y_n^r\\) and \\(y_C^r\\) using eq. (3) and take the completion with higher score via eq. (6) to produce the final completion\n$$\n    y^r = \\underset{y\\in\\{y_n^r, y_C^r\\}}{\\texttt{argmax}(F1(y))} \\tag{6}\n$$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Whilst the paper is difficult to follow at times, the motivation and the contributions of the work are clear. The proposed ASPIRO method provides a simple and versatile domain/task-agnostic pipeline for verbalization of data entries to relatively short sentences using a two-stage framework.\n* The authors measure the performance of their models along multiple automated metrics, including BLEU, METEOR, semantic similarity score using NUBIA, and PARENT F-1 score. These evaluation measures add more rigor and credibility to the presented findings.", "reasons_to_reject": "* The main experiments are exclusively conducted on the DART dataset. This limited scope raises concerns about the generalizability and robustness of the proposed method, as it lacks evaluation on other standard data-to-text datasets such as WebNLG (Castro Ferreira et al., 2020) and E2E NLG (Novikova et al., 2017). The absence of such diverse evaluations leaves readers questioning the broader applicability and performance of the ASPIRO approach on various data-to-text scenarios. Including experiments on other datasets would be crucial in addressing these concerns and validating the method's effectiveness across different domains and tasks.\n* Section III (Methods), the most critical section of the paper, is challenging to understand and follow. While I understand and acknowledge the constraints of limited space for method description, I believe the authors could have provided a clearer and more thorough explanation of their approach, covering all the involved steps. A comprehensive and well-articulated depiction of the method would have been useful in aiding the reader’s understanding of the contributions of this work.", "questions_to_authors": "* Question A: Have you conducted experiments on popular datasets like WebNLG and E2E NLG? If so, how does the proposed ASPIRO pipeline compare to baseline methods in terms of performance? \n* Question B: How does the ASPIRO method perform without the second step, i.e., the consistency validator? It would be valuable to include discussions on the significance of the consistency validator and its impact on the overall performance. \n* Question C: In Table, does \"(Kasner et al., 2023) fewshot-200\" mean that Kasner et al. used 200 input-output examples in their few-shot setup?", "paper_topic_and_main_contributions": "This paper introduces ASPIRO, a simple yet seemingly effective pipeline for data-to-text generation. ASPIRO consists of two main steps: (i) n-shot example generation and (ii) consistency checking.\n* (i) n-shot example generation: In this first stage, a language model generates potential input-output examples, given a test-time input. These examples are then subjected to scrutiny using a rule-based parser to ensure they meet specific predefined criteria.\n* (ii) Consistency checking: The second stage involves a consistency validator, which examines the output produced in the first stage. It identifies and addresses potential issues like hallucinations, ambiguities, and structural inconsistencies, aiming to correct them as much as possible.\n\nASPIRO's effectiveness is evaluated through experiments on the DART and Rel2Text datasets, showing a reduction in parsing errors and relatively high BLEURT scores. Compared to Kasner et al. (2023)’s method, ASPIRO shows competitive performance, though some improvements appear to be incremental. That said, the extent of ASPIRO's generalizability and robustness remains somewhat unclear due to the limited number of experiments conducted by the authors. Further investigations and evaluations across diverse datasets would provide a more comprehensive understanding of ASPIRO's capabilities and limitations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* Please rewrite Section III to improve its readability and presentation.\n* L025: RDF is not defined.\n* L037: Zero-shot → zero-shot [lowercase]\n* L043: Large Language Model (LLM) → large language model (LLM) [lowercase]\n* L101/L102: Change the @ symbol to the section symbol § (it is \\S in LaTeX)\n* L112: Extra comma after (3.1)\n* L124: No need to capitalize Large Language Model (also can use the acronym now)\n* Table 2: BLEU and METEOR should be capitalized. \n* L238: Pretrained Language Models → pre-trained language models\n* Tables 3 and 4: Not so clear what they represent and indicate.\n"}}, {"review_id": "wPE8hsClp5", "reviewer": "Reviewer_ShqH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very helpful review! We have conducted additional experiments and evaluations, which can be included in the final paper. Please see our response below and feel free to also read through the rebuttals to the other reviews on our paper for more details on performance and rewriting of the Methods section.\n\n## ad. Reasons to Reject:\n### 1) Include Cost and runtime\nYes, we agree. We did not give enough attention to evaluate the run times and costs in the original submission. As a reaction, we have conducted additional experiments and present the total cost as well as the 5-run-average run time (seconds) using different \"N\" in the N-shot-generator on the WebNLG dataset below (for the parsing error results on WebNLG, refer to rebuttal for review 4Jj1).\n\n#### Runtime costs (WebNLG)\nThe below table presents mean runtime cost in seconds from 5 experiment runs.\nAs our runs build up upon the results of previous runs (e.g. 1shot run will take already produced results from 0shot run and only rerun the ones, which had parsing errors), we also add a **Cummulative mean time** which adds up the total time it would require to run all shots of the respective experiment from scratch.\nThe dataset is **WebNLG**, containing 354 unique relations, meaning that there are in total 354 calls required to the 0x model (zero-shot call to the base LLM). \nBase LLM is GPT3.5-Turbo. \nFor simplicity, we use the same model for all shots of the N-shot Generator (1x, 2x, ...) and also for the Consistency Validator. \n\n| **Experiment **  | **Mean time (s)** | **Cummulative mean time (s)** |\n|------------------|:-----------------:|:-----------------------------:|\n| **0x**           |       377.5       |             377.5             |\n| **0x (with CV)** |       173.3       |             550.8             |\n| **1x**           |       142.4       |             519.9             |\n| **2x**           |       118.3       |             638.1             |\n| **3x**           |       159.3       |             797.5             |\n| **4x**           |       111.7       |             909.2             |\n| **5x**           |       108.1       |            1017.2             |\n| **5x (with CV)** |       138.8       |            1156.0             |\n\n#### Estimated API call costs\nIn total our costs of API calls were cca 191 USD. However, we did many redundant calls to the API during the development of ASPIRO. \nDuring our experiments, we mostly utilized the GPT3-davinci model, to keep the results consistent with what (Xiang et al., 2022) reported in their ASDOT paper. \nTo get a worst-case-scenario costs of ASPIRO, we calculate below a situation with base  GPT3.5-turbo (cost of $0.002 per 1000 tokens).\n\nEstimated experiments costs for Rel2Text, WebNLG and DART datasets.\nModel: GPT3.5-turbo\n\n| **Experiment**                  | **Total N-SHOT** | **Total CV (G3P5T)** | **Total Cost (G3P5T)** |\n|---------------------------------|:----------------:|:--------------------:|:----------------------:|\n| **REL2TEXT ASDOT_0xG3P5T (V4)** |       0.07       |         0.14         |          0.22          |\n| **REL2TEXT ASDOT_1xG3P5T (V4)** |       0.14       |         0.14         |          0.29          |\n| **REL2TEXT ASDOT_5xG3P5T (V4)** |       0.43       |         0.14         |          0.58          |\n| **REL2TEXT JSON_0xG3P5T (V4)**  |       0.14       |         0.14         |          0.28          |\n| **REL2TEXT JSON_1xG3P5T (V4)**  |       0.27       |         0.14         |          0.42          |\n| **REL2TEXT JSON_5xG3P5T (V4)**  |       0.81       |         0.14         |          0.96          |\n| **WEBNLG ASDOT_0xG3P5T (V4)**   |       0.11       |         0.23         |          0.34          |\n| **WEBNLG ASDOT_1xG3P5T (V4)**   |       0.23       |         0.23         |          0.45          |\n| **WEBNLG ASDOT_5xG3P5T (V4)**   |       0.68       |         0.23         |          0.91          |\n| **WEBNLG JSON_0xG3P5T (V4)**    |       0.21       |         0.23         |          0.44          |\n| **WEBNLG JSON_1xG3P5T (V4)**    |       0.42       |         0.23         |          0.65          |\n| **WEBNLG JSON_5xG3P5T (V4)**    |       1.27       |         0.23         |          1.50          |\n| **DART ASDOT_0xG3P5T (V4)**     |       0.46       |         0.92         |          1.38          |\n| **DART ASDOT_1xG3P5T (V4)**     |       0.92       |         0.92         |          1.84          |\n| **DART ASDOT_5xG3P5T (V4)**     |       2.76       |         0.92         |          3.68          |\n| **DART JSON_0xG3P5T (V4)**      |       0.86       |         0.92         |          1.78          |\n| **DART JSON_1xG3P5T (V4)**      |       1.73       |         0.92         |          2.65          |\n| **DART JSON_5xG3P5T (V4)**      |       5.18       |         0.92         |          6.10          |\n\nMultiply the above table by 10 to get runtime costs for GPT3-davinci model ($0.02 per 1000 tokens).\n\nThe table shows, that ASPIRO increases the costs up to 8 to 9-fold if we use 5-shot setting and Consistency Validation. Without Consistency Validation, the maximum cost of **N-shot setting is increased by N-times the initial (zero-shot) cost**. \n\n### 2) Incremental work \nWe believe that the work is valuable as the cost and run time of the conducted experiments are not negligible and results should be useful for future reference in NLG research. We acknowledge that further research on performance of ASPIRO with open source LLMs like Falcon or Llama 2 should be conducted to reduce these costs. Fine-tuning models such as Llama 2 for domain-agnostic data verbalization is also one more logical step to evaluate in further research.\n\n### 3) Open source models\nWe have thought of this option. We did experiments with Falcon LLM 7B during our preliminary research, but we would require additional time and compute power to analyse different open source models. Please see our results on Falcon 7B parsing errors below:\n\n| **Rel2Text parsing errors (total 226 samples)** | **Total Errors** | **Template Errors** | **Missing SUBJECT** | **Multiple SUBJECTs** | **Missing OBJECT** | **Multiple OBJECTs** | **Illegal PLACEHOLDER** |\n|-------------------------------------------------|:----------------:|:-------------------:|:-------------------:|:---------------------:|:------------------:|:--------------------:|:-----------------------:|\n| **a1_FALCON_7B(NONE)(0shot)(NONE)**             |        32        |          28         |          11         |           0           |          2         |          14          |            5            |\n| **a2_FALCON_7B(NONE)(0shot)(FALCON_7B)**        |        30        |          25         |          7          |           1           |          2         |          15          |            5            |\n| **b1_FALCON_7B(FALCON_7B)(1shot)(NONE)**        |        46        |          23         |          23         |           0           |         23         |           0          |            0            |\n| **b2_FALCON_7B(GPT3.5T)(1shot)(NONE)**          |        18        |          18         |          3          |           0           |          1         |          12          |            2            |\n\nYou can see that the FALCON_7B model does fairly well in zero-shot setup (a1) with only 28 templates being tagged as having parsing errors. Introducing Consistency Validation to the zero-shot setup (a2) reduces the parsing errors marginally from 28 to 25 templates. Conversely, reprompting FALCON (b1) to try and fix the parsing errors fixes 2 templates, but introduces more errors into the other 23, which is not beneficial.\n\nIn b2, we use GPT3.5-Turbo as the retry model, which shows promise in fixing 10 of the templates produced by a1. This shows the possible path of having open source model as \"cheap\" base model to produce templates and then use larger models such as GPT3.5T to fix the mistakes. \n\nWe plan to expand upon this in further work, using Llama2 as well.\n\n## Note on reproducibility\nWe did not mention it in the paper, but we did provide both working code as well as all result data in the supplementary files during the paper submission. We did not publish the code on GitHub as we feared it would clash with the anonymity requirements. We do have working code prepared for GitHub and we will add a reference to it in the final paper version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This setup is a useful tool for generating text from structured data in a low-resource setting.\n\nThe paper is well written and the pipeline is clearly described.\n\nThe experimental results show the utility of the setup and its ability to reduce erroneous output versus a simpler single-prompting approach.", "reasons_to_reject": "The complexity of the setup and the number of potential LLM calls leads me to believe it would be quite costly (as the authors admit in the limitations section). But they don't report on the actual cost. That would help bring the value of the proposed system into better focus. Additionally, run time would be useful to see, as this process seems like it could be quite time consuming. \n\nThe work largely build on previous work with an added method of repeating the prompting process for validation. That is important and valuable to show that this type of re-prompting helps, but it is relatively incremental.\n\nI would like to have seen experiments with open source models; given the number of prompting calls necessary, it would be valuable to know if this approach works using smaller, open-sourced models.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper reports on an improvement to the method of Xiang et al. (2022) for generating template sentences from RDF triples (or other similar structured formats) using LLMs. The key improvement of this paper is the design of their pipeline which incorporates several methods of validating the LLM's output - first, they generate a sentence using an LLM prompt with several examples and a target RDF triple. They then run this through a rule-based parser to check the validity of the output relative to the schema. If it is not valid, they then reprompt the LLM with an updated prompt up to N times (5 max in their experiments) to try to get the LLM to output a valid sentence. Once the output passes the rule-based parser, they pass it to an LLM-based consistency validator for further validation and possible correction. \n\nThis setup results in significantly reduced erroneous outputs, as shown by their experiments on DART and Rel2Text.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Your title is misspelled. Consistent, not \"consistant\" "}, "extra_text": {}}, {"review_id": "As8llYcr1M", "reviewer": "Reviewer_eDmS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and the comments and suggestions you provided. We would like to answer some of your concerns about this study below. Please also feel free to read the rebuttals for the other reviews on this paper for additional data and information.\n\n## ad. Reasons to Reject:\n### 1) Paper readability \n(\"*The paper does not read well; the text is cluttered with math formulations which obscure the main points of the paper.*\")\n#### Response\nWe have rewritten the Methods section in more complete and granular way, as it was indeed very cluttered. Please see it at the end of this rebuttal (formatting is off, as it is converted from latex to markdown).\n### 2) Ablation study\n(\"*The processing pipeline seems too heavy for the proposed task. Crucially, **no ablation study** is provided to assess the importance of individual components compared e.g. to simply prompting a single LLM with variety of prompts.*\")\n#### Response\nAs we mention in the **Operational costs** paragraph of the **Limitations** section, we believe that the consistency validator is not significantly beneficial to ASPIRO performance. We have conducted an ablation study on Rel2Text dataset, but due to the monthly cost hard-cap on OpenAI API it was not very thorough and we did not give it enough focus. Below is the result of our experiments. First 2 rows present GPT3.5Turbo as the base model for all parts of ASPIRO, while rows 3 and 4 use GPT3-davinci as the base model.\n\nNumber of parsing error counts without and with (**CV** ) consistency validation:\n\n| **error counts**                     | **Total Errors** | **Template Errors** | **Missing SUBJECT** | **Missing OBJECT** | **Illegal PLACEHOLDER** |\n|--------------------------------------|:----------------:|:-------------------:|:-------------------:|:------------------:|:-----------------------:|\n| **5xGPT3.5Turbo**                    |        12        |          11         |          2          |         10         |            0            |\n| **5xGPT3.5Turbo (CV GPT3.5Turbo)**   |        10        |          9          |          2          |          8         |            0            |\n| **5xGPT3-davinci**                   |        23        |          21         |          5          |         18         |            0            |\n| **5xGPT3-davinci (CV GPT3-davinci)** |        16        |          14         |          3          |         13         |            0            |\n\nAutomatic metrics without and with (**CV** ) consistency validation:\n\n| **automatic metrics for Rel2Text test set** | **BLEU** | **METEOR (%)** | **BLEURT** | **NB** | **SS** | **C (%)** | **I (%)** | **E (%)** | **PARENT_F1** |\n|---------------------------------------------|:--------:|:--------------:|:----------:|:------:|:------:|:---------:|:---------:|:---------:|:-------------:|\n| **5xGPT3.5Turbo**                           |   50.88  |      44.97     |   0.8215   | 0.8725 |  4.73  |    3.36   |    8.04   |   88.59   |     0.8910    |\n| **5xGPT3.5Turbo (CV GPT3.5Turbo)**          |   50.89  |      44.95     |   0.8213   | 0.8718 |  4.73  |    3.02   |    8.34   |   88.64   |     0.8908    |\n| **5xGPT3-davinci**                          |   50.96  |      44.69     |   0.8200   | 0.8680 |  4.71  |    4.92   |    7.59   |   87.48   |     0.8879    |\n| **5xGPT3-davinci (CV GPT3-davinci)**        |   51.32  |      44.77     |   0.8200   | 0.8702 |  4.72  |    3.35   |    7.70   |   88.94   |     0.8895    |\n\nThe above tables show, that CV is effective at reducing the Parsing error count slightly (12 to 10 and 23 to 16), but only shows marginal gains on statistical metrics.\nWe will additionally include these ablation results and the discussion on the importance of Consistency Validation in the appendix. \n### 3) Code and reproducibility\n(\"*Although the authors provide code / algorithm snippets throughout the paper, there is no mention of releasing their code upon publication which may hinder reproducibility.*\")\n#### Response:\n- We did provide both working code as well as all result data in the supplementary files during submission.\n- We did not include GitHub mention as we feared it would clash with the anonymity requirements. We do have working code prepared for GitHub and we will add a reference to it in the final paper version.\n### 4) Paper format\n(\"*Overall, would rather suggest transforming this paper into a long paper and presenting more principled experiments: comparing the proposed architecture with simpler baselines and performing and ablation study on indvidual components.*\")\n#### Response:\nThis is a valid point. We had problems with including all the results into 4 pages of content. We are discussing with the PCs the possibility of transforming the paper from short to long format in the final submission.\n\n# Methods (revised)\nThe proposed method (ASPIRO) revolves around the conversion of structured data samples into verbalisation templates using a two-stage pipeline: **$N$-shot Generator** (§subsec:n-shot-generator) and **Consistency Validator** (§subsec:consistency-validator). The pipeline processes structured data samples, wherein each sample comprises of one or more RDF triples which share the same relation. ASPIRO (see Figure `fig:ASPIRO-schema`) starts with an initial prompt to verbally articulate the structured data. This is equivalent to prompting a single LLM directly. If the zeroth attempt isn't accurate, it will retry a maximum of $N$ times, refining the previous completion based on parsing errors (§subsec:rule-based-parser). Subsequently, the outputs are validated for consistency, ensuring faithful and reliable verbalisations. We explain the individual stages and their sub-modules in the sections below. Refer to Figure `fig:ASPIRO-schema` for full pipeline and terminology and Figure `fig:ASPIRO-example` for a more illustrative example of ASPIRO pipeline on a single input.\n\n## $N$-shot Generator\n§subsec:n-shot-generator\n\n$N$-shot Generator further fractures into an LLM stack and a Rule-based parser. The LLM Stack is tasked with generating verbalisation attempts based on given initial prompt (see §subsec:app-prompt-asdot). It does so with the help of the Rule-based parser. This parser checks the generated completions for structural accuracy, ensuring they adhere to expected patterns.\n\n### LLM Stack\n§subsec:llm-stack\n\nThe LLM stack is a set of $N+1$ LLMs, indexed from $0$ to $N$, where $\\mathcal{L}_0$ is responsible for the initial completion, whereas each further retry shot initiated by the Rule-based parser (§subsec:rule-based-parser) increments the index by $1$. Each $L_n$ is instantiated separately and does not have to be the same model. Equation (1) shows the single completion for structured input sample $x$ at shot $n`.\n\n$$\ny_n = \\mathcal{L}_n(\\mathcal{T}(x)) \\tag{1}\n$$\n\nwhere $\\mathcal{T}$ is a given prompt and can be either $\\mathcal{T}_I$ (initial) or $\\mathcal{T}_R$ (retry).\n\n### Rule-based parser\n§subsec:rule-based-parser\n\nA purely algorithmic module, which validates $y_n$ against a set of conditions $\\{\\mathcal{C}\\}$ one by one. If $y_n$ does not pass the condition $C_i$, a respective parsing error is logged into set $\\mathcal{E}_n$. The rules for each given completion are as follows:\n\n$\\mathcal{C}_0$ ... has exactly one `<subject>` substring.\n\n$\\mathcal{C}_1$ ... has exactly one `<object>` substring.\n\n$\\mathcal{C}_2$ ... has no other `<...>` substrings.\n\nIf the parser identifies any error in the structure, the next LLM in the LLM stack is re-prompted with Retry Prompt (§subsec:app-prompt-retry) to generate new completion.\n\n## Consistency Validator\n§subsec:consistency-validator\n\nEven if the outputs from the $N$-shot Generator adhere to the structural patterns, they might still contain inaccuracies, such as hallucinated content. This module assesses the quality of the verbalisations, using the PARENT statistical metric [dhingra-etal-2019-handling]. If PARENT F1 score is too low, the module will utilise an LLM with specialised Consistency Prompt (§subsec:app-prompt-consistency) to improve the sentence.\n\n### PARENT$_{F1}$ threshold\n§subsec:parent-score\n\nTo gauge the quality of the completion $y_n$ from $N$-shot Generator, we set a minimal threshold ($\\mu$) for the PARENT score of $y_n$. The score is calculated using eq. (3) against artificially constructed table and reference.\n\nFirst, we construct the respective hypothesis, table and reference entries:\n\n$$ h = y_n\\texttt{.replace}([s, o], e) \\\\ t = \\langle e, r\\texttt{.split(\" \")}, e \\rangle \\\\ \\rho = r \\tag{2}\n$$\n\nwhere `<subject>` and `<object>` are replaced with `<entity>` to prevent penalizing order discrepancy between hypothesis and table.\n\nWe then calculate the PARENT F1 score using equation (3).\n\n$$\nF1(y_n) = \\texttt{PARENT}(h, \\rho, t) \\tag{3}\n$$\n\n### Consistency LLM\n§subsec:consistency-llm\n\nIf the calculated PARENT score from §subsec:parent-score is not sufficient, we call another LLM with prompt $\\mathcal{T}_C$ as in eq. (4).\n\n$$\ny_C = \\mathcal{L}_C(\\mathcal{T}_C(r, y_n)) \\tag{4}\n$$\n\nThe prompt $\\mathcal{T}_C$ is designed to guide $\\mathcal{L}_C$ to identify problems with the given completion, provide advice how to fix it and subsequently produce fixed completion in a structured json output. See §subsec:app-prompt-consistency for full version of the prompt.\n\n## Detailed Pipeline Formulation\n§subsec:formulation\n\nGiven a dataset of structured data samples $\\{x^r\\}_{r\\in\\mathcal{R}}$, where $x^r=\\{x^r_1, x^r_2, ..., x^r_m\\}$ and $x^r_j$ is a single RDF triple $x^r_j=\\langle s^r_j, r, o^r_j\\rangle$ with relation $r\\in \\mathcal{R}$, the pipeline for one $x^r$ is as follows:\n\n**Step 0** Set $n=0$ and $\\mathcal{T}^r_0=\\mathcal{T}_I(x^r)$.\n\n**Step 1** Calculate $y^r_n$ using eq. (1).\n\n**Step 2** Use §subsec:rule-based-parser to validate $y^r_n$ against all conditions ${\\mathcal{C}}$. If errors ($\\mathcal{E}^r_n$)  are found, run equation (5) and return to **Step 1**. Otherwise go to **Step 3**.\n\n$$\n    \\mathcal{T}^r_{n+1} = \\mathcal{T}_R(x^r, y^r_n, \\mathcal{E}^r_n) \\\\\n    n = n+1 \\tag{5}\n$$\n\n**Step 3** Use §subsec:parent-score and calculate $F1(y^r_n)$ via eq. (3). If the calculated $F1$ score is lower than our chosen threshold $0\\leq\\mu \\leq 1$, continue to **Step 4**. Otherwise, output current $y_n^r$ as the final completion $y^r$.\n\n**Step 4** Use §subsec:consistency-llm with prompt $\\mathcal{T}_C^r$ and calculate $y^r_C$ via eq. (4).\n\n**Step 5** Compute \\(F1\\) scores of \\(y_n^r\\) and \\(y_C^r\\) using eq. (3) and take the completion with higher score via eq. (6) to produce the final completion\n$$\n    y^r = \\underset{y\\in\\{y_n^r, y_C^r\\}}{\\texttt{argmax}(F1(y))} \\tag{6}\n$$", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The work shows that LLMs can be used for generating single RDF triples, reducing the need for training data for this task.\n- The work introduces multiple components for iterative validation of the model outputs, improving the controllability of the generation process.", "reasons_to_reject": "- The paper does not read well; the text is cluttered with math formulations which obscure the main points of the paper.\n- The processing pipeline seems too heavy for the proposed task. Crucially, no ablation study is provided to assess the importance of individual components compared e.g. to simply prompting a single LLM with variety of prompts.\n- Although the authors provide code / algorithm snippets throughout the paper, there is no mention of releasing their code upon publication which may hinder reproducibility.\n- Overall, would rather suggest transforming this paper into a long paper and presenting more principled experiments: comparing the proposed architecture with simpler baselines and performing and ablation study on indvidual components.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a method for verbalizing single RDF triples - a subtask of data-to-text generation. Unlike previous works which rely on templates or finetuned language models, the proposed method - ASPIRO - utilizes a processing pipeline based on prompting large language models (LLMs).\n\nThe pipeline consists of two components. In the first component, a LLM is prompted for producing a verbalization of the triple and a simple rule-based parser is used to either accept the verbalization or iteratively re-prompt a LLM given the original input and the parsing errors. The second component further prompts a LLM to improve the consistency of the generated verbalization.\n\nThe results show that the proposed LLM-based pipeline achieves competitive results on verbalizing single triple to the finetuned BART model proposed in Kasner et al. (2023).", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 189 - BERT -> BART\n- BLEU and METEOR scores are usually capitalized"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This study presents an approach for verbalizing structured data into concise template sentences, achieved by instructing Large Language Models (LLMs) to generate entity-agnostic templates while incorporating additional algorithmic parsing consistency checks.

**Pros**:
- The motivation behind the work and its contributions are clearly articulated.
- The proposed method yields significant performance improvements in data-to-text generation tasks.
- The authors use a diverse array of automated metrics for evaluation, enhancing the rigor and credibility of their results.

**Cons**:
- Two reviewers note that the paper's complexity is exacerbated by an abundance of mathematical expressions, which can obscure the paper's core message.
- The method is complex and requires numerous LLM API calls.
Decision: Accept-Findings