Faithful Low-Resource Data-to-Text Generation through Cycle Training
Year: 2023

Abstract:
Methods to generate text from structured data have advanced significantly in recent years, primarily due to fine-tuning of pre-trained language models on large datasets. However, such models can fail to produce output faithful to the input data, particularly on out-of-domain data. Sufficient annotated data is often not available for specific domains, leading us to seek an unsupervised approach to improve the faithfulness of output text. Since the problem is fundamentally one of consistency between the representations of the structured data and text, we evaluate the effectiveness of cycle training in this work. Cycle training uses two models which are inverses of each other: one that generates text from structured data, and one which generates the structured data from natural language text. We show that cycle training, when initialized with a small amount of supervised data (100 samples in our case), achieves nearly the same performance as fully supervised approaches for the data-to-text generation task on the WebNLG, E2E, WTQ, and WSQL datasets. We perform extensive empirical analysis with automated evaluation metrics and a newly designed human evaluation schema to reveal different cycle training strategies’ effectiveness of reducing various types of generation errors. Our code is publicly available at https:// github.com/Edillower/CycleNLG.

heading: 1 Introduction
text: A wealth of information exists in the form of structured knowledge, such as movie information databases or product catalogs, which we may want to verbalize for a variety of purposes, such as comparing two items, or presenting detailed descriptions in a natural language form suitable for conversational assistants. Recent work has tackled this data-to-text generation task using freely available
†The research was done during an internship at Amazon. *These two authors contributed equally to this work.
public datasets, most notably WebNLG (Castro Ferreira et al., 2020) and ToTTo (Parikh et al., 2020). However, there remain two major challenges. First, the volume of training data required for good performance, especially if it is not in a domain represented by one of the existing corpora, is very large. Second, multiple recent papers (Yang et al., 2022; Parikh et al., 2020), inter alia, point out that neural natural language generation (NLG) from structured data tends to produce multiple kinds of errors which limit the utility of these models in customer-facing applications. Hallucinations occur when NLG models inject nonsensical words or information not related to the input structured data, into the generated output text. For instance, an NLG model may claim a shirt’s color is “three”. Simple factual errors occur when an NLG model produces coherent but factually wrong output.
There are two threads of research to consider as we attempt to tackle these problems in the datato-text setting. The first is designing models that directly produce output more faithful to the input data. The second is designing models to detect and correct factual errors or hallucinations after the output text is generated. In both cases, prior research has generally assumed sufficient pairs of structured data and text as training data to achieve human-level performance on the task. While fact verification models can achieve very high performance, they generally do so when trained on large corpora of 100,000 examples or more. Since performance appears to degrade when evaluated on outof-domain data (Estes et al., 2022), this presents a significant limitation of fact-verification models. Similarly, corpora like WebNLG contain about 20,000 examples; this is probably too small to achieve human performance even under full supervision (Guo et al., 2020) but is large enough to make it prohibitive to generate domain-specific corpora of the size of WebNLG.
In spite of the above mentioned limitations, very
2847
few of the models developed for data-to-text and table-to-text tasks take advantage of the fact that the task of faithful text generation is fundamentally one of consistency between the data and the corresponding text. In fact, despite the WebNLG 2020 challenge being explicitly bi-directional, only three models competing in the challenge leveraged this idea of consistency.
To overcome the aforementioned limitations related to the lack of training data (especially out-of-domain data) and the consistency between structured data and text, we adopt a Cycle Training (Iovine et al., 2022a) approach. We assume unpaired data D, in the form of subject-predicateobject triples, and text T , which may or may not be from the same domain. We also make use of a small (100 samples) set of paired data and text, Dpr, Tpr. Cycle training makes use of two iteratively trained models, a forward model F : D → T and a reverse model R : T → D. Training is unsupervised, namely, we freeze one model and use it to transform one set of inputs, and train the other by using it to predict the original input from the output of the first model. Concretely, in one cycle, we freeze F , and train R by reconstructing the input D as R(F(D)). After one training epoch, we reverse the roles of the two models. Remarkably, even though the models are initially quite poor, this can converge to models with near-supervised performance, as we will show. Moreover, we show that this process ensures the faithfulness of the output text with respect to the input data, and vice versa, even with very little or no paired data.
We note that a previous data-to-text system, CycleGT, has used cycle training (Guo et al., 2020). We will discuss in detail the differences between CycleGT and our proposed approach in Section 2. Moreover, we examine in detail the conditions under which cycle training works well, with an emphasis on domains and the nature of the training text and structured data. We find that unsupervised cycle training outperforms low-resource fine-tuned models and can achieve near fully-supervised performance when initialized and post-tuned with a small amount of annotated data. We detail the results and findings in Section 5. Thus, to build on past research in self-consistent data-to-text generation, we make these novel contributions:
(i) We successfully apply cycle training to both the data-to-text and text-to-data models using only a pre-trained language model, T5, without recourse
to graph methods or other auxiliary models. (ii) We show that cycle training achieves nearly the same performance as supervised models for some domains.
(iii) We present an extensive empirical analysis on the conditions under which cycle training works well, and on the data-to-text faithfulness with respect to different types of generation errors.
(iv) We design a novel counting and ranking based annotation schema to more comprehensively evaluate the faithfulness of the generated text from the standpoints of correctness, faithfulness, data coverage, and fluency. Our schema improves upon the rating-based schema used for the WebNLG 2020 Challenge, in terms of objectiveness, consistency, precision and ease of evaluation.

heading: 2 Related Work
text: Multiple data-to-text and table-to-text tasks have been presented in the literature, such as WebNLG (Gardent et al., 2017a; Colin et al., 2016; Gardent et al., 2017b), DART (Nan et al., 2020), ToTTo (Parikh et al., 2020), and WikiTableT (Chen et al., 2021), which primarily consist of data from general-purpose sources like Wikipedia. Several large language models (Herzig et al., 2020; Liu et al., 2021; Yang et al., 2022) have been trained on large scale table-to-text corpora (Chen et al., 2019) to perform fact verification. However, these models may not perform well on specific domains they have not been trained on, such as ecommerce (Estes et al., 2022; Vedula et al., 2022). Therefore, we must either find a way to easily generate new data to train large data-to-text models, or use unsupervised methods. Recently, Xiang et al. (2022) attempted to augment training data using GPT-3 (Brown et al., 2020), and Su et al. (2021) employed an information retrieval system to build prototypes for the generation. Our work makes orthogonal contributions to these studies, as we directly utilize the underlying unpaired data and text of a target corpus without recourse to any additional information retrieval or generation systems. Further, the above-mentioned data-to-text tasks have been evaluated primarily on automatic word- or ngram-level metrics such as BLEU (Papineni et al., 2002) or METEOR (Banerjee and Lavie, 2005), with minimal (and mostly subjective) evaluation of faithfulness. In this work, we design a novel annotation schema to perform a more comprehensive evaluation of the faithfulness of the generated text
to the input data.
Cycle training (Zhu et al., 2017; Zhou et al., 2016) relies on two models which are essentially inverse transforms of each other that are used to create “cycles”, which should return identical output to the input given. There are two distinct forms of cycle training. The first form (Zhou et al., 2016) aims to learn to transform from one input form to another, e.g., to learn rotations of a car in one image to another. The second is the use of a “cycle consistency loss” as an auxiliary loss to some other task, e.g., in generative adversarial networks performing style transfer on images (Zhu et al., 2017). NLG typically relies on models which are autoregressive and non-differentiable. This precludes the direct use of cycle consistency losses (Guo et al., 2020; Pang and Gimpel, 2019; Iovine et al., 2022a). Nonetheless, we can still use cycle training via an alternating training strategy where we freeze one model and train the other, and vice versa (Lample et al., 2017; Pang and Gimpel, 2019). In this work, we train solely using cycle consistency. Cycle training has been recently applied to language processing tasks. In one text-to-text application, Iovine et al. (2022b) use a similar unsupervised methodology to perform bidirectional text transformations for converting keyword search queries to natural language questions, and vice versa. It has also been used for Named Entity Recognition in the absence of large annotated text (Iovine et al., 2022a). In this case, one model extracts entities, and the inverse model creates text from those entities. The approach is limited by the fact that there are many ways to realize sentences with the same entities. Put differently, there is no strong requirement of cycle consistency, and this will become even more apparent as we analyze the conditions under which cycle training works well in data-to-text tasks.
To the best of our knowledge, the only work to explicitly call out the self-consistency requirement of data-to-text generation tasks is the CycleGT model (Guo et al., 2020) developed for data-totext generation on the WebNLG dataset. One key advantage of cycle training is that it need not rely on any supervision, and instead relies primarily or solely on the self-consistency of inputs and outputs. However, CycleGT relies on a pre-existing NER model to extract entities from the output text. The authors then train an inverse model to predict the links between entities and predicates. Should the entities not be recognized by their NER system, the
model will fail overall; this is not an uncommon situation in applications such as online shopping (Estes et al., 2022; Vedula et al., 2023), where entities are complex or change frequently (Malmasi et al., 2022). In principle, a separate NER model could be built using cycle training, as in CycleNER (Iovine et al., 2022a), but the CycleGT authors did not do so. In this work, we design a simple approach using pre-trained language generation models, fine-tuned for both data-to-text and text-to-data generation cycles.

heading: 3 Methodology

heading: 3.1 Backbone Models
text: The pre-requisite of cycle training is having two mutually inverse models. We adopt T5, an evidently strong-performing model according to the WebNLG 2020 challenge (Castro Ferreira et al., 2020; Agarwal et al., 2020; Guo et al., 2020), as our backbone model for both forward generation, (F : D → T that performs RDF-to-text generation) and reverse generation, (R : T → D that performs text-to-RDF generation). T5 is a large sequence-to-sequence model pre-trained with the unsupervised span-mask denoising objective and several supervised text generation tasks like summarization and translation (Raffel et al., 2020). We linearize the RDF triples of each sample into a sequence d that denotes the subject, predicate, and object of each triple by the [S], [P], and [O] tags respectively. Therefore, both RDF-to-text and text-to-RDF can be treated and trained as sequenceto-sequence generation tasks. We further train or optionally fine-tune the T5 backbone models, as detailed in Section 4, with the teacher forcing (Williams and Zipser, 1989; Lamb et al., 2016) learning objective for task-specific generation. This means that for the training of the auto-regressive decoder, we do not propagate the model decoded next token but force each input to be the correct gold token for training.

heading: 3.2 Cycle Training of the Backbone Models
text: Iterative Back-Translation (IBT) (Hoang et al., 2018) has been reported as an effective training schema that enforces cycle consistency for various NLP tasks (Guo et al., 2020; Iovine et al., 2022a). We apply this idea to iteratively cycle train our models. This consists of the Data-Text-Data (DTD) cycle that enforces the self-consistency of data, and the Text-Data-Text (TDT) cycle that similarly en-
forces the self-consistency of text. As shown in Figure 1, for the DTD cycle, the Data-to-Text model takes the linearized triples d as input and generates the associated intermediate text t̂. Sequentially, the Text-to-Data model is trained with the objective of reconstructing d with the supplied t̂. The reconstruction loss Ld′ is the averaged negative log likelihood shown below where di denotes the i-th token of sequence t and |d| is the sequence length:
Ld′ = − 1|d| ∑|d|
i=0 log p(di|d0, ..., di−1, t̂) In a reverse manner, for the TDT cycle, the Text-toData model first takes text t as input and generates the associated linearized triples d̂. Sequentially, the Text-to-Data model is trained with the objective of reconstructing t with the supplied d̂. The reconstruction loss Lt′ is the averaged negative log likelihood shown below where ti denotes the i-th token of sequence t and |t| is the sequence length:
Lt′ = − 1|t| ∑|t|
i=0 log p(ti|t0, ..., ti−1, d̂) Due to the non-differentiable procedure of generating discrete intermediate outputs of tokens, the reconstruction loss can only propagate through the second model of each cycle, namely the Text-toData model of the DTD cycle and the Data-to-Text model of the TDT cycle. Therefore, the training of the two models can only proceed with the alternation of the TDT cycle and the DTD cycle so that both models’ performance may gradually improve.

heading: 4 Experimental Setup

heading: 4.1 Data and Baselines
text: We experiment on existing data sources that have annotated pairs of data triples and reference texts.
WebNLG (Colin et al., 2016; Gardent et al., 2017b; Castro Ferreira et al., 2020) is a well-established dataset that has supported multiple challenges on four tasks: RDF-to-English (Text), RDF-toRussian (Text), English (Text)-to-RDF, and Russian (Text)-to-RDF. Each WebNLG sample consists of a set of subject-predicate-object triples and up to three associated human-written reference texts that faithfully express and verbalize the information contained in the triple set. We use the English data from the most recent 3.0 version of the WebNLG corpus, from the WebNLG+ 2020 challenge.
DART (Nan et al., 2020) is a large-scale datato-text dataset that unifies and builds upon multiple data resources including E2E (Novikova et al., 2017), WikiSQL (WSQL) (Zhong et al., 2017), WikiTableQuestions (WTQ) (Pasupat and Liang, 2015), and WebNLG (Gardent et al., 2017a). To better facilitate our experiments and evaluations on different domains, we separately utilize the humanannotated portion of E2E, WTQ, and WSQL from DART. To align the data formats in accordance with WebNLG, we also drop some WSQL and WTQ samples that contain non-conventional structural tags. The DART dataset hereafter refers to the cleaned, WebNLG-excluded, and human-annotated portion of E2E, WTQ, and WSQL.
Table 1 shows detailed dataset statistics. When the data is used for cycle training, we follow previous work and split all the paired samples into one separate corpus of shuffled text, and another separate corpus of shuffled triple sets. For the linearized sequences, as shown in Figure 1, we: (1) prefix the string “Generate in English:” to the input sequence of the RDF-to-text model and pre-
fix the string “Extract Triples:” to the input of the text-to-RDF model; (2) convert camel-cased or snake-cased subjects, predicates and objects to regular strings; and (3) normalize accented characters.
Fine-tuning large pre-trained language models, such as BERT (Devlin et al., 2019), BART (Lewis et al., 2020), and T5 (Raffel et al., 2020) has been proven to be effective in achieving new state-of-theart performance on numerous tasks. Fine-tuning refers to the supplemental training of a pre-trained model on a dataset of the target task and domain. We detail and perform the following three baseline fine-tuning strategies in this work:
Fully supervised fine-tuning: We fine-tune T5 with the entire in-domain (with respect to the test set) data as the supervised baseline. Low-resource fine-tuning: We fine-tune the T5base model with 100 randomly selected sets of triples and their associated reference texts to formalize a low-resource supervised baseline. We deem 100 annotated samples to be a small enough amount, that is easily achievable with a relatively low human annotation effort. Low-resource fine-tuning with additional pretraining: When using text from the target domain for cycle training, the teacher forcing algorithm naturally raises the probability of generating the target domain tokens, which may result in performance gains in token matching metrics (Section 5.1). To study the influence of using in-domain text, we further pre-train the T5 model with in-domain text and an unsupervised span-mask denoising objective prior to the low-resource fine-tuning process. As our main objective is to probe a training strategy orthogonal to the model structure, we only include the above three baselines to control the model structure, data pre-requisites, and parameter sizes.

heading: 4.2 Comparing Cycle Training Strategies and Pre-requisites
text: We explore two different training strategies evaluating the effectiveness and generalizability of cycle training under different data constraints.
Unsupervised cycle training: As the most constrained low-resource scenario, in unsupervised cycle training we directly employ the IBT schema to cycle-train the forward model and reverse model with unpaired text and triple sets in turns. Low-resource cycle training: In this setting, a small amount of paired text and triple sets are accessible. For fair comparison and consistency, we utilize the same subset of data as the low-resource fine-tuning baseline described in Section 4.1. The low-resource paired data is leveraged through precycle fine-tuning, which first trains the forward and reverse model with the paired data before employing the IBT schema to cycle-train the two models.
Guo et al. (2020) and Iovine et al. (2022a) vaguely state that the latent content or entity distribution of the text corpus and the data corpus must have some uncertain degree of overlap to make the cycle training approach work. To empirically assess this pre-requisite condition, we apply unsupervised cycle training with the same size of text and data corpus at different matching levels, as a rough approximation of overlap of the latent content or entity distribution. Specifically, we randomly select half of the WebNLG triplets as the data corpus. We purposefully select five equal-sized text corpora that contain 0%, 25%, 50%, 75%, and 100% of the originally related reference text; and complementarily include 100%, 75%, 50%, 25%, 0% of unrelated reference text respectively.

heading: 4.3 Training Parameters
text: We use the T5-base model which has 12 layers, a hidden size of 768, 12 self-attention heads, and 220M parameters. We use the AdamW optimizer with linear weight decay, a max input length of 256, a learning rate of 3e-4, and an effective batch size of 256. At inference time, we decode with the beam search algorithm using 4 beams and a generation length varying between 3 tokens and 256 tokens. We train each model up to 50 epochs with a delta of 0.05 basis points and a patience of 5 epochs as the early stopping criteria. We select the best model by
the validation set’s METEOR score - the ranking metric of the WebNLG 2020 challenge, and we report the aforementioned model’s performance on the test set. We repeat each experiment 5 times with different random seeds and report the average and standard deviation of each metric.

heading: 5 Results and Discussion

heading: 5.1 Automatic Evaluation
text: We assess each system/strategy with five widelyused automatic metrics that measure the generation quality from three different aspects: tokenmatching, semantic similarity, and faithfulness. ROUGE (Lin, 2004) is a recall-oriented metric that calculates the overlapping n-grams (ROUGEN for N-grams) and word sequences (ROUGE-L) between the reference text and generated text. BLEU (Papineni et al., 2002) is a precisionoriented metric calculating overlapping n-grams between the reference text and generated text. METEOR (Banerjee and Lavie, 2005) computes the unigram match between the reference text and generated text based on the tokens’ surface form, stemming, synonyms, and paraphrase similarities. BertScore (Zhang et al., 2020) measures the semantic similarity of the reference text and generated text via the utilization of the contextual embeddings from BERT for the calculation of the cosine similarity of best-matching token pairs.
PARENT (Dhingra et al., 2019) is an entailmentbased token-matching metric that calculates the F1 score based on entailed precision (an n-gram is correct if it occurs in the reference text or entailed by the input data) and entailed recall (recall against the reference text input data, adjusted by a weight parameter). It measures the faithfulness of the generated text with respect to the input data.
Table 2 displays the performance of multiple data-to-text generation approaches under various settings. We observe that unsupervised cycle training generally falls short of the fully-supervised finetuning method’s performance. When compared with the low-resource fine-tuning method, it scored higher on WebNLG and WTQ but performed worse on E2E and WSQL, where the performance gap on WSQL is larger. We attribute such divergence to the difference in the number of unique predicates and vocabulary. Cycle training should be able to improve the model’s generalizability and robustness through exposure to larger amounts of diverse text and structured data, and through its capability of gradually learning different data-totext associations. For datasets like E2E and WSQL, their smaller vocabulary size and number of unique predicates imply that a small amount of annotated samples might cover a great deal of the datasets’ underlying variation. This leads to a strong lowresource fine-tuning performance that has smaller
performance gaps with the fully-supervised counterparts, and overshadows the unsupervised cycle training method.
However, when a small amount of annotated data is made available for initializing the cycle training, the low-resource cycle training strategy significantly improves the generation performance over the low-resource fine-tuning method, and achieves competitive performance with respect to the fully-supervised method. Such an improvement is consistent across all four datasets and five types of evaluation metrics. Notably, when applied to multi-domain and open-domain datasets (WebNLG, WTQ, and WSQL), low-resource cycle training generated texts that have better faithfulness to the input data, evident from the PARENT score, compared to the fully-supervised fine-tuning approach. Compared with the setting that applies additional pre-training, it is evident that cycle training works beyond simply raising the probability of generating target domain tokens.
As for the experiments on cycle training with unpaired datasets at different overlapping levels, the results in Table 3 show that performance sharply increases at the beginning with the increase of overlapping levels and then turns to flatten at around the 50% overlapping level. This suggests that when the size is the same, the unpaired data corpus and text corpus used for cycle training need to have at least 50% entities (or say, latent information) overlap to achieve performance at an ideal level. We deem 50% as a reasonable level since many related but unpaired texts and structured data (e.g., content and infoboxes from Wikipedia, product specification tables and descriptions from online shopping platforms, etc.) may have higher information overlap. Hence, based on our experimental results, we believe that low-resource cycle training is a universally applicable approach that can effectively learn from vast unpaired structured data and texts with minimal human effort.

heading: 5.2 Human Evaluation
text: To quantitatively compare generated text with respect to correctness, faithfulness, data coverage, and fluency, we develop a new counting and ranking-based annotation schema, and use it to conduct human evaluation. Our schema features better objectiveness, consistency, and precision compared to the 0-100 rating-based schema used for the WebNLG 2020 Challenge. We define the following measures (full annotation guidelines, including disambiguation examples, and screenshots of the annotation interface available in Appendix A):
Count of Factual Errors (FE) measures the factual correctness of the generated text with respect to the entities (subject and object) and predicates of the input triplets. Factual errors are information in the generations that contradict the information in the input subject-predicate-object context. For each attempted predicate given in the input triplets, the annotator is asked to increase the factual error count if the subject and/or object of the predicate’s associated expression doesn’t match facts from the input. Count of Hallucination Errors (HE) measures the relevance of the generated text with respect to the input triplets. Hallucination errors occur when words or phrases in the generation cannot be inferred from the input subject-predicate-object triplets, for instance, because the value does not make logical sense, or because the predicate of the expression is not present in any triple. Unlike FEs, HEs add information not present in the triplets or reference, but do not directly contradict the triplets. The annotator is asked to increase the HE count if a piece of information contained in the generated text is not presented in, or cannot be reasonably inferred by the input triplets. For better consistency and less ambiguity, a reasonable inference is defined as a piece of information contained in the generated text that isn’t present in the input triplets but is present in the reference text. Count of Information Misses (IM) measures the information coverage of the generated text with
respect to the predicates given in the input triplets. For each predicate given in the input triplets, the annotator is asked to increase the IM count if the generated text does not attempt to express the predicate. Fluency Preference (FP) measures the quality of the generated text in terms of the grammar, structure, and coherence of the text. The annotator is asked to compare the fluency of pairs of generated texts within a batch, to compile the final ranking that reflects the annotator’s subjective preference. The fluency comparison and ranking only considers the grammar, structure, and coherence of the text independent of IM, FE, and HE.
In terms of the training time required to perform
the task accurately, we collected the error annotations (FE, HE, IM) from two domain experts and the fluency annotations from crowd-sourced workers respectively via an annotation tool built on the Appen1 platform. To enforce the annotation quality and foster future research on explainable automatic error analysis, we ask the domain experts to mark the token(s) that constitute an FE or HE, and to select the triple(s) that constitute the IM before counting the respective errors. The domain experts independently annotate the same set of 204 randomly sampled generations with a resulting agreement (Cohen’s kappa score (Artstein and Poesio, 2008)) of 0.74 for FE, 0.69 for HE, and 0.85 for IM, which is very satisfactory given the complexity of the task. For the relatively more subjective fluency ranking task, we use the average of three crowd-sourced native English speakers’ judgments for each generation. As generating longer text for larger triple sets is more difficult than generating for smaller triplets, we normalize the counts of FE, HE, and IM by the number of their input triples. Therefore, the FE, HE, and IM we report in Table 4 can be interpreted as the probability of making such errors per input data triple. We show an example of our error analysis in Table 5, and provide additional examples in Appendix B.
Our human evaluation suggests that lowresource cycle training consistently reduces factual errors, hallucination errors and information misses. From Section 5.1, cycle training presents a larger performance gain when applied to datasets that have more variations in terms of underlying relations and surface realizations. When looking together with Table 2, the human evaluation of errors and information coverage correlates better with the PARENT score, which confirms PARENT’s capability of measuring faithfulness. It is also evident from the annotation results that all three evaluated data-to-text generation models are more likely to make hallucination errors over factual errors, which calls for more future effort to alleviate hallucinations. In terms of the generated texts’ fluency, lowresource cycle training is able to improve over the low-resource fine-tuning method but still cannot consistently beat the fully-supervised approach.

heading: 6 Conclusions
text: In this work, we demonstrated the application of cycle training for data-to-text generation. We sys-
1https://appen.com/
tematically investigated the effectiveness of cycle training across different domains, and the application of pre-cycle fine-tuning in low-resource settings. We showed that our approach substantially improved data-to-text generation performance in low-resource settings, achieved competitive performance compared to fully-supervised models, and also improved the faithfulness of the generated text through a reduction in factual errors, hallucinations and information misses, even when compared to fully supervised approaches. We also designed a schema for effective human evaluation of data-totext generation, that improves upon prior work and encourages more objective and consistent reviews of faithfulness.

heading: Limitations
text: We recognize that our annotation and analysis methods can require considerable human labor, that can limit the amount of annotated data we can collect. Also, despite cycle training being generally accepted as a model-agnostic approach, we were not able to test a wide variety of backbone models due to resource constraints. In addition, though we relaxed the entity constraints and made cycle training for data-to-text generation end-to-end, the nondifferentiability problem remains unsolved. The intermediate outputs generated by the first model of each cycle are assumed to be correct. This is a weak assumption that may propagate misleading training signals to the second model of each cycle, particularly in the early stage of the training.
To address these limitations, future work may focus on the following directions: 1) building differentiable cycle training models; 2) exploring au-
tomated error detection methods and building models that may utilize such signals; and 3) assessing different backbone models, including large language models like GPT-X, with the cycle training approach.

heading: Acknowledgements
text: First and foremost, we extend our appreciation to Prof. James Caverlee for his unwavering support that was vital for the completion of this work. We gratefully acknowledge the contributions of the following individuals for their expert advice as well as their participation in our preliminary human annotation study, which helped us a lot in refining our experiments, annotation guidelines and annotation interface: Dr. Giuseppe Castellucci, Dr. Besnik Fetahu, Prof. Eugene Agichtein, Dr. Saar Kuzi, Jason Ingyu Choi, Dr. Zhiyu Chen, Dr. Tuan M. Lai, Lingbo Mo, and Yicheng Wang. We also would like to express our gratitude to the three reviewers and the meta reviewer for their constructive suggestions.

heading: Appendix

heading: A Annotation Guidelines
text: In this section, we include descriptions of the human annotation task performed in this work.
For this annotation task, the annotators will be provided a set of input triplets in the subjectpredicate-object structure, and the annotators will be asked to provide their judgement of four modelgenerated text snippets associated with the input triplets. Our target is to annotate the 1) Count of Factual Errors, 2) Count of Hallucination Errors, 3) Count of Information Misses, and 4) Fluency Preference for the generations. We use two different Appen interface-pages: one for the annotation of the three types of error counts, and one for the annotation of Fluency Preference.

heading: A.1 Annotation of Error Counts

heading: A.1.1 Count of Factual Errors (FE)
text: Count of Factual Errors (FE) measures the factual correctness of the generated text with respect to the entities (subject and object) and predicates of the input triplets. Annotation Instruction: Factual errors are information in the generations which contradict the information in the subject-predictate-object context. For each attempted predicate given in the input triplets, the annotator is supposed to increase the count if [the subject and/or object of the predicate’s associated expression does not match the facts suggested by the input triplets]. Examples: (See Table 6)

heading: A.1.2 Count of Hallucination Errors (HE)
text: Count of Hallucination Errors (HE) measures the relevance of the generated text with respect to the input triplets. Annotation Instruction: Hallucination errors occur when words or phrases in the generation cannot be inferred from the subject-predicate-object triplets, for instance because the value doesn’t make logical sense, or because the predicate of the expression isn’t present in any triple. Distinguished from FEs, HEs invent information not in the triplets or reference, but do not directly contradict the triplets. The annotator is supposed to increase the count if [a piece of information contained in the generated text is not presented in or can not be reasonably inferred by the input triplets]. For better consistency and less ambiguity, reasonable inference is defined as a piece of information
contained in the generated text isn’t presented in the input triplets but is presented in the reference text. Examples: (See Table 7)

heading: A.1.3 Count of Information Misses (IM)
text: Count of Information Misses (IM) measures the information coverage of the generated text with respect to the predicates given in the input triplets. Annotation Instruction: For each predicate given in the input triplets, the annotator is supposed to increase the count by 1 if [the generated text did not attempt to express the predicate]. Examples: (See Table 8)

heading: A.1.4 Annotation Interface for Errors
text: The annotation task is presented batch-by-batch. Each batch contains one shared input triplet and three model-generated text snippets (in random order) with respect to the input triplets. The annotators will see the input triplets data and the reference ground-truth data at first. Please keep in mind that the ground-truth data is just a reference for the convenience of better understanding the input triplets and the boundary of "reasonable inference" and they may not be perfect. To begin with, we ask the annotators to provide token level annotations of FE and HE. The “Context” is the input triplets shown before. The annotators can click the [ grey-rounded i ] button at the upper-right conner to see information regarding the use of the annotation tool. The annotators can also click the [grey-rounded i] button next to the tag to see a recap of its definition. Annotations of overlapped tokens are permitted. After finishing up the token-level FE and HE annotation, please provide the count of FE and the count of HE respectively. Next, the annotators need to identify if there’s any missed information in the generation. If "Yes", the annotators will be asked to check the IMs. See Figure 2 and Figure 3 for screenshots of the annotation interface for FE, HE, and IM.

heading: A.1.5 Fluency Preference (FP)
text: Fluency Preference (FP) measures the quality of the generated text in terms of the grammar, structure, and the coherence of the text. Annotation Instruction: The annotator is supposed to perform pairwise fluency comparison of the generated texts within a batch to compile the final ranking that reflects the annotator’s subjective preference. The fluency comparison and ranking
shall only consider the grammar, structure, and the coherence of the text without the consideration of IM, FE, and HE. Examples: Since FP is a relatively more subjective measure that asks for overall preference, we only provide some contrasting examples for the three aspects of fluency.
• Grammar: Generation A is better than B because B is grammatically incorrect/influent.
– Generation A: 108, written by karen maser, has 2.12 million U.S. viewers.
– Generation B: 108 U.S. viewers million is 2.12, written by karen maser.
• Structure: Generation A is better than B because the pieces of information in A are more naturally connected and expressed.
– Generation A: Andrew Rayel is a member of the Bobina band that plays trance music.
– Generation B: Andrew Rayel is an associated band/associated musical artist with Bobina. His genre is Trance music.
• Coherence: Generation A is better than B because She speaks modern Hebrew is more logically and consistently connected with the pre-
vious sentences compared to Modern Hebrew is spoken in Israel.
– Generation A: Liselotte Grschebina was born in Karlsruhe, Germany and trained in the School of Applied Arts in Stuttgart. She speaks modern Hebrew.
– Generation B: Liselotte Grschebina was born in Karlsruhe, Germany. She studied at the School of Applied Arts in Stuttgart. Modern Hebrew is spoken in Israel.

heading: A.1.6 Annotation Interface for FP
text: The annotators may see two to three generations, and the annotators are asked to perform pairwise comparison and rank the generations by their grammar, structure, and coherence without considering information coverage and factual errors. The annotators should start with 1 for the highestranked/most-fluent text of the generations within the batch. Ranking tie is permitted, but note this is a ranking task, so the annotators will need to check the numbers in a normal ranking manner. If the annotators see two generations [A, B], and A is better than B, then the annotators should select 1 for A and 2 for B instead of 3 for B. If the annotators see three generations [A, B, C], and A is identical to B, B is better than C, then the annotators should select 1 for A, 1 for B, 3 for C instead of 2 for C. See Fi-
-gure 4 for a screenshot of the annotation interface for FP.

heading: B Additional Result Analysis for WebNLG and DART
text: In this section, we show some additional samples of text generated by multiple models trained under different cycle settings, and the errors made by these models for the WebNLG dataset in Table 9 and DART dataset in Table 10.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 7
A2. Did you discuss any potential risks of your work? Not applicable. Our work is based on publicly available datasets and the method won’t introduce additional risks.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Section 5
3 B1. Did you cite the creators of artifacts you used? Section 4 and Section 5
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The artifacts we used are publicly available and accessible. The license can be found on their respective GitHub or project page.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4 and Section 5
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. We use existing datasets that have no personal or offensive information.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4
C 3 Did you run computational experiments? Section 4 and Section 5
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4 and Section 5
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We used the default settings that are consistent with previous work
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Section 5
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Section 5
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Section 5
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? We didn’t discuss it in the paper but the use of data was made clear to the annotators
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Ethics review is not required due to the nature of our data
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? The demographic and geographic characteristics are irrelevant to our work, and such information was not collected from the annotators.

heading: Future Work
text: To address these limitations, future work may focus on the following directions: 1) building differentiable cycle training models; 2) exploring au-
tomated error detection methods and building models that may utilize such signals; and 3) assessing different backbone models, including large language models like GPT-X, with the cycle training approach