Compositional Generalization for Data-to-Text Generation
Authors: Xinnuo Xu
Ivan Titov
Mirella Lapata
Venue: EMNLP 2023 Findings (2023)
Keywords: Compositional Generalization
Data-to-Text Generation
Natural Language Generation
Clustering
Reinforcement Learning
Benchmark
TL;DR: A clustering-based approach aiming at improving the compositional generalisation ability of the models on Data-to-text generation tasks.

Abstract: Data-to-text generation involves transforming structured data, often represented as predicate-argument tuples, into coherent textual descriptions. Despite recent advances, systems  still struggle when confronted with unseen combinations of predicates, producing unfaithful descriptions (e.g.,hallucinations or omissions). We refer to this issue as compositional generalisation, and it encouraged us to create a benchmark for assessing the performance of different approaches on this specific problem. Furthermore, we propose a novel model that addresses compositional generalization by clustering predicates into groups. Our model generates text in a sentence-by-sentence manner, relying on one cluster of predicates at a time. This approach significantly outperforms T5-baselines across all evaluation metrics. Notably, it achieved a 31% improvement over T5 in terms of a metric focused on maintaining faithfulness to the input.

Compositional Generalization for Data-to-Text Generation
Xinnuo Xu', Ivan Titov and Mirella Lapata' ILCC, School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam muu3ed.accuu {ititov, mlap}@inf ed. ad..ac.uk
Abstract
Input Tuples
Output Text Description
Data-to-text generation involves transforming structured data, often represented as predicate- argument tuples, into coherent textual descrip- tions. Despite recent advances, systems still struggle when confronted with unseen com- binations of predicates, producing unfaithful descriptions (e.g., hallucinations or omissions). We refer to this issue as compositional gen- eralisation, and it encouraged us to create a benchmark for assessing the performance of different approaches on this specific prob- lem. Furthermore, propose novel model that addresses compositional generalization by clustering predicates into groups. Our model generates text in a seetennceby-sentenee man- ner, relying on one cluster of predicates at time. This approach significantly outperforms T5 baselines across all evaluation metrics. No- tably, achieved 31% improvement over in terms of a metric focused on maintaining faithfulness to the input.'
Willam Anders
Retroment
1989-09-00
Willamm
Apollo
commander
Frank
Saptember tal, 1969, was crew member on Apolla 8 served under commander Frank Borman Apoloo 8 was operated by NASA with Duzz Aldnn backup plac.
Willam Anders
member of Apolla 8
Apollo
backup pilot
Apollo operator
NASA
Argument
Predicate
Arguiennt
Figure An example from WebNLG dataset (Gardent al., 2017). DTG aims at transforming the input struc- tured data (left) into coherent textual description (right).
Chen et al., 2020) has focused on evaluating the compositional generalization (CG) abilities of DTG models. These studies created few-shot training splits using established benchmarks by reducing the number of training examples or limiting the number of distinct predicate combinations in the training set through random selection. However, these arbitrary selections overlook the practical ef- fort required for annotating different examples. For example, annotating examples with larger number of input tuples requires more time and effort.
1 Introduction
Data-to-text generation (DTG) (Gardent et al., 2017; Dušek et al., 2020) aims to accurately gener- ate textual descriptions from input tuples; the tuples should encompass all the information needed for generating the description regardless of the narra- tive order. Typically, as shown Figure 1, each tuple consists of two arguments and one predi- cate that conveys their relationship. Given the large number of pre-defined predicates, it is time- consuming to collect amples for each potential combination of them. Thus, models must have the ability to generalize and handle examples with previously-unseed predi- cate combinations. We refer to this generalization scenario as compositional generalization.
We introduce test environment based on Gar- dent et al. (2017). During training, models are exposed to examples with fewer input tuples, while in the testing phase, examples with more input tuples are presented. To make it even more chal- lenging, we combine CG with few-shot learning by reducing the number of training examples for each predicate combination to one. We also incorporate CG with domain adaptation by evaluating the mod- els on unseen domains. Our results demonstrate that the SoTA pre-trained language models (LMs; Raffel et al. 2020; Kale and Rastogi 2020b) fail generalize effectively in our experimental setup.
training ex-
To tackle this issuc, we propose clustering- based method (Figure 2) that utilize the graph weights learned from training data decompose unfamiliar predicate compositions into smaller groups during inference. Each group consists predicate combinations encountered by the model
Prior research (Mehta et al., 2022; Xu et al., 2021; Kale and Rastogi, 2020a; Peng et al., 2020;
CCode available at: gi thub. wtchub.coo///.e com com/XinnuoXu/CG_no "ee follow the format specified in Gardent et al. (2017).

---

Input Tuples
Predicato Decomposition
Text Generation based the Predicate Decomposiiion
Willam Anders
refirmmet
1998 09-009-11
Apclo 8
commender
Frank Borman
operator
retiament
Willam Anders
Apolio 8
commander Frank Borman
rotroment 1968-09 0988009-01
Apollo
tecdup pilol
Akain
Willam Anders member Apollo
Apollo
operator
NASA
member
Willam Andirs
Appllo
Apollo 8
baakkpp plloi Buzz Aldinn
Apollo 8 operator NASA
Seckkup pilo
commander
Predicate Composition
Willa Anders, who retred on Stptember 1st. 1869, was crew member on Apolo 8 and served under commanderr Frank Bowmar.
Apoll was oporated by NAASA with Buzz Ateno as barkup pllot
Chatering
Figure 2: Framework our proposed inference procedure. We introduce a set of clustering basen methods that leverage the graph weights learned during training decompose the unseen predicate compositions into familiar groups. For each group, we gather the input tuples associated with predicates in that group, and generate a sentence describe them. The final text description is created by combining the generated sentences from all the groups.
during training. Then, individual sentence descrip- tions are generated separately for each group, and combined to form the final description for the input. In contrast to previous studies that primarily rely self-tr self-training to improve CG in DTG (He et al., 2020; Heidari et al., 2021; Li et al., 2021; Mehta al., 2022), as well using data augmentation to improve CG in various tasks such as seman- tic parsing (Andreas, 2020; Qiu et al., 2022; Fang et al., 2023), our method solely relies on small training sets and does not require any additional human-annotated automatically labeled, or unla- beled data. In the CG-centric testing scenario, we observe significant improvements across all metrics in the benchmark compared the vanilla T5 model (Raffel et al., 2020; Kale and Rastogi, 2020b). A faithfulness based metric (Dušek and Kasner, 2020) shows an impressive gain of 31 % over T5. A similar trend is seen when combin- ing the CG challenge with few-shot learning and domain adaptation. Our contributions are:
Train on CGFull-k
Train on CGOneShot-k
Seen test
Compositional Generalization
Compositional Generalization
Few-shot Learning
Unseen test
Compositional Generalization
3
Compositional Generalization
Domain Adaptation
Domain Adapiation
Few-shot Learning
Figure 3: Four evaluation scenarios.
CG focused Benchmark
Existing benchmarks (Ratnaparkhi, 2000: Liang et al., 2009; Mairesse et al., 2010; Banik et al., 2013; Wen et al., 2015; Lebret et al., 2016; Wen et al., 2016; Gardent et al., 2017; Wiseman et al., 2017: Novikova et al., 2017: Parikh et al., 2020) have provided important test-bed for DTG mod- els. We specifically choose WebNLG 2017 (Gar- dent et al., 2017) to build our benchmark upon (reasons are shown in Appendix B) and primarily focus on assessing the models' capability for CG. We present two sets of training splits and create four distinct testing scenarios with different diffi- culty levels by incorporating the training splits with the seen and unseen test sets offered in WebNLG.
We create a benchmark to assess the CG ability of DTG models and generate four testing scenarios of varying difficulty by combining CG with few- shot learning and domain adaptation.
We present an innovative architecture that uses a clustering algorithm to decompose the text de- scription generation for unfamiliar input predicate combinations into smaller, familiar ones.
2.1 Training Sets
CGFULL consists of set of independent train- ing splits (CGFULL--),, ranging from 2 to 7. CGFULL-k exclusively consists of examples where the number of input tuples is equal to or less than k. We excluded CGFULL-5 and -6 here due to the marginal increase in the amount of data within these two sets. Note that, CGFULL-7 represents the full training set of WebNLG 2017.
We show that our method produces outputs that are not only more faithful but also exhibit greater resemblance human- references compared vanilla pre-trained LMs while tested on the proposed benchmark.
We also introduce an intrinsic evaluation framework for inspecting input decomposition.
htttp:: thub. com htt.c///hiiggoo/////ennggrrreee master/data/y1.6.

---

CGONESHOT
CGFULL
Training Splits
-2
-3
-7
-2
-4
# Examples
553
987
1477
1867
2043
2203
7712
11350
14744
18101
% of WebNLG2017 train
3%
5%
8%
10%
11%
12%
43%
63%
81%
100%
Table 1: The amount of examples within splits in collection CGFULL and CGONESHOT.
CGONESHOT is a one-shot version of CGFULL. We define predicate composition combination of the input predicates that is order agnostic. In CGFULL, the same predicate composition is found in multiple training examples paired with different arguments. To present a more challenging situation, we generate split CGONESHOT-K by randomly se- lecting one example for each predicate composition in the corresponding CGFULL-K split.
2.4 Evaluation Metrics
Generation Evaluation focuses on evaluating the generated text w.r.t its similarity to human- authored reference sentences. We adopt BLEU (Papineni et al., 2002), a token-level exact matching metric that is incorporated in the WehNLG.
Faithfulness Evaluation tests the generated text is faithful to the input tuples (Wen et al., 2015; Reed et al., 2018; Dušek and Kasner, 2020). Un- faithful generations contain hallucinations (genera- tions with extra or incorrect information) or omis- sions (generations missing important input infor- mation) (Dušek et al., 2019). We adopt PARENT (Dhingra et al., 2019), an entailment- met- ric, where a higher score indicates a lower occur- rence of hallucinations; and OK-percent (Dušck and Kasner, 2020), a natural language inference- based metric representing the proportion of system generations free from hallucinations or omissions.
The statistical details presented in Table 1. Models will undergo separate training on each indi- vidual splits in both CGFULL and CGONESHOT.
2.2 Validation and Test Sets
The validation and test set in the original WebNLG 2017 remain unchanged. The test set consists of two categories: The seen category includes exam- ples from 9 domains present in training. while the unseen category consists of examples from 5 new domains with newly defined predicates and unseen entities not present in training. Note that, both vali- dation and test sets contain examples with different numbers of input tuples, ranging from to
3 Clustering based CG Methods
The framework of our approach is shown in Figure 2. For clarity, we denote the input tu- ples and their corresponding predicates X {X1, X2, Xw) and P { P2.... Pv] re- spectively, where P. the predicate of the 7th tuple X,. N represents the total number of tuples. The output text is denoted as Y = (Yi, Y2 YM}. where Y, represents the jth sentence in the output. Fine-tuning pre- pre-trained LMs (Kale and Rastogi, 2020b; Ribeiro et al., 2021), aim to maximize the log likelihood of generating the ground truth text Y given the linearized input tuples X, denoted log p (Y xX), during training. However, these mod- els face challenges in generalizing to unseen predi- cate combinations. To overcome this, an intuitive approach is to decompose these unseen combina- tions into smaller groups, ensuring that the combi- nation in each group has been seen during training; then, generate sentence from each group individu- ally and combine them to form the final description.
2.3 Evaluation Scenarios
We create four evaluation scenarios (Figure 3) by pairing training splits with test sets. When trained on CGFuLL-k, the models are exposed to numer- ous examples with predicate combinations of up to k predicates. When tested on the WebNLG seen category, their CG abilities evaluated they en- counter novel combinations consisting of a greater number of predicates. To further intensify the chal- lenge, the models are tested on the unseen cate- gory, requiring them to demonstrate both CG and adaptability to predicate combinations from new domains. They also need to handle newly intro- duced arguments. On the other hand, the models trained on CGONESHOT-K have only seen ample for each combination of up to k predicates. When tested on the seen set, their CG abilities are assessed with few-shot learning skills. When eval- uated on the unseen set, the examination of their capabilities further extends to domain adaptation.
We denote the decomposition (C,, C2. Cm}. where C; represents the jth predicate group responsible for generating sentence Yy. Since DTG tasks require to include all the input
C

---

information in the output without repetition, the de- composition must fulfill C₁ U2 CM P and Vi,j 1 < j & M, C,C - Ø. The text generation is then broken down into a set of parallel steps. Each step aims at creating a single sentence to describe the tuples associated with the predicates in one of the groups:
C
silver annotations for the training of 7. For each training example in DTG', we calculate the BLEU score for each input tuple w.r.t each sentence in the reference output. Afterwards, the tuple is aligned with the sentence that achieves the highest BLEU score. This process yields collection of tuple groups, where tuples within a group are described in the same sentence. By removing the arguments from each tuple, we obtain annotated predicate decomposition from each DTG training example. Now, we introduce two methods to obtain matrix ? using these annotated predicate decompositions:
p XY = (XIN)
(I)
where X is subset of X. The tuple X; E X iff its predicate P. E C An alternative represen- tation of a predicate decomposition involves the use of a matrix. Given a set of tuples with their corresponding predicates, we construct a fully con- nected undirected graph, denoted as G = (V, E), where the predicates are represented as nodes in V. In turn, a decomposition can be considered titioned graph derived from the original graph G. We encode the partitioned graph
Numerical Weight Prediction determines Yij by analyzing individual occurrence co-occurrence of two predicates P., P3 in the training data:
# (P..P) #(P)+
par-
where # (Pi, P3) denotes the frequency of both predicate P: and Pj being mentioned in the same sentence throughout the corpus. # (P.) represents the frequency of predicate P. appearing in the cor- pus However, this approach has a limitation: if either predicate not included in the training set, the weight will always be zero. This becomes chal- lenging when transitioning to a new domain, since most weights in the matrix y will be zero.
binary matrix M, where M.j sigmifies that the predicates P. and Pj belong to the same group, while Mij = indicates that they belong to different groups.
Unfortunately, the annotated ground truth de- compositions are unavailable in the majority of DTG training sets. Therefore, the training objective becomes maximizing the marginal log likelihood of the output text Y w.r.t. the latent M.
Neural Network based Prediction solves this problem by introducing a small scale transformers- based neural network?. The model takes two tok- enized predicates concatenated as input, including a "[CLS]" token attached to the beginning. The embedding of the "[CLS]" token from the final transformer layer is then fed into classification head. This head predicts whether the two predi- cates should be described in the same sentence (1) or not (0). Elements in ? can be written as:
log (Y x) log PM- X))
As the number of imput tuples increases, exploring all possible decompositions for each example is intractable. Following Kim et al. (2017); Deng et al. (2018), we approximate the marginal likelihood as:
log PM (Y |M, x) z log (Y) (M))) M, X)
This results in the stochastic decomposition vari- able M being replaced with the deterministic value M (MM) M. Assuming that M follows a Bernoulli distribution B(y), with each element within the matrix being independent, we can rep- resent the distribution of M as Mjj ~ B(1g)) Thus, the expectation of the binary matrix M is the Bernoulli parameter matrix y. In Section 3.1 we demonstrate two training methods to predict
"is sigmoid (W (Transformers (P,, P3)) + b)
where W is a linear transformation. b is the bias. For classifier training, synthetic data is generated using the automatically annotated predicate decom- position. Positive examples are formed by pairing any two predicates within the same group, while negative examples consist of pairs from different groups. In cases where there is only one predicate
3.1 Training
Inspired by Su et al. (2021) and Moryossef et al. (2019), we propose an automatic way for creating
"Check Appendix for the data preprocessing details. "To calculate # (P.), we discard the examples with only a single tuple as input. ? "Appendix D shows the hyper-parameters.
For the input tuple subsets, we also have x ...U C = X and vi, j: & < & M, X 0 X = Ø.

---

Algorithm 1 DTG with Predicate Decompostion
icate clusters. For each cluster, we calculate the occurrence of its predicates being described in the first sentence across all training examples, and choose the one with the highest frequency as the first cluster. Next, we select the subsequent cluster by identifying the one with the highest number of unique arguments observed in the previous cluster. We repeat this step until all clusters are sorted. Fi- nally, we utilize the fine-tuned T5 model produce sentence for each cluster following the order, and concatenate these generated sentences form the final output describing the input tuples.
Require; Input tuples X; Their predicates P: Trained predi- cate clustering model PC; Fine-tuned generator T5.
1: for ls 4- 1 X do
2: C PCPPC.PP
3: if FfflectiveCluster() then
4:
C 4- C and & f- k; Break
5: end if
6: end for
7: C + Son(C)
8: for + 1 to & do
9: Y, ())
10: end
11: Y < (X1)
group in the imput, indicating single-sentence out- put, each input predicate is randomly paired with a predicate from the dataset that is not part of the in- put for the negative examples' creation. The model is trained with the Cross Entropy loss.
3.3 REINFORCE-enhaneed Decomposition
Deterministic approaches heavily rely on automat- ically annotated predicate decompositions. How- ever, the annotator based on exact token match- ing is weak at detecting paraphrasing, resulting in misalignment of tuples to the wrong sentence. To address this, we propose REINFORCE (Glynn, 1990; Williams, 1992) based approach that reduces reliance on silver annotations.
To train the text generation model, we use the annotated tuple groups introduced earlier in this section instead of relying the predicate decompo- sitions predicted by the introduced models. Since each tuple group aligned sentence, we take the tuples as input and the sentence as output to fine-tune setence-level T5 for text generation.
We first simplify the marginal distribution in Eq 2 using Jensen's 's Incquality. Since the logarithm function concave, we have:
log PM- (Y]M,X) EM--(Mix log M)
3.2 Testing
Our goal to train the parameters & in Ps (M(X) to optimize this lower bound. The gradient of is:
The testing procedure is outlined in Algorithm 1. Initially, we aim to obtain predicate decomposi- tion C for given set of input tuples. To accom- plish this, we begin by estimating the expectation of the binary matrix M, i.e. ?, using the models introduced in Section 3.1. Afterwards, we iter- ate through all possible values for the number of predicate groups & (ranging from 1 to (X) and em- ploy a clustering algorithm (specifically, spectral clustering in this study) over the matrix 79. If the minimum weight between two predicates within the same cluster exceeds threshold 10, halt the exploration. Our objective is minimize the number of predicate clusters and ensure that each cluster does not contain unfamiliar predicate pairs. To enhance the coherence of the generated text, we implement a simple method to arrange the pred-
(MID [log (Y(M,X) Ve log (M[X)
However, directly sampling binary matrix from the Bernoulli distribution M B(y) does not guar- antee that the matrix can be transformed into a valid set of clusters C. Therefore, we propose a method to replace the sampling process in the forward pass. We first sample a binary matrix M from the Bernoulli distribution B(?). Next, we perform element- wise multiplication between this matrix and 7, denoted as M y. Finally, we apply the spectral clustering algorithm the resulting matrix to obtain discrete clusters Il As this pro- cess is not differentiable, akin to the concept of the straight-through estimator (Bengio et al., 2013), 3), we perform a backward pass through the sampling step by computing the probability p& (MM[)) using the sampled predicate decomposition C:
*In experiments, we ohserved decrease in the BI.UU metric when using sentence-level T5 models for text genera- tion. Consequently, we substitute them with standard tuples- paragraph T5 models that fine-ture mnedt unaligned input- output data in each training split.
Pe (M
IIII (1 799) 1-k
To improve the coherence of the generated texts, we adjust y by assigning O to 7/y
corresponding triples of predicate
and P, do not share any arguments.
"'hhe number of clusters corresponds to the numher of sentences in the ground truth text description.
The threshold is determined using the validation set.

---

where kay E {0,1}. ka- = 1 means that predicate P. and P belong to the same cluster in C, while ki = 0 indicates that they are assigned to different clusters. By using this approach, the gradients can be propagated through the Bernoulli distribution. We compute My E (0, using the transformers classifier structure discussed in Section 3.1. In order to speed up convergence, we initialize the parameters using the classifier that has been trained in the deterministic approach.
CGONESHOT
-2
-3
-6
-7
T5
44.94
49.43
54.39
56.86
58.49
58.98
OK-per PARENT BLEU
CG-Ra
47.01
49.03
49.32
51.06
50.39
51.08
CG-Nu
47,46
52.91
55.37
57.81
56.32
58.07
CG-RL
47.58
53.17
57.06
57.41
58.08
58.44
T5
46.03
50.61
53.75
54.33
55.14
56.12
CG-Ra
50.43
53.14
55.37
55.39
56.04
56.75
CG-Nu
52.23
53.26
55.24
55.39
55,90
56.92
CG-RL
52.11
$2.84
54.33
55.24
55.47
56.52
T5
36.56
47.79
64.16
68.07
74,46
78.37
CG-Ra
59.22
68.38
77.34
79.09
81.26
81.15
CG-Nu
64.88
62.92
71.68
73.02
79.92
80.23
CG-RI. 63.65
61.28
68.92
74.67
77.24
79.40
In turn, we align each cluster in the sampled decompositon C with a sentence from the ground truth text description utilizing the Hungarian al- gorithm (Kuhn, 1955). The cost matrix for the algorithm is derived from the negative BLEU score between a tuple group and a sentence. Then, employ the fine-tuned T5 generator (Section 3.1) as the reward model to evaluate the sampled predicate decomposition. The reward is calculated based on Eq 1, which involves multiplying the likeli- hood of generating each sentence in the ground truth, conditioned on its corresponding aligned tu- ple group. Since REINFORCE is prone to high variance (Zaremba and Sutskever, 2015; Li et al., 2016), we propose a baseline log logp(V( (V)C, X), to the reward model. C denotes randomly generated predicate decomposition. We achieve this by ran- domly assigning each tuple to a cluster. !2
Table 2: Performance of models evaluated in scenario 2 (refer to Figure 3), ie. trained on CGONESHOT-K and tested on SEEN category. The top pop-perfrmmnn system is highlighted in bold, while the second best system for Ok- -ekpeereentt is underlined. CG-Ra, CG-Nu are short for CG-Random and CG-Numerical, respectively.
CGONESHOT
-2
-3
-5
-6
-7
TS
35,80
39.79
44.57
45.06
47,43
47.69
OK-per PARENT BLEU
CG-Ra
39.19
40.00
40.57
41.51
42.46
41.87
CG-Nu
35.31
36.83
37.66
38.21
39.83
39.13
CG-RL
36.06
37.94
39.86
40.73
42.38
42.84
T5
37.79
40.18
44.96
46.59
47.86
48 14
CG-Ra
41,90
43.41
46.19
48.25
48.30
48.12
CG-Nu
44.48
44.15
46.78
48.17
48.82
48.58
CG-RL
44.67
43.87
46.64
48.72
49.22
49.19
T5
34.34
41.19
53.42
55.22
57.46
64.3T
CG-Ra
52.41
55.33
63.86
64.76
66.55
70.15
CG-Nu
70,82
69.02
71.38
71.49
71.60
72.62
CG-RI.
67.90
68.01
68.24
71.16
69.25
72.50
Table 3: Performance of models evaluated in scenario 4 (refer to Figure 3), Le. trained on CGONESHOT-K and tested on UNSEEN category. CG-Ra, CG-Nu are short for CG-Random and CG-Numerical, respectively.
4 Experiments and Results
We compare our methods (CG-Numerica,, CG-NN, CG-RL) against fine-tuning 15 (Kale and Ras- togi, 2020b) using the benchmarks introduced in Section 2. Additionally, we include another base- line called CG-Random. determines the number of groups, ranging from I to X and assigns each predicate to one of the groups randomly. Compar- ing to CG-Random allows us to gain insights into the impact of our methods on predicate decomposi- tion and how it affects the text generation quality. To enhance readability given the large number of experiments conducted, we present a portion of the results that will be thoroughly discussed this section. Table 2 and 3 show models performance in testing scenario 2 and 4 (refer to Figure 3), re- spectively. For scenario 1 and 3, the results can be
found in Table 10 and 11 in Appendix F.
4.1 Case Study on Pre-trained LMs
In all four testing scenarios, we observe a signif- icant decline in both generation performance and faithfulness when T5 is trained on the splits with fewer input tuples. This suggests that pre-trained LMs are not well-suited for CG tasks. Additionally, we highlight the performance of T5 models trained using CGONESHOT-77, CGFULL-2 and -7 in Ta- ble 4. Comparing CGFULL-7 and CGONESHOT-7, we observe that limiting the number of training ex- amples for each predicate combination to one does not significantly hurt the performance. However, when comparing CGFULL-7 with CGFULL-2, a significant decrease in performance is observed. This drop occurs even though there is a smaller reduction in training data. These findings highlight the difficulty of our benchmark and emphasizes the
12If any clusters end up without tuples, we repeat the pro- cess until all clusters contain at least one uple.
15 "Mehta et al. (2022) found that simply increasing the model size does not effectively bridge the CG gap. Due to resource constraints, we specifically focus on T5-hase. Hyper- parameters for fine-tuning is shown in the released code.

---

T5
ONESHOT-7
FULL-2
FULL-7
is noticeable decrease in the OK-percent scores when tested using the seen set (Table 2). This is because predicate compositions in the in-domain test set are more likely to have been seen in the training set. Thus, our models tend to decompose input predicates into fewer groups. However, CG- random selects the number of groups randomly, resulting in a higher average group number (Ta- ble 16 in Appendix 1). This allows CG-random achieve slight gains in Oe-peneent but comes the cost of an 8- 8-point decrease BLEU compared to our approaches. Our study does not promote this unnecessary decompositions that may lead to unnatural text descriptions. These findings indi- cate that breaking down predicate compositions into smaller groups generally results in more faith- ful generations. However, when compared the random approach, learned decompositions produce texts that are closer human expressions.
#Example
2203
7712
18101
BLEU
Unseen Seen
58,98
52.54
65.01
PARENT
56.12
54.31
62.17
OK-per
78.37
43.87
78.27
BLEU
47.69
44.94
58.98
PARENT
48.14 14
46.03
56.12
OK-per
64.31
36.56
78.37
Table 4: Performance of T5 models trained on CGONESHOT-k (short as ONESHOTT) and CGFULL k (short FUI.I).. # Example represents the number of training examples in each training splits.
importance of models possessing CG abilities.
4.2 Results of Proposed Approaches
Our objective is to identify methods that excel across all of the four distinct testing scenarios in- troduced in Section 2.3.
Our approaches vs. T5 Our approaches demon- strate superior performance compared to T5 in all four testing scenarios (refer to Figure 3), measured across all three metrics, except for BLEU in the domain adaptation scenario (Table 3). This is due the presence of new predicates in the out-of- domain test set. Our approaches tend to decompose unseen predicate compositions into smaller groups for text generation, which deviates from human an- notations. On average, the number of sentences in the descriptions generated by humans, T5, and our approaches are 1.35, 1.4, and 2.0 respectively (see Table 16 in Appendix I). This divergence is penal- ized when evaluating with reference- based metrics like BLEU. However, our study encourages the decomposition behavior it allows for the genera- tion of faithful texts while maintaining a reasonable level of similarity to the Human-writted references. Another observation is that, across all four scenar- ios, when trained solely on examples with fewer input tuples, the performance advantage of our ap- proaches over T5 becomes more pronounced. For example, when trained with CGFULL-2 and tested the seen set (Table 10 in Appendix F), the best performed CG-based approach outperform TS points on BLEUU, 4.2 points on PARENT and 31.2 points on OK -e-peecent. These findings highlight the effectiveness of our approaches in enhancing the CG capability of vanilla preitrained LMs, partic- ularly when trained on a very limited number of examples with simple predicate compositions.
CG-Numerical vs. CG-RL In this section, we directly compare CG-Numerical to CG-RL since CG-RL is extension of CG-NN. The comparison between CG-NN and CG-RL can be found in Ap- pendix F. CG-RL exhibits better performance than CG-Numerical terms of BLEU score across all scenarios, particularly when evaluated on out-of- domain examples (Table and 11). The results for metrics PARENT and OK-percent are com- parable between the two approaches, except that CG-Numerical consistently outperforms CG-RL. in terms of OK percent when tested in unseen do- mains (Table 3). The reason for this is that CG-RL utilizes neural networks encode tokenized pred- icates, enabling some level of knowledge transfer when encountering out-of-domain predicates that consist of tokens seen in the in-domain predicates. However, CG-Numerical is unable to process out- of-domain predicates, resulting in a higher number of decomposed clusters. In fact, this number (2.4) is even higher than that of CG-random (1.8) (see Table 16 in Appendix I). Consequently, this con- tributes decrease BLEU.
In-depth Analysis and Discussion
5.1 Qualitative Evaluation
For the qualitative evaluation, randomly lected 30 inputs from the seen test set, each contain- ing five or more tuples. All models were trained using CGFUL.I-2 and tasked with generating text descriptions for these inputs. Building upon the metrics used in prior works (Mehta et al., 2022;
Our approaches vs. CG-random Our ap- proaches generally outperform CG-random, par- ticularly in terms of BLEU scores. However, there

---

CGFULL-2
Gr
Re
Ha
Om
Few-shot
0.5%
1%
5%
10%
T5
2.0
2.0
1.8
0.13
BART
38.29
40.77
50.30
54.23
CG- CG-Random
2.0
1.17
1.8
1.17
FT-KGPT"
BLEU
22.30
25.60
41,20
47.90
CG-RL
2.0
1.1
1.93
1.67
CBST
38.74
44,40
54,98
58.78
CG-RL
39.31
42.74
51.05
52.96
Table 3: Human evaluation with the metrics of Grammar, Repetition, Hallucination, and Omission. A higher score indicates better performance. Models are trained using CGFULL-K and tested on the SEEN set.
BART
33.08
31.95
38.98
40.03
CG-RL
37.31
34.75
40.12
41.20
BART
OK
29.50
31.56
44.75
48.63
CG-RL
47.19
48.44
50.94
52.35
Table 7: Models performance on few-shot settings. Sys- tems marked with are from previous work. To ensure fair comparison with CBST, for this set of experiments CG-RL is trained on BART. Note that the numbers in this table cannot directly compared to those in the previous tables due to the inclusion of additional do- mains in the WebNLG 2020 training set.
CGFULL
-3
4
-7
CG-Random
0.4087
0,3859
0.4355
0,3602
CG-Numerical
0.4487
0.5235
0.5703
0.60664
CG-RL
0.4755
0.6217
0.6251
0.6433
Human
0.7006
Table 6: Predicate decomposition performance evalu- ated using NMI. Models are tested under scenario 1, i.e. trained using CGFULL-K and tested on the SEEN set.
ing the Normalized Mutual Information (NMI) between the hypothesis and the reference.
Chen et al., 2020), we introduced four metrics: grammar (Gr), repetition (Re), hallucination (Ha), and omission (Om). Each system was rated on a scale of 0 (bad), 1 (moderate), and (good) for each metric. Detailed evaluation standards can be found in Appendix E. The results are presented in Table 5. CG-RL outperforms T5 and CG-Randde R--Randdom in hallucination and omission, showcasing its abil- ity generalize from 2-tuple examples multiple tuples with accurate and faithful descriptions. In contrast, the generated texts from CG-RL exhibit more repetition compared to T5. One prominent issue observed in CG-RL is the repetitive sentence structure, such as generating multiple sentences that start with the same entity. On the other hand, the majority of texts generated by T5 are single- sentence, indicating that it suffer less from the rep- etition problem. principle, we have the option to reintroduce conditional relationships berween the generated sentences to mitigate the repetition entities in the proposed CG-RL. Cherry-picked examples are shown in Table 8. Randomly-picked examples can be found in in Appendix A.
Additionally, require each model to generate predicate decompositions for every input example based on the number of clusters present in the lected reference. We evaluate the generated clusters by computing the NMI metric w.r.t. the reference. The results are shown in Table 6. When the models are trained using CGFULLL- and tested on the in- domain dataset, both of the proposed methods show superior performance compared to with CG-RL outperforming CG-Numerical. How- ever, while not falling too far behind, none of these models achieve the same level of correlation as hu- mans. Similar trends can be observed in the results for the other three testing scenarios, which can be found in Table 17 in Appendix I.
5.3 Test on Existing Few-shot Benchmarks
Chen et al. (2020) proposed few-shot splits for WebNLG 2020 by randomly selecting a certain portion of examples from the training set. We com- pare the performance of our best-performed sys- tem CG-RL with pre-trained BARG and two prior works, FT-KGPT (Chen et al., 2020) and CBST (Ke et al., 2022), on these splits. FT-KGPT fine- tunes a shoolledge-gooundee I Language Model that data
5.2 Predicate Decomposition Performance
The WebNLG 2017 v1.6 test sets include human- annotated decompositions. For each input example, we randomly select two annotated decompositions from its references, ensuring they have an equal number of predicate clusters. One of them is cho- sen as the reference, while the other becomes the hypothesis. Examples with only one predicate clus- ter as output are excluded, resulting in total of 160 testing examples. We evaluate the decomposition correlation among human annotators by measur-
has been pre-trained on 7M uulle--to-sentenn collected from Wikipedia pages. CBST is BART- based approach that is tuned using a self-training style, on 0.37M structured data without paired texts collected from GenWiki. The results in Table 7
I'N quantifies mutual information between clusters normalized a value between and 1. A score indicates no mutual information, while a score of signifies perfect correlation.
Addiiioonlll. we present the results of our other proposed approaches in Table 18 in Appendix I.

---

<SUB> acharya institute of technology <PRED> president <OBJ> "b.m. reddy":
<SUB> acharya institute of technology <PRED> city <OBJ> hangalore;
Input
<SB>> acharya institute of technology <PRED> established <OBJ> 2000;
<UBB> acharya institute of technology <PRED> country <ORI> "india":
<SUB> acharya institute of technology <PRED> campus <OBJ> "in soldevanahalli, acharya dr. sarvapalli
radhakrishnan road, hessarghatta main road, hangalore- 560090.";
<SUB> acharya institute of technology <PRED> aftiliation <OBJ> visvesvaraya techooolo technological university
T5
acharya institute of technology india has its campus soldevanaballi, acharya dr. sarvapalli radhakrishnan road, hessarghattu main road, bangalore- 560090.
CG-RL
acharya institute of technology india was established in 2000. it affiliated with visvesvaraya technological university. acharya institute technology's 's campus located soldevanahalli, acharya dr. sarvapalli radhakrishnan road, hessarghatta main road, bangalore 560090. bm reddy the president of acharya institute technology.
<SUB> turkey <PRED> leaderName <OBJ> ahmet davutoglu;
<SUB> turkey <PREDs capital <OBJ> ankara;
<SUB> turkkue-eeeeeee <PRED> largestCity <OBJ> istanbul;
Input
<UB>> atatürk monument (izmir) <PRED> material <OBJ> "bronze";
<SUB> turkey <PRED> currency <OBJ> turkish lira;
<UB>> atatürk monument (izmir) <PRED> inaugurationDate <OBJ> "1932-07-27":
<SUB> atatürk monument (izmir) <PRED> location <OBI> turkey;
"T5
the atatürkk monument, made of bronze, located in istanbul, turkey, where ahmet davutoglu is the leader. the atatürk monument (izmur) is located in turkey where ahmet davutoglu is the leader. the inauguration date of the atatürk monument (izmir). made of bronze. is 1932-07-27, ankara is the capital of turkey where the currency is the turkish lira. istanbul is the largest city in turkey.
"CGRRL"
<SUB> spain <PRED> language <OBl> spanish language;
<SUB> ajoblanco <PRED> region <OBI> andalusia;
Input
<SBB> andalusia <PRED> leaderName <OBI> susana díaz;
<SUB> ajoblance <PRED> country <OBJ> spain;
<SUB> spain <PRED> ethnicGroup <OBJ> spaniards
"TTs "C--L
ajoblanco is from andalusia where spaniards are an ethnic group.
ajoblanco originates from the country of spain where spaniards are ethnic group. susana diaz is the leader of andalusia where ajoblanco from. spanish is language spoken in spain.
Table 8:
Cherry-picked examples of input and
texts. Models are trained on CGFULL-2
show that CG-RL outperforms BART across all splits and metrics, except for BLEU the 10% split. FT-KGPT and CBST only reported BLEU scores. In the 1%, 5%, and 10% splits, CBST outperforms our approach. However, we acknowl- edge that leveraging task-specific pretraining and self-training--ere -baed tuning techniques on our text generator can potentially enhance the few-shot gen- eration performance.
of LLMs on DTG. Across benchmarks including WebNLG, both studies find that GPT-3 and Chat- GPT achieve lower BLEU scores compared fine- tuned smaller scale models. In addition, LLMs still face challenges in comprehending the seman- tic relationships between entities, and the generated text often includes hallucinations. Thus, we did not include LLMs in this study. Due to the limited com- putational resources, we focused our performance testing T5-base. However, it is important to eval- uate the models in different sizes, including LLMs. We anticipate that tasks with longer inputs/outputs. such as multi-docuument summarization, may derive even greater benefits from the proposed "solving CG through learning to decompose" idea. In the future, we aim to extend this idea to other tasks.
6 Limitations and Future Work
We have constructed our benchmark exclusively us- ing WebNLG 2017, it exhibits several favorable characteristics (see Appendix B). Nonetheless, would be advantageous to expand the benchmark to include data from a more diverse range of resources. Additionally, we recognize the importance of in- cluding multiple languages in the benchmark. The multilingual divisions introduced in WebNLG 2022 were not included in our benchmark due to the pres- ence of training data generated automatically using translation models, which resulted in noisy data. the future, we aim to expand our benchmark to include high-quality multilingual data resources. Recent work (Axelsson and Skantze, 2023; Yuan and Färber, 2023) explored the shro-shot ability
Acknowledgments
Lapata gratefully acknowledges the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1), Titov is supported by the Dutch National Science Foundation (NWO Vici VI.C.212.053). We thank Ido Dagan, An- dré Martins, Matthias Lindemann, Antonio Miceli- Barone, Parag Jain, Laura Beltzz-Belrachiii, Yao Fu, Xue Gong, for inspiring discussions.

---

References
compositional generalization: case study in open intent detection. arXiv preprint arXiv:2308. 13517.
Jacob Andreas. 2020. Good-enough compositional data augmentation. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7556-7566, Online. Association for Computational Linguistics.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Brrez-Beltrachiii. 2017. Creating training corpora for NLG micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers), pages 179-188, Vancouver, Canada. Association for Computational Linguistics.
Agnes Axelsson and Gabriel Skantze. 2023. Using large language models for zero-shot natural language generation from knowledge graphs. arXiv preprint arXiv:2307.073/2.
Peter W Glynn. 1990. Likelihood ratio gradient estima- tion for stochastic systems. Communications of the ACM, 33(10):75-84.
Eva Banik, Claire Gardent, and Eric Kow. 2013. The KBGen challenge. In Proceedings of the 14th Eit- ropean Workshop on Natural Language Generation, pages 94-97, Sofia, Bulgaria. Association for Com- putational Linguisties.
Junxian He, Jiatao Gu, Jiajun Shen, and Mare' Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. In International Conference on Learning Representations.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computa- tion. ArXiv, abs/1308 303.3
Peyman Heidari, Arash Einolghozati, Shashank Jain, Soumya Batra, Lee Callender. Ankit Arun, Shawn Mei, Sonal Gupta, Pinar Donmez, Vikas Bhardwaj. Anuj Kumar, and Michael White. 2021. Getting to production with few-shot natural language genera- tion models. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 66-76. Singapore and Online. Association for Computational Linguisties.
Wenhu Chen, Yu Su, Xifeng Yan, and William Yang Wang. 2020. KGPT:
wwwlledgegroundeed pre- training for data-to-text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 8635- 8648, Online. Association for Computational Lin- guistics.
Mihir Kale and Abhinav Rastogi. 2020a. Template guided text generation for task task-oriented dialogue. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6505-6520, Online, Association for Computa- tional Linguistics.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. 2018. Latent alignment and vari- ational attention. Advances neural information procexsing systems, 31.
Mihir Kale and Abhinav Rastogi. 2020b. Text-to-text pre-training for data-to-text tasks. In Proceedings of the 13th International Conference on Natural Lan- guage Generation, pages 97-102, Dublin, Ireland. Association for Computational Linguistics.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang. Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1884-4895, Florence, Italy. Asso- ciation for Computational Linguistics.
Pei Ke, Haozhe Ji, Zhenyu Yang, Yi Huang, Jun- lan Feng. Xiaoyan Zhu, and Minlic Huang. 2022. Curriculum based self aff-training makes better sew- learners for data-to-text generation. In Proceedings of the Thirty-Firv International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4178-4184 International Joint Conferences on Artificial Intelli- genee Organization. Main Track.
Ondrej Dušek, David M. Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural lan- guage generation. In Proceedings of the 12th Interna- tional Conference on Natural Language Generation, pages 421-426, Tokyo, Japan. Association for Com- putational Linguistics.
Yoon Kim, Carl Denton, Luong Hoang, and Alexan- der M. Rush. 2017. Structured attention networks. In International Conference on Learning Representa- tions.
Ondfej Dušek and Zdenčk Kasner, 2020. Evaluating semantic accuracy of data-to-text generation with nat- ural language inference. In Proceedings of the 13th International Conference on Natural Language Gen- eration, pages 131-137, Dublin, Ireland. Association for Computational Linguistics.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly, 2(1-2):83-97.
Ondfej Dušek. Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the state-of-the- thetthrttt of end-to end-to-end natural language generation: The e2e nlg challenge. Comput. Speech Lang.. 13
Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with ap- plication to the hiography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1203-1213, Austin, Texas, Association for Computational Linguistics.
Yihao Fang. Xianzhi Li, Stephen Thomas, and Xi- aodan Zhu. 2023. Chatgpt as data augmentation for

---

Jiwei Li. Will Monroe, Alan Ritter, Michel Galley, Jian- feng Gao, and Dan Jurafsky. 2016. Deep reinforce- ment learning for dialogue generation. arXiv preprint 1rXivv1606.01544
generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1173-1186, Online. As- sociation for Computational Linguisties.
Xintong Li. Symon Stevens- Guille, Aleksandre Maskha- rashvili, and Michael White. 2021. self-training for compositional neural NLG in task-oriented dialogue. In Proceedings of the 14th International Conference on Natural Language Generation, pages 87-102, Ab- erdeen, Scotland, UK. Association for Computational Linguistics.
Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task- oriented dialog. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 172-182, Online. Association for Computational Lin- guistics.
Percy Liang, Michael Jordan, and Dan Klein. 2009, Learning semantic comresoondnnces with less super- vision. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th In- ternational Joint Conference Processing of the AFNLP, pages 91-99, Suntec, Sin- gapore, Association for Computational Linguistics.
Linlu Qiu, Peter Shaw. Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022. Improving compositional generalization with latent structure and data augmentation. In Proceed- ings of the 2022 Conference of the North Ameri- can Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4341-4362, Seattle, United States. Association for Computational Linguistics.
Natural Language
François Mairesse, Milica Gašić, Filip Jurčfček. Simon Keizer, Blaise Thomson, Kai Yu, and Steve Young. 2010. Phrasc-based statistical language generation using graphical models and active learning. In Pro- ceedings of the 48th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1552- 1561, Uppsala, Sweden. Association for Computa- tional Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Adwait Ratnaparkhi. 2000. Trainable methods for sur- face natural language generation. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics.
Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, and Emmma Strubell. 2022. Improving compositional generalization with self tra-iig for data-to-text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4205- 421 2219, Dublin, Ireland. Association for Computational Linguistics.
Lena Reed, Shereen Oraby, and Marilyn Walker. 2018.
Can neural generators for dialogue learn sentence planning and discourse structuring? In Proceedings of the 11th International Conference on Natural Lan- guage Generation, pages 284-295, Tilburg Univer- sity. The Netherlands. Association for Computational Linguistics.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. bte-bby-teep: Separating planning from realization neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2267-2277, Minneapolis, Min- nesota. Association for Computational Linguistics.
Leonardo F. R. Ribeiro, Martin Schmitt, Hinrich Schütze, and Iryna Gurevych. 2021. Investigating pretrained language models for graph-to-text genera- tion. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 21 211-227, Online. Association for Computational Lin- guistics.
Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end- to-end generation. In Proceedings of the 18th An- nual SIGdial Meeting on Discourse and Dialogue, pages 201-206, Saarbrücken, Germany. Association for Computational Linguistics.
Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021. plan-then-generate Con- trolled data-to-text generation via planning. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 895 995-909, Punta Cana. Do- minican Republic. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Tsung- Hsien Wen, Milica Gašié, Nikola Mrkšić, Lina M. Lojas-Barahona Pai-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text

---

pages 120-129, San Diego, California. Association for Computational Linguistics.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Pei- Hao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural lan- guage generation for spoken dialogue systems. In Proceedings of the 2015 Conference Empirical Methods Natural Language Processing, pages 1711-1721, Lisbon, Portugal. Association for Com- putational Linguistics.
Ronald Williams. 1992. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Reinforcement learning. pages 5-32.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in cata-to-document generation. Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Xinnuo Xu, Guoyin Wang, Young-Bum Kim, and Sungjin Lec. 2021. AugNLG: Few-shot natural lan- guage generation using self-traines data augmenta- tion, Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume Long Papers), pages 1183-1195, Online. Association for Computa- tional Linguistics.
Shuzhou Yuan and Michael Färber. 2023. Evaluating generative models for graph-to-text generation. arXiv preprint arXiv:2307. 147122277......
Wojciech Zaremba and llya Sutskever. 2015. Rein- forcement learning neural turing machines. ArXiv, 105/1505.00521.

---

<SUB> agra uirport <PRED> locution <OBl> india;
<SUB> agra airport <PRED> runwayl.ength <ORI> 1818.0;
Input
<UB>> agra airport <PRED> operatingOrgamisttionn <OBI> indian air force;
<UB>> agra airport <PRED> elevation Seaeeieheaaaaeeeeeeelllaallalaa Leeelleevee__eeeeeeeeeeee_eeee ove__(nn_ metres) <OBJ> 167.94;
<SB>> agra airport <PRED> ICAO_ LeoatLootiion Identifier
Location__dentiffer <OBJ> "viag"
""T CG-RL
agra airport operated by the indian air force and has a runway length of 1818.0 metres. agra airport is located in india and its icao location identifier is viag. agra airport, operated by the indian air force, has a runway length of 1818.0. agra airport is elevated 167.94 metres above sea level.
<SUB> attica, indiana <PRED> isPartOf <OBI> united states;
<SUB> attica, indiana <PRED> areaTotal <OBI> 4.14 (square kilometres):
Input
<SUB> attica, indiana <PRED> AlovationAbove Twweeeeheeaaeeeeeee <OBJ> 166.0;
<SUB> attica, indiana <PRED> isPartOf_ <OBI> logan township, fountain county, indiana;
<SUB> attica, indiana <PRED> populationDensii <OB1> 783.1 (inhabitants per square kilometre);
"TT
attica, indiana, is part of logan township. fountain county, indiana, which has a population of 783,1 inhabiitntts per square kilometre.
CG-RL
attica, indiana is part of the united states, attica, indiana, is part of logan township, fountain county, indiana. attica, indiana has population density of 783.1 inhabitants per square kilometre. attica, indiana is located at 166.0 above sea level. the total area of attica, indiana is 4,14 square kilometres.
<SUB> denmark <PRED> leaderName <OBI> lars lokke rasmussen;
<SUB> curopean university association <PRED> hcadquarters <OBJ> brussels;
SUB> schnol of business and social sciences at the aarhus university <PRED> country <OBJ> denmark;
Input
SUB> denmark <PRED> leader'Title <OBJ> monarchy of denmark;
<UB> school of business and social sciences at the aarhus university <PRED> affiliation <OBI> european
university association;
<UB>> denmark <PRED> religion <OBJ> church of denmark;
"Ts
the school of business and social sciences the aarhus university is affiliated with the curopean university association which has its headquarters in brussels.
CG-RL
the school of business and social sciences the aarhus university is affiliated with the european university association and is located in denmark. the leader of denmark lars lokke rasmussen. demmark's 's leader title is the monarchy of denmark. the headquarters of the european university association are brussels. the religion of denmark church of denmark.
<SBB> buzz aldrin <PRED> birthPlace <OBI> glen ridge, new jersey;
<UB>> buzz aldrin <PRED> nationality <OBJ> united states;
SUB> buzz aldrin <PRED> ras_sclected_be
by_NASA <OBJ> 1963;
Input
SUB> buzz aldrin <PRED> was_a_crew _tt____cre_mmemeer___ of <OBI> apollo 11;
SUB> buzz aldrin <PRED> occupation <OBJ> fighter pilot:
<UB>> buzz aldrin <PRED> almaMater <OBJ> "massachusetts institute of technology. sc.d. 1963"
T5 CG-RL
buzz aldrin, who was member of the apollo 11 crew, was horn in glen ridge, new jersey. buzz aldrin, who was born glen ridge, new jersey, was selected by nasa 1963, buzz aldrin, us citizen, graduated from massachusetts institute technology, sc.d. 1963. buzz aldrin, fighter pilot, was a crew member of apollo 11.
<SUB> turkey <PRED> leaderTitle <OBJ> president of turkey:
<SUB> turkey <PRED> leader <OBI> ahmet davutoglu;
<SB>> atatürk monument (izmir) <PRED> designer <OBI> pietro canonica;
Input
<UR>> turkey
turkey <PRED> capital <OBJ> ankara;
SUB> atatürk monument (izmir) <PRED> material <OBJ> "bronze";
<UB> atatürk monument (izmir) <PRED> inaugurationDaate <OBJ> "1932-07-27";
<SB>> atatürk monument (izmir) <PRED> location <OBJ> turkey;
T5 "CG-RL""
the atatürk monument in turkey, made of bronze, was inaugurated on 27th july, 1932.
the ätatürk monumeni (izmir) is located in turkey where ahmet davutoglu is the leader. the atatüick monument in izmir, made of bronze. was inaugurated on 27th july 1932. ankara is the capital of turkey. the leader of turkey is known as the president of turkey.
<SUB> buzz aldrin <PRED> birthPlace <ORI> glen ridge, new jersey; <SUB> buzz aldrin <PRED> was ___ __s____eww__________________ >_t__mmeee________ of <OBI> apollo 11;
<SUB> buzz aldrin <PRED> status <OBJ> "retired";
Input
<SB>> buzz aldrin <PRED> nationality <OBJ> united states;
<SB>> buzz aldrin <PRED> almaMater <OBI> "massachusetts institute of technology, sc.d. 1963";
<UB>> apollo 11 <PRED> backup_pilas <OBJ> william anders;
<SUB> apollo 11 <PRED> operator <OBJ> nasa:
T5 ""GGRL""
buzz aldrin, who was crew memher of nasa's apollo 11, was horn in glen ridge, new jersey.
buzz aldrín was born in glen ridge. new jersey. buzz aldrín is from the united states. "buzz äldrín, who graduated from massachusetts institute of technology in 1963 with a doctorate in science, is now retired. buzz aldrin was crew member of apollo 1.. apollo 11 was operated by nasn. the backup pilot of apollo 11 was william anders.
Table 9: Randomly-picked examples of input and system-generater texts. Models are trained CGFULL-2 and tested with examples equal or more than tuples as input.

---

A Examples of system outputs
B Reasons of choosing WebNLG
We choose WebNLG because it offers examples with different input/output sizes. The training set covers multiple domains with diverse predefined predicates. It also includes out-of-domain test examples, allowing us to create challenging scenarios that involve both compositional generalization and cross- domain generalization. Moreover, several faithfulness evaluation metrics have been shown be highly correlated with human judgments on the WebNLG dataset.
Data Preprocess for Tuple-to-Sentence Alignment
Concretely, first preprocess the training examples by tokenizing the predicates in the input tuples" resolving coreferences in the output texts!? and splitting the texts into sentences
D Transformer-based Weight Predication
The dimension of the model is 128; the dimension of the feedforward is 256; number of heads is 4; number of layers is 2; dropout is 0.1. Other details will be shared with the code.
E Human Evaluation Standards
Grammar Correctness
2: The generated text is free of grammar mistakes.
1: The generated text may contain minor grammar mistakes, but they do not significantly affect understanding.
The generated text difficult to understand due to significant grammar mistakes.
Repetition
2: The generated text is natural and free of repetition.
1: The generated text may contain minor instances of repetition, but it is overall natural.
0: The generated text cxhibits significant repetition, such as mentioning input triples multiple times or using repetitive sentence structures (e.g., starting sentences with the same entity).
Hallucination
2: The generated text dose not contain any hallucinations.
1: Less than half of the generated text hallucinated.
0: More than half of the generated text is hallucinated.
Omission
The generated text fully covered all the input information.
1: The generated text misses tuple from the input.
0: The generated text misses more than one tuple from the input.
/httts::/gittubb.. cwm hww.///gggingggaeeeeeeeeaa....o
#hhttps://www nltk. http:///nttkooog////n...... tokenize, html.

---

F Extended Main Experiment Results
This section presents an overview of the experimental results. The performance of all models can be found in Table 10 and Table 11. The results for testing scenario 1 and 2 can be found in the left and right sections of Table 10, respectively. Similarly, the results for testing scenario 3 and 4 can be found in the left and right sections of Table 11, respectively.
CG-NN vs. CG-RL CG-NN and CG-RL show similar performance overall. CG-RL tends outperform CG-NN on faithfulness metrics when trained on examples with more input tuples, and on BLEU when trained on examples with fewer tuples. This trend is particularly evident when trained on CGONESHOT (right part of Table 10 and 11).
CGFULL
CGONESHOT
SEEN
-2
-2
-3
-7
52.54
58.70
62.63
63.01
44,94
49.43
54.39
56.86
58.49
58.98
CG-Random
OK-pereent PARENT BLEU
53.27
53.20
54.15
54.77
47.01
49.03
49.32
51.06
50.39
51.08
CG-Numerical
53.40
59.21
62.03
64.77
47.46
52.91
55.37
57.81
56.32
58.07
CG-NN
53.87
61.36
62.75
65.04
47.28
52.79
55.73
57.60
58.68
59.14
CG-RI
54.56
61.02
61.78
65.01
47.58
53.17
57.06
57.41
58.08
58.44
54.31
60.26
61.67
62.17
46.03
50.61
53.75
54.33
55.14
56.12
CG- Randan
56.95
$9.00
60.02
60.31
50.43
53.14
55.37
55.39
56.04
56.75
CG-Numerical
58.75
60.40
61.39
60.93
52.23
53.26
55.24
55.39
55,90
56.92
CG-NN
58.44
60.39
60.67
60.75
52.09
52.87
54.96
55.54
55.40
56.39
CG-RL
58.52
60.47
61.07
60.68
52.11
52.84
54.33
55.24
55.47
56.52
T3
43.87
68.07
76.32
78.27
36.56
47.79
64.16
68.07
74.46
78.37
CG Random
69. 69.00 00
78.58
81.67
82.39
59.22
68.38
77.34
79.09
81.26
81.15
CG-Numerical
74.97
79.61
79.92
79.30
64.88
62.92
71.68
73.02
79.92
80.23
CG-NN
75.08
76.93
78.78
78.78
65.09
63.54
71.27
74.25
76.11
79.20
CG-RL
74.25
76.00
79.81
78.99
63.65
61.28
68.92
74.67
77.24
79.40
Table 10: Models performance second best system for Ok-percent is underlined.
category.
The top-performing system is highlighted in bold, while the
FUFULL
CGONESHOT
UNSEEN
-2
-3
-7
40.83
47.14
48.64
50.01
35.80
39.79
44.57
45.06
47.43
47.69
CG- Random
OK-pen percent PARENT BLEU
43.17
44.20
42.24
44.00
39.19
40.00
40.57
41.51
42.46
41.87
CG-Numerical
38.78
40.46
39.39
40.52
35.31
36.83
37.66
38.21
39.83
39.13
CG-NN
39.78
43.99
44.81
47.69
35.65
37.42
39.81
40.99
43.23
42.95
CG-RL
40.42
43.61
41.81
46.21
36.06
37.94
39.86
40.73
42.38
42.84
42.92
49.9 49.46
49.60
51.75
37.79
40.18
44.96
46.59
47.86
14.14
CG- "G-Random
46.33
49.16
49.17
49.60
41.90
43.41
46.19
48.25
48.30
48.12
CG-Nur CG-Numerical
48.60
49.25
49.20
49.67
44.48
44.15
46.78
48.17
48.82
48.58
CG-NN
48.19
49.98
49.68
50.85
44.40
44.14
46.85
48.65
48.94
48.84
CG-RL
48.22
50.01
49.30
50.75
44.67
43.87
46.64
48.72
49.22
49.19
T5
43.43
56.90
68.07
67.68
34.34
41.19
53.42
55.22
57.46
64.31
CG- Random
64.09
67.56
73.40
76.32
52.41
55.33
63.86
64.76
66.55
70.
CG-Numeriaal
77.22
74.97
77.55
78.90
70.82
69.02
71.38
71.49
71.60
72.62
CG-NN
76.66
74.07
72.84
74.41
70.03
69.02
69.36
70,93
66.67
70.48
CG-RL
74.19
75.08
73.51
76.99
67.90
68.01
68.24
71.16
69.25
72.50
Table 11: Models performance on unseen category.

---

G T5-La Results
CGFULL
CGONESHOT
SEEN
-2
-3
-7
-2
-3
-4
-6
-7
T5
52.54
58.70
62.63
65.01
44,94
49.43
54.39
56.86
58.49
58.98
CG- CG-Random
OK- OK- pereeenet PARENT BLEU
53.27
53.20
54.15
54.77
47.01
49.03
49.32
51.06
50.39
51.08
CG-Numerical
53.40
59.21
62.03
64.77
47.46
52.91
55.37
57.81
56.32
58.07
CG-NN
53.87
61.36
62.75
65.04
47.28
52.79
55.73
57.60
58.68
59.14
CG-RL
54,56
61.02
61.78
65.01
47.58
53.17
57.06
57.41
58.08
58.44
T5
54.31
60.26
61.67
62.17
46.03
50.61
53.75
54.33
55.14
56.12
CG-Random
56.95
59.00
60.02
60.31
50.43
53.14
55.37
55.39
56.04
56.75
CG-Numerical
58.75
60.40
61.39
60.93
52.23
53.26
55.24
55,39
55.90
56.92
CG-NN
58.44
60.39
60.67
60.75
$2.09
52.87
54.96
55.54
55.40
56.39
CG-RL
58.52
60.47
60.68
52.11
52.84
54.33
55,47
$6.52
43.87
68.07
78.27
36.56
47.79
64.16
68.07
74,46
78.37
CG-Random
69.00
78.58
81.67
82.39
59.22
68,38
77.34
79.09
81.26
81.15
CG-Numerical
74.97
79.61
79.92
79.30
64.88
62,92
71.68
73.02
79.92
80.23
CG-NN
75.08
76.93
78.78
78.78
65.09
63.54
71.27
74.25
76.
79.20
CG-RL
74.25
76.00
79.81
78.99
63.65
61.28
68.92
74.67
77.24
79.40
Table 12: T5-Large based models performance while the second best system for Ok-percent underlined.
category. The top rop-performing system is highlighted in bold.
CGFULL
CGONESHOT
UNSEEN
-2
-2
-7
T3
40.83
47.14
48.64
50.01
35.80
39.79 79.79
44.57
45.06
47.43
47.69
CG-Random
OK-percent PARENT BLEU
43.17
44.20
42.24
44.00
39.19
40.00
40.57
41.51
42.46
41.87
CG-Numerical
38.78
40.46
39.39
40.52
35.31
36.83
37.66
38.21
39.83
39.13
CG-NN
39.78
43.99
44.81
47.69
35.65
37.42
39.81
40.99
43.23
42.95
CG-RI.
40.42
43.61
41.81
46.21
36.06
37.94
39.86
40.73
42.38
42.84
42.92
49.46
49.60
51.75
37.79
40.18
44.96
46.59
47.86
48.14
CG- Random
46.33
49.16
49.17
49.60
41.90
43.41
46.19
48.25
48.30
48.12
CG-Numerical
48.60
49.25
49.20
49.67
44.48
44.15
46.78
48.17
48,82
48.58
CG-NN
48.19
49.98
49.68
50.85
44.40
44.14
46.85
48.65
48.94
48.84
CG-RL
48.22
50.01
49.30
50.75
44.67
43.87
46.64
48.72
49.22
49.19
T5
43.43
56,90
68.07
67.68
34.34
41.19
53.42
55.22
57.46
64.31
CG- Random
64.09
67.56
73.40
76.32
52.41
55.33
63.86
64.76
66.55
70.15
CG-Numerical
77.22
74.97
77.55
78.90
70.82
69.02
71.38
71.49
71.60
72.62
CG-NN
76.66
74.07
72.84
74.41
70.03
69.02
69.36
70.93
66.67
70.48
CG-RI.
74.19
75.08
73.51
76.99
67.90
68.01
68.24
71.16
69.25
72.50
Table T5- T5-Large based models performance unseen category.

---

H T5-Small Results
CGFULL
CGONESHOT
SEEN
-2
-3
-7
52.54
58.70
62.63
65.01
44.94
49.43
54.39
56.86
58.49
$8.98
CG- Random
OK- -eeppeeent PArENT BLEU
53.27
53.20
54.15
54.77
47.01
49.03
49.32
51.06
50.39
51.08
CG-Numerical
53.40
59.21
62.03
64.77
47.46
52.91
55.37
57.81
56.32
$8.07
CG-NN
53.87
61.36
65.04
47.28
52.79
55.73
57.60
58.68
59.14
CG-RL
54.56
61.02
61.78
65.01
47.58
53.17
57.06
57.41
58.08
58.44
T5
54.31
60.26
61.67
62.17
46.03
50.61
53.75
54.33
55.14
56.12
CG-R Random
56.95
59.00
60.02
60.31
50.43
53.14
55.37
55.39
56.04
56.75
No-Numeriial
58.75
60.40
61.39
60.93
52.23
53.26
55.24
55.39
55.90
56,92
CG-NN
58.44
60.39
60.67
60.75
$2.09
52.87
54.96
55.54
55.40
56.39
CG-RL
58,52
60.47
61.07
60.68
52.11
52.84
54.33
55.24
55.47
56.52
T5
43.87
68.07
76.52
78.27
36.56
47.79
64.16
68.07
74.46
78.37
CG- Random
69.00
78.58
81.67
82.39
59,22
68.38
77.34
79.09
81.26
81.15
CG-Numerical
74.97
79.61
79.92
79.30
64.88
62.92
71.68
73.02
79.92
80.23
CG-NN
75.08
76.93
78.76 78
78.78
65.09
63.54
71.27
74,25
76.
79.20
CG-RL
74.25
76.00
79.81
78.99
63,65
61.28
68.92
74.67
77.24
79.40
Table TS-Small based models performance on seen eategory. while the second best system for Ok-percent underlined.
The top-performing system is highlighted bold,
CGFULL
CGONESHOT
UNSEEN
-2
-3
-7
-2
-3
-4
-5
-6
-7
40.83
47.14
48.64
50.01
35,80
39.79
44.57
45.06
47.43
47.69
CG- Random
OK OK- -eeperennt PARENT BLEU
43.17
44.20
42.24
44.00
39.19
40.00
40.57
41.51
42.46
41.87
CG-Numerical
38.78
40.46
39.39
40.52
35.31
36.83
37.66
38.21
39.83
39.13
CG-NN
39.78
43.99
44.81
47.69
35.65
37.42
39.81
40.99
43.23
42.95
CG-RL
40.42
43.61
41.81
46.21
36.06
37.94
39.86
40.73
42.38
42.84
T5
42.92
49.46
49.60
$1.75
37.79
40.18
44.96
46.59
47.86
48.14
CG-Random
46.33
49.16
49.17
49.60
41,90
43.41
46.19
48.25
48.30
48.
CG-Numete No-merrrica
48.60
49.25
49.20
49.67
44,48
44.15
46.78
48. 17
48.82
48.58
CG-NN
48.19
49.98
49.68
50.85
44,40
44.14
46.85
48.65
48.94
48.84
CG-RL
48. 22
50.01
49. 30
50.75
44.67
43.87
46.64
48.72
49.22
49.19
T3
43.43
36.90
68.07
67.68
34.34
41.19
33.42
53,22
57.46
64.31
CG-Random
64.09
67.56
73.40
76.32
52.41
55.33
63.86
64.76
66.55
70.15
CG-Numerical
77.22
74.97
77.55
78.90
70.82
69.02
71.38
71.49
71.60
72.62
CG-NN
76.66
74.07
72.84
74.41
70.03
69.02
69.36
70.93
66.67
70.48
CG-RI.
74.19
75.08
73.51
76.99
67.90
68.01
68.24
71.16
69.25
72.50
Table 15: 75-Small based models performance on unseen category.

---

I
Extra Experiment Results
CGFULL
CGONESHOT
Avg #Clusters
-2
-3
-7
-2
-3
-5
-6
T5
1.07
1.19
1.46
1.38
1,05
1.09
1.26
1.4
1.44
1.42
CG- Random
UNSEEN SEEN
2.04
2.03
2.00
2.04
1.98
1.98
2.02
2.03
2.00
2.00
CG-Numerical
2.05
1.59
1.25
1.13
2.01
1.53
1.34
1.27
1.23
1.22
CG-NN
2.00
1.36
1.11
1.06
2.01
1.53
1.3
1.28
1.09
1.08
CG-RL
1.91
1.38
1.23
1.07
1.99
1.47
1.22
1.28
1.16
1.13
T5
1.02
1.15
1.40
1.32
1.01
1.02
1.26
1.29
1.37
1.4
CG- Random
1.88
1.85
1.87
1.88
1.90
1.89
1.82
1.87
1.82
CG-Numerical
2.61
2.46
2.43
2.41
2.62
2.47
2.43
2.43
2.42
2.42
CG-NN
2.44
2,00
1.56
1.45
2.37
2.00
2.05
1.79
1.74
CG-RL
2.34
2,04
1.63
2.24
2.09
Table 16: The average number of groups each model generates for test examples. The average number of sentences for human written references are 1.37
1.35 seen and unseen test set respectively.
CGFULL
CGONESHOT
NMI
-2
4
-7
-2
-5
-6
-7
CG-Random
40.87
38.59
43.55
36,02
39.43
38.43
40.74
39.06
37.20
35.76
CG-Numerical
UNSEEN SEEN
44.87
52.35
57.03
60.64
46.72
55.71
59.79
62.60
61.81
60.65
CG-NN
47.08
58.87
65.10
66.24
48.03
58.97
65.49
64.40
66.18
66.18
CG-RI
47.55
62.17
62.51
64.33
48.08
56.82
65.21
63.17
64.56
63.39
CG- CG-Random
44.18
39,43
40.59
37.01
41.20
39.71
37.21
39.11
38,55
38.34
CG CG-Numerical
41.78
39.04
41.17
41.83
38.20
45.10
41.19
47.76
42.57
41.63
CG-NN
50.95
53.17
55.30
52.60
52.63
52.26
52.40
54.59
55.57
52.03
CG-RI.
47.77
49.66
47.20
57.28
52.73
53.79
53.57
54.01
49.65
Table 17: Predicate decomposition performance of all models evaluated using NMI. The NMI among human annotated references are 70.06 and 67.31 for seen and unseen test set respectively. Since we can not control the amount of sentences the vanilla T5 generates, we discard the T5 for this experiment.

---

Few-shot
0.5%
1%
5%
10% %
BART
38.29
40.77
50.30
54,23
FT-KGPT"
BLEU
22.30
25.60
41.20
47.90
CBST
38.74
44.40
54.98
58.78
CG- Random
39,93
42.50
48.31
49.50
"G-Numerical
39.43
43.37
51.12
53.96
CG-NN
39.63
43.96
50.9
53.95
CG-RL
39.31
42.74
51.05
52.96
BART
33.08
31.95
38.98
40.03
CG-Random
OK-percent PARENT
36.39
33.76
38.80
39.69
CG-Numerical
37.50
34.68
40.22
40.84
CG-NN
37.48
34.59
39.87
40.93
CG-RL
37.31
34.75
40.12
41.20
BART
29.50
31.56
44.75
48.63
CG-Random
41.44
43.56
52.50
51.94
CG-Numerical
47.63
45.31
50.00
51.50
CG-NN
47.31
44.37
50.06
51.25
CG-RL
47.19
48.44
50.94
Table 18: Models performance few-shot settings. Systems marked with * are from previous work. To ensure fair comparison with CBST, for this set of experiments all CG- models are trained on BART. Note that the numbers in this table cannot be directly compared to those in the previous tables due the inclusion of additional domains in the WebNLG 2020 training set.


Reviews:
[{"review_id": "ORNeEkJLgm", "reviewer": "Reviewer_1UpM", "summary": "", "questions": "Question A: Given the distinct sentence generation process in CG methods, repetition becomes an evident concern. When entities recur across sentences, the use of pronouns is minimized, leading to redundancy. This issue might escalate when crafting lengthier content, like full-length documents. How do the authors propose to tackle this repetition challenge?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's suggestions.\n\n## RE-1: Limited to a single model size\n\nWe will incorporate an additional subsection within Section 5 (In-depth Analysis and Discussion), dedicated to investigating how our predicate decomposition approach  interacts with model size (T5-small, T5-base, T5-Large). We are including partial outcomes from **T5-Large** in this reply. Due to the time constraints of the rebuttal stage, we were unable to carry out all experiments. The concluded outcomes **exhibit a very similar trend** to those of the T5-base model. Complete experimental findings for both T5-Small and T5-Large will be available in the final edition of this paper.  *(Note: OK-percent Omi stands for the amount of omissions observed in the system generations. The lower OK-percent Omi the better)*\n\n* Performance of T5-Large-based models evaluated in scenario 2 (refer to Figure 3, corresponding to the Table 2 in the paper), i.e. trained on CGOneShot-k and tested on SEEN category. \n|CGOneShot-Seen|Models| -2 | -3 | -4 |\n| -------- | -------- | ------- | ------- | -------- |\n|BLEU|T5-Large | 46.55 | 50.12 | 55.82 |\n||CG-Random | 46.89 | 48.27 | 50.15 |\n||CG-Numerical | 47.73 | 53.11 | 56.09 |\n||CG-NN | 47.59 | 52.73 | 56.73 |\n||CG-RL | running | running | running |\n|PARENT|T5-Large | 47.95 | 49.94 | 54.44 |\n||CG-Random | 51.44 | 51.95 | 55.16 |\n||CG-Numerical | 53.32 | 52.97 | 55.53 |\n||CG-NN | 53.25 | 52.72 | 55.44 |\n||CG-RL | running | running | running |\n|OK-percent|T5-Large | 38.41 | 50.36 | 66.43 |\n||CG-Random | 59.73 | 67.66 | 78.17 |\n||CG-Numerical | 67.97 | 65.60 | 74.87 |\n||CG-NN | 67.04 | 65.71 | 73.43 |\n||CG-RL | running | running | running |\n|OK-percent (Omi)|T5-Large | 53.14 | 42.53 | 29.25 |\n||CG-Random | 33.57 | 25.33 | 16.89 |\n||CG-Numerical | 26.16 | 27.81 | 27.81 |\n||CG-NN | 27.19 | 27.60 | 22.66 |\n||CG-RL | running | running | running |\n\n* Performance of T5-Large-based models evaluated in scenario 4 (refer to Figure 3, corresponding to the Table 3 in the paper), i.e. trained on CGOneShot-k and tested on UNSEEN category. \n|CGOneShot-Unseen|Models| -2 | -3 | -4 |\n| -------- | -------- | ------- | ------- | -------- |\n|BLEU|T5-Large | 38.05 | 41.44 | 46.75 |\n||CG-Random | 41.13 | 41.96 | 42.55 |\n||CG-Numerical | 36.88 | 38.66 | 39.61 |\n||CG-NN | 36.97 | 38.82 | 41.96 |\n||CG-RL | running | running | running |\n|PARENT|T5-Large | 41.18 | 43.12| 46.72 |\n||CG-Random | 44.26 | 45.26 | 46.54 |\n||CG-Numerical | 45.77 | 45.73 | 47.07 |\n||CG-NN | 45.76 | 45.61 | 47.54 |\n||CG-RL | running | running | running |\n|OK-percent|T5-Large | 36.48 | 43.32 | 56.45 |\n||CG-Random | 52.97 | 57.80 | 66.22 |\n||CG-Numerical | 66.78 | 63.97 | 70.59 |\n||CG-NN | 66.33 | 63.52 | 70.15 |\n||CG-RL | running | running | running |\n|OK-percent (Omi)|T5-Large | 46.91 | 41.75 | 31.99 |\n||CG-Random | 32.44 | 27.83 | 22.78 |\n||CG-Numerical | 19.87| 24.24 | 19.30 |\n||CG-NN | 19.87 | 23.57 | 19.64 |\n||CG-RL | running | running | running |\n\n* Performance of T5-Large-based models evaluated in scenario 1 (refer to Figure 3, corresponding to the CGFull section in Table 10 in the paper), i.e. trained on CGFull-k and tested on SEEN category. \n|CGFull-Seen|Models| -2 | -3 | -4 |\n| -------- | -------- | ------- | ------- | -------- |\n|BLEU|T5-Large | 46.55 | 50.12 | 55.82 |\n||CG-Random | 46.89 | 48.27 | 50.15 |\n||CG-Numerical | 47.73 | 53.11 | 56.09 |\n||CG-NN | 47.59 | 52.73 | 56.73 |\n||CG-RL | running | running | running |\n|PARENT|T5-Large | 47.95 | 49.94| 54.44 |\n||CG-Random | 51.44 | 51.95 | 55.16 |\n||CG-Numerical | 53.32 | 52.97 | 55.53 |\n||CG-NN | 53.25 | 52.72 | 55.44 |\n||CG-RL | running | running | running |\n|OK-percent|T5-Large | 38.41 | 50.36 | 66.43 |\n||CG-Random | 59.73 | 67.66 | 78.17 |\n||CG-Numerical | 67.97 | 65.6 | 74.87 |\n||CG-NN | 67.04 | 65.71 | 73.43 |\n||CG-RL | running | running | running |\n|OK-percent (Omi)|T5-Large | 53.14 | 42.53 | 29.25 |\n||CG-Random | 33.57 | 25.33 | 16.89 |\n||CG-Numerical | 26.16 | 27.81 | 21.63 |\n||CG-NN | 27.19 | 27.6 | 22.66 |\n||CG-RL | running | running | running |\n\n* Performance of T5-Large-based models evaluated in scenario 3 (refer to Figure 3, corresponding to the CGFull section in Table 11 in the paper), i.e. trained on CGFull-k and tested on UNSEEN category.\n|CGFull-Unseen|Models| -2 | -3 | -4 |\n| -------- | -------- | ------- | ------- | -------- |\n|BLEU|T5-Large | 38.05 | 41.44 | 46.75 |\n||CG-Random | 41.13 | 41.96 | 42.55 |\n||CG-Numerical | 36.88 | 38.66 | 39.61 |\n||CG-NN | 36.97 | 38.82 | 41.96 |\n||CG-RL | running | running | running |\n|PARENT|T5-Large | 41.18 | 43.12 | 46.72 |\n||CG-Random | 44.26 | 45.26 | 46.54 |\n||CG-Numerical | 45.77 | 45.73 | 47.07 |\n||CG-NN | 45.76 | 45.61 | 47.54 |\n||CG-RL | running | running | running |\n|OK-percent|T5-Large | 36.48 | 43.32 | 56.45 |\n||CG-Random | 52.97 | 57.8 | 66.22  |\n||CG-Numerical | 66.78 | 63.97 | 70.59 |\n||CG-NN | 66.33 | 63.52 | 70.15 |\n||CG-RL | running | running | running |\n|OK-percent (Omi)|T5-Large | 46.91 | 41.75 | 31.99 |\n||CG-Random | 32.44 | 27.83 | 22.78 |\n||CG-Numerical | 19.87 | 24.24 | 19.3 |\n||CG-NN | 19.87 | 23.57 | 19.64 |\n||CG-RL | running | running | running |\n\n\n## RE-2: 4-way OK-percent metric results\nIndeed, the proposed methods receive low omission scores across all testing scenarios when employing the OK-percent metric. We will include the full 4-way OK-percent results in the final version of the paper. We only report the omission scores here in the response.\n\n* Evaluation scenario 2 \n|CGOneShot-Seen|Models| -2 | -3 | -4 | -5 | -6 | -7 |\n| -------- | -------- | ------- | ------- | -------- | -------- | -------- | -------- |\n|OK-percent (Omi)|T5-base | 50.77 | 44.59 | 32.03 | 28.63 | 22.45 | 17.51 |\n||CG-Random | 45.62 | 35.32 | 25.75 | 21.22 | 18.64 | 15.96 |\n||CG-Numerical | 44.9 | 33.88 | 23.48 | 20.8 | 16.27 | 15.04 |\n||CG-NN | 45.42 | 33.68 | 23.58 | 19.67 | 16.58 | 15.55 |\n||CG-RL | 46.14 | 33.57 | 23.17 | 19.05 | 16.89 | 15.24 |\n\n* Evaluation scenario 4 \n|CGOneShot-Unseen|Models| -2 | -3 | -4 | -5 | -6 | -7 |\n| -------- | -------- | ------- | ------- | -------- | -------- | -------- | -------- |\n|OK-percent (Omi)|T5-base | 50.28 | 48.04 | 37.82 | 37.49 | 33.33 | 28.28 |\n||CG-Random | 36.36 | 31.65 | 24.8 | 24.47 | 25.7 | 21.77 |\n||CG-Numerical | 20.76 | 20.88 | 18.97 | 21.1 | 19.53 | 19.3 |\n||CG-NN | 21.44 | 20.31 | 20.76 | 22.56 | 25.59 | 22.67 |\n||CG-RL | 44.11 | 39.06 | 28.84 | 30.75 | 27.5 | 24.58 |\n\n* Evaluation scenario 1 \n|CGFull-Seen|Models| -2 | -3 | -4 | -7 |\n| -------- | -------- | ------- | ------- | -------- | -------- |\n|OK-percent (Omi)|T5-base | 50.15 | 27.7 | 18.13 | 16.07 |\n||CG-Random | 26.78 | 50.77 | 11.12 | 12.26 |\n||CG-Numerical | 39.96 | 16.27 | 14.42 | 15.04 |\n||CG-NN | 18.95 | 17.92 | 16.17 | 15.55 |\n||CG-RL | 20.19 | 19.57 | 14.73 | 15.55 |\n\n* Evaluation scenario 3 \n|CGFull-Unseen|Models| -2 | -3 | -4 | -7 |\n| -------- | -------- | ------- | ------- | -------- | -------- |\n|OK-percent (Omi)|T5-base | 52.15 | 36.25 | 16.07 | 27.38|\n||CG-Random | 40.07 | 26.6 | 22.11 | 21.44 |\n||CG-Numerical | 39.96 | 24.92 | 22.22 | 22.67 |\n||CG-NN | 36.03 | 25.36 | 21.21 | 21.89 |\n||CG-RL | 37.26 | 26.26 | 21.55 | 21.89 |\n\n\n## RE-3: The repetition challenge\nAt present, sentence generation for each predicate group operates autonomously. Consequently, the generation of a new sentence occurs without any awareness of the previously generated sentence, leading to repeated entities. In principle, we can **reintroduce conditional dependencies** between generated sentences. Nonetheless, this approach necessitates careful and thorough experimentation. \nIn addition, there exists rapid solutions. For instance, a rule-based system could be developed to identify recurring entities and then assign a likelihood for substitution with references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The evaluations are carefully structured, grounded in predicate configurations. This approach establishes a solid benchmark that future investigations can reference and build upon.\n* The paper introduces multiple predictors for predicate decomposition: numerical-based, neural network-driven, and those enhanced by REINFORCE. A random predictor also serves as a baseline in the comparison.\n* The paper provides a comprehensive analysis of the experiment results, showing intricacies and broader implications of the CG methods.", "reasons_to_reject": "* The study is limited to a single model size, leaving open questions about the method's efficacy in bridging compositional generalization gaps in larger models.\n* While the CG method seems to exhibit potential in minimizing hallucinations or omissions for longer text, it appears prone to repetition issues, even in WebNLG content that spans an average of 1 to 2 sentences. Effective solutions to this concern aren't elaborated upon.\n* The human evaluation (as seen in Table 5) implies that the CG method notably diminishes omissions, given the low scores associated with T5 outputs. This improvement would be more evident if reflected in the OK-percent metric. An inclusion of the 4-way OK-percent metric results would be a valuable addition.\n", "questions_to_authors": "Question A: Given the distinct sentence generation process in CG methods, repetition becomes an evident concern. When entities recur across sentences, the use of pronouns is minimized, leading to redundancy. This issue might escalate when crafting lengthier content, like full-length documents. How do the authors propose to tackle this repetition challenge?", "paper_topic_and_main_contributions": "The paper delves into the compositional generalization challenges faced by current Data-to-Text models, especially concerning predicate structures. The evaluations revolve around predicate structure-controlled instances, sourced from the WebNLG benchmark. The authors present a methodology that begins with sentence planning -- clustering input predicates where each cluster maps to a subsequent sentence -- followed by the text generation. Various variants of this method are implemented, tested, and contrasted with relevant baselines. The experimental findings indicate that breaking down the predicate structure into smaller chunks heightens the faithfulness of the output and bolsters the model's domain adaptability.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2G7q78I8Q", "reviewer": "Reviewer_GZ7d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers comments.\n\n## RE-1: Limited to a single model size, T5-base\nPlease refer to our response **RE-1: Limited to a single model size** to Reviewer-1.\n\n## RE-2: Incomplete work without LLMs\nFirst, it is essential to emphasize that conducting experiments on models of smaller scales still remains valuable. While LLMs are powerful models, they are not universally suitable for all application scenarios, such as [mobile devices](https://www.computerweekly.com/news/366545122/Qualcomm-Meta-collaborate-to-enable-on-device-AI-applications-using-Llama-2) with constrained GPU resources. \n\nMeanwhile, we are not aware of any studies corroborating claims that Data-to-Text tasks like WebNLG are easy for LLMs, and that compositional generalization poses no challenges for them. Recent work (published in July and August, yet to undergo peer review) explores the zero-shot capabilities of LLMs on Data-to-Text tasks: https://arxiv.org/pdf/2307.07312.pdf, https://arxiv.org/pdf/2307.14712.pdf. Across benchmarks including WebNLG, both studies find that GPT-3 and ChatGPT achieve lower BLEU scores compared to fine-tuned smaller scale models. In addition, large-scale LLMs still face challenges in comprehending the semantic relationships between entities. Furthermore, the generated text often includes instances of hallucinations or irrelevant information.\n\nWe will discuss both studies in the future work section. \n\n## RE-3: Distillation based approaches\n\nData Augmentation serves as one approach to tackle compositional generalization problems, but it is not the sole solution. Other methodologies, such as input decomposition, also warrant investigation. \n\nKnowledge distillation, such as Data Augmentation, aims to enhance a model's proficiency in text generation by enlarging the constrained pool of human-annotated training data. Conversely, our approach takes an alternative perspective, focusing on the efficient utilization of the limited yet high-quality annotated resources available.\n\nIn contrast to knowledge distillation, our methods necessitate minor adjustments in data preparation, model architecture, and training mechanisms. However, in comparison to our approaches, knowledge distillation also exhibits limitations:\n\n* Generating possible predicate permutations is a laborious process. Within the WebNLG 2017 dataset, each domain typically encompasses an average of 20 distinct predicates. With a maximum of 7 tuples per input, the potential permutations of predicates within each domain, accounting for predicate order, can be calculated as 20 + 20^2 + 20^3 + 20^4 + 20^5 + 20^6 + 20^7. It's important to note that both generating augmented data and fine-tuning a smaller model on this extensive dataset consumes a substantial amount of time.\n\n* Data Augmentation assumes that LLMs possess the capability to produce silver-standard data that resembles human writing and remains faithful to the input. However, as previously noted, recent investigations have not provided substantial evidence to support this particular capability.\n\n* In contrast to the data augmentation strategy, our suggested input decomposition approach is more intuitive and closely aligned with human behaviour. In tasks like Data-to-Text generation, humans aren't required to encounter every conceivable combination of predicates and entities. Through simple demonstrations, when presented with a set of tuples, humans can efficiently break them down into groups and cohesively describe them within multiple sentences.\n\nWe plan to incorporate a discussion that addresses the distinctions between our approach and knowledge distillation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Compositional generalization is an important area of data-to-text generation, having a targeted benchmark to assess this capability is great. It allows testing compositional generalization, few-shot learning, domain adaptation and their combinations.\n", "reasons_to_reject": "As already mentioned in the limitations section, the paper uses only T5-base, which is a fairly small model.\nFor very large models, WebNLG is a fairly easy task, and they will probably already be excellent at compositional generalization. Given LLMs are now SoTA and ubiquitous, the paper seems incomplete without evaluating them.\n\nIn the presence of LLMs and distillation based approaches, its not clear if the decomposition based approach offers additional value. For example, you could create all/many combinations of input predicates, use an LLM to get outputs and then train a smaller model on this silver data. Compositional generalization would be much less of an issue, since the small model has been trained on a much larger set of combinations.", "questions_to_authors": "", "paper_topic_and_main_contributions": "A benchmark to assess the compositional generalization of data-to-text systems is proposed. Authors also present a technique that decomposes unseen structured data combinations into smaller, seen ones. The decomposition idea also used to develop evaluation criteria.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5yi7mhUAcN", "reviewer": "Reviewer_Yzid", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's positive feedback. We agree with the reviewer that a notable advantage of our approaches lies in its avoidance of the need to acquire extensive datasets. Instead, it centers on the efficient utilization of available resources, which are often modest in scale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They present a well written and clear paper that shows clearly how their cluster-based compositional generalization method was developed. The intuition to Decompose unseen predicate combinations into smaller groups, ensure that a combination in each group has been seen during training, then generate a sentence from each group individually first, followed by an aggregation, is quite brilliant!\n\nAdditionally, it is always advantageous to have improved performance not dependent on more training examples, and their cluster-based generation method requires small training sets, with no need for further labeled or unlabeled data.\n\nFinally, their approach performs very well when evaluated on T5 baselines, and the method of evaluation as well as the discussion of results is clear and relevant.\n", "reasons_to_reject": "I find no reason to reject this paper.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents work in the area of data-to-text generation, which involves the generation of descriptions from tuples (comprising arguments and predicates) that should contain all Information required for accurate text generation. The problem space is the issue of unseen predicate combinations, which requires a model to generalize. Specifically, the authors aim to address compositional generalization, where data-to-text systems produce unfaithful descriptions when faced with unseen predicate combinations.\n\nTheir work makes two contributions: ((1) an evaluation method for assessing how existing data-to-text generation algorithms handle compositional generalization; and (2) a novel approach to handling compositional generalization by clustering predicates into groups. Their evaluation method involves training with fewer input tuples than are used during testing; reducing to a single input example for each predicate combination; as well as testing on unseen domains. Under their test environment, they find that SoTA pre-trained language models handle compositional generalization poorly.\n\nThey then present an approach to address compositional generalization by clustering predicates into groups. The groups are composed of decomposed unfamiliar predicate combinations that are clustered based on graph weights learned during training. Text is then generated, sentence by sentence, individually for each group of predicates at a time, and then combined to form the final output.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Caption for Algorithm 1, the spelling of \"decomposition\"\n"}, "extra_text": {}}, {"review_id": "aZw9IV9G6E", "reviewer": "Reviewer_7idM", "summary": "", "questions": "Question A: There are sorted and grouped predicates in the training set of WebNLG. Have you considered using them in training?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer's efforts.\n\n## RE-1: CG-Random is enough\n\nIt is important to highlight that our paper introduces **four** distinct testing scenarios. Our objective is to identify methods that excel across **all of them**. We assume a method is performant if it can generate text which resembles human writing (as evaluated by BLEU) **and** is also faithful to the input (as evaluated by PARENT and OK-percent). \n\nWe next emphasize why CG-Random underperforms according to these criteria:\n\n* In the context of domain adaptation (scenarios 3 and 4), CG-Random displays worse yet somewhat comparable performance to our proposed methods based on BLEU and PARENT metrics, while it is substantially inferior (by **more than 10 points**) according to OK-percent metrics. Please see the lower sections in Table 3 and Table 11). This indicates that our methods produce considerably fewer omissions and hallucinations, while maintaining human-like fluency. \n\n* Regarding in-domain training (scenarios 1 and 2), CG-Random performs slightly worse according to PARENT metrics. This performance gap is much wider when looking at BLEU scores (our methods outperform CG-Random with a margin of **nearly 10 points**). Please see the upper section of Table 2 and Table 10. The sole metric on which CG-Random surpasses our methods is OK-percent, although the difference is marginal. In Section 4.2 (Lines 470-490), we discuss why this is the case and highlight that CG-Random's excessive decompositions might result in faithful generation, but the  approach also leads to unnatural and redundant textual descriptions. \n\nIn summary, across **all four testing scenarios**, our proposed methods strike a **better balance** between human-likeness and faithfulness compared to CG-Random without introducing significantly greater complexity. The fundamental generation model remains consistent with T5. When compared to CG-Random, the CG-Numerical method simply requires tallying of the co-occurrences of two predicates within the training set. Similarly, CG-NN only mandates a compact classification network, comprising less than 20 lines of code in its forward pass. As for CG-RL, it can be understood as the integration of the original, unaltered T5 generator with the newly introduced classifier.\n\nIn the final version of the paper, we will discuss the limitations of the CG-Random approach more thoroughly.\n\n\n## RE-2: Use the annotated grouping information in training\n\nPredicates are indeed grouped in the WebNLG 2017 v1.6 dataset. In section 5.2, we utilize this annotation in the test set to assess the decomposition. We assume that such high quality annotations are not usually available for training, and thus design our models around this premise.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is intuitive to decompose predicates into groups first (trying to ensure at least one combination pair has been seen) and then generate text for each group. The results show improvements over T5 under different settings\n\nThe benchmark (especially the few-shot one) can be useful in this field. \n\nAblation study and analysis experiments (along discussions) are clear and in detail.", "reasons_to_reject": "The CG-Random is performing quite well in many scenarios. This brings doubts on the effect of the complex predicate composition process.\n", "questions_to_authors": "Question A: There are sorted and grouped predicates in the training set of WebNLG. Have you considered using them in training?", "paper_topic_and_main_contributions": "This paper focuses on assessing the compositional generalization ability on the Data-to-Text generation task. The authors propose a benchmark based on WebNLG 2017 to specifically assess the compositionality in four different scenarios. The authors present an innovative framework (predicate decomposition) which demonstrates improved performance compared to T5. Beyond the standard BLEU evaluation, this paper brings more attention to the faithfulness evaluation (PARENT and OK-percent). ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper investigates the challenges of compositional generalization in Data-to-Text models, particularly in relation to predicate structures. The authors propose a new methodology that begins with sentence planning, whereby input predicates are clustered (each cluster mapping to a subsequent sentence), followed by text generation. The approach is evaluated using predicate structure-controlled instances from the WebNLG benchmark. The experimental results indicate that decomposing the predicate structure into smaller units improves the faithfulness of the output and enhances the model's domain adaptability.

Main Contributions:

The authors present a novel evaluation methodology centered on predicate configurations, providing a robust benchmark for future research.
The paper introduces multiple predictors for predicate decomposition, including numerical-based, neural network-driven, and those enhanced by REINFORCE. A random predictor also serves as a baseline.
The authors propose an innovative framework (predicate decomposition) which demonstrates improved performance compared to T5.
The paper provides a comprehensive analysis of the experiment results, highlighting the nuances and broader implications of the compositional generalization methods.
The authors create a benchmark for assessing compositional generalization, few-shot learning, and domain adaptation, which can be useful in this field.
Reasons for Acceptance:

The paper is well-written and clearly explains the development and implications of their cluster-based compositional generalization method.
The proposed method requires smaller training sets and does not require additional labeled or unlabeled data.
Their approach performs well when evaluated against T5 baselines, and the method of evaluation and discussion of results is clear and relevant.
The introduction of an evaluation method and a novel approach to address compositional generalization fills a gap in the current research.
Reasons for Rejection:

The study is limited to a single model size, leaving questions about the method's efficacy in bridging compositional generalization gaps in larger models.
The proposed method seems prone to repetition issues, particularly in WebNLG content that spans an average of 1 to 2 sentences, and effective solutions to this concern aren't elaborated upon.
The CG-Random performs quite well in many scenarios, raising questions about the necessity and effectiveness of the complex predicate composition process.
Decision: Accept-Findings