ReTAG: Reasoning Aware Table to Analytic Text Generation
Authors: Deepanway Ghosal
Preksha Nema
Aravindan Raghuveer
Venue: EMNLP 2023 Main (2023)
Keywords: Table to Text Generation
Table Understanding
Structured Reasoning

Abstract: The task of table summarization involves generating text that both succinctly and accurately represents the table or a specific set of highlighted cells within a table. While significant progress has been made in table to text generation techniques, models still mostly generate descriptive summaries, which reiterates the information contained within the table in sentences. Through analysis of  popular table to text benchmarks (ToTTo (Parikh et al., 2020 and InfoTabs (Gupta et al., 2020) we observe that in order to generate the ideal summary, multiple types of reasoning is needed coupled with access to knowledge beyond the scope of the table. To address this gap, we propose ReTAG, a table and reasoning aware model that uses vector-quantization to infuse different types of analytical reasoning into the output. ReTAG achieves 2.2%, 2.9% improvement on the PARENT metric in the relevant slice of ToTTo and InfoTabs for the table to text generation task over state of the art baselines. Through human evaluation, we observe that output from ReTAG is upto 12% more faithful and analytical compared to a strong table-aware model. To the best of our knowledge, ReTAG is the first model that can controllably use multiple reasoning methods within a structure-aware sequence to sequence model to surpass state of the art performance in multiple table to text tasks. We extend (and  open source 35.6K analytical, 55.9k descriptive instances) the ToTTo, InfoTabs datasets with the reasoning categories used in each reference sentences.

ReTAG: Reasoning Aware Table to Analytic Text Generation
Deepanway Ghosal 1.2
Preksha Nema? Aravindan Raghuveer*
1 DeCLaRe Lab, Singapore University of Techhologg and Design, Singapore 2 Google Research, India
Srepanway_ghossee Sathhoal@@mmmilllsuu.edd.......d (preksh, araghuveer) @ google.com
Abstract
Section Title Spin Boldak (Cilimtt)
Month
Jan
Apr
Jul
Dec
The task of table summarization involves gen- erating text that both succinetly and accurately represents the table or a specific set of high- lighted cells within table. While significant progress has been made in table to text gen- eration techniques, models still mostly gener- ate descriptive summaries, which reiterates the information contained within the table sen- tences. Through analysis of popular table to text benchmarks (ToTTo (Parikh et al., 2020) and InfoTabs (Gupta et al., 2020)) we observe that in order to generate the ideal summary, multiple types of reasoning is needed coupled with access to knowledge beyond the scope of the table. To address this gap, we pro- pose RETAG, a table and reasoning aware model that uses vector-quantization different types of analytical reasoning into the output. RETAG achieves 2.2%, 2.9% im- provement the PARENT metric in the rel- evant slice of ToTTo and InfoTabs for the ta- ble to text generation task over state of the art baselines. Through human evaluation, we ob- serve that output from RETAG is upto 12% more faithful and analytical compared to strong table-aware model. To the best of our knowledge, RETAG is the first model that can controllably use multiple reasoning methods within structure- aware sequence to sequence model to surpass state of the art performance in multiple table to text tasks. We extend (and open source 35.6K analytical, 55.9k descrip- tive instances) the ToTTo, InfoTabs datasets with the reasoning categories used in each ref- erence sentences.
Average HCg
(3.9
28.1882.)
40.2(104.4
(23
6.9 (44.4)
(1)
20.2
(59.2)
9.248.4)
Average Low "C (P)
0.0 (32)
(54.1)
23.4 (74.0)
0.9 (33.6)
Descriative
Sppp Bolcak July has an average temperature of 31.8 and
Januurr has un over temperature of 6.9%
Tablezin Spin Boldak, the lowest
January with
everage temperature 6.9 (64.4 "O and the highest is July with a maximum of 31.8"c.
BeTAG
Commonsense: Spin Julyy average temperature 31.8 1 (8922
coolest month, January..uyy........ is the coldest 6.9CC
Figure 1: A descriptive summary simply states in words the numerical values highlighted in the cells. With RETAG we can produce different analytical summaries by controlling which reasoning categories used: Tabular (Tabular Common Sense Reasoning)
beyond the selected cells and / or employs non- trivial reasoning. Being able to generate a high quality analytical summary is critical property to improve the performance of Table Text Mod- els. We show later in our analysis that ToTTo (Parikh et al., 2020) and Infotabs (Gupta et al., 2020), two widely used table to text benchmarks, roughly 27% and 44% of the reference summaries are analytical, respectively. In literature, there are five popular categories of reasoning used in analyt- ical summarization: 1. Tabular Reasoning where information in the table beyond the selected cells is used (Chen al., 2020d: Saha et al., 2022; Zhao et al., 2022b; Chen et al., 2022), Numerical Rea- soning that uses math operations on/across cells (Geva et al., 2020a; Dua et al., 2019; Amini et al., 2019), 3. Temporal Reasoning that imparts knowl- edge about different units of time (Qin et al., 2021), 4. Commonsense Reasoning to apply world knowl- edge (Talmor et al., 2019; Chen al., 2020a) and 5. Entity Awareness to distill knowledge about entities and their surface forms (Liu et al., 2018; Moiseev et al., 2022; Guu et al., 2020).
infuse
1 Introduction
In the task of Table to Text Generation (Yang et al., 2021), the output summaries usually are of two types: Descriptive and Analytical. A descriptive summary is formed with only information con- tained within the selected cells of the table and nothing else (Refer Table 1). On the other hand, an analytical summary is one that uses information
Most table to text models, use structure-aware architectures or pretraining strategies to internally equip the model with one or more types of analyt-

---

ical reasoning capabilities. We draw attention to two practical scenarios where reasoning categories should not be statically baked into a table to text model and more inference time control is needed. 1. Dataset Diversity: Tables like financial charts that consists of pre-dominantly numbers usually need numerical and temporal reasoning. On the other hand, tables like biographic information would need entity and common-sense knowledge more often. Since the same model would be used to summarize diverse tables, becomes essential to be able to pick the appropriate reasoning categories based on the input table.
the highlighted cells. To the best of our knowledge. there is no prior generative table to text model that is capable of controlling and composing multiple reasoning categories.
In this paper, we make the following contribu- tions to address the problems described above.
We formulate new summarization task called Reasoning Aware Table to Text motivated by failure situations that arise in real-world settings. To the best of our knowledge we are the first to propose this problem formulation. (Section 3)
We present RETAG, vector quantized approcah in encoder-decote table to text models to gen- erate rich analytical summaries with controllable multi-category reasoning. The use of RETAG im- proves the performance of state-of-the-ard table to text models on the ToTTo and InfoTabs by 2.2%, 2.9%, respectively, in the PARENT metric. (Sec- tions 4.5)
2. Usage and Context Diversity: Depending on the context of usage, basketball match score- card table can be summarized in two distinet ways. For avid sports experts, temporal and tabular rea- soning can be used summarize interesting pat- terns in the table. On the other hand, for a newspa- per article the focus would shift to entity knowledge and crisp tabular summary of match results.
We release an extended version of the ToTTo and the InfoTabs dataset that are annotated with the following popular reasoning categories: numerical, temporal, commonsense, table reasoning and en- tity knowledge defined in (Gupta et al., 2020) (Section 6).
Following from the examples above, we argue that explicit control is needed dynamically pick the reasoning categories. during inference time. To the best of our knowledge, there no prior liter- ature on table to text summarization with explicit timerence-time reasoning control.
Related Work
Next, we study some key patterns in the human generated references in ToTTo (Parikh et al., 2020) and Infotabs (Gupta et al., 2020). The ToTTo and Infotabs validation set have 7700 and 196 tables with at most 3 reference summaries per table. We observe that close to 16.4% and 62.2% of the ta- bles have summaries that use more than 2 reasoning categories simultaneously in ToTTo and Infotabs respectively. We term this as Multi-Categon Rea- soning. Such an example is shown in Figure 1. The actual table in the dataset lists the monthly temperatures recorded in town Spin Boldak- we show only a sample here for illustration. We ob- serve that to generate the ideal reference from the highlighted cells requires the model to posses two reasoning skills. First, Tabular Summarization: to compute the maximum and min values over the daily mean field. Second, commonsense reasoning to map the notion of maximum to warmest and that of minimum coolest. Therefore, attain human level accuracy on benchmarks like ToTTo and In- fotabs single generative model should have i) the knowledge of multiple reasoning skills ii) the abil- ity to combine more than one skill to summarize
Among recent advances Table To Text Models, two directions have led to significant improvement in analytical reasoning performance: Table-Aware Ar- chitectures and Analytical Reasoning Pretraining 1. Table-Aware Architectures: Novel model ar- chitectures (Liu et al., 2021; Gong et al., 2020; Xing and Wan, 2021; Li et al., 2021) have been pro- posed to efficiently encode tabular data and hence better use the implicit semantic structure of rows and columns. A large number of works have been introduced to incorporate table structure using pre- training strategies. (Herzig et al., 2020; Andrejczuk et al., 2022: Xing and Wan, 2021; Yin et al., 2020) introduce a pretraining strategy with specialized objectives for structured data: such as Masked Col- umn Prediction(MMPP) Adjacent Cell Prediction (ACP) and so on. Some of the above works, also use special row and column embedding encode the table structure in the input. (Liu et al., 2021) learns table structure using the task of neural SQL execution over tables in their pretraining strategy. (Dong et al., 2022) presents an claborate survey for table based pretraining strategies. It has been

---

shown that for analytical reasoning understanding table structure is a key ingredient. Therefore in our work we use TAPEX (Liu et al., 2021) as our backbone architecture.
Reasoning Type
ToTTo
InfoTTabs
Simple Lookup
1543
74
Numerical
168
43
Multirow
118
14
Temporal
59
23
2. Analytical Reasoning Pretraining: (Zhao et al., 2022b) introduces new pretraining strat- egy with synthetic data augmentation using tem- plates for very specific operations in numerical and temporal reasoning. (Zhu et al., 2021) also incorpo- rates numerical and table operations in the model by appending symbolic reasoning operator selec- tion on top of RoBerTa (Liu et al., 2019). (Zhao et al., 2022a; Chen et al., 2021) performs arithmetic reasoning through program generator module that converts the reasoning operation to executable program to derive the answer. Also, (Chen et al., 2022) disentangles computation with reasoning. by converting numerical reasoning aspect to an exe- cutable program. (Lei et al., 2022) models numeri- cal question answering as expression tree generator task which helps in better alignment among ques- tion, table and paragraph.
Quantification
12
36
Entity Type
36
9
Commonsense
72
31
Lexical Reasoning
11
28
Named Fintity
3
2
Coreference
5
7
Subjective
5
4
Syntactic Alterations
20
Ellipsis
1
Negation
0
Table 1: Breakdown of categories 1750 randomly sampled instances ToTTo and 180 randomly sampled in- stances in InfoTabs.
categories as described in (Gupta et al., 2020). For (ToTTo, InfoTabs) the breakdown of the 14 categories are given in Table 1.
We observe that three categories (Ellipsis, Nega- tion, Subjective) are very rare in both datasets. Due to the sparsity of training data in these categories, we do not consider them for the modeling task. The categories (Coreference, Lexical reasoning, Syn- tactic Alternations) are well covered by linguistic pre-training datasets and hence we do not explicitly try to model them either. We focus on the remain- ing categories. Due similarity of the categories and coalesce training data, we combine (Numer- ical Reasoning, Quantification) into one category. Similarly we combine Entity Type and Named En- tities into one category leaving us with a total of six categories.
To the best of our knowledge, none of exist- ing techniques provide controllability with multi- category reasoning. This the focus of our work. We discuss other prior literature on table to text datasets and controllability in Appendix G.
3 Problem Formulation
In this section, through systematic human eval- uation first justify the choice of six reasoning categories (five analytical and the descriptive) that we introduce in Section 1. Next, we proceed to for- mally state the problem of Reasoning-Awand Table to Text Generation.
Please note that the above methodology of con- structing the six categories is an extension (Gupta et al., 2020) and not core contribution of our work. It is presented as a simple heuristic to form a viable set of non-overlapping, useful cat- egories each with enough training data. The core contribution of our work is agnostic to the choice of actual categories themselves. Choosing a com- prehensive and concise taxonomy for table to text reasoning is outside the scope of our work.
3.1 Analytical Reasoning Categories
InfoTabs (Gupta et al., 2020) was among the first to systematically present a comprehensive taxonomy for analytical reasoning with 14 categories. For the purposes of addressing data sparsity and non- overlap of reasoning categories we simplify the 14 categories proposed in (Gupta et al., 2020) into categories as we explain below.
Informed by the above analysis, we annotated the entire dataset of InfoTabs with the six cate- gories. The training datasct for ToTTo is large (120K samples), therefore we using a filtering heuristic (refer Appendix-A) annotate a subset of the training set. The test set of ToTTo not di- rectly available as it is an online benchmark. Hence we annotated only the validation dataset for ToTTo.
We sample roughly 20% of instances from the validation set of each dataset (1750 instances in ToTTo and 180 instances in InfoTabb respectively). The annotation below was done by team of trained human experts and the details of the process is discussed in further detail in Section
All instances were now annotated with the 14

---

Reasening Specific Cadehoski
Input Table
Encoder
Decoder
Output Sentence
Q
Figure 2: RETAG consists of three modules: Encoder, Vector Quantized Based Codebooks and Decoder. The output of the eadebooks element-wis added the encoder output before being decoded.
Please note that no hyper-paramett tuning was done using the ToTTo validation set and was used for performance measurement alone.
category, we use pretraining strategy to better learn the reasoning categories from structured ta- bles and free-form text data ($4.3).
For any table-to-text model, basic property re- quired is the efficient understanding of table struc- ture. To infuse this aspect, we the pretrained TAPEX model (Liu al.. 2021) as the encoder- decoder backbone in our framework. TAPEX a BART (Lewis et al., 2020) model fine-tuned on the task of neural SQL execution over tables. Authors show that this is an effective pretraining strategy for various table related downstream tasks.
3.2 Problem Statement
Let R. be the set of six categories defined in Sec- tion I: Deseriptive, Tabular, Numerical, Temporal, Common-Seni and Entity reasoning. Given a ta- ble T, set of cells }Cij] contained within T and reasoning category set r C R, the task of reasoning aware table to text is to generate a summary that uses all the information contained in (Cij). applies analytical reasoning r' and then produces a truthful summary S confirming to
It is important note that model architec- ture contributions are not limited to TAPEX, and thus we integrate the same modules T5 (Raffel et al., 2020a) based models and benchmark their performance later in the paper.
We use several automatic metrics to measure the relative quality the generated sentences com- pared the reference: BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), semantic similarity with sentence embeddings (Reimers and Gurevych, 2019) using the MPNet model (Song et al., 2020); (ii) Between the tables and generated sentences: PARENT (Dhingra et al., 2019).
4.1 Preliminaries
We denote the encoder and decoder as E and D. We use the question to pass the reasoning tags in the input: Generate a sentence with TAG reason- ing based on the following table? for analytical generation; Generate descriptive sentence based on the following table? for the descriptive gener- ation task. We concatenate with the linearized table to form the input x. The encoded vector is E(x) E R N in x, and H the latent dimension. For our base model, the decoder D generates the corresponding output sentence y DxE(x)).
three human eval metrics de- fined by the following questions:
In addition
1. Reasoning:
Does the generated summary use all reasoning categorics required in r?
2. Faithful: Is the generated summary hallucina- tion free (i.e., faithful to the table T?)
x where is the number of tokens
3. Coverage: Are all cells in [Cij used in the generated summary?
4 Proposed Model
4.2 Vector Quantized Reasoning
We propose our model RETAG for controllable multi-skill reasoning table to text generation. We embed the two key aspects required to generate alytical sentences: control reasoning categories, and being reasoning aware. First, better model each reasoning category with control, we use a vec- tor quantization (VQ) strategy in RETAG ($4.2 and $4.4). Second, to precisely model each reasoning
Our primary objective is to incorporate reasoning level control in model RETAG However, of the main challenges of generating analytical sen- tences is learn category specific aspects, which can be used perform interaction between these categories for complex reasoning. To achieve this, we sandwich a vector quantization module between

---

the encoder and the decoder. Each reasoning cat- egory has its own codebook on which the vector quantization operation is performed.
egories we use extends beyond performing infer- ences on specific tables. Therefore, we explore pretraining strategies with various free-form and structured-datt based datasets having the specific reasoning components. We collect the following datasets: (i) numerical and textual data (ND, TD) from Geva et al. (2020b), DROP (Dua et al., 2019), MathQA (Amini et al., 2019) for numer- ical reasoning, (ii) LogicNLG (Chen al., 2020b), Logic2Text (Chen al., 2020e) for numerical rea- soning and table knowledge, (iii) WIKIBIO (Liu et al., 2018) for table reasoning and entity knowl- edge, (iv) CommonsenseQQ (Talmor et al., 2019), PIQA (Bisk et al., 2020), Social-IQA (Sap et al.. 2019) for commonsense reasoning, and (v) MC- TACO (Zhou et al., 2019) for temporal reasoning. We have total of 276k instances from the above datasets spanning over the five reasoning categories, We formulate a seq-to-seq text generation task and pretrain our model (encoder, decoder, code- book) on the reasoning- dataset. We detail the model training strategy in Section 4.4.
We use the encoded representation E(x) to in- tervene reasoning specific knowledge from code- books to create new reasoning aware representa- tion which is then passed the decoder D.
codebook denoted as с E RK*H which a latent embedding space of size K, with H being the dimension of each of the codes Oj in e (Van Den Oord et al., 2017). We have a codebook cº for each of the five reasoning categories For com- pleteness, we also have separate sixth codebook for the descriptive category. We use the codebooks extract reasoning based representation for the en- coded (x)) using quantization process Q. It maps each token vector in E(x) to the nearest code of for the given category codebook c":
Q' 3'(E(r).) = ch, =n=l..... where, k = argmin, [xx)) c/ll
())
Now, to model multi-category reasoning for gen- erating analytical sentences, we propose the follow- ing weighted summation technique:
4.4 Reasoning Control based Fine-tuning
To further improve upon reasoning based represen- tations, we add classifier network M on top of the residual features u and ud, which classifies it into analytical and descriptive classes. This classifica- tion constraint helps the model to broadly learn the difference between analytical and descriptive sen- tence. We term this strategy CI (Classification of Intermediate activations). We show later that this classification strategy helps in improved generation for both descriptive and analytical sentences. The overall loss function for a batch consisting of both analytical and descriptive references, is as follows:
Q"(x)
1s
w (c))
(2)
Here, w² represents scalar weights for each rea- soning category, predicted from the last layer of the encoder E, through an additional head layer. The binary labels 1a simplify the equation so that the codebooks used are restricted only to the specified reasoning labels. Furthermore, we add a residual connection between the reasoning based represen- tation and the original encoder representation. We then pass the resultant vector to the decoder to generate the analytical sentence y
L -(8 logy" + 8 + [l))(M((") log(M(("" "lea
u Q"(xi (x) E(x) y D(u")
(3)
[||g][E(x]] E(x)
Q(x) (. [$|[E(x) ag (())))))
We also generate the descriptive sentence yd in a similar manner using a residual connection be- tween the encoded vector and the quantized repre- sentation from the descriptive codebook. For case of understanding, we refer to Q"(x) Q(x).
The loss function consists of four components: Generative Loss: The first term is the cross- entropy loss over the vocabulary for generating the gold descriptive and analytical sentences yd and y Classifier Loss: The second term is the cross-entropy loss from the classification constraint. We denote and + as the descriptive and analyt- ical class. Codebook Loss: gradiett (sg) in the third term required for training the code- books as the quantization process with argmin op- eration in Equation (1) is non-differentiable. Van
4.3 Reasoning-Aware Pretraining
In order to generate analytical sentences with our proposed architecture, is crucial that the code- books are rich in representing each of the reason- ing categories efficiently. The five reasoning cat-

---

Overall
Performance
Analytical Performance
Descripdive Performance
Dataset
Model
Strategy
B1
B4
R-I.
Sim
PAR
B1
B44
ROUGG
Sim
PAR
B4
ROLGE
Sim
PAR
68.91
22.78
59.24
86.51
63.12
65.60
19:45
53.62
84.12
60.11
70.18
24.12
87.66
66.15
T5 TXPEX
No TAGS
69.83
22.67
59,69
87.17
63.31
65,47
19.77
54.36
84,51
60.05
71.00
24.94
61.61
88.
66.27
69.92
22.57
59.18
87.13
63.51
65.71
18.60
53.33
84.29
60.34
71.20
24.26
61.28
88.
66.20
T5 TS
69.18
22.77
59.31
87.11
64.12
66.05
19.77
54,00
84.66
61,46
70.51
24.08
61.05
87.99
65.99
ToTTo
TxGS
70.37
23.62
59.70
87.11
61.41
66.88
20.33
54.62
84.69
61.25
71.84
25.05
61.52
87.99
66.55
TAPEX
70.79
22.63
39,07
86,93
61.91
66.82
18.25
52.91
83.89
61.76
72:43
24-49
61.29
88.03
66.89
TS FLAN TS
70.05
22.34
59.01
87:27
61.65
66.43
18.57
53,15
84.19
61.86
71.54
23.94
61.12
88.31
66.75
RETAG
70.46
22.56
39,10
86,99
61.85
66.91
18.86
53.06
84.00
62.01
71,90
24.12
61.27
88.07
67.12
TAPEX
71.24
23.03
60.20
86.81
65.22
67.90
19.04
53.08
83.84
62.57
72.59
24.69
61.14
87.88
67.32
TS PLAN TS TΔPEX
40.64
17.23
37.26
65:42
23.26
36.44
4.60
32.63
65.22
13:04
12.54
22.17
40.97
65.58
33.30
No TAGSS
39.43
17.00
36.32
65,009
24.09
35.61
4.01
31.17
65.13
11.72
41.02
21.62
40.47
66.06
32.12
44.04
16.82
63.11
25.01
42.01
5.45
32.79
62.69
14.86
21.24
38.77
63.46
32.20
T3
41.54
17.34
38.90
66:55
25.75
43:41
7.85
38.12
69.06
17.28
40.77
20.61
39.52
64.52
32.56
InfoTabs
FLAN TS
39.82
17.02
38.54
66.74
76.77
38
6.47
37.05
68.40
16.90
40.25
20.95
65.40
34:71
TΛPEX
44.60
17.28
36.96
64.13
25.52
41.67
6.16
33.97
64.36
16.92
46.07
22.10
39.37
63.94
32.31
42.60
17.78
64.13
26.58
39.67
6.16
33.97
16,40
22.10
39.37
63.94
34.78
FLAN TS TnPEX
RETAG
39.93
11.96
32.61
62.03
25
30.64
3.12
27.76
60.55
15.48
46.55
18.02
36.52
63. 63.23
33.96
45.15
17.71
37.32
26.97
45.01
6.80
34.46
65.52
17.75
39.62
35.05
Table 2: Automatic evaluation results for analytical and descriptive sentence generation with in the Infotabs and ToToo datasets. Sim denotes the cosine similarity from sentence embedding model. B1, B4, R-L, PAR denotes BLEU1, BLEU4, ROUGE-L. PARENT scores respectively. Reasonig awareness (using TAGS or RETAG ) consistently improves the performance of all the models.
Den Oord et al. (2017) used straight- esti- mator to approximate the gradient by copying the gradient from the decoder input to the encoder out- put. We use the common term Q(x) to represent the quantized vector for both analytical, descriptive instances. Commitment Loss: The fourth term scaled by hyperparameter 33 helps the encoder out- put to commit to the closest codebook vector. For codebook pretraining in Section 4.3, we use the loss terms (3), (4) in the first stage, and then (1), (3), (4) in the second stage.
ally mentioned at the beginning or end of the table string. We assume that the highlighted cells would be indicated in either of these ways within the lin- earized table I. We use the strategy devised in Liu et al. (2021) for linearizing the table. Additionally we are also given the reasoning categories r and the corresponding output sentence y.
5.1 Main Results
We use the following models in our experiments: T5-Large (768M parameters) (Raffel et al., 2020b), FLAN T5- L5rLer (768M parameters) (Chung et al., 2022) and T^PEx (406M parameters) (Liu et al., 2021). We use the models in three different ways:
5 Experiments
We evaluate the performance of RETAG as fol- lows: (i) Performance comparison against strong basclines (ii) Ablation study of design choices used in RETAG (iii) Effectiveness of Vector Quantiza- tion (iv) Reasoning category wise quantitative anal- ysis and (v) Human evaluations for faithfulness and reasoning control. Since our primary contri- bution is multi-categon reasoning, we bench- mark RETAG 's performance for ToTTo valid and InfoTabs test datasets which have heterogeneous reasoning categories. Hence, we do not evaluate against datasets that are specific to one type of rea- soning, such as LogicNLG, Numerical-NI.C. GG, etc.
We use only x to directly generate y. No in- formation about r is consumed by the model. This is the usual seq to--t--se baseline. We de- note this strategy as No TAGS in Table 2.
2. We use information about r as part of the ques- tion Then we train the models to generate y from x. The category information is thus used as part of the input string x. We deonte this as TAGS in Table 2.
3. We use information about r with the codebook selection strategy. This is our proposed RE- TAG method with pretraining as mentioned carlier in Section and Section 4.3.
We use the following notations for our experi- ments: given question g with a linearized table i, we concatenate them to form the input (as men- tioned in Section 4. 1). The table may contain some highlighted cells, which can be enclosed within special indicators such as <hi> or can be addition-
We observe that the models with the TAGS or RETAG approach generally outperform the models with No TAGS. We observe this result
ToTTo:

---

#CBs
CI Loss
ToTTo
InfoTabs B4
Pretraining
ToTTo B4
InfoTabs
R-L
No
68.17
20.89
56.78
43.87
16.09
35.64
No Pretraining
69.8
21.66
58.25
44.49
16:75
36.13
Yes
68.75
21.54
57.30
44.25
16.52
35.71
With Pretraining
71.24
23.03
59.00
48.95
17.71
37.32
No
70.54
22.71
58.49
45.38
17.32
57.26
Yes
71.24
23,03
$9.00
45.95
17.71
37.32
Table 4: Results the overall ToTTo validation and In- fo test set with different pretraining strategies for the TAPEX RETAG model.
Table 3: Results on the overall ToTTo validation and In-
foTabs test set with different number of codebooks (CBs) and the classifier loss for the TAPEX RETAG model.
performance.
1. Number of Codebooks In Table 3, we present the performance of RETAG for two and six code- book setup with category specific quantization. The six codebook setup is the model originally intro- duced in Section 4. We now combine the five rea- soning categories into one analytical category and keep the descriptive category as the other category. This results in the two codebook setup, which we then benchmark with the TAPEX RETAG model.
across most of the evaluation metries. In particular the TAPEX RETAG model achieves around 1% improvement in BLEU-I, ROUGE-L and around 2% improvement over the No TAGS models for the overall performance in the validation set. The performance improvement in the analytical set is slightly more prominent compared to the descrip- tive set in the BLEU-1 and PARENT metrics. We postulate that the tag-based distinction between analytical and descriptive control improves the per- formance for descriptive sentences, as the model gets a clear signal of when to describe the content versus when to reason.
We observe that six codebook setup consistently outperforms two codehook setup across the various metrics for both ToTTo and InfoTabs datasets (first and third rows in Table 3). We conclude that cat- egory specific codebooks provide more flexibility and capacity in the model with better control over the generations.
We further study the importance of augmenting the input table with reasoning categories for fine- tuning the models in the Tags group of results. We observe that it leads to increment in performance across the BLEU-1 and PARENT metrics for the overall set. In the analytical set, the improvement in performance is around 1 1% across for BLEU-1 and PARENT for all the models.
2. Intermediate Activation Classification In Ta- ble 3 also study the effect of classifying the residual features u and u into analytical and de- scriptive classes with the CI classification loss (Sec- tion 4.4), We observe that the RETAG performance improves consistently with CI constraint for both the two and six codebook setups on both the ToTTo and Info dataset.
InfoTabs We achieve considerable improvement with the TAGS and RETAG approach for over- all performance and analytical set performance in Infotabs. TAPEX model has superior perfor- mance over the T5 family model in No TAGS as the TAPEX is trained table corpora on table un- derstanding tasks. However, the performance the comparatively poorer T5 and FLAN-T5 model significantly improved with the use of categorical information. It re-iterates the importance of adding reasoning based control in various models. Our pro- posed TAPEX RETAG model still outperforms the TAPEX TAG model by more than 1% for BLEU-1 and PARENT for overall set, and around 3% and 1% for BLEU and PARENT for the analytical set.
3. Codebook Pretraining Strategies In Table 4, study the efficacy of the codebook pretraining stratey. All the RETAG models here have six code- books and the CI constraint enabled. As the name suggests, in the no pretraining strategy, we do not perform any pretraining on the model and directly tine-tun it for table-to-tex task. In the with pre- training strategy, we start with random initializa- tion of the codebooks and pretrain the codebooks along with the encoder and decoder of the models on the ensemble of reasoning dataset mentioned earlier Section 4.3. We found that the the pre- training strategy is considerably more effective for the final tasks ToTTo and InfoTabs across all the evaluation metrics, as reported in Table 4.
5.2 Ablation Studies
RETAG consists of three main components: the codebooks, the classification objective differen- tiate analytical and descriptive and the pretraining technique of the codebooks. In this section, we study the effect of these three components on the
5.3 Codebook Analysis
In this section, we analyze the effectiveness of the codebooks for analytical generations as follows.

---

Numerical
Commonsense
Temporal
Table
Entity
2 Category
Category
Dataset
Model
Strategy
B1
R-L
B1
R-1.
B1
R-L.
R-L
B1
R-L.
R-L.
B1
R-L.
No TAGS
64.73
49.10
67.81
54.21
69.63
58.29
65:67
50.86
64.97
50.73
65.83
51.17
65.31
47.12
ToTTo
TAPEX
TAGS
65.12
49.75
68.25
55.05
69.78
58.98
65.94
51.41
65.05
50.70
66.28
51.70
65.08
47.72
RETAG
66.28
49.96
68.47
54.75
70.77
58.98
67.20
51.31
67.10
50.95
67.57
51.36
63,47
48.03
TAGS
40.47
29,99
43.71
41.63
31.38
30.63
41.80
28.94
InfoTabs
TΑPEX
TAGS
42.35
30.93
45.80
36.01
43.11
30.73
44,95
33.27
43.52
34,25
42.24
31.38
45.76
33.52
RETAG
42,90
30,78
46,88
36.45
44.47
31.43
45.10
33.36
44.63
35.26
43.39
32.39
45.13
33.18
Table 5: Category-win wise automatic evaluation results. We observe improvement in performance for each category and combination of categories after the introduction of category specific information with the TAGS and RETAG approach.
ToTTo
Infotabs
Label Type
I-Category
2-Category
3-Category
I-Category
2-Category
3-Category
B1
R-I.
B1
R-I.
B1
R-I.
B1
R-I.
B1
R-L
B1
R-L
Random
65
50.93
64.28
50.05
32.23
46.37
40.29
33.61
39,51
29.01
39.38
30.57
Gold
68.26
54.37
67.57
51.36
65.47
48.03
45.51
35.76
43.39
32.39
45.13
33.18
Table 6: Effectiveness of the codehooks for generating analytical sentences with the TAPEX RETAG model. Choosing random analytical labels during inference leads to drop in performance, showing that the codebooks learn meaningful representations of the reasoning categories.
1. Category- Wise Performance We evaluate re- sults across reasoning categories in Table 5. We pick out instances having, just single category an- notated and report the results for them in the L.H.S of Table 5. helps us in analyzing the effect on each category without the involvement others. We note that the six codebooks in TAPEX RETAG helps improve performance across all the five rea- soning categories in comparison to the baseline model. The TAPEX model with TAGS also helps in improving the performance over the TAPEX model without any tags.
We conclude that RETAG models capture reasoning-speciffe information in each codebook through pretraining, which it uses effectively for both single and multi-category analytical sentence generation.
5.4 Human Evaluation
We sample 500 instances from the ToTTo valida- tion set and generate their corresponding analytical sentences from four different models specified in Table 7. Given an instance of table and the reason- ing categories, ask the annotators evaluate the the sentence on three questions described in Section 3.2: Reasoning. Faithful and Coverage.
2. Multi-Categon Reasoning: We also study TAPEX RETAG for complex analytical sentences that involves two more reasoning categories. We report average results for instances having two or three categories in the R.H.S of Table TAPEX RETAG consistently outperforms the baseline by 2% in BLEU1 score for ToTTo and around 4% in BLEU1 and 3% in ROUGE-L for the InfoTabs dataset.
We ask the annotators to provide a label between yes (score 1), partially (score 0.5), or no (score 0), We compare TA PEX RETAG against the TAPEX model without tags for human evaluation to quan- tify faithfulness and reasoning control. We collect 3 ratings for each sample and compile the majority label results in percentage scale in Table We ob- serve that with explicit reasoning control. TAPEX RETAG generates 13% more faithful and 12% het- ter coverage on the reasoning categories on analyt- ical sentences as compared to TAPEX No TAGS. We found very good inter rater agreement for the human evaluation task. The Fleiss' kappa score between the three annotators for human evalua- tion were as follows: 0.5011 on the reasoning cate- gories, 0.5583 on the faithfulness, and 0.6722 on the coverage of highlighted cells. Some examples of RETAG output can be found in Appendix E.
3. Random Reasoning Labels In Table 6, we study the performance when random reasoning cat- egories are sent to the model during inference. We observe that, the evaluation scores drop by sig- nificant margins across sentences with single and multiple categories across both the datasets. This shows, that the codebook encodes meaningful and distinct representations for each reasoning category. We show in Appendix F that RETAG also able beat SOTA baselines for Table QA task on a dataset called Turning Tables (Yoran et al., 2022) where the reasoning categories are provided.

---

Human Evaluation Faithful
8 Limitations
Model
Reasoning
Coverage
TAPEX No TAGS
51.0
76.0
69.4
Some of the limitations of our work are as fol- lows. First, the dataset curation and performance evaluation was restricted datasets in the English language, and does not extend non-dominant languages. Second, several advanced methods have been introduced for numerical reasoning. Our current strategy to incorporate reasoning is data- centric. However, we would like to emphasize that the explicit reasoning control is complementary to the existing methods and in future works, advance methods infuse reasoning side our method. Third, to gain explicit reasoning control for newer domain/reassonin category, in- volves few examples to annotated to bootstrap the model using our method. Fourth, although RETAG is designed for multiple skill reasoning, in future work we will also benchmark RETAG against reasoning specific datasets such as Logic- NLG.
TAPEX RETAG
63.2
89.3
77.6
Two Codebooks
53.9
90.6
78.9
No Pre-training
61.1
91.3
79.9
Table 7: Human evaluation results for analytical sentence generation in ToTTo. Scores are shown in scale.
6 Reasoning Category Tagged Dataset Release
be used along-
As explained in Section 3.1, we will release 31.4K analytical instances and 50.6K for the ToTTo train and validation set. We will also release 4.2K an- alytical instances and 5.3K descriptive sentences over the entire InfoTabs dataset. This section ex- plains the human labeling methodology and the corresponding performance metrics.
We prepared detailed annotation instructions, qualifying questions and trained a pool of 14 crowd- source annotators. The annotators are based in In- dia and are fluent in English. The annotators were paid at rates above required by local wage laws.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Sencel-Kedziorsk. Yejin Choi, and Hannaneh Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 2357-2367.
We instructed the annotators to choose one or more of the five reasoning categories for analyti- cal sentences. We instructed them keep the five reasoning categories and the Descriptive category exclusive i.e. sentence is descriptive only when it does not use any of the other five reasoning cate- gories. Three annotators labeled every instance and we keep only those label voted by atleast two raters. The annotators reached high consensus agree- ment the task. 86.81% of ratings had all three raters agree the binary class for categorizing be- tween descriptive and analytical. 75.12% all three raters agreed on the exact same set of categories for choosing the analytical categories.
Ewa Andrejezuk, Julian Martin Fisenschlos, Francesco Piccinno, Syrine Krichene, and Yasemin Altun. 2022. Table-to-text generation and pre-training with tabt5.
Junwei Bao. Duyu Tang, Nan Duan, Zhao Yan, Yuan- hua Lv, Ming Zhou, and Tiejun Zhao. 2018. Table- to-text: Describing table region with natural lan- guage. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
7 Conclusion
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 34, pages -432-7439.
In this paper, we presented the case for Reasoning- aware table-to-text models. We introduced RETAG, a vector quantized approach for encoder-decoder table-to-text models with explicit control across reasoning categories. RETAG beats SOTA mod- els for ToTTo and InfoTabs datasets for analyti- cal and descriptive sentence generation. We will also release close 35.6K instances of reasoning category tagged analytical abd 55.9k instances of descriptive table to text data.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural lan- guage generation from opeu- tables. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942, Online. Association for Computational Lin- guistics.

---

Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020b. Logical natural lan- guage generation from open-domain tables. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942.
evaluating table-to-text generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884-4895.
Haoyu Dong. Zhoujun Cheng. Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dongmei Zhang. 2022. Table pretraining: A survey on model architectures, pretraining ob- jectives, and downstream tasks. arXiv preprint arXiv:2201.0974,
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020c. Logical natural lan- guage generation from open-domain tables. CoRR, abs/2004.10404.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019, Drop: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long and Short Papers), pages 2368 2378.
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and Tabfact: A large-scale dataset for table-based fact verification. CoRR, abs/1909.02164.
William Yang Wang. 2019.
Mor Geva, Ankit Gupta, and Jonathan Berant, 2020a. Injecting numerical reasoning skills into language models. CoRR, abs/2004,0442
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020d. HybridQA: A dataset multi-hop question answer- ing over tabular and textual data. In Findings of the Association for Computational Lingwistics: EMNLP 2020, pages 1026-1036, Online, Association for Computational Linguistics.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020b. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties. pages 946-958.
Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-text generation with table structure reconstruction and content match- ing, In Proceedings of the 28th International Con- ference on Computational Linguistics, pages 1978- 1988, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R. Routledge, and William Yang Wang. 2021. Finqa: A dataset of numerical reasoning over financial data. CoRR, 109/2100.
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020c. Logic2text: High- fidelity natural language generation from logical forms. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 2096-2111
Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as data. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 2309-2324, Online, Association for Computational Linguistics.
Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020f. Logic2text: High- fidelity natural language generation from logical forms. CoRR, 0e//2004414599
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Ming-Wee Chang. 2020. REALM: retrieval- augmented language model pre-training. CoRR, abs/2002.08909.
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Lc, and Jason Wei. 2022. Scaling instruction- instrectionfiietunee language mod- els.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisensch- los. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 4320 4333.
Fangyu Lei. Shizhu He, Xiang Li, Jun Zhao, and Kang Liu. 2022. Answering numerical reasoning ques- tions in table-text hybrid contents with graph-based encoder and tree-based decoder.
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming- Wei Chang, Dipanjan Das, and William Co- hen. 2019. Handling divergent reference texts when
Mike Lewis, Yinhan Liu, Naman Goyal, Mar- jan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.

---

2020. Bart: Denoising sequence-tor to -e-ueueeece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIME- DIAL: temporal commonsense reasoning in dialog. CoRR, abs/2100.04571.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020a. Exploring the lim- its of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. 21:1-67.
Tongliang Li, Lei Fang, Jian-Guang Lou, and Zhoujun Li. 2021. TWT: Table with written text for con- trolled data-to-text generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1244-1254 1254, Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020b. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Re- search, 21(140):167.
Chin- Yew Lin. 2004. Rouge:
A package for automatic evaluation of summaries. In Text summarization branches out, pages 81.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin. Weizhu Chen, and Jian MannGuang Lou. 2021. Tapex: Table pre-training via learning neural sql executor. In International Conference on Learning Representations.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
Tianyu Liu, Kexiang Wang, Lei Sha, Baobao Chang, and Zhifang Sui. 2018 Table-to-text generation by structure-aware seq2seq learning. In Thirty-Second AAAI Conference on Artificial Intelligence.
Swarnadeep Saha, Xinyan Velocity Yu, Mohit Bansal, Ramakanth Pasunuru, and Asli Celikyilmaz. 2022. Murmur: Modular multi-step reasoning for semi- structured data-to-text generation.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1692.
Maarten Sap. Hannah Rashkin, Derek Chen, Ronan
Le Bras, and Yejin Choi. 2019. Social iqa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP- pages 4463- 4473.
Fedor Moiseev, Zhe Dong, Enrique Alfonseca, and Martin Jaggi. 2022. Skill: Structured knowledge in- fusion for large language models.
Linyong Nan, Chiachun Hsich, Ziming Mao, Xi Vic- toria Lin, Neha Verma, Rui Zhang, Wojeiech Kryšcińsski, Hailey Schoelkopf, Riley Kong, Xian- gru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong. Dragomir Radev, and Dragomir Radev. 2022. FeTaQA: Free-form Table Question Answering. Transactions of the Association for Computational Linguistics, 10:35-49.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Yan Liu. 2020. Mpnet: Masked and permuted pre- training for language understanding. Advances in Neural Information Processing Systems, 33:16857, 16867.
Yixuan Su, Zaiqiao Meng, Simon Baker, and Nigel Collier. 2021a. Few-shot table-to-text genera- tion with prototype memory. arXiv preprint arXiv:2108.12511
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic eval- uation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- lational Linguistics, pages 311-318.
Yixuan Su, David Vandyke, Sihui Wang. Yimai Fang, and Nigel Collier. 2021b. Plan-then generate: trolled data-to-text generation via planning. In Find- ings of the Association for Computational Linguis- tics: EMNLP 2021, pages 890-80999 -09, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
-lan-then-generate:: Con-
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra. Diyi Yang. and Dipanjan Das. 2020. ToTTo: controlled table-to- text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (FMNLP), pages I 1173-1186, On- line. Association for Computational Linguistics.
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Taka- mura. 2021. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing ( (Volume 1:
Matt Post. 2018 A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.

---

Long Papers), pages 1451-1465, Online. Associa- tion for Computational Linguistics.
Fengbin Zhu, Wenqiang Lei. Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A question answer- ing benchmark on hybrid of tabular and textual content in finance. CoRR, abs/2105.07624.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguisties: Human Language Technologies. Volume 1 (Long and Short Papers).
A Dataset Details and Filtering Strategy
pages 41 4158.
ToTTo (Parikh et al., 2020) is an open domain table-to-text dataset where the task is as follows: given table with set of highlighted cells, gen- erate a sentence constrained upon the highlighted cells. The tables are collected from Wikipedia and metadata information (table title, section title con- taining the table) are available. We observe that many reference sentences in the dataset are an- alytical, as they are based on different forms of reasoning, as opposed to simply stating the high- lighted cells verbatim. We compute fuzzy string match between the annotated reference and the highlighted cells metadata of the table. We iden- tify ~63k instances (out of total 120k in ToTTo training set) using various constraints: (i) reference contain non-stop words not present in the table; (ii) fuzzy match < 80%, (iii) fuzzy match < 85% and presence of superlatives, comparatives, or nu- merics (largest, faster, third, etc.). We annotate a small pilot set and surmised that ~ 40% of the 63k instances analytical and these 63k instances would contain most of the analytical instances in ToTTo training set. In addition we also use the full validation set for annotation, as mentioned previ- ously in Seciton 6.
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neu- ral discrete representation learning. Advances in neural information processing systems, 30.
Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. 2022. Robust (controlled) table-to-text gen- eration with structure-wware equivariance learning.
Xinyu Xing and Xiaojun Wan. 2021. Structure-aware pre-training for table-to-text generation. In Find- ings of the Association for Computational Linguis- ties: ACL-IJCNLP 2021, pages 2273-223 2278. Online, Association for Computational Linguistics.
Yang Yang. Juan Cao, Yujun Wen, and Pengzhou Zhang. 2021. Table to text generation with accurate content copying. Scientific reports, 11(1):1-12.
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Se- bastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. arXiv preprint 2005:2005.0831
Ori Yoran. Alon Talmor, and Jonathan Berant. 2022. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume I: Long Papers), pages 6016-6031, Dublin, Ireland. Association for Com- putational Linguistics.
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang 2022a. MultiHierit: Numerical reasoning over multi hierarchical tabular and textual data. Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 6588-68 6600, Dublin, Ireland, Association for Computational Linguistics.
Info (Gupta et al., 2020) is a dataset for table natural language inference (NLI), where the tables (collected from Wikipedia info-boxes) are consid- ered semi-structured premise and human written sentences are provided hypothesis. The task is predict whether the sentence is an entailment, con- tradiction, or neutral w.r.t the table. We compute the fuzzy match between the linearized table and the entailment instances and identify: (i) 4.9k in- stances with fuzzy match < 75% as potentially an- al alytical, and (ii) 1.7k instances with fuzzy match > 80% as potentially non-analytical. To keep the the two categories somewhat balanced, we additionally generate 2.9k potentially non-analytical instances using a keyword to sentence seq2seq model trained on the WikiTableTex. dataset (Bao et al., 2018). In total, we have 9.5k instances to annotate.
Yilun Zhao, Linyong Nan, Zhenting Qi. Rui Zhang, and Dragomir Radev. 2022b. Reastap: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples.
Ben Zhou, Daniel Khashabi, Qiang Ning. and Dan Roth. 2019. "going on a vacation" takes longer than "going for a walk": ; A study of temporal com- monsense understanding.
Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369.

---

B Reasoning Categories
List VFLAFF recurds
We show some examples of table, sentence pairs and the corresponding reasoning categories Figure 3 7. Some of these examples show how multiple reasoning categories could be combined to generate an analytical sentence. We also show example of descriptive sentence in Figure 8.
Seriio Take Mar caset grees
Bank Player
Club
Coreer span
North Melboume
2016
<26
Nichoel Taass
lawthor
1972-1991
Kevrin Bartiett
Richmond
1955-1083
Rescher Essendoon
2015
383
Robert Harvey
St #idd
1988
2006
Sentence
Treck held wes trodien by Nart-Mill Malher mork Rooe Flurnry.
bolder antil
Esref Apak
Secise Tall Achieaanors Nooe
Figure 5: An example of temporal reasoning in the above table
Year Competition
Verue
Position Notes
Represercing Turkey
2000
Jumer Champtondhip
Santogo,
69.5
2001
Meditorranonn Gainzs
Taris, Turisia
6th
71.06
Burch Stadiums
2003
Eimppon U23 Champtonanips
Bydgemzzz Poland
2nd
76. 93.52. m
Aecdes Sule
Olyppc Games Universide
Greece
3rd
79.51
Too Vlay
lamir,
2nd
2005
Mediterrancón Gamzss
Atrerla, Spaln
1st
77. 7.888
World Champ crahlips
Hela Firland
17th (4)
73.04 m
Earopean Championships
Gothenburg, Sweden
19th (4)
70.17 m
2007
2003
Olympic Games
Sering Chena
16th
24.45
Way wnew
2009
World Champ crahips
Barln, Germary
27th 04) 70.70 m
2012
Clympic Carses
London, Uniied kingtom
17th 0g)
23.47
Figure 6: An example of numerical and table reasoning in the above table.
Europein Championihtts
Aneterdam, Nacherlands
2016
Olympic Garees
Ro de Janetro, Brazi
24th (4) 70.
blamic Seliciuiit Games
Reiss, Azerisajjan
1st
74.13 m
Lumpeen Chempionehpss
Francesco Bugnaia
World Champ orchiee
London, United Kingtom
10th
73.35
2016 E. Eumpeen Championentps
Tertin, Germary
17th (n) 77 70
Rocia
fort
Now
Seasss Cin
Noteroycle Team
Humber face Wh
Todua
Pele
Tue Po Pioe
Rendencz
2015
FTB Hucd
San Cate farl
190
77.4 patso
gold
3015
Janindr
Pull opprr Nanmerra Tran
2017
take
5CY Rechi Tran VI46
42
10
0
8
174 30
Figure 3: An example of commonsense reasoning in the above table.
2015
M952
Sales
Sor
30.
Masd couppro Ragual Mani ucing Troe
Swaziland the 1992 Winter Olympies
Figure 7: An example of numerical and temporal reasoning the above table.
Section Tifle Alpin dking
Table Section Fest Home
Race
Athlete
Event
Time
Time
Rank
Trey Johnson
Mat'ss Super-G
1:29.39 79
Netth Frasor Hen's glant sialom 1/21.93 1:19.83
2:41.76 65
Section Title: College Table Section Monr
War''s stalom
DNF
DNF
Year
Team
GP
G5
MPG
FCN
3P%
FT%.
RPG
APG
SPG
BPG PPG
Sentence
2003 04 Alcom State
29
28.5
.414
.415
781
2.4
2.7
55
11.0
Koth Praver finsbol the scond nas a faster drnc of minue and 19.25 seconds which led him pasition.
2005 Jackson State
32
29
34.8
.455
441
750
4.8
2.3
1.4
23.5
2006-07 07
Jackson State
35
35
.144
.33
4.5
2.6
27.1
.499
21.0
Figure 4: An example of numerical reasoning in the above table.
Sentence
Trey Jeiason evenged 23.5 poirts 4.8 rebousda, and 2.5 asios in 32 gamees
C Experimental Setup and Computational Resources
Figure 8: An example of descriptive sentence in the above table.
We use beam search generate outputs from the our generative models. We used beam length of 10 used. All models were trained with the A optimizer with a learning rates of 1e-6,
3e-6, 5e-6, le-5. We used Quadro RTX 8000 GPU for our experiments. We train all our models for 10 epochs, which takes 3 hours for ToTTo and 1.5

---

Category
Table
TAPEX No TAGS
TAPEX RETAG
Numerical,
2008) Oc- cupation: Actress, businesswemann Years active: 1994 present
Jessica Spouse(s]:
Alba married
Jessica
Commossense,
Cash Warren since 2008.
in her profession for over two decades.
Temporal
Commonsense,
Trile: David Foster Genres: pop rock, classical. gospel. R&B Occupation(S:: Music executive, record producer, musician, composer, songwriter, arranger Years
David Faster has been perform- ing for over farty years.
David Foster has been perform- ing since the 1970s.
Temporal
1971-present...
Numerical
Titl: Dead Poets Society Directed by: Peter Weir I Running 128 minutes Country: United States Language: English| Budget: $164 million Box office: $235.9 million...
Dead Poems Society is an Amer- Hollywood film.
Dead Poems Society made over million dollars at Boe.
Commmomentse
Title: Hindi Medium Directed by: Saket Chaudhary I Release
Hindi Medium was released in
Hindi Medium was releused in the summer of 2017.
date: May 2017 2017-05-199 (India), April 2018 2018- 04-04 (Chisa)
the year its release.
Table Generated examples for some samples from the InfoTTahs datasct. The predictions generated for TAPEX RETAG were controlled by the corresponding categories listed in the left-most column.
hours for InfoTabs. The TAPEX and T5 family models have 406M and 768M parameters, respec- tively. The codebooks in RETAG contribute to the additional 3M parameters.
G
Extended Related Work
G.1 Table-Aware Pretraining
With the success of pre-trained language models on unstructured text, a large number of works have been introduced to incorporate table structure using similar pre-trained strategies. (Herzig et al., 2020; Andrejczuk al., 2022: Xing and Wan, 2021,?; Yin et al., 2020) introduce a pretraining strategy with specialized objectives for structured data: such as Masked Column Prediction(MCP)) Adjacent Cell Prediction (ACP) and so on. Some of the above works, also use special row and column embedding to encode the table structure in the input. (Liu al., 2021) learns table structure using the task of neu- ral SQL execution over tables in their pretraining strategy. (Dong et al., 2022) presents an claborate survey for table based pretraining strategies. For any table-specific reasoning task, understanding ta- ble structure forms the basis, therefore in this work we use TAPEX (Liu et al., 2021) as our base model. To the best of our knowledge it is the state of the art for ToTTo dataset.
D Evaluation Metrics
We used the sacreBLEU implementation (Post, 2018) for computing BLEU scores. For ToTTo, we used the originally released code by authors for computing the PARENT metric. For InfoTabs, we used a lambda weight of 0.1 for computing the PARENT metric.
E Example of Generations
We show some instances of generated analytical sentences for the InfoTabs dataset in Table 8. We show generations from the TAPEX without tags model and our proposed TAPEX RETAG model in the table. the first example, given the numerical, temporal reasoning categories, TAPEX RETAG is able to infer that "she has been acting for over 20 years" and "is still active" through commonsense reasoning.
F Results on Turning Tables
G.2 Datasets
Turning tables (Yoran et al., 2022) is a table based QA dataset with reasoning categories annotated. We observe that our TAPEX RETAG approach outperforms TAPEX for this task as well, when we use the reasoning categories mentioned in their work.
Newer dataset encompasses (Chen et al., 2020c): consists of diversified sentences across symbolic operations (such as max, min, etc.); (Chen et al., 2020d; Zhu et al., 2021; ; Chen et al., 2021; Zhao et al., 2022a) introduce table and text based QA tasks that primarily involve numerical and table rea- soning along with interactions between table and given text predict the correct answer. (Nan al., 2022) is Table-QA dataset that involves complex reasoning. (Suadaa et al., 2021; Chen et al., 2020f, 2019) also consist of references that incorporate numerical and table reasoning.
Datuset
Model
FM
F1
Overall
B4
R-I.
Turning
TAPEX
52.61
34.00
$2.62
RETAG
47.44
53.52
42.97
34.74
53,12
Table 9: Automatic evaluation results for question answering Turning Tables datasct. Scores shown in % scale.

---

G.3 Controllability in Structured Data
There are various works (Parikh et al., 2020; Su et al., 2021b; Wang et al., 2022) that focus on the controlling content while generating from struc- tured data. (Li et al., 2021) uses prefix set of tokens to better control the topic of the generated text. (Su et al., 2021a) extracts free-form proto- types from large knowledge base control the structural formation of the generated text. One of the recent works. To the best of our knowledge, ours one of the first works to explicitly model for control on various reasoning aspects for structured data to text generation.


Reviews:
[{"review_id": "I42paaitAY", "reviewer": "Reviewer_CsWy", "summary": "", "questions": "- Did you consider use ChatGPT to aid the data construction (e.g., to annotate the reasoning category)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review of our work. Please find the responses below:\n\n**Reject 1 and Question 1:**\n\nWe tried experimented with a large 24B  instruction-tuned large language model (similar to chatGPT )  in a few-shot setting and found that they have difficulties understanding the table structure and processing long tables (commonly found in ToTTo). The results are as follows: \n\n| Dataset  | Model                | Overall | Overall | Analytical | Analytical | Descriptive | Descriptive |\n|----------|----------------------|:-------:|:-------:|:----------:|:----------:|:-----------:|:-----------:|\n|          |                      |   BLEU  |  PARENT |    BLEU    |   PARENT   |     BLEU    |    PARENT   |\n| ToTTo    | Instruction LLM (24B)|  38.73  |  35.17  |    34.15   |   30.96    |    39.76    |    37.29    |\n| ToTTo    | ReTAG with PreTraining        |  55.18  |  54.37  |    48.32   |    45.84   |    56.45    |    55.72    |\n\nWe use the following few shot prompt for the LLM:\n\n```bash\nYou are proficient in understanding tables.Given the highlighted cells in the table and reasoning category, generate a summary focusing on the highlighted cells using the given reasoning category. \n\nIf no reasoning category is given, generate a descriptive sentence. The table is given as a sequence of tokens arranged row wise. The highlighted cells are marked with [hl].\n\nInput: title: julius erving; section: regular season; col : year | team | gp | gs | mpg | fg% | 3p% | ft% | rpg | apg | spg | bpg | ppg row 1 : [hl] 1972–73 [hl] | [hl] virginia (aba) [hl] | 71 | nan | 42.2 | 0.496 | 0.208 | 0.776 | 12.2 | 4.2 | 2.5 | 1.8 | [hl] 31.9 [hl] \nReasoning Category: No Reasoning Category\nSummary: in the aba, julius erving averaged 31.9 points per game in the 1972–1973 season.\n\nInput: title: simon williams (athlete); section: international competitions; col : year representing the great britain and england | competition representing the great britain and england | venue representing the great britain and england | position representing the great britain and england | event representing the great britain and england | notes representing the great britain and england row 1 : 1992 | olympic games | barcelona, spain | 28th (q) | discus throw | [hl] 53.12 m [hl]\nReasoning Category: numerical, table, external knowledge\nSummary: simon williams was well down on his best there, however, managing only 53.12 m in the qualifying round.\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses an important gap in table summarization by incorporating multiple types of reasoning and access to knowledge beyond the table, which is a novel approach.\n2. The human evaluation results show that RETAG's output is up to 12% more faithful and analytical compared to a strong table-aware model, indicating the practical usefulness of the approach.\n3. The authors extend and open-source 32K tagged instances of the ToTTo and InfoTabs datasets with reasoning categories used in each reference, which is a valuable contribution to the research community.", "reasons_to_reject": "- This paper does not compare with current ChatGPT models with appropriate reasoning prompting commands, which may be a more simple method for current data-to-text generation task.", "questions_to_authors": "- Did you consider use ChatGPT to aid the data construction (e.g., to annotate the reasoning category)?", "paper_topic_and_main_contributions": "This paper addresses the problem of table summarization with multiple types of reasoning and access to knowledge beyond the table's scope.\n\nThe main contributions of this paper are:\n\n1. The introduction of RETAG, a vector-quantized encoder-decoder table-to-text model with explicit control across reasoning categories. \n\n2. Through human evaluation, the authors show that RETAG's output is up to 12% more faithful and analytical compared to a strong table-aware model.\n\n3. The extension and open-sourcing of 32K tagged instances in the ToTTo and InfoTabs datasets with the reasoning categories used in each reference.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Feng Nie, Jinpeng Wang, Jin-Ge Yao, Rong Pan, and Chin-Yew Lin. 2018. Operation-guided Neural Networks for High Fidelity Data-To-Text Generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3879–3889, Brussels, Belgium. Association for Computational Linguistics.\n\nThis paper also discuss reasoning over data-to-text task, which should be discussed in the related work."}}, {"review_id": "AHg4FIka4P", "reviewer": "Reviewer_5oq8", "summary": "", "questions": "There are a series of T5 and FLAN T5 models with different sizes. Have you compared RETAG with a larger T5 and FLAN T5?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We  thank the reviewer for their appreciation of the strengths of the paper and for the detailed and constructive comments that we address below. \n\nWe were also able to increase the annotated ToTTo corpus from 21.7K to 37K sentences. Hence the total dataset size that we will release will be 47K sentences. \n\n**Reject-1a:** \n* We first clarify that all baselines (including TAPEX) were fine tuned exactly like how ReTAG was fine tuned. Specifically the datasets used to fine tune (ToTTo, InfoTabs) and the data format (linearized table and reasoning  tags) was identical as ReTAG.  \n* However we acknowledge that the suggested ablation experiment could be very helpful. In this experiment, we retrain TAPEX with exactly the same pre-training datasets, fine tuning datasets and format as used for ReTAG. Therefore the only difference here would be the vector quantized reasoning codebooks and the accompanying loss function.  Specifically,  we took the TAPEX checkpoint and pre-trained it on the same reasoning corpora with reasoning tags using an identical training process that we used in ReTAG. We then fine-tuned the reasoning enhanced TAPEX model on the ToTTo and InfoTabs dataset with reasoning tags in the inputs. We found that ReTAG outperforms this model on both the datasets. The full results are as follows:\n\n| Dataset  | Model                | Overall | Overall | Analytical | Analytical | Descriptive | Descriptive |\n|----------|----------------------|:-------:|:-------:|:----------:|:----------:|:-----------:|:-----------:|\n|          |                      |   BLEU  |  PARENT |    BLEU    |   PARENT   |     BLEU    |    PARENT   |\n| ToTTo    | TAPEX-Tagged with Pre Training |  53.83  |  53.15  |    46.99   |    44.42   |    55.11    |    53.79    |\n| ToTTo    | ReTAG with Pre Training        |  55.18  |  54.37  |    48.32   |    45.84   |    56.45    |    55.72    |\n| InfoTabs | TAPEX-Tagged with Pre Training |  37.98  |  34.95  |    34.78   |    23.41   |    38.09    |    43.20     |\n| InfoTabs | ReTAG with Pre Training        |  38.82  |  36.58  |    35.19   |    24.01   |    40.90    |    45.36    |\n\n\nNote that the ReTAG model without pre-training also outperforms the TAPEX model without pre-training. The results for ReTAG without pre-training are already provided in the paper in Table 3. We will compile the results in a single table for easier comparison. We chose TAPEX in the rebuttal since it has the best performance among the chosen baselines. We are happy to report numbers of this ablation experiment for the other 3 baselines as well. \n\n**Reject-1b**\n\nThe Fleiss' kappa score between the three annotators for human evaluation are as follows:    \nReasoning: 0.5011  \nFaithfulness: 0.5583  \nCoverage of Highlighted Cells: 0.6722  \nThese show very good inter rater agreement for the human evaluation task. \n\n**Reject 2:**  \nThanks for pointing them out. We will improve the writing and the presentations of the paper to make the draft more clear.\n\n**Question 1:**    \nWe found that the performance of large versions of FLAN-T5 are as follows:\n\n| Dataset  | Model                | Overall | Overall | Analytical | Analytical | Descriptive | Descriptive |\n|----------|----------------------|:-------:|:-------:|:----------:|:----------:|:-----------:|:-----------:|\n|          |                      |   BLEU  |  PARENT |    BLEU    |   PARENT   |     BLEU    |    PARENT   |\n| ToTTo    | FLAN-T5 Large UnTagged  |  54.67  |  53.75  |    47.74   |    45.13   |    56.38    |    55.81    |\n| ToTTo    | FLAN-T5 Large Tagged |  55.31  |  54.11  |    47.86   |    45.41   |    56.97    |    56.12    |\n| ToTTo    | ReTAG        |  55.18  |  54.37  |    48.32   |    45.84   |    56.45    |    55.72    |\n| InfoTabs | FLAN-T5 Large UnTagged  |  36.47  |  34.92  |    32.61   |    23.79   |    38.27    |    43.98    |\n| InfoTabs | FLAN-T5 Large Tagged |  37.72  |  35.43  |    34.65   |    24.38   |    39.14    |    44.77    |\n| InfoTabs | ReTAG        |  38.82  |  36.58  |    35.19   |    24.01   |    40.90    |    45.36    |\n\nThe FLAN T5-Large model has 783M parameters which is nearly two times of ReTAG, which has 407M parameters. Even with half the number of parameters, ReTAG is able to almost reach the same performance as FLAN T5 for ToTTo and for InfoTabs  even surpass it. For ToTTo, RETAG is able to outperform FLAN-T5 Large on the analytical slice. Combining this observation with the previous ablation on effectiveness of vector quantization, we expect the result to improve with larger models. We are happy to train larger FLAN-T5 checkpoints with our codebook architecture in the camera ready version, if accepted. \n\nAlso as part of response to reviewer CsWy, we have reported performance with a 24 Billion instruction-tuned language model. We show that RETAG is able to outperform the LLM 24B in a few shot setting. \n\nAlso, we would like to point out that the usage of tags (reasoning awareness in Contribution-1) helps improve the performance of FLAN T5 tagged over FLAN-T5 untagged.  Hence this experiment also shows that reasoning awareness can improve the performance of table summarization models as noted by the reviewers in reasons to accept.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper studies the controllable data-to-text generation task, which can be a valuable contribution to the standard task setting.\n2. They collect a human-annotated dataset for categorizing the table summaries into different descriptive and analytical types and will publically release the dataset.\n3. The experimental results suggest the effectiveness of their proposed methods.", "reasons_to_reject": "1. The experimental setting is not comprehensive enough to support the claim of this paper. Specifically, \n-  The baseline models they used for comparison are either smaller or of the same size, which can lead to unfair comparison.\n-  A similar concern is that the model proposed in this work (RETAG) is pre-trained on multiple related datasets and fine-tuned on the specific input format (i.e., table summary format requirement + linearized table) proposed in this work. However, while the author did say that the baseline models (not including TAPEX) are fine-tuned on the datasets used in the evaluation, there is still a discrepancy between the training process of RETAG and baseline models because (a) the baseline models are not pre-trained on the related datasets as RETAG is; (b) the authors did not clearly state whether the baseline models are fine-tuned on the same modified input format.\n- Following the previous point, the authors did not fine-tune TAPEX using the same process they used for fine-tuning RETAG, while this could have been the most fair comparison.\n- The inter-annotator agreement of the human evaluation conducted in this paper is missing.\n\n2. Personly I think the writing of this paper could have been improved. I found it very difficult to follow this paper and there are multiple formatting/writing issues that constantly increase the difficulty of understanding this paper. Just to give a few examples\n- In line 110 the authors used the terminology \"Tabular Summarization\", while in line 052 they used \"Tabular Reasoning\" (I think they meant the same thing)\n- It is probably better to format the content from line 219 to line 225 as a table. \n- The notation of Eq.1 and Eq.2 is confusing. The authors seem to be overloading the term Q^r().", "questions_to_authors": "There are a series of T5 and FLAN T5 models with different sizes. Have you compared RETAG with a larger T5 and FLAN T5?", "paper_topic_and_main_contributions": "This paper proposes methods for improving table-to-text model performance by adding a reasoning-aware module to the baseline model. With this objective, they \n\n(1) introduce a controllable table-to-text dataset by adding an output format requirement (e.g., Temporal Reasoning) as part of the input to the existing dataset;\n\n(2) collect human annotations for categorizing the table summaries into different descriptive and analytical types; \n\n(3) propose the idea of modifying the encoded input latent representation with codebooks for each of the reasoning types defined in this work;\n\n(4) empirically demonstrate that their model can outperform the baseline models on the original table-to-text datasets as well as their modified datasets.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Please see the reasons to reject section."}, "extra_text": {}}, {"review_id": "z0jWJOV9Um", "reviewer": "Reviewer_fnkP", "summary": "", "questions": "Question A: The claim in the abstract that \"While significant progress has been made in table to text generation techniques, models still mostly use information contained within the table to generate the summary.\" Your problem statement in Section 3.2 is also to generate a summary also using all the information contained in C{ij}, rather than any new information that is not contained within the table. The extra component of your problem statement is that the summary needs to confirm to a subset of R. However, in your introduction section and also Tables 5 & 6, your contributions seem to include an explicit control to pick the reasoning categories. So the selection of the subset R itself is important and should be a task in the problem statement?\n\nQuestion B: In Abstract: RETAG achieves 1%, 2.7% improvement on the PARENT metric in the relevant slice of ToTTo and InfoTabs for the table to text generation task over state of the art baselines. In Section 1: RETAG outperforms state-of-the-art table to text models on the ToTTo and InfoTabs by 1.8% and 2.2% in BLEU respectively. This information can be found from Table 1 anyway. Is there any reason why different metrics are used to highlight the contributions in different sections?\n\nQuestion C: Although there are 14 annotators, each instance is labelled by 3 annotators and you only keep those labels voted by at least two raters. Since each label has a score (e.g., partially 0.5), why is it better to keep those labels voted by at least two raters instead of keeping all but averaging their scores? Or do you mean those annotated categories voted by at least two raters?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We  thank the reviewer for their appreciation of the strengths of the paper and for the detailed and constructive comments that we address below. \n\nWe were also able to increase the annotated ToTTo corpus from 21.7K to 37K sentences. Hence the total dataset size that we will release will be 47K sentences. \n\n\n\n**Reject 1 and Question A about Section 3.2:**\n\n\nWe agree with the reviewer that the problem statement can be clarified further in both the abstract and Section 3.2. The clarification we would like to add is that.. “ The task of reasoning-aware table to text is to generate a summary that uses information contained i {Cij}, combine it with world & common sense knowledge and produce a truthful summary S conforming to r” \n\nWe also respectfully submit that the task of predicting {R} is while an important problem but is hard to control for in a summarization setting. For instance, in the original ToTTo paper, the references were produced by modifying Wikipedia sentences and hence could have used any reasoning category. In other words, the set of valid reasoning categories that can be used to summarize a table can be very large and hard to determine apriori. To confirm this we also built a classifier to predict the reasoning category for ToTTo and as expected  the task was quite difficult: weighted F1: 53.91%, exact match: 49.14% for multi-label classification in ToTTo.\n\nHowever, when {R} has to be predicted for QA dataset where R can be predicted from the question, we were able to build a classifier for the Turning Tables QA dataset (Appendix-F in the paper). For Turning Tables the classification performance is much higher at 88% F1.\n\n**Reject 2 and Question B:** \nNo, there was not any specific reason to highlight the different metrics in the different sections. The metrics are equally important. This was an oversight and we will highlight them both in the abstract. \n\nWe respectfully submit that 1% and 2.7% improvement in PARENT on the well known ToTTo and InfoTabs benchmarks are not an insignificant improvement [1]. This observation also correlates with the significant BLEU (2 points +) improvement and significant improvement on human eval (12%).  We also have added two new baselines (please refer response to Reviewer  CsWy and 5oq8) and the performance improvement of RETAG is consistent. \n\n[1]: https://arxiv.org/abs/2004.14373\n\n**Question C:** The partial score of 0.5 is used only for the human evaluation of model generated outputs. For example, if the gold reasoning categories are {Numerical, Commonsense} and the model generated output has only Numerical reasoning then a partial score of 0.5 is given.\n\nDuring the main annotation process, we indeed kept the categories voted by at least two raters. Thus, in the above example the gold categories are {Numerical, Commonsense} implies that at least two raters voted Numerical and at least two raters voted Commonsense.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Using multiple reasoning categories (with an explicit control of the categories) improves the quality of table summaries based on automatic and human evaluations.\n\n* Approximately 32K instances of reasoning category tagged, table-to-text data (tagged ToTTo & InfoTabs) will be released.", "reasons_to_reject": "* The problem statement does not appear to be entirely clear with respect to the description in the abstract (See Question A below).\n\n* 1% and 2.7% improvements over the baselines seem insignificant, though 12% improvement is observed based on human evaluation.\n\n* Typos/grammatical issues and inconsistencies make the paper not easy to read.", "questions_to_authors": "Question A: The claim in the abstract that \"While significant progress has been made in table to text generation techniques, models still mostly use information contained within the table to generate the summary.\" Your problem statement in Section 3.2 is also to generate a summary also using all the information contained in C{ij}, rather than any new information that is not contained within the table. The extra component of your problem statement is that the summary needs to confirm to a subset of R. However, in your introduction section and also Tables 5 & 6, your contributions seem to include an explicit control to pick the reasoning categories. So the selection of the subset R itself is important and should be a task in the problem statement?\n\nQuestion B: In Abstract: RETAG achieves 1%, 2.7% improvement on the PARENT metric in the relevant slice of ToTTo and InfoTabs for the table to text generation task over state of the art baselines. In Section 1: RETAG outperforms state-of-the-art table to text models on the ToTTo and InfoTabs by 1.8% and 2.2% in BLEU respectively. This information can be found from Table 1 anyway. Is there any reason why different metrics are used to highlight the contributions in different sections?\n\nQuestion C: Although there are 14 annotators, each instance is labelled by 3 annotators and you only keep those labels voted by at least two raters. Since each label has a score (e.g., partially 0.5), why is it better to keep those labels voted by at least two raters instead of keeping all but averaging their scores? Or do you mean those annotated categories voted by at least two raters?", "paper_topic_and_main_contributions": "Most table summarization models use information contained in the table to generate a summary. This paper proposes ReTAG that utilizes multiple categories of reasoning for table summary generation. It shows that with an explicit control of the reasoning categories, the proposed model improves the quality of table summaries based on automatic and human evaluations.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* There are many typos/grammatical issues throughout the paper  (e.g., \"multiple types of reasoning ARE needed\", \"Refer TO Table 1\", \"at least\", those labels\").\n\n* Inconsistent hyphenations (e.g., reasoning-aware vs reasoning aware) and casing for terms (e.g., Table to Text vs table to text)."}}]

Author rebuttal:

Meta-review: The authors present a new table-to-text generation dataset along with a model and a credible evaluations. Their human annotator results are well constructed with reasonable reliability. The authors also seem to have allayed some of the concerns from reviewers in their rebuttal.

I encourage the authors to follow reviewer  5oq8's advice and suggestions to increase the clarity of writing in the paper. There's
Decision: Accept-Main