QTSumm: Query-Focused Summarization over Tabular Data
Authors: Yilun Zhao
Zhenting Qi
Linyong Nan
Boyu Mi
Yixin Liu
Weijin Zou
SIMENG HAN
RUIZHE CHEN
Xiangru Tang
Yumo Xu
Dragomir Radev
Arman Cohan
Venue: EMNLP 2023 Main (2023)
Keywords: Summarization
Text Generation
Reasoning over Structured Data
TL;DR: This paper presents a new task known as query-focused table summarization and investigates the state-of-the-art text generation models for this new task.

Abstract: People primarily consult tables to conduct data analysis or answer specific questions. Text generation systems that can provide accurate table summaries tailored to users' information needs can facilitate more efficient access to relevant data insights. 
Motivated by this, we define a new query-focused table summarization task, where text generation models have to perform human-like reasoning and analysis over the given table to generate a tailored summary. We introduce a new benchmark named QTSumm for this task, which contains 7,111 human-annotated query-summary pairs over 2,934 tables covering diverse topics.
We investigate a set of strong baselines on QTSumm, including text generation, table-to-text generation, and large language models. 
Experimental results and manual analysis reveal that the new task presents significant challenges in table-to-text generation for future research.
Moreover, we propose a new approach named ReFactor, to retrieve and reason over query-relevant information from tabular data to generate several natural language facts. Experimental results demonstrate that ReFactor can bring effective improvements to baselines by concatenating the generated facts to the model input.
Our data and code are publicly available at https://github.com/yale-nlp/QTSumm.

QTSUMM:
Query-Focused Summarization over Tabular Data
Yilun Zhao'
Zhenting Qi
Linyong Nan'
Boyu Mi Xiangru Tang!
Yixin Liu'
Weijin Zou'
Simeng Han'
Ruizhe Chen
Yumo Xu
Dragomir Radev Arman Cohan 1.5
Yale University Harvard University
$Zhejiang University
School of Informatics, University of Edinburgh
*Allen Institute for AI
yilun. whwo@yale.eeeeeooooo. edu
Abstract
Diariz
Tansedorr
Find Blane
Candifates
People primarily consult tables to conduct data analysis or answer specific questions. Text gen- eration systems that can provide accurate ta- ble summaries tailored to users' information needs can facilitate more efficient access to relevant data insights. Motivated by this, define new query-focused table summariza- tion task, where text generation models have perform human-like reasoning and analysis over the given table to generate tailored sum- mary. We introduce new benchmark named QUMMMM for this task, which contains 7,1 auman-annotated query-summant pairs over 2,934 tables covering diverse topics. We inves- tigate set of strong baselines QTS including text generation, table-to-text genera- tion, and large language models. Experimental results and manual analysis reveal that the new task presents significant challenges table-to- text generation for future research. Moreover, we propose a new approach named REFAC- TOR, to retrieve and reason over query-relevant information from tabular data to generate sev- eral natural language facts. Experimental re- sults demonstrate that REFACTOR can bring improvements to baselines by concatenating the generated facts to the model input. Our data and code are publicly available https: / //1i thub. com/ yale-nlpel nlp/ mwtpnnlpp////oeeeeee Summm
Nerh Careliza
Maker Anic
1954
Kopubkian
Eril Anterwen (3) 54.8%
Wiler Trus 63
Carelizz
Price
Desoonsik
Dardd
74.4%
Careiss
Hoverd Coble
Repiliioon
(Tt)
Mike Memror 30.1%
Nerth Careimaa?
Mike Momaene
1996
Deromnts
Tow Bounee (8)-45 435
North Carelana
Lerry Kinei
2008
Dessornts
Raduadd itaduce (81) 54.1%
Kisuii (1)
Coolin
Millery
Repuiilion:
31-0%
Whs wer thu podkiars for rice Norh Coniialls 4t
Dovid Democitti OrAnnendo Repubécoti
condools 2810 Dewid Tinls This weeiinn 1996.
soakingg 26 1%
showed
Dawid
the voling neadt which represaraali exction North Cartina
contoittion why?
The race the Nott Candin IIn sisiit was the most competiive as the Demoorrds bocuthee Vinn Milis was by a ein eorgin with only a 000% otheneio betwees tim and hi Repustionn challage Deed Roowe Burtarnors this cloction nos in eaanli of with ess toe pancert
umong of Nort Gandl desiau 2012 tha
An example of QTSUMM. Given the nu- merous data points in the table, different users may be interested in various aspects for their own information- seeking or prcision-making purposes. The system needs to perform human-like reasoning and analysis over rele- vant table regions to generate a tailored table summary.
Figure 1:
tables containing various statistics to develop game strategies and make team adjustments. However, effectively accessing and comprehending the infor- mation contained within a large and complex table can be time-consuming for users (Hurst, 2000; Pa- supat and Liang, 2015; Pujara et al., 2021; Nan et al., 2022a). Text generation systems that can accurately summarize provided table according to users' information needs have the potential greatly enhance data analysis and expedite the pro- cess of obtaining data insights.
1 Introduction
the era of data-driven decision-maken tabular data plays a crucial role in facilitating data analysis, serving as a concise and structured representation of information (Kukich, 1983; Pasupat and Liang, 2015; Chen et al., 2020c; Zhu et al., 2021; Zhao et al., 2022a; Tang et al., 2023). People often con- sult tables extract valuable insights and make informed decisions. For example, sales managers typically explore large tables with specific business questions to gain insights about clients and pro- cesses. Sports coaches will analyze performance
Existing work and datasets on table-to-ted gener- ation (Parikh et al., 2020; Chen et al., 2020a; Cheng et al., 2022b; Lebret et al., 2016; Moosavi et al., 2021: Suadaa al., 2021) have mainly focused on converting tabular data into coherent statements, aiming to present the structured data human- readable format. However, these approaches have overlooked the fundamental goal of addressing

---

users' inforiation-sseekngg purposes. Table-to-text generation systems should adopt more flexible and interactive approach that allows people to ob- tain a user-customized summary tailored to their information needs (Dang, 2006; Xu and Lapata, 2020; Zhong et al., 2021; Xu and Lapata, 2022; Zhou et al., 2023), as illustrated in Figure 1. While table question answering (QA) (Pasupat and Liang, 2015; lyyer et al., 2017; Zhong et al., 2018; Chen al., 2020c; Nan al., 2022b) has made signif- icant progress in answering fact-based questions, the primary focus of their approaches is on extract- ing relevant facts or entities from the table and composing short-form answers. Nevertheless, in real-world scenarios, users often have more com- plex and diverse information needs that extend be- yond simple fact retrieval. They expect models to perform human-like reasoning and provide trust- worthy explanations or analyses that accompany the extracted insights.
To improve both text generation systems for QT- SUMM, we propose REFACTOR. Given a user query, REFACTOR can retrieve and reason over query-relevant facts from the source table to gener- ate multiple data insights natural language sen- tences. Our results illustrate that directly concate- nating the original input sequence with REFAC- TOR's generation can bring effective improvements to state-of-the baseline systems. We conclude our main contributions as follows:
We propose a new query-focused table sum- marization task, and construct a large-scale benchmark, QTSUMM, comprising 7,111 query- summary pairs collected in real-world situations. Strict quality control measures are employed to ascertain the high quality of the dataset.
We conduct a systematic study of state-of-the-arr models on QTSUMM, and illustrate that they are still far behind expert performance, motivating future research on this new table-to-text task.
With comprehensive consideration of the real- world information needs of users when consulting tabular data, we propose new task, query-focused table summarization. In this task, the model is required to generate a user-customized summary given the table and user query. To enable research this area, we construct human- -anottadd table- to-text generation dataset named QTSUMM', that contains 7,111 query-summary pairs over 2,934 Wikipedia tables covering diverse topics. Table 1 compares QTSUMM with previous table-to-text generation datasets. To the best of our knowledge, QTSUMM is the first dataset that tackles tasks of generating user-customized table summaries based on real-world scenarios.
We present REFACTOR for the efficient retrieval and reasoning of query-relevant facts from tables. It demonstrates significant enhancements pertain- ing to state-of-the-art text generation baselines.
2 Related Work
Table-to-Ter Text Generation As illustrated in Ta- ble I, existing work and datasets on table-to-text generation typically pose the problem either single-sentence generation task (Chen al., 2020a; Parikh et al., 2020; Cheng et al., 2022b; Liu et al., 2022a), or a generic summarization task (Le- bret et al., 2016; Moosavi et al., 2021; Suadaa et al., 2021). In the single-sentence generation task (Parikh et al., 2020; Chen et al., 2020a; Cheng et al., 2022b), the focus is on generating fluent and faithful descriptions using provided table regions as a control for text generation. Nevertheless, using table regions for controlling text generation does not align with real-world scenarios, where people refer to tabular data for information-seeking pur- poses. The generic table summarization tasks (Le- bret et al., 2016; Moosavi et al., 2021; Suadaa et al., 2021) aim to create concise and informative sum- maries based the content of given domain- specific table (i.e., sports scientific). In contrast, the tables in QTSUMM cover diverse topics. Fur- thermore, considering the numerous data points in the table, various users may be interested in dif- ferent aspects for their own informationnseeking
We provide a comprehensive evaluation of cur- rent statc-of-the-art models, including text genera- tion (Lewis et al., 2020; Raffel et al., 2020; Chung et al., 2022), table-to-text generation (Liu et al., 2022b; Zhao et al., 2022b; Jiang et al., 2022), and large language models (Touvron et al., 2023a,b; Zheng et al., 2023; Jiang et al., 2023a; Xu al., 2023; OpenAl, 2023). Our results and analysis from different perspectives reveal that the existing models struggle in solving this new task, highlight- ing the challenges the models face when perform- ing human-like reasoning and analysis to generate summary tailored users' information needs.
Wee released the dataset at https:// /huggingfacce co/ datassess//eea settss/alleeeeee syale-nlp//ee /rrumm using "gated repos- itories" to protect the data from automatic crawling (Jacovi
ct al., 2023).

---

Words /
Rich Analysis
Daraset
Table Source
Tables Statements
Explicit Control
Statement
& Reasoning?
Single-sentence
Iuble-to-Tet
ToTTo (Parikh et al., 2020)
Wikipedia
83,141 83,141
17.4
Table region
LOGICNLG (Chen et al. 2020a)
Wikipedia
7,392 36,960
14.2
Table regions
HiTab (Cheng al., 2022b)
Statisties web
3,597 10,672
16.4
Tahle regions reasoning operator
Generic Table Summarization
RoToWIRE (Lebret et al., 2016)
NBA gumes
4 4,953
SciGien (Moosavi 2021)
Sci- SeiePa
1,338/ 1,338
116.0
NumericNLG (Suadan al., 2021)
Sci-Paper
1
1,355
94.2
x
Table Question Answering
FeTaQA (Nan al., 2022b)
Wikipedia
10,330 10,330
18.9
Queries rewritten from ToTTo
x
Query-Focused Table Summarization
QTSUMM
Wikipedia
2,934/
7,111
68.0
Queries from real-world scenarios
Table 1: Comparison between QTSUMM and existing table-to textext generation datasets.
purposes, making it challenging to create a generic summary that encompasses all the salient informa- tion within the table. Therefore, in this paper, we propose and investigate query-focused summarization. FeTTaQA (Nan al., 2022b) is a table QA dataset that collects queries by rewriting ToTTo's (Parikh et al., 2020) statements into questions and uses the same state- ments as the answers. In comparison with FeTaQA, the queries in QTSUMM were annotated under real- world scenarios, making them more natural and better-reflccting users' actual information needs.
to generate summaries from documents that are tai- lored to specific user queries (Dang, 2006). Despite its potential real-world applications, QFS remains a challenging task due to the lack of large-scale train- ing data. Existing works have attempted to address this issue by leveraging distant NLP resources, in- cluding question answering (Xu and Lapata, 2020) and paraphrase identification (Su et al., 2020), and generic summarization (Xu and Lapata. 2022; Zhou et al., 2023). Recently, Zhong et al. (2021) adopted QFS for meeting summarization and proposed human-annotater benchmark over meeting tran- scripts. Similar to text, effectively accessing and comprehending the information contained within large and complex table can be time-consuming for users, while QFS remains unexplored table- to-text generation. In this work, we extend QFS to this new modality for more effective information- seeking and decision-making purposes.
task setting related
Reasoning Over Tabular Data Enhancing the table reasoning capabilities of models is essential for a variety of tasks related tables, such as ta- ble question answering (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2018; Zhao et al., 2023d), table fact verification (Chen et al., 2020b), and table-to-text generation (Chen et al., 2020a; Cheng et al., 2022b). One prevalent approach is pre-training models with table-text joint reasoning data (Herzig et al., 2020; Liu et al., 2022b; Zhao al., 2022b: Liu al., 2022a; Jiang et al., 2022; Dong et al., 2022: Cheng al., 2022a; Xie al., 2022). Nevertheless, these models generate text in an end-to-end manner, resulting in reduced explain- ability and difficulties in handling more complex reasoning, such as arithmetic calculation. There- fore, propose REFACTOR, which can retrieve and generate query-relevant facts from tables as intermediate results for model input (Zhou al., 2022: Zhao et al., 2023b), mitigating the implicit reasoning processes of text generation models.
Query-Focused Table Summarization
3.1 Problem Formulation
We formally define the proposed query-focused table summarization task as follows. The input is user query Q, and table T. The table T W U (Ti Rr.j CT} has Ry rows and Cr columns, with W being the table title and Tıy being the textual content in the (i. cell. The task objective of QTSUMM is to generate paragraph- long textual summary Y = (y1, (y1.32.... ./.....,hh given the user query Q and source table T:
Query-Focused Summarization Initially formu- lated as document summarization task, QFS aims
Y argmaxx I Punlues Q,T:O), (1)

---

where 0 denotes the parameters of neural text generation model, and y denotes the i-th tokens in the generated summary.
Property
Value
Unique Tables
2,934
Query-Summary Pairs
7,111
Rows per Table (Kedian/Avg)
10/ 11.8
Columns per Table (Median/Avg)
6/6,6
3.2 Data Collection Principles
Table Title Leugth (Median/AAvg)
7/7.6
At high level, the goal of the data collection pro- cess is to obtain high-quality user queries and cor- responding paragraph-long summaries grounded on the tabular data. We outline our key criteria for designing benchmark to thoroughly evaluate the table-to-text summarization capabilities models. To achieve this, we first design three principles for annotating good query-summary pair:
Query Length (Xedian/Avg)
22//2.3
Relevant Rows (Yedian/Awa)
3.3.8
Summary Length (Xedian/Avvg)
63 68.0
Training Set Size (Table/ (Tablle/Summary) Development Set Size (Table/ Summazy)
2,055/4,98 /49981 (70%)
439 1,052 (15%)
Test Set Size (Table/Summary)
1,078 (15%)
Table Basic statistics of QTSUMM dataset.
as query responses, we avoid queries that can be answered in a short sentence (e.g.. "Which country held the 2022 FIFA World Cup?").
Comprehensivenes::
The tailored summary should provide enough details and analysis of the source table respond to the user query, ful- filling user's information need.
Query-Focused Summary Annotation Given table and user query, ask another annotator to only information from the source table write a paragraph- summary that satisfies the user's information need. We encourage annotators to produce sophisticated summaries that 1) contain as much information from the table as possible, and 2) involve more types of reasoning over multiple relevant table regions. To further encourage high quality annotations, we adopt the "two channel col- lection" design (Chen et al., 2020b), in which the annotators would be paid 60% more if their sum- maries are manually verified to exhibit adequate complexity. We also require the annotators to anno- tate the row indices of relevant table regions that are referenced the written summary, allowing future researchers to quantify how well the summaries are grounded in the table in their work.
Attributablity & Faithfulness: The query should be answerable using only information from the source table. The summary should be grounded on the source table, and not contain any unfaithful or nonsensical text.
Fluency: Both the user query and its correspond- ing table summary should be coherent and fluent.
3.3 QTSUMM Annotation Pipeline
To ensure that QTSUMM annotation fulfills the aforementioned principles, carefully design annotation pipeline consisting of following steps:
Source Table Collection QTSUMM uses ta- bles from LOGICNLG (Chen et al., 2020a) and ToTTo (Parikh et al., 2020) datasets source ta- bles, as these tables are crwaled from Wikipedia and covers diverse domains and topics. We fil- ter out tables that are 1) too large or too small, 2) with only string-type columns, or 3) with hierar- chical structures (e.g., containing more than one table header). Then we randomly sample 2,000 candidate tables from LOGICNLG and ToTTo, respectively, for the query-summary annotation.
Multi-Round Validation We conduct multi- round validation protocol to ensure that the anno- tated data fulfills the aforementioned annotation principles. We first assign query annotators to val- idate each summary against their corresponding queries, and fix the mistakes if there are any. Then we check 1) whether a query-summman tain adequate information and complex aggregation by examining the length of the summary, and 2) whether the information in summary is essential in responding the user query. We manually revise pairs that do not meet the above standard.
pair con-
User Query Annotation Given a table, the an- notators are required to read its content, and deter- mine whether the table is informative and intelligi- ble to common web users. Then they were asked three queries, assuming they are users seeking certain information from the table. We require each query to be answerable us- ing information only from the table. Morcover, as this work focuses on paragraph-long summaries
come up with
3.4 Annotation Quality Control
Table describes the basic statistics of QTSUMM. In addition to the multi-round validation, we care- fully design several quality control approaches,

---

comprising expert annotation and numerous anno- tation de-biasing designs, to ensure the high quality QTSUMM annotations.
Dtera
3.5%
Educvtion
2,5%
Sporss
Ssience
27.0%
To help improve the annota- tion process, five experts with professional expe- rience the text summarization tasks are invited conduct the internal annotation. They are asked to provide feedback regarding the task instructions and the user experience of the annotation interface, based on which we iteratively modify the annota- tion guideline and interface design. In the stage of external annotation, enroll 17 graduate students majoring in STEM fields (10 females, and 7 males). We do not the crowd-source annotation plat- form such as Mechanical Turk as our preliminary study indicates that annotators on MTTurk fail to annotate high-quality query-summary data. Before starting the official annotation process, each anno- tator is given two-hour training session to learn the annotation requirements and interface.
Expert Annotators
8.0%
Pollcc
R00%
Celebrity
Figure Domain distribution QUSUMM tables.
Kappa /95%
Annotation Quality
%65 2 4 Agree
Table Infernativenees
84,9
0.77 (0.7/ 0.82]
[0.72,
Meuningfulness
93.2
0.89
0.84/ (0.79, 0.89)
Query Complexity
91.4
0.87
0.81 [0.75, 0,87]
Query Fluency
97.2
0.94
0.92/ [0.90, 0.94]
Relevant Rows Correctuess
89.7
0.85
0.83 [0.79, 0.88)
Summary Comprehensiveness
[0.90, 0.96]
Summary Faithfulness
91.6
0.90
0.88 [0.84, 0.92]
Annotation De-biasing We observed several kinds of annotation bias during our internal annota- tion, and we proposed countermeasures as follows for annotation de- de-biasing:
Summary Fluency
96.1
0.93
0.89 (0.86, 0.92]
Table Human evaluation over 200 samples of QT- SUMM. Three internal evaluators were asked to the samples scale of 1 to 5. We report 1) percent of samples that have an average score to indicate the annotation quality of QTSUMM; and percent of agreement and Randolph's 's Kappa with 95% CI (Ran- dolph, 2005) to show the inter-annotator agreement.
Source Table Diversity: During internal annota- tion, found that many tables in LOGICNLG have similar content. For example, there are around 200 tables describing the results football games, with identical table headers. To ensure the diversity source tables, we keep only one table for each unique table header.
quality and inter-annotator agreement (Table 3).
Query Diversity: When annotating queries, an- notators may prefer simpler ones, resulting in low frequently monitor the diversity of queries for each annotator. An- notators are also encouraged to craft queries that are either creative or require complex reasoning in summarization, resulting in a doubled payment to compensate them for the extra time.
3.5 QTSUMM Evaluation
We develop a comprehensive approach for evaluat- ing QTSumm, incorporating both automated and human evaluation. We adopt following popular automated evaluation metrics:
query diversity. Therefore,
BLEU (Papineni et al., 2002) computes the geo- metric average of the precision over output text's n- grams. We used SacreBLEU (Post, 2018) that pro- duces comparable and reproducible BLEU scores. ROUGE (Lin and Hovy, 2003) measures the word overlap between the candidate and reference sum- maries. We reported F1 score for ROUGE-L (longest common subsequences).
Supporting Fact Position: We found that annota- tors prefer raise queries regarding the first few rows of each table. To deal with such bias regarding supporting fact positions, certain rows for each table the annotation inter- face. We require the annotators to write queries whose summaries should cover at least two rows of the highlighted regions.
randomly highlight
METEOR (Banerjee and Lavie, 2005) is based generalized concept of unigram matching between the machine perhiiee--ruueed translation and hroducproduued reference translations.
We also report the human evaluation scores and inter-evaluator agreements over 200 sampled query- summary pairs. QTSUMM has high annotation
BERTScore (Zhang et al., 2020) computes the sim-

---

ilarity between the reference and generated sum- mary using contextual word embeddings.
REFACTOR
Fact Generation
MUN
14:00
Fact Banking
TAPAS-Acc (Herzig et al., 2020; Liu et al., 2022a) is a reference-free metric that uses TAPAS (Herzig et al., 2020) fine-tuned on the Tab- Fact dataset (Chen et al., 2020b) as the backbone to evaluate the faithfulness of generation.
Amrinc Couy
126
Coouare Naae solanal Profi
CVC
10.00
20101
13/8
The Doepay hew. was bdkary
11390
A/ Coner
a- Comag, No Henuoor
The
Deary: Which
hisher
longo
pottt the DI and Cas indestry and how
difenncc terween apple
soue coequri 12: the moe prstiable
AutoACU (Liu et al., 2023a) is an interpretable and Sference-bbssed summarization evaluation system that exhibits better alignment with human judge- ments. The A2CU first extracts atomic content units (ACUs) from the generated summary and then evaluates them against reference. A3CU is an ac- celerated version of A2CU that directly computes the similarity between two text without extracting ACUs, but with the similar evaluation target. We use Fl score of A3CU for evaluation.
Baseline Models (e.g. g., Flan-TS, PLOG)
Error-prone rewsoring
National Pstroleum tamta lishtd prsit the profraita corpany
topur
$4,575
Walmart
millee X
Goop..
Explici and faitiful reasoning by RiFACTOR
Witha the dl und tos induay, Siaees Graad was ligho wott- Sigts pade cored vaige Caeq is rean lover Apyle so1.ig suoce poiti Stacpe Crrag.
Modak vr, NIPACTOR inpur
Figure 3: Enhancing fine-tuned models with the pro- posed REFACTOR. After generating and selecting the top-n query-relevant facts obtained through various rea- soning operations (e.g., numerical comparison, count- ing), these facts are concatenated with query and table data as the model input both fine-tuning and inference stage. REFACTOR can mitigate the error-prone implicit reasoning issues of end-to-end text generation systems. For LLM in zero- few-shot setting, we provide gener- ated facts within the prompts (Figure 5 in Appendix A).
For human evaluation, the summaries from dif- ferent models were evaluated by experts from three criteria (i.e., comprehensiveness, faithulness, and fluency) that have been discussed in Section 3.2. Each summary was scored from (worst) to (best) for each criteria, with the final score averaged across different evaluators.
4 REFACTOR
ble. Specifically, we define 6 types of table reason- ing operations (e.g., numerical operation, counting. and conjunction) that are necessary for the QT- SUMM task, as shown in Table in the Appendix. For each reasoning operation, the fact generator (adopted from Zhao et al. (2022b)) takes table and a query as input. It produces multiple facts based on the fact template. Each fact template in- cludes several placeholders that need to be filled with information retrieved from the table. Specifi- cally, column col and cell value val are indexed to specify the column and cell name, respectively. Some templates also regulate that the selected col- umn and cell value must be date or number type. OP OPERATOR corresponds to operators that are in- stantiated according to the specific reasoning rea- soning. And CONDITION : i can be 1) a cell value from the i-th column; or 2) number/temporal comparison statement if the i-th column is date or number type. After substituting all the placeholders in the provided template, the fact generator will pro- grammatically return the executed resserrrrrssteeeeeeeeeeee and form one fact. Once facts for a [table, query) pair are collected from different fact generators, we pass them to the Fact Ranking process.
QTSUMM requires models to perform human-like reasoning in generating summaries that provide comprehensive and precise analysis of the source table to fulfill the user's information need. How- ever, existing end-to-end text generation models rely on error-prone implicit reasoning processes for generating text, leading to diminished explainabil- ity and challenges in addressing user queries that necessitate complex human-like reasoning (Zhou et al., 2022: Zhao et al., 2023b). To address this, we present REFACTOR, to retrieve and reason over query-relevant information from tabular data to generate several NL. data insights (i.e., facts) as explicit reasoning results. As shown in Figure 3, the generated facts is concatenated to the model input to mitigate the implicit reasoning issues, en- hancing the comprehensiveness and faithfulness of generated summary. We next discuss the imple- mentation of REFACTOR.
4.1
Fact Generation
Given the user query and source table, REFACTOR will generate several candidate facts by executing various forms of human-like reasoning over the ta-

---

4.2 Fact Ranking
5.1.3 Large Language Models
Llama-2' (Touvron al., 2023a,b) is open- source large language model trained on large-scale and publicly available datasets.
Given the query and source table, each fact gen- erator will be utilized generate several query- relevant facts, resulting
large number of candi- date facts in total. Therefore, we need to rank the generated facts to seleet the most relevant ones. We use the QA encoding model (Reimers and Gurevych, 2019) to obtain the embedding of the query and each generated fact. Then, we select the top-n generated facts with the highest cosine similarity to the query embedding. In practice, assign n as mar( row colummaum 5),
Vicuna" (Zheng al., 2023) tuned from Llama-1 with instruction-folloiinn data, exhibiting better instruction-following capabilities.
Mistral (Jiang et al., 2023a) a 7-billion- parameter LLM that outperforms Llama-2-13B across most of popular evaluated benchmarks.
Lemur (Xu et al., 2023) is tuned from Llama-2 with instruction -oiructtion-folowiingg data, exhibiting compet- itive natural language and coding capabilities.
2
and ensure that the number of selected facts from each type of reasoning operation does not exceed 3. The selected facts, which are handy and readily available for end-to-end text generation systems, are then concatenated into the model input.
GPT (Brown et al., 2020; OpenAI, 2023) is pow- erful large language model which is capable of generating human--lie text and performing a wide range of NLP tasks in a few-shot setting.
5.2 Experimental Setup
5 QTSUMM Experiments
The specifics of input data serialization and LLM prompting examples are discussed in Appendix A. All experiments were conducted on an NVIDIA RTX A6000 48GB cluster. We selected the 1 large version for all fine-tuned baseline models, whose weights are publicly available at HuggingFace, For each fine-tuming experiment, ran 15 epochs with batch size of 128. The best fine-tuning check- points were selected according the validation loss. The experiments for open-sourced LLMs were conducted using vLLM framework (Kwon et al., 2023). We used gpt-3. 5- turbbo-0613 for GPT-3.5 and gpt 4- 0613 for GPT-4 via the OpenAl APIs7. For LLM hyperparameter settings, set temperature as 1.0, Top P as 1.0, and maxi- mum output length as 256.
5.1
Baseline Systems
We evaluate the following three types state-of- the-art baseline systems on QTSUMM:
5.1.1 Text Generation Models
BART (Lewis et al., 2020) pre-trained denois- ing autoencoder with transformer- based architec- ture and shows effectiveness in NLG tasks.
T5 (Raffel et al., 2020) demonstrates effectiveness in NLG tasks by treating all NL problems as text- to-text tasks during pre-training stage.
Flan-T5 (Chung et al., 2022) enhances T5 by scal- ing instruction fine-tuning and demonstrates better human-like reasoning abilitics than the T5.
5.1.2 Table-to-Text Generation Models
5.3 Main Results
TAPEX (Liu al., 2022b) continues pre-training the BART model by using a large-scale corpus of synthetic SQL query execution data. It shows better table understanding and reasoning abilities.
We draw following conclusions based on the auto- mated and human evaluation results (Table 4 & 6).
Importance of table structure understanding Table-to-text generation models achieve better per- formance than their corresponding text-generation
ReasTAP (Zhao et al., 2022b) enhances the table understanding and reasoning abilities of BART by pre-training synthetic Table QA corpus.
hhttps://hggg nngf co/meta- lama/ 11ama httsssieeebchaeeeeee
OmniTab (Jiang et al., 2022) uses the same back- bone as TAPEX, and is further pre-trained on col- lected natural and synthetic Table QA examples.
"We only evaluate Vicuna (ht //tuggingface
co/ Imsys/vi wwwiwwiunna------ shot settings, as some examples under the two-shot setting might exceeds its maximum length limit.
3) under zero- and one-
"mist aasstralai/isssralllll 7 Instruct-v0.1 'https: http:/::::///ggggggggf//aa co Opent.mmur/
WW released the model weights of evaluated fine-tuned models at HuggingFace (https: : /hugg acc..ce.e. co/ yale-nlp p/ (model_nanee name: finetuned qt summ).
Lemur-70b-chat-y comp/pp.cc/pppppp//

---

Type
Model
Backhone
Avg Len
BLEU
ROUGE-L
METEOR
BERTScore
TAPAS-Ace
A3CU
Ground Truth
67.8
T5-large
61.8
20.3
38.7
40.2
89.6
75.1
43.5
Text Geacration
Flan- -S-arge
TS
74.1
19.9
39.8
42.5
89.8
83,9
46,3
lage
60.1
21.2
40.6
43.0
90.6
77.1
48.0
10. REFACTOR
61.4
21.5 (40.3)
41.0 (40.4)
43.1 (10.1)
90.1 (-0.5)
79.4 (+2.3)
48.6 (-0.6)
ReasTAP
BART
61.4
22.5
41.9
44.3
90.8
80.6
51.9
TAPEX
BAKT
70.1
23.1
42.1
45.6
90.6
87.8
52.0
fine-tuning
OmmiTah
22.4
80.2
REFACTOR
BART
22.5 (+0.1)
(-0.2)
44.8 (+0.1)
90.7
80.3 (+0.1)
54.0 (+0.9)
LJama-2- 2-138
64.3
14.6
25.5
30.9
86.8
76.6
28.6
Llama-2-7B
110.3
13.3
31.3
42.5
88.8
78.1
37.3
Mistral-7B
Llama-2
98.4
13.8
31.7
41.4
89.1
73.0
37.5
s:. REFACTOR
Llama-2
99.0
13.8 (+0.0)
31.4 (-0.3)
41.5 (+0.1)
88.7 (-0.4)
(41.))
37.7 (+0.2)
Vicuna-333
Lemur-70B
Llama-2
102.3
13.3
30,9
39.9
87.8
82.8
40.8
L.M 20m0 shot
Llama-2-70B
91.5
17.2
35.2
44.1
89.8
85.7
45.7
w.. REFACTOR
98.1
17.0 (-0.2)
34.7 (.0.5)
44.6 (+0.5)
90.0 (+0.2)
82.3 (-3.4)
46.3 (+0.6)
GPT-3.5
82,5
21.1
40.7
49.1
91.1
89.7
55,5
sn. REFACTOR
85.4
20.6 (-0.5)
39,8 (-U.9)
49.2 (+0.1)
90.5 (-0.6)
89.9 (+0.2)
55.9 (+0.4)
GPT.4
REFACTOR
19.6 (-0.2)
(-0.)
(-0.3)
87.1
92.3 (+0.0)
57.5 (+0.0)
Llama-2-11
13.8
81.4
26.5
Mistral-7B
Llama-2
13.7
31.5
71.7
36.7
REFACTOR
Llama
94.2
14.1 (+0.4)
31.8 (+0.3)
40.7 (-0.)
88.9 (+0.0)
72.2 (+0.5)
38.2 (+1.5)
Llama-2-7B
105.0
13.6
32.3
42.5
89.1
75.3
38,5
Lemur-70BB
Llama-2
86.5
14.3
31.5
38.3
88.1
81.3
39.8
Vicuma-33b
Llamu-1
75.0
19.3
37.0
43.8
90.1
78.4
45.3
LLM I-shot
Llama-2-778
18.2
37.3
46.2
86.5
REFACTOR
(-0.1)
(-0.3
(+0.0)
(+0.1)
86.7 (+0.2)
(+0.2)
GPT-3.5
88.0
20.2
40.0
49.7
90.9
91.7
55.6
REFACTOR
85.2
20.3 (+0.1)
39.8 (-0.2)
50.0 (+0.3)
91.2 (+0.3)
92.2 (+0.5)
57.0 (+1.4)
GPT-4
92.1
19.0
39.9
51.2
91.0
94,3
60.1
w. REFACTOR
89.4
19.5 (+0.5)
40.0 (+0.1)
51.4 (+0.2)
91.3 (+0.3)
93.7 (-0.6)
61.3 (+1.2)
1llma-2-13B
17.5
81.2
Mistral-7B
Llama-2
14.9
32.7
89.
72.8
38.4
Llama-2-78
99.3
14.0
33.2
42.3
89.0
77.9
39.6
L.M 2-shor
Lemur-70B
Llama-2
82.7
15.0
32.0
38.5
88.4
81.6
40.6
Llama-2-70B
Llama-2
87.3
19.0
38.0
46.4
90.4
87.3
49.1
GPT:3.5
89.8
20.0
39.9
50.0
90.9
93.2
56.2
GPT-4
90.1
19.5
93.3
Table 4: Automated evaluation results on the QTSUMM set, involving three types of baseline systems with and without REFACTOR. We used chat instruct version for each type of LLMs. Within each experimental setting, we used A3CU (F-score) as the ranking indicator of model performance. Due to the budget constraints, for all LLM REFACTOR experiments, we randomly selected 200 samples.
Examples
Eror Types
Representative Question
Explanation
200 Difficulty in parsing cell values via rule-
The relevant numeric- or time-type columns hard parse (e.g., multiple numbers and within cell), thus REFACTOR fail generate related facts.
based methods
17 200
Complex user query causes difficulty in ranking related facts
Analyze the correlation between size of the geographical area of Gmina type and its popula- tion?
REFACTOR employs the QA encoding model for fact ranking. However, it struggles to understand complex information needs from users, such "correlation between A and B", and might conse- quently rank irrelevant facts higher.
13 200
Unsupported soning operations
rea-
Who are the top three coaches with the highest win percent- ages? Analyze their perfor- mance in the 2019-2020 season.
The table only contains "wins" and "overall games" columns. Models must compute winning per- centages independently. However, REFACTOR does support such rate calculations
//0
Other errors
141 /200
Successful cases
Table 5: Case study on REFACTOR'S failure cases.

---

Model
Faithfulness Compre.
Fluency
We identify four common mistakes that current text generation models are likely to make (i.e., halluci- nation, factual incorrectness, user intent misun- derstanding. and repetition), providing detailed examples and explanations for each type of com- mon mistake in Table 8 in the Appendix.
BART
3.26
3.67
4.56
w. REFACTOR
3.37 (+0.11)
3.72 (+0.05)
4.59 (+0.03)
OmniTab
3.30
3.58
4.52
W. REFACTOR
3.45 (+0.15)
3.69 (+0.11)
4.52 (+0.0)
1-shot Mistral-7B
2,98
3.77
4.65
W. REFACTOR
3.12 (+0.14)
3.82 (+0.05)
4.52 (-0.13)
1-shot Lluma-2-70B
3.08
3,82
4.69
REFACTOR
3,36 (+0.28)
3,99 (+0.17) 4.66 (-0.03)
5.5 REFACTOR Analysis
O-shot GPT-3.5
3.65
4.66
ut REFACTOR
3.84 (+0.19)
4.03 (+0.09)
4.74 (+0.08)
We also undertake human evaluation examine the efficacy of REFACTOR in generating query- relevant facts from tabular data. Specifically, we randomly sample 200 examples from QTSUMM validation set, and ask two human evaluators to evaluate each fact generated by REFACTOR, deter- mining its relevance to the query. 56.4% generated facts (528 out of 937) are labeled as "relevant", sug- gesting an adequate coverage of REFACTOR. To delve deeper into this, we also conduct a case study examining the failure cases, specifically those ex- amples where less than two facts were annotated as "relevant". We identified three kinds of common failure cases: (1) difficulty in parsing cell values via rule-based methods, (2) complex user query causes difficulty in ranking related facts, and (3) unsup- ported reasoning operations. We provide detailed examples and explanations in Table 5.
O-shot GPT 1
3,92
4.12
4.84
w. REFACTOR
4.08 (+0.16)
4.15
(+0.03)
4.70 (-0.14)
1-shot GPT-3.5
3,84
4.20
4,86
W. REFACTOR
3,95 (+0.11)
4.27 (+0.07)
4.84 (-0.02)
1-shor GPT-1
4.11
4.32
4.88
w. REFACTOR
4.08 (-0.03)
4.35 (+0.03) 4.76 (-0.12)
Table 6: Human evaluation results (Likert Scale Scor- ing) of selected baselines on the test set. Five experts are enrolled evaluate 50 predictions for each model.
backbones, demonstrating the importance of con- sidering table structure for the QTSUMM task.
Importance of reasoning and analysis Among text generation models, Flan-T5, which enhances T5 through scaled instruction fine-tuning, outper- forms T5. Moreover, LLMs with improved reason- ing capabilities (i.e., Llamm-2-70B and GPT-4) also achieve better performance. These findings indi- cate the significance of reasoning and analytical skills in handling the QTSUMM task.
Conclusion
This paper defines a new query-focused table sum- marization task, and constructs a large-scale bench- mark, QTSUMM. We investigate a set of strong baselines, including text generation, table-to-text generation, and large language models. Experi- mental results and manual analysis reveal that the new task presents significant challenges in table- to-text generation. Moreover, we propose novel approach named REFACTOR, to retrieve and rea- son over query-relevant information from tables, improving the faithfulness of generated summary.
Mismatch between automated and human eval- uation Despite receiving low scores in popular automated evaluation metrics such as BLEU and ROUGE, GPT-* cxhibit better performance than state-of-the-art fine-tuned models in human evalua- tion. This finding underscores the need for future research to investigate the development of auto- mated evaluation metrics for the QTSUMM task that better align with human judgments (Zhang and Bansal, 2021; Liu et al., 2023a; Jiang et al., 2023b).
Acknowledgements
Effectiveness of REFACTOR As assessed by human evaluation, baseline systems employing REFACTOR typically yield better performance, es- pecially in faithfulness-level. This suggests the efficacy of REFACTOR in enhancing the reasoning process in text generation.
We would like to dedicate this paper to the memory of Dr. Dragomir Radev. Dr. Radev's leadership. guidance, and expertise were instrumental in shap- ing the direction and quality of this project. We ap- preciate the efforts of all annotators in constructing QTSUMM and conducting human evaluation. We are grateful the Google TRC program for their support. We would also like to thank the anony- mous reviewers and action editors for constructive discussions and feedback.
5.4 Error Analysis
For a deeper understanding the query-focused ta- ble summarization task on QTSUMM, we conduct an error analysis to illustrate existing challenges.

---

Limitations and Future Work
ToTTo (Parikh al., 2020) datasets, which are publicly available under the MIT license and CC BY-SA 3.0 license", respectively. They both per- mit us to compose, modify, publish, and distribute additional annotations upon the original dataset.
The baseline systems provided have a restricted maximum number of tokens they can accommo- date (c.g., 1024 for all examined fine-tuned mod- els), which prevents them from generating sum- maries for large tables that, when converted into sequence, exceed the maximum number of to- kens. To handle large tables (e.g., with more than 300 table cells), future work can apply neural mod- els (Herzig et al., 2020; Liu et al., 2022b) to first filter out those query-irrelevnt rows or columns. Moreover, this paper demonstrates the effective- ness of using intermediate results obtained from ex- plicit reasoning operations to mitigate the implicit reasoning issues. However, the proposed REFAC- TOR utilizes template-base based method generate facts. Although such complate-based approach can ensure the factual correctness of generated facts, as discussed in Section 5.5, it might not cover all crucial facts for some complex user query. We believe following directions warrant further explo- ration: (1) Complex query decomposition. Our case study reveals that the TAPEX- fact rank- ing module struggles with comprehending complex questions. To address this, future research could in- vestigate LLM chain-of- of-touhf methods to break down complex questions into more understandable and actionable sub-questions. (2) Tool usage. The predefined and template-based execution modules in the REFACTOR fact generation phase have their limitations. Recent studies (Schick et al., 2023; Lu et al., 2023; Paranjape et al., 2023; Gou et al., 2023; Qiao al., 2023) highlight the impressive abilities of L.LMs in making and utilizing tools for problem-solving. would be intriguing to explore if LLMS can produce executable programs from scratch to derive query-relevant insights. (3) Ex- plainable automated evaluation. In Section 5.3, a discrepancy between automated and human eval- uation results is observed. Such discrepancies are concerning, developers might opt for suboptimal systems for real-world applications if they solely rely on automatic metrics for comparing and rank- ing different text generation systems. Therefore, a more reliable and explainable automated evalua- tion system is required (Zhang and Bansal, 2021; Liu et al., 2023a,b; Jiang et al., 2023b).
For the external annotation of QTSUMM, we hired 17 graduate students majoring in STEM ma- jors. We regard 1) creating three queries for table, and validating the corresponding summaries annotated by others, and 2) composing a query- focused summary response as a unit task. And wc paid around $1.5 for each unit task. For cre- ative annotation rewards, we paid additional $0.5 for a query, and $1.5 for a summary. Averagely, an annotator can finish 7 unit tasks per hour after training and practicing. And the hourly rates are in the range of $9 and S13 based on the different working speed (above the local average wage of similar jobs). We recommended that annotators complete a maximum of 30 unit tasks per day or- der to reduce pressure and maintain a comfortable pace. In total, the approximate working hours to annotate QTSUMM dataset was 1,400 hours. The whole annotation work lasted about 40 days.
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with im- proved correlation with human judgments. ceedings of the ACI. Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sammarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguis- tics.
Pro-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems. volume 33, pages 1877-1901. Curran Associates, Inc.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural lan- guage generation from open-domain tables. In Pro-
Ethical Consideration
"nttps://opensouree" org/ wttgg//cceeee///e /MIT 'htpps: / vecoomonss wwwrmmoonsosoorroooooo wrt//1/cn////////////////// censes/ by-sa/3.0/
The source tables in QTSUMM were collected from LOGICNLG (Chen et al., 2020a) and

---

ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7929- 7942, Online. Association for Computational Lin- guistics.
Zhibin Gou, Zhihong Shao. Yeyun Gong, Yelong Shen, Yujiu Yang. Nan Duan, and Weizhu Chen. 2023. Critie: Large language models can self-correct with tool-interactive critiquing.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-based fact verification. In Inter- national Conference Learning Reprexentations (ICLR), Addis Ababa, Ethiopia.
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. pages 4320 Online. Association for Computa- tional Linguistics.
Wenhu Chen, Hanwen Zha. Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Wang. 2020c. Hybridqa: dataset of multi-hop question answering over tabular and textual data. Findings of EMNLP 2020.
Matthew F. Hurst. 2000, The interpretation of tables in texts.
Mohit lyyer, Wen-tau Yih, and Ming- Wei Chang. 2017. Search- neural structured learning for sequen- tial question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1821- 1831, Vancouver, Canada. Association for Computa- tional Linguistics.
Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, and Dongmei Zhang. 2022a. Using formulas for numerical-reasoning aware table pretraining. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 1150- 1166, Dublin, Ireland. Association for Computational Linguistics.
FORTAP:
Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contami- nation by evaluation benchmarks.
Zhoujun Cheng. Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian- Lou, and Dongmei Zhang, 2022b. HiTab: A hicrarchical table dataset for question answering and natural language generation. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1094-1110, Dublin, Ireland. Association for Computational Linguisties.
Albert Q Jiang, Alexandre Sablayrolles. Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, et al. 2023a. Mistral 7b. arXiv preprint 2xXx:2331006880
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li. Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scal- ing instruction-fine finetuned language models. ArXiv. abs/2210. 14416.
Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. 2023b. Tiger- score: Towards building explainable metric for all text generation tasks.
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neu- big, and Weizhu Chen. 2022. OmniTab: Pretraining with natural and synthetic data for few-shot table- based question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 932-942, Seattle, United States. Association for Computational Lin- guistics.
Hoa Trang Dang. 2006. DUC 2005: Evaluation of focused summarization systems. In Pro- ceedings of the Workshop on Task-Focused Summa- rization and Question Answering, pages 48-55, Syd- ney, Australia. Association for Computational Lin- guistics.
Karen Kukich. 1983. Design knowledge-basee report generator. In 21 Annual Meeting of the As- sociation for Computational Linguistics, pages 145- 150, Cambridge, Massachusetts. USA. Association for Computational Linguistics.
Haoyu Dong, Zhoujun Cheng, Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dong- mei Zhang. 2022. Table pre-training: A survey on model architectures, pre-training objectives, and downstream tasks. Proceedings of the Thirty-First International Joint Conference on Artificial Intel- ligence, IJCAI-22, pages 5426-5435. International Joint Conferences on Artificial Intelligence Organi- zation. Survey Track.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu. Joseph E. Gonzalez, Hao Zhang, and lon Stoica. 2023. Effi- cient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.

---

Rémi Lebret, David Grangier, and Michael Auli. 2016. Neural text generation from structured data with ap- plication to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 1203-1213, Austin, Texas. Association for Computational Linguisties.
Robust data-to-text with replacement detection. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6903-6917, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
2022a, R2D2:
Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryáciński. Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022b. FeTaQA: Free-form table question answering. Trans- actions of the Association for Computational Linguis- tics, 10:35-49,
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to te-euueeee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics, pages 1-7107 Online, Association for Computa- tional Linguistics.
OpenAL. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
Chin- Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North American Chapter of the Association for Computational Lin- guistics, pages 150-157.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Ao Liu, Haoyu Dong, Naoaki Okazaki, Shi Han, and Dongmei Zhang. 2022a. PLOG: Table-to-logic pre- training for logical table-to-text generation. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 5531- 5546, Abu Dhabi, United Arab Emirates. Association for Computational I Linguistics.
Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke 7ettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi- step reasoning and tool-use for large language mod- els.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022b. TAPEX: Table pre-training via learning neural SQL executor. In International Conference on Learning Representations.
Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann. Man- aal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipan- jan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP)), pages 1173-1186, Online. As- sociation for Computational Linguisties.
Yixin Liu, Alexander R. Fabbri, Pengfei Liu, Yilun Zhao, Linyong Nan. Ruilin Han. Simeng Han, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. 2023a. Revisiting the gold stan- dard: Grounding summarization evaluation with ro- bust human evaluation.
Panupong Pasupat and Percy Liang. 2015. Composi- tional semantic parsing on stmi-structured tables. In Proceedings of the 53rd Annual Meeting of the As- sociation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1470- 1480, Beijing, China. Association for Computational Linguisties.
Yixin Liu, Alexander R. Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu. Caiming Xiong, and Dragomir Radev, 2023b. Towards interpretable and efficient automatic reference- based summariza- tion evaluation.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1 186- 191, Belgium. Brussels. Association for Computa- tional Linguistics.
Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. Chameleon: Plug-and-play com- positional reasoning with large language models.
Jay Pujara, Pedro Szekely, Huan Sun, and Muhao Chen. 2021. From tables to knowledge: Recent advances in table understanding. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. KDD '21, page 4060-4061, New York, NY, USA. Association for Computing Machinery.
Nafise Sadat Moosavi, Andreas Rücklé Dan Roth, and Iryna Gurevych. 2021. Scigen: dataset for reasoning-as aware aware text generation from scientific ta- bles. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track (Round 2).
Shuofei Qiao, Honghao Gui, Huajun Chen, and Ningyu Zhang. 2023. Making language models better tool learners with execution feedback.
Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, and Dragomir Radev.

---

Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (174)
Naman Goyal, Anthony S. Hartshorn, Saghar Hos- seini, Rui Hou, Hakan Inan, Marein Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Pumit Singh Koura, Marie- Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molyhog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, llyann Zarov, Yuchen Zhang, An- gela Fan, Melanie Kambadur, Sharan Narang, Aure- lien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models.
Justus Randolph. 2005.
Free-marginal multirater kappa (multirater [free]]: An alternative to fleiss' fixed marginal multirater kappa. Advances in Data Analysis and Classification. 4.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP),, pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong. Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida 1. Wang. Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caim- ing Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zemtemoyer and Tao Yu. 2022. Unified- skg: Unifying and multi-tasking structured knowl- edge grounding with text-to-text language models. EMNLP.
Timo Schick, Jane Dwivedi-YY Yu., Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools.
Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham Barezi, and Pascale Fung. 2020. CAiRE- COVID: A question answering and query-focused multi-document summarization system for COVID- 19 scholarly information management. In Proceed- ings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online. Association for Compu- tational Linguistics.
Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi. Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Ling- peng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. 2023. Lemur: Harmonizing natural language and code for language agents.
Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021. Towards table-to-text generation with numer- ical reasoning. Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1451-1465, Online. Association for Computational Linguistics.
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fine query focused multi-document summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3632-3645. Online. Association for Computa- tional Linguistics.
Yumo Xu and Mirella Lapata. 2022. Document sum- marization with latent queries. Transactions of the Association for Computational Linguistics, 10:623- 638.
Xiangru Tang, Yiming Zong, Jason Phang, Yilun Zhao, Wangchunshu Zhou, Arman Cohan, and Mark Ger- stein. 2023. Struc-bench: Are large language models really good at generating complex structured data?
Shiyue Zhang and Mohit Bansal. 2021. Finding a bal- anced degree of automation for summary evaluation. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 6617-6632, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. ArXiv, abs/2302.15 13971.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. In International Conference on Learning Representations.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu. Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami,
Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022a. MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 6588 -600, Dublin, Ireland. Association for Computational Linguistics.

---

Yilun Zhao, Boyu Mi, Zhenting Qi. Linyong Nan, Ming- hao Guo, Arman Cohan, and Dragomir Radev. 2023a. OpenRT: An open-source framework for reasoning over tabular data. In Proceedings of the ō Annual Meeting of the Association for Computational Lin- guisticy (Volume 3: System Demonstrations/ pages 336-347, Toronto, Canada. Association for Compu- tational Linguistics.
TaCube: Pre-computing data cubes for answering numerical-reasoninn questions over tabular data. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2278-2291, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
Yijie 7hou, Kejian Shi, Wencai Zhang. Yixin Liu, Yilun Zhao, and Arman Cohan. 2023. Odsum: New bench- marks for open domain multi-document summariza- tion.
Yilun Zhao, Linyong Nan, Zhenting Qi. Rui Zhang, and Dragomir Radev. 2022b. ReasTAP: Injecting ta- ble reasoning skills during pre-training via synthetic reasoning examples, In Proceedings of the 2022 Con- ference Empirical Methods in Natural Language Processing, pages 9006-9018 8, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang. Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on hybrid of tabular and textual con- tent finance. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3277-3287, Online, Association for Computational Linguistics.
Yilun Zhao, Zhenting Qi, Linyong Nan, Lorenzo Jaime Flores, and Dragomir Radev. 2023b. LoFT: Enhanc- ing faithfulness and diversity for table-to-text gener- ation via logic form control. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 554- 561, Dubrovnik. Croatia, Association for Computa- tional Linguistics.
A Implementation Details
Input Data Serialization The input contains a user query, and corresponding table data. For text generation and large language mod- els (Section 5.1.1 & 5.1.3), we followed re- cent works on table-to-text generation (Liu et al., 2022b; Xie al., 2022; Zhao et al., 2023c,a) to flatten the table data as T= [HEADER] :h, [ROW] n:rn. where h is table header, r; is the i-th table row. For text gener- ation models, [HEADER] and [ROW] are special tokens indicating the region of table headers and rows respectively; while for L.LMs, we set them empty strings. We also separated headers or cells in different columns using vertical bar I- In this way, the flattened table input be fed directly into text generation models. For table-to-texi gen- eration models (Section 5.1.2), we followed their original data processing methods to input the query and table data.
Yilun Zhao, Haowei Zhang, Shengyun Si. Linyong Nan, Xiangru Tang, and Arman Cohan. 2023c. Large lan- guage models are effective table-to-text generators, evaluators, and feedback providers.
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi. Wenlin Zhang, Xiangru Tang, Boyu Mi, and Dragomir Radev. 2023d. RobuT: A systematic study of table QA robustness against numan-annotaan versarial perturbations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6064- 6081, Toronto, Canada. Association for Computa- tional Linguistics.
[ROW] 1 1:r1,
..., ...,....,
ad-
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging Ilm-as-a-judee with mt-bench and chatbot arena. arXiv preprint 23Xiv:2306.088 05685.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021. QMSum: new benchmark for query- based multi-domain meeting summarization. ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5905-5921, Online. Association for Computational Linguistics.
Pro-
Victor Zhong, Caiming Xiong, and Richard Socher. 2018. Seq2SQL: Generating structured queries from natural language using reinforcement learning.
Fan Zhou, Mengkang Hu, Haoyu Dong, Zhoujun Cheng. Fan Cheng, Shi Han, and Dongmei Zhang. 2022.

---

Reasoning
Example of Fact Templates
Example Fact
Conjunction
The col that have CONDTION executed_resuliee
The Player Name that have Country is Canada are
Cozey Conners, Nick Taylor, Adam Svenssen.
Counting
executed_resuit colll have col :2 CONDITION:2.
2 Game have Attendance greater than 10,235,
Temporal or Numerical Order
The co1:1 ordered by col:3 3 are executed_ peccuted_results.
The Company ordered by Sales are Apple, Nvidia, Google, (...abbreviate..)
The col:1, with col:2 CONDITTON:2;
ordered by col: are executed
Temporal or Numerical
The col:1 that col:2 CONDITICN: are executed_results.
The institutions that Founded year car- lier than 1860 are Adrian College, Michigan State University.
Comparison
Numerical
The OPERATOR of col:l with col:2
The sum Earning with Point is greater 140 430, 327.
Operation (Sum, Avg)
COND1TION:2 executed results.
Numerical
The difference between val:1 and val:2
The difference between China and Canada in Gold 16.
Operation
col executed results.
(Diff)
Table 7: 6 reasoning operations, along with fact template and examples, defined for the fact generation process of REFACTOR. Variable names indicate permissible instantiations. col denotes column name, val denotes cell value, and executed results denotes the execution results of the function. OPERATOR, is instantiated according to the specific reasoning operation, e-g., "Numerical Operation", JCPERATTOO "orratiio","PERAATTAAA TOR is replaced with "sum" or "average"; CONDITION can be 1) cell value from the i-th column, or 2) number/temporal comparison statement (e.g. "later than 1967") if the i-th column of number or date type.
Top Company
Rank Company Name Industry| Revenue ($ Milian) Profit ($ Millin))
Employaes Headquarters
Table Titl:: Company 2012
/aamart Retail $689,151 $13,510 2,300,0000 United States
2 State Grid Electricity $388,818 $5,580 896,360) China
Table:
3] Amazon Retail| $386,064 $21, 2,8,00000 United
Rank Company Nama Industry/ Revenue ($ Millin) Praft ($ Million)
4 National Petroleum Dil and gas $283,966 ($4,575 1,242.245 China
Employees Headquerters
5 Sinopec Group |
gas $283,
1633 833|
Walmart Retail $559,151 $13,510 2,300,000 United States
Apple Elecronics $274,515 $57, 147,000] United Stakes
Grid| Electricity $386,818 $35,580 (996366
7 CVS Health Healthcare $268 708 $7,179 225,,000 United States
Amazzon Retail $388,086 064 $21,331 1,296,000 ,00| United States
UnitedHealth Healthcare $257,
/330,00
United
National Petroloum Oli and gas $283,958 |$4,575 1,242,245 China
9 Toyota Aulomollvv | $258,722 $21,180 /86,263) Japan
Sinopac Group
ges $283,728 205
China
Volkswagen Automotive $253. |100 $10,104 662,575 Germany
8 Apple Electronics| $274,515 $57,511 147,000| United States
CVS Health Heaathaaee ($268.708 ,,,7 256,500 United States
Relevant facts table:
UniledHeslth Healthcare $257.141 141 515,403 333,000 United States
1. Company ordered
Apple, Toyota, UnitedHealth, Walmart, Velkawagen, CVS Health, Sinopec Group,
Toyata /Automotive $256,722 $$21. 366,283 Japan
10 Volkewagen /Automotive $253 965
662.575 Germany
Grid,
Petrolaum.
2. The Company Name, with Industry is and ordered by Profit
Using the information from table summartze paragraph-long Which company the highest
Milion) Sinopec Group, National Petroloum.
to the following profit the Oll and Gas industry, and how does compare to the most proftiable company
The of with Industry
gas 10780.
4. The difference between Apple and Sinapec Group in Profit is 51306.
5. The average Profit with Industry and gas 5380.
Using the information from tab and relevant facts table, summarize paragraph-long response following query: company eame the highest profi. in the Gil and Gas industry, and how does it compare profitable company
Figure 4: An example of L.LM zero-shot prompt prefix wo. REFACTOR for the QTSUMM task.
Figure example of LLM shot prompt prefix w. REFACTOR for the QTSUMM task.

---

Query
System output
Reference
Error analysis
Q: Which driver per- formed better, Tracy or Haberfeld, and why is that?
Paul Tracy performed hetter than Mario Haherfeld, as ev- idenced their results from the 2004 Centrix Financial Grand Prix of Denver. In the race, Tracy finished in second place, while Haberfeld placed in eighth. Additionally, Tracy be- longs to Forsythe Racing, which has had a more successful track record compared Haberfeld's Walker Racing. Additionally, Tracy was able maintain his position in the race for the full 90 laps, whereas Haberfeld was only able make it Based on these results, it is clear that Trucy was more competitive compared to Mario Haberfeld.
Paul Tracy is more competitive. Tracy finished the race sec- ond place with time +7 ,466 seconds, whereus Haberfeld fin- ished in cighth place with time of +24.141 seconds. This demonstrates that Tracy was able to navigate the track much faster than Haberfeld and place higher in the race. Additionally, Tracy eamedd points while Haberfeld carned only 15 points, which confirms that Trucy is stronger competitor, In conclu- sion, Tracy is the more competi- tive driver.
Error Type: Hallucination
Explanation:
Mention facts that are not shown in the table.
Analysis:
There is no information indicat- ing that Forsythe Racing has a more successful track record compared to Walker Racing. and the model incorrectly stutes that Haberfeld completed only 86 laps instead of the correct 90.
laps.
Q: According to the vot- ing result, which rep- resentative election in North Carolina districts was the competi- tive, and why?
The most competitive represen- tative election in North Carolina districts was the one for the 8th district, in which incumbent Larry Kissell (D) lost re-election to Richard Hudson (R). This was the only race which was de- cided by less than 10 percentage points, with Hudson winning 54.1% 1% of the vote to Kissell's 45.9%. The close margin of the race demonstrates that both cun- didates were competitive, and it was an intense contest until the very end.
The race in the North Carolina 7th district was the most com- petitive, as the Democratic in- cumbent Mike McIntyre won slim margin. with only 0.2%% difference hetween him and his Republican challenger David Rouzer. Furthermore, this election was only one among all North Carolina districts in 2012 that resulted in margin of victory within less than 1%.
Error Type: Factual Incorrectness
Explanation:
Present information that contra- diets the source table.
Analysis:
The most competitive repre- sentative election should be the North Curolina 7th district, ruther than the 8th district, in which the difference between the two rivals was only 0.2
@: Analysc the coun- tries in East Asia with Human Development In- dex (HDI) higher than 0.8.
The countrics in East Asia with Human Development Index (HDI) higher than 0.8 are Hong Kong, Japan, Macau, Mongolia, and South Korea. Hong Kong has highest HDI at 0.898, fol- lowed by Japan at 0.901. Macau has the lowest HDI at 0.653, and North Korea has the highest at 0.897. All of these countries have populations greater than 1.8 million, with the highest pop- ulation density being in Jupan at 337 people per km square.
The countries in East Asia with HDI higher than 0.8 Hong Kong (PRC), Japan, and South Korea. Hong Kong has an HDI of 0.898 and a popu- lation density per square kilo- meter of 6390. Japan has an HDI of 0.901 and a population density per square kilometer of 337. South Korea has an HDI of 0,897 and a population density per square kilometer of 500. All three countries have compar- utively high population density which is likely to he a factor in their high HDI.
Error Type:
User Intent Misunderstanding
Explavation:
Include information that is irrelevant to the user question.
Analysis:
The query does not ask for country with lowest HDI or any country with HDI lower than 0.8. The system output also displays many other errors such us hallucination (confusing South Korea with North Korea) and factual incorrecteess (Japan has highest HDI).
@: What are the build- ings that have been the tallest in Portland, Oregon hetween 1927 and 1965? Summarize basic information of these buildings.
The buildings that have been the tallest in Portland, Ore- gon between 1927 and 1965 are the Public Service Build- ing. the Hilton Portland Hotel, and the Harrison West Condo- minium Tower. The Public Ser- vice Building is located 920 Sw Sixth Avenue and has been the tallest building in Portland, Oregon. It stands 220 feet (67 meters) and has 16 floors. The Hilton Portland Ho- tel is located at 921 Sw Sixth Avenue and has been the tallest building in Portland, Oregon.
Between 1927 and 1965, the tallest buildings in Portland, Oregon were Public Service Building, Hilton Portland Ho- tel, and Harrison West Condo- minium Tower. Specifically, Public Service Building was built in 1927. was 220 feet 67 meters tall, had 16 floors, and held the title of tallest building until 1962. Hilton Portland Ho- tel was built in 1963, feet / 73 meters tall, had 22 floors, und held the title of tallest building until 1965. Harrison West Condominium Tower was built in 1965, was 256 feet 78 meters tall, had 25 floors, and held the title of tallest building until 1969.
Error Type: Repetition
Explanation:
Generate repetitive information.
Anolysis:
The information of these build- ings being the tallest in Portland, Oregon has been mentioned repetitively throughout the system output, while the system fail to also distinguish them (until which yeur each of them was the tallest respectively).
height of
241
Table 8: Case study for common errors made by Flan- Man-re-- wo. REFACTOR, The colored text highlights problematic parts of the system output.


Reviews:
[{"review_id": "bcoQ9DV5pK", "reviewer": "Reviewer_PL53", "summary": "", "questions": "Mentioned in previous section", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the time you’ve taken to review our work and for the constructive feedback!\n\n### A. Concerns about Paper Structure\n> The paper is somewhat hard to read due to too much going on in the main paper (several abbreviations, metrics, systems, disparate contributions and referenced details with having to jump to Appendix to verify some details that are better fit in the main paper).\n\nWe greatly appreciate the time you took to thoroughly read our paper and examine the details in the Appendix. In the revised version, we will condense and move content from \"Appendix A.2 - Details of Automated Evaluation Metrics\", \"Appendix B.1 - Implementation Details of Fact Generator\", and \"Appendix C.1 - Details of Baseline Systems\" to the main sections of the paper.\n\n> For example, just keeping the main dataset and the Refactor method makes a good story rather than the eval benchmark, and section 6.\n\nThanks for your suggestion! We will consider it when preparing the final version.\n\n### B. Questions about Limitations of Proposed Method\n> For the templates used for fact extraction from tables (Figure 7), there is not much discussion on what is potential downside. As in, what types of reasoning questions may be missed due to reliance on these templates and thoughts on how to overcome them.\n\nThanks for the insightful comments! We appreciate the opportunity to explain and extend this point:\n\n***B.1 We Add a New Case Study on ReFactor’s Failure Cases***\n\nIn section 5.4, we conducted human evaluation on 200 QTSumm examples for assessing the coverage of facts generated by ReFactor. We revealed that 56.4% facts are annotated as “relevant”. To delve deeper into this, we have added a case study examining the failure cases, specifically those examples where two or fewer facts were annotated as “relevant”. The details of this case study are as follows:\n\n| #Examples | Error Types                                                   | Representative Question                                                                                            | Explanation                                                                                                                                                                                                                                                  |\n|--------------|---------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n|       24 / 200      | Difficulty in parsing cell values via rule-based methods      | Difficulty in parsing cell values via rule-based methods                                                           | The relevant numeric- or time-type columns are hard to parse (e.g., multiple numbers and text within one cell), thus ReFactor fail to generate related facts.                                                                                                |\n|      17 / 200      | Complex user query causes difficulty in ranking related facts | Analyze the correlation between the size of the geographical area of a Gmina type and its population?              | ReFactor employs the employs the QA encoding model for fact ranking. However, it struggles to understand complex information needs from users, such as the \"correlation between A and B,\" and might consequently rank irrelevant facts higher. | \n|      13 / 200      | Unsupported reasoning operations                              | Who are the top three coaches with the highest win percentages? Analyze their performance in the 2019-2020 season. | The table only contains “wins” and “overall games” columns. Models must compute the winning percentages independently. However, ReFactor does not support such rate calculations                                                                             |\n|       5 / 200      | Other errors                                                  |                                                                                                                    |                                                                                                                                                                                                                                                              |\n|      141 / 200     | Successful cases                                              |                                                                                                                    |                                                                                                                                                                                                                                                              |\n\n&nbsp;\n\n***B.2 Thoughts on Future Work***\n\n**The Motivation Behind ReFactor is Compelling.** As discussed in section 4, the motivation of proposing ReFactor is to address the _error-prone implicit_ reasoning issues of current end-to-end text generation models. \nWe believe that the high-level ideas of \"employing plug-and-play, explicit reasoning modules to extract and reason over query-relevant information as explicit and intermediate reasoning results\" is promising for future extensions. Nonetheless, we recognize the inherent limitations of ReFactor, particularly its reliance on predefined and template-based reasoning modules.\n\n**Potential Directions for Future Enhancements.** In the limitation section, we have discussed the potential of applying LLMs with 1) tool learning and 2) multi-step chain-of-thought on generating reasoning-required and query-tailored summary. Given the surge of pertinent research in recent months, we believe the following directions warrant further exploration. \n1. _Decomposing complex questions into fine-grained sub-questions using chain-of-thought._ Our case study reveals that the TAPEX-based fact ranking module struggles with comprehending complex questions. To address this, future research could investigate LLM chain-of-thought methods to break down complex questions into more understandable and actionable sub-questions.\n2. _Generating and utilizing executable programs for fact generation._ The predefined and template-based execution modules in the ReFactor fact generation phase have their limitations. Recent studies [1-2] highlight the impressive abilities of LLMs in making and utilizing tools for problem-solving. It would be intriguing to explore if LLMs can produce executable programs from scratch to derive query-relevant insights.\n3. _Composing insights into paragraph-long analysis using chain-of-thought._ How to use chain-of-thought to compose salient information and generate summary [3] is still underexplored in the community.\n\nWe will include and extend above points in the limitation section of the revised version.\n\n&nbsp;\n\n&nbsp;\n\n### References\n1. Schick, Timo, et al. [\"Toolformer: Language models can teach themselves to use tools.\"](https://arxiv.org/pdf/2302.04761.pdf) arXiv preprint arXiv:2302.04761 (2023).\n2. Cai, Tianle, et al. [\"Large language models as tool makers.\"](https://arxiv.org/pdf/2305.17126.pdf) arXiv preprint arXiv:2305.17126 (2023).\n3. Wang, Yiming, Zhuosheng Zhang, and Rui Wang. [\"Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method.\"](https://arxiv.org/pdf/2305.13412.pdf) ACL 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper is well-written and the proposed problem is a well-motivated extension to\nrecent research directions on tables-to-text generation as well as query focused summarization\n\nB. The dataset creation process is reliable (multi round validation) and when shared will be a valuable add for further\nstudy on this topic.\n\nC. The method of generating facts relevant to a query and further collating them forms\nan explainable way to generate a reliable query focused summary.\n", "reasons_to_reject": "A. The paper is somewhat hard to read due to too much going on in the main paper\n(several abbreviations, metrics, systems, disparate contributions and referenced details\nwith having to jump to Appendix to verify some details\nthat are better fit in the main paper).\nFor example, just keeping the main dataset and the Refactor method makes a good story\nrather than the eval benchmark, and section 6. In particular,\nI could not follow much of section 6.2  and what Table-5 with Kendall Tau values \nis meant to highlight.\n\nB. For the templates used for fact extraction from tables (Figure 7), there is not much\ndiscussion on what is potential downside.  As in, what types of reasoning\nquestions may be missed due to reliance on these templates and thoughts on how to overcome them.\n", "questions_to_authors": "Mentioned in previous section", "paper_topic_and_main_contributions": "The paper presents\n(1) a new dataset as well a novel variation of the table-to-text generation problem\nby formulating the query-focused summarization of table data where the summaries require human-like reasoning/explanations\nrather than facts (different from previous works)\n(2) a meta-evaluation benchmark to help in the design of metrics for evaluating the above problem\nand (3) a method to summarize table data in response to queries by making use of extracted query-relevant\nfacts that are ranked\nand further combined to generate a summary.\n\nExperiments are presented on this novel dataset with their proposed approach as well as\nmodifying existing state-of-the-art baselines on closely related problems and the\nresults quite support the claims made in the paper regarding the importance of\ntable structure, analytical reasoning, and factual correctness.\n\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a1EjZSpxvu", "reviewer": "Reviewer_vhSV", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive feedback! We greatly appreciate the time you took to thoroughly read our paper and examine the details in the Appendix. We will relocate \"Appendix B.1 - Implementation Details of Fact Generator\" to the main body of the paper in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Great motivation for building the proposed benchmark: Learning how to summarize structural information with personalized query is very important to let models learn to follow the instructions and provide proper feedback. I’m impressed by Table 3 which contains very comprehensive evaluation and further shows the necessity of having this benchmark.\n\nComprehensive investigation on the quality of generated instruction data: I'm glad to see that the authors utilize different metrics and many latest models like GPT-4 and Tulu to evaluate the performance. The comprehensive evaluation helps us better understand the effect of the task difficulty and proposed methods. \n\nGood quality control for the benchmark construction: This paper also describes the details of quality control when constructing the dataset. Multi-round validation is helpful for further improving the quality. Also, to make the dataset not that easy, the authors introduce some mechanism to create instances involving multi-hop reasoning.", "reasons_to_reject": "Lack of details for Refactor in main paper: For Section 4, it is very hard for me to understand the details of Refactor method. Although the authors put description in Appendix B, I may still think generally describing methods with pure text in main paper is not helpful for readers to understand the details. It is better to move them to the main paper, so that readers will know the formulation (e.g., each step’s input/output) more clearly.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This research introduces a novel task in the field of text generation, focusing on creating tailored summaries for tables based on user queries. The authors propose QTSumm, containing 5,625 query-summary pairs from a diverse set of 2,437 tables. The study explores various strong baselines and evaluation methods, revealing significant challenges in both generating accurate table-to-text summaries and evaluating them. The authors propose a new approach called REFACTOR, which retrieves and reasons over query-relevant information from tabular data to generate natural language facts. Experimental results demonstrate that REFACTOR effectively enhances state-of-the-art models and evaluation metrics by incorporating generated facts into the model input.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pNRUZIMdbg", "reviewer": "Reviewer_Pktq", "summary": "", "questions": "Why not compare with other TableQA/Table-to-text generation methods in experiments?\nWhy only test LLM in the 2-shot setting? Why not use more examples?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the time you’ve taken to review our work!\n\n### 1. Concerns about Novelty of QTSumm Benchmark\n> Compared with summarization, it is more like a TableQA/Table-to-text generation dataset/task. Rather than saying that this article defines a new task, it is more like giving the old task a new name. Better to revise the title and submit it to the Question Answering and Natural Language Generation track.\n\n***Why Submit to Summarization Track***\n\nAs discussed in the related work section (line 222-242), query-focused summarization is a crucial sub-field of summarization. The primary objective of this field is to generate summaries from source input that are tailored to user queries. This aligns with the task setting of QTSumm.\n\n***Comparing QTSumm with Existing Table-Relevant Tasks***\n\nThe distinctions between QTSumm and existing Table-to-Text generation / Table QA datasets and tasks have been highlighted in introduction (lines 56-82 and table 1) and the related work (lines 160-200) sections. We copy and organize them as follows, _with corresponding line indices_, for your convenience:\n\n1. Differences from Table-to-Text Tasks in _Human-like Reasoning and Analysis_:\n   - Existing table-to-text generation tasks (i.e., ToTTo, LogicNLG, and HiTab) focus on converting the highlighted table regions into a _human-readable_ and _coherent single sentence_ (line 60-62, 169-171).\n   - QTSumm emphasized more on human-like reasoning. It focuses on generating _paragraph-long_ and _query-tailored data analysis_ with _human-like reasoning_ process (line 62-68, 171-175).\n2. Differences from Table Summarization Tasks in _Addressing Information-seeking Purposes_\n   - QTSumm _aligns more closely with real-world scenarios_ of how humans consult tabular data. Considering the numerous data points and diverse user's information needs, it becomes challenging to create a _generic_ summary that encompasses all the salient information within a table (line 181-186).\n   - Existing table summarization tasks (i.e., RotoWire, SciGen, and NumericNLG) use _domain-specific_ tables as source input; while the tables in QTSumm are from Wikipedia and cover _diverse topics_ (line 175-180).\n3. Difference from Short-form Table QA Tasks in _Addressing More Complex Information Needs_\n   - Existing Table QA tasks (i.e., WikiTableQuestions, WikiSQL, SQA, and HybridQA) focus on retrieving and composing short-form answers (typically one word) that cover a _single_ fact (line 74-77).\n   - QTSumm _aligns more closely with real-world scenarios_, where real-world users often have more complex information needs that expect models to perform _human-like reasoning_ to provide _paragraph-long_ analyses (line 77-82).\n4. Difference from FeTaQA in _Human-like Reasoning and Analysis_\n   - Most questions in FeTaQA focus on surface-level facts and single-sentence answers (e.g., “Which country held the 2022 FIFA World Cup?”), whereas QTSumm requires models to perform human-like reasoning for implicit data insights (line 196-200).\n   - FeTaQA is collected by rewriting ToTTo’s statements into questions and uses the same statements as the answers; while queries in QTSumm is annotated by human annotators under real-world scenarios, making them more natural and better-reflecting users’ actual information needs (line 188-195).\n\n***QTSumm Introduces Distinct Challenges in Both Text Generation and Automated Evaluation***\n\nAs highlighted in the paper (line 22-25, 144-153), the above unique features of QTSumm introduce new challenges in:\n1. Text generation (Section 5) - Models must comprehend users' information needs and perform human-like reasoning and analysis over structured tables.\n2. Automated evaluation (Section 6) - Automated metrics need to reliably assess the faithfulness and comprehensiveness of summaries that are query-tailored and reasoning-required.\n\n&nbsp;\n\n### 2. Concerns about Experiment Design\n> Why not compare with other TableQA/Table-to-text generation methods in experiments?\n\nWe have already evaluated and compared the latest work in TableQA/Table-to-text generation methods (i.e., TAPEX, UnifiedSKG, ReasTAP, PLOG) in section 5 and Table 3. As presented in [1], these models are the state-of-the-art methods in both Table-to-Text generation (i.e., LogicNLG) and Table QA (i.e., WikiTableQuestions, WikiSQL, SQA, and FeTaQA) tasks.\nIf the reviewer has additional specific models they’d like to see, we’d be happy to consider adding.\n\n> Why only test LLM in the 2-shot setting?\n\nWe have already reported the performance of LLMs in both _0-shot_ and 2-shot settings, with and without the proposed ReFactor method, in section 5 and Table 3.\n\n&nbsp;\n\n### 3. Concerns about Method Novelty\n> Compared with previous work on TableQA and Table-to-text generation, the novelty of the proposed method is very limited.\n\n**ReFactor is Plug-and-Play, Effective and Adapative** To the best of our knowledge, ReFactor is the first plug-and-play method in table-to-text generation research that can be easily adopted to different types of fine-tuned and large language models (as demonstrated in section 6). Furthermore, in Section 6, we also demonstrate the effectiveness of ReFactor in improving existing table-to-text automated evaluation metrics, making the metric more aligned with human evaluation.\n\n**ReFactor is Proposed for Resolving Common Issues Regarding Implicit Reasoning.** The state-of-the-art TableQA/Table-to-Text methods (i.e., TAPEX, UnifiedSKG, ReasTAP, PLOG) are end-to-end models following the “pre-training and then fine-tuning” paradigm. They generate text in an end-to-end manner, resulting in reduced explainability and difficulties in handling more complex reasoning, such as arithmetic calculation (line 201-221, 379-385, 1212-1226).\nSuch \"error-prone implicit reasoning\" and \"reduced explainability\" issues motivate us to propose our method, ReFactor, which generates several faithful and query-relevant NL facts serving as _explicit_ reasoning results (line 376-393).\n\nWe acknowledge that there are some _task-specific_ methods in Table QA [2-3] that use executable logical forms such as sql query to retrieve and obtain a _single fact_ as the final answer. But we are the first to explore extracting, ranking, and composing multiple facts from tabular data into paragraph-long and query-tailored summary via _explicit_ and _plug-and-play_ reasoning modules. If the reviewer is aware of any specific work that closely resembles ours, we would be happy to review and differentiate our approach accordingly.\n\n&nbsp;\n\n### 4. Concerns about Writing\n> The writing of this manuscript still needs improvements. Many redundant paragraphs can be simplified, and the logic of writing can be clearer.\n\nThanks for your suggestions! We will further carefully proofread and improve the manuscript in the revised versions.\n\n&nbsp;\n&nbsp;\n\n### Reference\n1. Zhao, Yilun, et al. [\"OpenRT: An Open-source Framework for Reasoning Over Tabular Data.\"](https://aclanthology.org/2023.acl-demo.32.pdf) ACL 2023 demo.\n2. Herzig, Jonathan, et al. [\"TaPas: Weakly supervised table parsing via pre-training.\"](https://aclanthology.org/2020.acl-main.398.pdf) ACL 2020.\n3. Nan, Linyong, et al. [\"FeTaQA: Free-form table question answering.\"](https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00446/109273/FeTaQA-Free-form-Table-Question-Answering) TACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This manuscript builds a valuable Table Summarization dataset, which is close to real-world scenarios. \nThe proposed method achieves the state-of-the-art performance on their dataset.", "reasons_to_reject": "1) Compared with summarization, it is more like a Table-to-text generation dataset/task. Rather than saying that this article defines a new task, it is more like giving the old task a new name. \n2) Compared with previous work on TableQA and Table-to-text generation, the novelty of the proposed method is limited\n3) The writing of this manuscript still needs improvements. Many redundant paragraphs can be simplified, and the logic of writing can be clearer.", "questions_to_authors": "Why not compare with other TableQA/Table-to-text generation methods in experiments?\nWhy only test LLM in the 2-shot setting? Why not use more examples?", "paper_topic_and_main_contributions": "This manuscript propose a benchmark for query-focused table summarization.\nThey also propose table summarization approach named REFACTOR.\nAuthors find the gap between automatic and human evaluation results and propose QTSUMM-EVAL to investigate the performance of automatic metrics.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new query-focused table summarization task. Unlike other tasks such as Table QA or Table-to-text, the proposed task requires both the adaptation to user queries and the necessary reasoning for summarization.

* Reviewers all agree on the value of the dataset, especially commending the transparency and properness of the construction.
* The authors also presented comprehensive experiments to contrast different SOTA methods for neighbor tasks (i.e., table QA, table summarization).
* At the same time, however, the clarity on the proposed Refactor needs improvement. While some details are presented in the Appendix, the authors are expected to retain the main information in the body of the paper.
Decision: Accept-Main