QATCH: Benchmarking SQL-centric tasks with Table Representation Learning Models on Your Data
Authors: Simone Papicchio
Paolo Papotti
Luca Cagliero
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Table Representation Learning
TRL
Benchmarking tool
Checklist
Question Answering
QA
Semantic Parsing
text-to-SQL
Proprietary Data
Tabular Data
Relational Tables
Cross-task performance metrics
Automated testing
Large Language Model
ChatGPT
LLM​
TL;DR: QATCH is a toolbox for testing Table Representation Learning models on unseen data, automating checklists for Questions Answering and Semantic Parsing tasks, and evaluating performance with cross-task metrics.

Abstract: Table Representation Learning (TRL) models are commonly pre-trained on large open-domain datasets comprising millions of tables and then used to address downstream tasks. Choosing the right TRL model to use on proprietary data can be challenging, as the best results depend on the content domain, schema, and data quality. Our purpose is to support end-users in testing TRL models on proprietary data in two established SQL-centric tasks, i.e., Question Answering (QA) and Semantic Parsing (SP). We present QATCH (Query-Aided TRL Checklist), a toolbox to highlight TRL models’ strengths and weaknesses on relational tables unseen at training time. For an input table, QATCH automatically generates a testing checklist tailored to QA and SP. Checklist generation is driven by a SQL query engine that crafts tests of different complexity. This design facilitates inherent portability, allowing the checks to be used by alternative models. We also introduce a set of cross-task performance metrics evaluating the TRL model’s performance over its output. Finally, we show how QATCH automatically generates tests for proprietary datasets to evaluate various state-of-the-art models including TAPAS, TAPEX, and CHATGPT.

QATCH: Benchmarking SQL-centric tasks with Table Representation Learning Models on Your Data
Simone Papicchio Politecnico di Torino Turin, Italy
Paolo Papotti EURECOM Sophia Antipolis, France
Luca Cagliero Politecnico di Torino Turin, Italy
Abstract
Table Representation Learning (TRL) models are commonly pre-trained on large open-domain datasets comprising millions of tables and then used to address downstream tasks. Choosing the right TRL model use on proprietary data can be challenging, the best results depend on the content domain, schema, and data quality. Our purpose is support end-users in testing TRL models on proprietary data in two established SQL-centric tasks, i.e., Question Answering (QA) and Semantic Parsing (SP). We present QATCH (Query-Aided TRL Checklist), a toolbox to highlight TRL models' strengths and weaknesses relational tables unseen at training time. For an input table, QATCH automatically generates a testing checklist tailored to QA and SP. Checklist generation is driven by SQL query engine that crafts tests different complexity. This design facilitates inherent portability, allowing the checks be used by alternative models. We also introduce a set of cross-task performance metrics evaluating the TRL model's ' performance over its output. Finally, show how QATCH automatically generates tests for proprietary datasets to evaluate various state-of-the-arr models including TAPAS, TAPEX, and CHATGPT.
Introduction
Table Representation Learning (TRL) models are getting increasing attention for their ability to support various NLP downstream tasks involving tabular data (Badaro et al. 2023 Dong et al. 2022). Such models are built using large open-domain datasets during pre-training and can then fine-tuned with labelled examples for the target task.
Motivation. In most settings, companies aim at adopting a pre-trained model to reduce costs. However, despite the abundance of such methods, is still challenging to select and manage a TRL model for proprietary data. Models use different pre-training tasks and datasets, work under different assumptions, and the top performing model for existing benchmark when tested on proprietary data. Indeed, the performance of models fine-tuned on benchmark data is necessarily replicable on proprietary data.
necessarily the best one
In corporate setting, there are at least three scenarios where given TRL model needs to be evaluated against proprietary datasets:
Comparison: Compare TRI. models fine-tuned on private examples to see which one performs best. Validation: As crafting examples is expensive, verify when the quality meets the requirements. Maintenance: Fine-tuned models need to be re-calibrated to avoid data and conceptual shifting, continuous evaluation helps the identification of this issue.
Challenges. Supporting these scenarios with accurate evaluation is a not obvious task. For the first scenario, consider an engineer who selecting the TRL model for a Question Answering (QA) task
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

over their enterprise tabular data. Several options are available and they start assessing the models for a zero-shot setting, where the model is used as-is. As labelled examples are not available, the engineer has to craft some test data to assess how the model is performing on their proprietary tables, e.g., write pairs of questions in natural language (NI.) and the expected data results.
The task above has cost that increases with the number of tables and tests crafted by the engineer. Moreover, the process above applies for all the target tasks that the engineer wants to support. For example, they would have to repeat the exercise for the same model and datasets to evaluate its suitability for Semantic Parsing (SP, aka Text2Sq)), where, given a question in NL and table schema, the model returns SQL query.
Creating tests is only half of the story. Test results must be evaluated, i..., the data in the QA model's output Da should "match" the expected output manually crafted in the test Dr. A simple equality test fails short, as the tuples (or values) in Da can be in different order w.r.t. those in Dr- Different order for records and attributes does not matter in the relational model, so the test should be independent from this difference. While existing systems are evaluated with a naive accuracy metric that handles these issues, many problems that make this matching hard are ignored. First, they do not test results for relational data integrity, i.e., if tuple and attribute relationships are satisfied. Second, consider tuples in Dar that are similar to the expected ones in Dr except null values, or that differ in the order when the question requires that results are sorted these issues are ignored by existing metrics. The same observations apply for the Semantic Parsing task. as it is a common practice to evaluate the produced SQL scripts on their results once executed (Li al. (023).
Automatic Assessment. QATCH (Query- Aided TRL Checklist) addresses the challenges in eval- uating TRL. models for proprietary data over SQL-centric tasks. QATCH tests TRL. models by automatically generating and evaluating testing checklists of varying complexity for QA and SP.
Given proprietary table, such as the one in Figure ! (step QATCH produces questions in NL such as the one in the figure (step are passed, together with the input table, to the TRL models to evaluate (step 3 ). The models' output is then consumed by tool (step to produce an assessment of the results (step
"Name and the most expensive product?
Product
Price
QATCH
The questions
M1
M2
iMask
$3500
xPhone
$1100
iMask
Ywatch
$500
M1:
5
TC
iMask $3400
Proprietary table
M2:5
1
>>Phoe511000
At the heart of QATCH, query generation algo- rithm crafts tests covering a comprehensive range of features enabled by the expressive power of SQL. Our objective is to assess the capability of the models concerning question complexity. To achieve this, we generate the corresponding NL questions using templates designed to maintain clarity and precision. This approach ensures that we do not inadvertently test the models' capacity to handle complex textual content, our primary focus is on their performance in relation to question complexity.
Figure I: QATCH takes a table as input and returns metrics for TRL. models MI and M2,
QATCH also introduces cross-task performance metrics that evaluate the models' results while accounting for the nuances the comparison of tabular outputs. To address the complexities in evaluating model outputs, our metrics consider numerous factors when comparing databases, including the cardinality of the results, presence of null values, missing or extra cell values, and sorting when required by the NL questions. By taking these aspects into account, QATCH provides a robust and insightful evaluation model performance.
Our work differs from existing rigid benchmarks, such as Spider Yu et al. 2018 which cannot capture how well a model performs on proprietary data. Instead, QATCH supports end-users in corporate settings by facilitating decisions on TRL model comparison, validation and maintenance.
Our contributions are summarized follows:
We present QATCH, a toolbox for automatically generating and evaluating test checklists tailored to proprietary data and SQL-centric tasks. Code, data, and results are available at https::/ github. . wtt. wwwp:///pppccccco///aaaaaa
We exploit a SQL query generator craft a testing checklist for multiple TRL models and two established tasks, i.e., Question Answering and Semantic Parsing.
2

---

We introduce robust performance metrics that capture the nuances in comparing model outputs, such as cardinality, null values, missing or extra cell values, and sorting requirements. By executing tests of increasing complexity over four TRL models and 8 proprietary datasets, we report insights into the models' capacity to handle two SQL-centrie tasks on tabular data.
2
Related work
Table Representation Learning. Table Representation Learning (TRL) refers the process of developing and training neural models to capture the underlying structure, semantics, and relationships in tabular data. TRL used for tasks both natural language processing (NLP) and data management. Such models support data-driven systems that surpass the limitations of traditional declarative specifications based on first-order logic and SQL.
Examples of tasks that use these models include answering questions in natural language (Katoggaamis-Memmrrkke
and Koutrika] [02T Herzig al. 20200; Liu et al. (02T). fact-
checking (Chen et al. [2020; Yang and Zhu| [2021; Aly et al. (0211. semantic parsing (Yin et al.
2020 Yu al. (021), 1) table retrieval (Pa et al. [0221 Kostič et al., 2021, Glass et al.) 2021) table
comprehension Suhara et al. 202T Du et al. (02T). and table content prediction (Deng ct al. 2020 lida al. 2021). These models are built with different architectures (encoder only, decoder only, encoder+decoder), but our approach is agnostic to this aspect and evaluates any model that can satisfy the input/output requirements of a SQL-centric task, i.e., the output may be obtained with a SQL query. In this work, we focus on two tasks that satisfy these requirements: Question Answering (QA) and Semantic Parsing (SP).
Question Answering. In the context of free text, QA aims to retrieve passages containing the answer to given question. In the tabular data setting. QA involves returning the cells that answer a given query, with the input consisting of a question and a table (Herzig et al. 2020 Liu et al. (022)) There are two levels of complexity tabular QA tasks, Simple QA focuses on lookup queries on tables, while more complex QA tasks require aggregation operations and numerical reasoning.
Semantic Parsing. SP in the tabular data setting involves generating declarative query in SQL over the table's s schema, given a question and a table as input (Yin et al. 20200] Liu et all] [02T]] Yu et al.) 2021; Li et al. 2023). The purpose of SP is retrieve the answer the question by producing an interpretable query rather than directly obtaining the answer. Unlike QA, where the focus is on finding the answer cells, SP emphasizes the generation of a structured query.
Other tasks, such as Table Metadata Prediction (TMP) and Table Content Population (TCP), are also widely covered by TRL models, but they do not follow the task requirements. TMP focuses on predicting inter-table metadata, such as column types, headers, cell types, and table types, well as intra-table relationships, such as equivalence between columns and entity linking/resolution (Deng
al.) 2020 Cappuzzo al. 2020p. Mcanwhile, TCP addresses the recovery of corrupted cell content and aims to impute missing cell values in an input table (Tida et al.) 2021 This serves to enhance and improve the consistency of table information. However, those are traditional DB problems over tabular data and there exist some relevant benchmark solutions relying on metadata creation (Arocena
al. 2016).
Previous Benchmarking Approaches. Several benchmarks have been developed for QA and SP models for measuring model performance on fixed datasets. Examples include those closer to QA (Pasupat and Liang, 2015) Chen et al., 2020) and those for SP (Yu et al. 2018 Gkini et al. 2021). WikiTableQuestionn. is designed for evaluating table comprehension tasks, with a dataset consisting of questions about tables from Wikipedia. TabFact focuses on fact-checking and comprises statements about tables that can be labeled as "true" or "false". Spider, and similar efforts, are corpora of databases, each with set of SQL queries to test SP. For each query, they provide the paraphrases of the query as a NL question. These datasets have played a role in advancing TRL models across various tasks. However, they inherently limited due to their fixed set of crafted examples and lack test generation for different TRL models given only the proprietary datasets.
Moreover, previous approaches have relied on simple accuracy metrics, which test if the produced values are included in the ground truth. This "execution accuracy" is adopted also for SP as is more precise than just comparing SQL scripts (Li et al. 2023 Yu et al. 2018). However, this metric fails to
3

---

QATCH
TRL Model
QATCH
NL
Generate tests
Question
Predict
Compute metrics
Model Output
[0.70.8.0220..0... Tests Evaluation
Proprietary Table
Ground Truth
Figure 2: Workflow of QATCH: given proprietary relational tables as input, it generates tests for Question Answering and Semantic Parsing; tests are executed on given TRL model; the evaluation metries are computed between predictions (model output) and ground-truth results.
capture nuances in comparing data instances. For example, does not measure whether values from the same tuple in the ground truth output appear in the same manner in the model output. As result, this simplistic metric overlooks aspects of TRL. models' performance and motivates our proposal.
In terms of test generation, our work took inspiration from an existing effort in measuring the quality of traditional text language models (Ribeiro et al. 2020). Unlike QATCH, Ribeiro al. (2020) focuses on behavioral testing of NLP models across multiple tasks. Conversely, we leverage the expressive power of SQL to evaluate the test model outcomes directly on tabular data.
3 The QATCH Toolbox
QATCH's automatically generates and evaluates test checklists on TRL models based on the three- step process depicted in Figure
1. QenCH-Generate. It generates set of queries tailored to proprietary data. For each query formulates both the SQL declaration, its free-text version, and the expected ground truth consisting of table instances. The SQL declaration expresses the logical complexity of the query and reflects the presence/absence of specific features peculiar to relational data model.
2. TRL Model Prediction. processes the tests for various TRL models and tasks. The toolbox supports alternative TRL models for both the QA and the SP tasks.
3. QATCH-Evalutte, evaluates the models outputs according to a set of cross-tast task performance metrics. Separately for each model, it provides end-users with the results for a testing checklist to gain insights into TRL models' strengths and weaknesses.
We describe the tasks covered in the toolbox next. We then explain the test generation. Finally, we discuss the performance metrics.
3.1 Question Answering and Semantic Parsing based TRL models
Question Answering aims at retrieving the answer to given question Q, where both Q and A are expressed in natural language. Here focus retrieving the content necessary formulate the answer from a proprietary relational table T encoded by a TRL model M.
We specifically address two sub-tasks tailored to Table Representation Learning Models, i.e. Question Answering from TRLs (QA-TRL, in short) and Semantic Parsing based on TRLs (SP-TRLs).
Question Answering from TRLs. Let T be a relational table and let ST and 1T be T's schema and instance respectively. Executing a SQL declaration SQL on T entails retrieving (ground truth) result RSQL derived from Ty:
(1) In QA-TRL we do not query the original table T, but rather the TRL model M that encodes T. This allows us pose the query natural language and obtain the result directly from the model.
RSQl QUERY QUERY(SQL, Sr. Ir)
Let SQL be a SQL. declaration and let Q be one of its free-text reformulations, i.e., question expressed in natural language that is semantically equivalent SQLq- The goal of QA-TRL is to ask question Q on M (which encodes T) to retrieve a result RQ
Ro = QA-TRI(M, Q)
(2)

---

Category
SQL declaration
Free-Text question
Project SELECT [CI...., Cp FROM (T)
Show [Ci...., Cn. in table (T)
Distinct SELECT DISTINCT (Ci, Cx FROM (7) Show the different (Ci...., ,n in table [7)
Select SELECT # FROM (T) WHERE {c;) (op) [ Show data of table [t] where [ vva]]
Order by SELECT * FROM (T) ORDER BY {ca} [ord] Show data for table {7) in (ord) order by (c;
Table 1: Templates for queries in SQL and natural language. i' the target relational table. c; E ST (1 S i n) is an attribute of the T's schema, op (=, !=, <. 2, 5) is logical operator, VAL; is an arbitrary value for attribute e; occurring in Ir. ord is the order of visualization of the tuples in the output (i.e., ascending or descending).
s.t. RQ is equivalent (or mostly similar, in the worst case) to the expected outcome RsQL from T.
Semantic Parsing using TRLs. Given a free-text question Q and relational table T, the goal of SP is map the corresponding SQL declaration SQL.Q.
In SP-TRL we leverage the TRL model M, which encodes the table schema Sr, perform the Tex:2SQL transformation.
SQLO SP-TRL(Q,M)
Performance Evaluation. Testing TRL models for QA-TRL and SP-TRL provides complementary information about their ability encode complex instance- and schema-level relations holding in tabular data. Instead of verifying the syntactic overlap between free-text questions and SQL declarations, we evaluate results' consistency at the tuple level. Specifically, in QA-TRL we measure how similar the result Ro returned by the query Q and the expected outcome RsQr. are in terms of tuple characteristics and cell values. Similarly, in SP-TRL we compare the tuples retrieved by the execution of the output query SQLQ on T with those contained in the expected outcome RSQL.
QATCH Key Steps. QATCH automatically generates SQL declarations SQL-q and free-text ques- tions Q from proprietary table T. Based on the table schema and instances, a query generator first returns a set S of SQL declarations of varying complexity.
S QUERYGENERATOR
Then, for every query SQLQ S, QATCH generates template corresponding of its free-text versions.
Q = T
(5)
The separation of queries and questions allows us to disentangle the characteristics of the output from the linguistic capabilities of the models. Free-text questions and SQL declarations are used to test the TRL models on the QA-TRL an SP-TRL tasks.
3.2 Automatic generation of testing checklists with SQL queries
Given a relational table T, QATCH relies on SQL query generator to produce queries of varying logical complexity for TRL model's 's assessment. Each test query consists of triple (Q, SQLQ 18t) where SQLO is the input query formulated as a SQL declaration, Q is one of the possible free-text reformulation of the question, and got is the corresponding ground truth.
Table enumeraats the templates used to generate SQLQ first and then Q based on proprictary data. They encompass SQL projections on the table schema (Project), selections on the table instance and sorting criteria at the tuple level (Distinct and Order by [I The variable in the templates are automatically filled up by the tool according to the values the schema and active domain on the given T.
(Select), and more complex cardinality-ban
The SQL generator automatically crafts multiple versions of the input tests, each one incorporating different feature. For example, for the Project template, QATCH explores the following three cases: (1) SELECT ALL, which selects all the attributes in the table schema; (2) SELECT-RANDOM.
Since existing QA models work single tables, currently disregard Join operations.
5

---

which selects n. random subsets of attributes, each subset of cardinality I,..., where n is the number of attributes in the table schema; (3) SELECT SATT-AAD-OOL which starts from fixed partition Ps ST of the table schema and incrementally adds one random attribute at time. The SQL. generator also mixes categories to create more complex queries, e.g., using projection and selection in the same test.
The motivation behind the proposed templates is to examine the TRL model's capability to handle various algebraic operations that may be directly or indirectly applied the relational schema (based on the targeted task). Our goal is not to stress the complexity of the linguistic expression, such as ambiguity (Veltii al. 2023), bur rather analyze the logical complexity of the question according to the relational data model.
3.3 Cross-task performance metrics
QATCH summarizes the outcomes of the tests with set of cross-task performance metrics defined on the comparison of the model output against the expected ground truth. The comparison reflects the similarity between the corresponding instances and the ability of TRL models to handle specific data issues. Specifically, given Q and peelfgtt as the sets of distinct pairs respectively occurring in
Q
IQ and Q'
it supports the following metrics:
Cell Precision. The fraction table cells cell, in the output instances that are relevant to the input query. The higher is the score, the more predicted elements are in the target. However, it does not measure how many target cells are in the prediction (measured by cell recall).
[[oll, Teeff.at
C-PR=
[[oelt, ceft, Teel
Cell Recall. The fraction of table cells that are relevant to the input query that successfully retrieved. The higher is the score, the more target cells are in the prediction. does not measure how many prediction cells are in the target (measured by cell precision).
[[oell e Tooff I mott.art
Q
C-REC= |eelll s oall.ot
H
Tuple constraint. The fraction of ground truth tuples in the query output. It is one if the expected and produced outputs have both the same schema, the same cardinality and the same cell values, zero otherwise. However this hard constraint does capture all the cases (see example below).
!Q no Q
T-Cons=
110
Tuple cardinality. The ratio of output and ground truth cardinality. It is "softer" constraint w.r.t. the tuple constraint since it does not consider neither the schema nor the cell values. However, has to be analysed with cell precision and cell recall be meaningful.
UO IIQ
T-CARD=
Tuple Order: the Spearman rank correlation coefficient (Katsaounis, 2003) between the vector representations of the ranked lists of returned and ground truth tuples. This non -araameeetiic test is computed only for queries with the Order-Bi By clause.
Figure 3 shows examples of the performance metrics a toy dataset. Given the ground truth result (target) with three tuples over two attributes, we report the metric values for five predictions, coming either from QA from the execution of query in SP. Notice that:
Tuple constraint/cardinaltyy and cell-precision/reaall range between 0 (no matches) and 1 (all matches), while the tuple order ranges between (opposite rank) and I (same rank).

---

[[apple, red), [pear, green). [banana, yellow]l
TUPLE CONSTRAINT CARDINALITY
TUPLE ORDER
PRECISION
RECALL
PREDICTION
[[apple, red), [banana, yellow]]
PREDICTION
[[apple]. [pear], [banana]]
1.0
0.5
0
1.0
0.5
PREDICTION
l[grape, bluej, [pineapple, yellow]]
PREDICTION
[[pear, green). [apple, red), [banana, yellow]. [peach]]
6/7
1.0
1.0
3/4
0.75
PREDICTION
[[eaa red]. (apple. yellow], [banana, green]]
Figure 3: QATCH's metrics are computed between the model output (prediction) and expected ground-truth results (target), The target is the answer of the NL question "Show me all the data" over table with three tuples and two attributes.
The tuple constraint ignores partial tuple matches (c.g., [APPLE, RED] is different from [APPLE] in Prediction 2), which is captured by cell recall.
When the number of expected tuples is zero, tuple cardinality takes either value zero, the number of predicted tuples is greater than zero, one, if no tuples are predicted.
To explain the rationale behind the proposed metrics, let us consider the following target values (al, bl, cl). The first prediction, denoted by "output 1", is composed of two tuples (al, bl, cl), (al, bl, cl). Instead, the second prediction, denoted by "output 2", has the following two tuples: (al, bl), (cl). "Output I" has cardinality two instead of one whereas "output 2" has an incorrect schema. In both cases the tuple constraint returns zero even part of the output cells match. Notice also that even if the outputs' cardinality is the same cell precision and recall show relevant differences in the returned values.
Our Cell Precision is equivalent to the Execution Accuracy reported most QA and SP papers. We remark that while informative, this single metric does not capture cases where the output is incorrect, such as Prediction in Figure
Experimental Evaluation
TRL models. We test six TRL models and one Large Language Model, i.c., CHATGPT OOenAI 2023). For QA-TRL, we report for TAPAS (Herzig al. 2020), TAPEX (Liu al. 2021 and OMNITAB (Jang et al. 2022). TAPAS leverages the transformer architecture to pre-train on large- scale datasets and fine-tune learning in the pre-training a neural SQL executor over a synthetic corpus of SQL queries and their execution outputs. OMNITAB exploits pretraining approach that uses both natural and synthetic data to learn reasoning over multiple table elementes. We use TAPAS and TAPEX fine-tuned on the WTQ dataset for QA.
specific tasks using labeled examples. TAPEX exploits the idea of
For SP-TRL, we report for RESDSQL (Li al. 2023). GAP Shi et al. 202I). UNIFIEDSKG (Xie ct al. 2022). RESDSQL is based on seq2seq architecture with ranking skeleton-aware decoding framework. UNIFIEDSKC implements the encoder-decod model based on T5, We use UNIFIEDKGG and RESDSQL with the T5 large setting. GAP employs generative models to create pre-training data, enabling the simultaneous learning of natural language utterances and table schemas.
ranking-enhanced enhanced encoding and
(text-to-text)
CHATGPT is language model employing a decoder-only transformer setup: by leveraging chat interface, it allows to specify range of tasks (including QA and SP) with the in-context adaptation technique. This technique leverages model's s ability to understand and consider the immediate context of conversation or text. Further details to reproduce the results can be found in the appendix.

---

Category
Table Name
# rows
categorical cols
numerical cols
Example cols
cales-transactions
500k
3
ProductNo, Date
ECOMMERCE
Fitness-trackers
565
3
Brand Name, Display DaysSinceRequest, Velocity6h
Account-fraite
1M
26
FINANCE
Late- pate-p
2466
6
InvoiceDate, Disputed trtbps, # oldpeak pgr. rfstime education, fnlwgt cap-shape, ring-type
Heart-attack
303
11
MEDICINE
Breast-cancer
686
6
Adult-census
32.6k
6
MISC
Mushrooms 8.1k
23
0
Table 2: Information for the proprietary tables used in the experiments.
Datasets. We generate tests for proprietary tables that we have selected from Kaggle to maximize variety both in terms of content (four categories) and in terms of data properties (different sizes and arity), as detailed in Table 2 These datasets are available online, thus possibly "seen" by the models, but not available for question answering and semantic parsing tasks, i.e. they do come with natural language questions or queries. The tables are used in this paper as sample to show the benefit of automatic test generation with QATCH, as new corpus with a fixed set of questions and tables. We also report results for the widely used Spider benchmark, this is the standard for QA and SP tasks evaluation, using its tables and its questions. This comparison aims to emphasize the discrepancies between the quality results obtained using Spider and the practical application of models on proprietary tables. More details on the data processing phase can be found in the appendix.
suggesting
them
As models limited in the input, for every table sample subset of rows and attributes that can be executed within the model context (e.g., about 4k tokens for CHATGPT).
Category
Model
Cell precision
Cell recall
Tuple cardinality
Tuple constraint
Tuple order
Avg
PROPRIETARY
DATA
TAPAS-LARGE-WE
0.71
0.12
0.53
0.05
0.33
0.35
ECOMMERCE
TAPEX-LARGE-WTE
0.40
0.06
0.18
0.01
0.40
0.21
OMNITAB
0.20
0.01
0.14
0.00
0.50
0.17
CHATGPT 3,5
0.44
0.24
0.20
0.10
0.42
0.28
T'PAS-LARGE-WTO
0.72
0.12
0.48
0.05
0.38
0.35
FINANCE
TAPEX-LARGE-WE
0.52
0.06
0.16
0.01
0.48
0.25
OMNITAB
0.30
0.02
0.13
0.00
0.50
0.19
CHATGPT 3.5
0.71
0.52
0.38
0.21
0.48
0.46
LARAS-LARGE-WO
0.72
0.16
0.57
0.09
0.34
0.38
MEDICINE
TAPEX-LARGE-WY
0.37
0.04
0.15
0.0
0.44
0.20
OMNITAB
0.29
0.01
0.12
0.0
0.50
0.18
CHATGPT 3.5
0.77
0.46
0.22
0.12
0.70
0.45
TAPAS-LARGE-WTO
0.67
0.12
0,34
0.04
0.29
0.29
MISCELLANEOUS
TAPEX-LARGE-WTO
0.48
0.10
0.25
0.01
0.44
0.26
OMNITAB
0.12
0.02
0.13
0.01
0.50
0.17
CHATGPT 3.5
0.76
0.67
0.36
0.16
0.50
0.49
EXISTING BENCHMARK DATA
TAPAS-LARGE-WTS
0.64
0.42
0.53
0.30
0.64
0.51
TAPEX-LARGE-WTC
0.62
0.45
0.54
0.21
0.51
0.47
Spider
OMNITAB
0.30
0.24
0.53
0.23
0.52
0.36
CHATGPT 3.5
0.74
0.77
0.86
0.66
0.75
0.76
Table 3: Results for QA-TRL models: average of all tests multiple tables. ChatGPT version:
ChatGPT 3.5-tubbo-0613

---

Category
Model
Cell precision
Cell recall
Tuple cardinality
Tuple constraint
Tuple order
Avg
PROPRIETARY DATA
RESDSQL
0.91
0.89
0.92
0.81
1.00
0.90
GAP
0.84
0.80
0.81
0.73
0.97
0.83
ECOMMERCE
UNIFIEDSKG
0.71
0.71
0.69
0.69
1.00
0.76
CHATGPT 3.5
0.98
0.98
0.99
0.95
1.00
0.98
RESDSQL
0.90
0.87
0.95
0.77
1.00
0.90
GAP
0.79
0.78
0.76
0.74
1.00
0.81
FINANCE
UNIFIEDKGG
0.79
0.76
0.74
0.67
0.98
0.79
CHATGPT 3.5
0.96
0.96
0.99
0.90
1.00
0.96
RESDSQL
0.86
0.75
0.94
0.67
0.95
0.83
GAP
0.77
0.73
0.73
0.67
0.59
0.70
MEDICINE
UNIFLEDSGG
0.72
0.69
0.70
0.66
0.95
0.74
CHATGPT 3.5
1.00
1.00
0.98
0.99
1.00
0.99
RESDSQL
0.94
0.90
0.90
0.77
1.00
0.90
GAP
0.82
0.78
0.73
0.69
1.00
0.80
MISCELLANEOUS
UNIFIEDKG
0.74
0.69
0.68
0.59
0.98
0.73
CHATGPT 3.5
0.98
0.98
0.98
0.91
1.00
0.97
EXISTING BENCHMARK DATA
RESDSQL
0.93
0.93
0.97
0.84
0.99
0.93
GAP
0.95
0.95
0.96
0.91
0.96
0.95
Spider DEV
UNIFIEDKG
0.81
0.82
0.82
0.80
1.00
0.85
CHATGPT 3.5
0.93
0.96
0.97
0.92
0.90
0.94
Spider TRAIN
CHATGPT 3.5
0.90
0.92
0.92
0.88
0.97
0.92
Table 4: Results for SP-TRL models: average of all tests on multiple tables. CHATGPT version: CHATGPT 3.5-turbo-0613. UNIFEDSKG and RESDSQL evaluated with the T5 large.
Results. Tables and show the results for QA-TRL and SP-TRL, respectively. All metric results averaged over all tests and over tables grouped by category; report detailed results for every dataset and every test category in the appendix.
For the QA-TRL task, Table reports the results achieved on the proprietary tables and on the Spider benchmark to enable a comparison between the two groups. models show promising results in terms of Cell precision, which is the metric closest to the used in previous papers. However, the other metrics show lower values, with all models struggling in preserving the intra-tuple value relations in their output (i.c., low Tuple constraints). Also, in almost all experiments the models tend to remove duplicates, even when the Distinct clause not present. This evident with the low scores for Cell recall and Tuple Cardinality. Finally, none of the models is able return results according to the Order by requirements (i.e., low Tuple order scores).
Results for the SP task in Table show that, in general, the TRL models performs much better in this task. The most problematic aspect the preservation the tuple structure for the returned values (Tuple constraint).
Tests can be generated for specific dataset using QATCH in just a few seconds. The testing times varies based on the data, particularly the size of the table. For instance, comparing prediction and ground truth containing 1000 tuples and 20 attributes takes less than second.
As we do not control the infrastructure of OpenAI, we do not report exact execution times for CHATGPT. On average, it takes few seconds to execute test, but the distribution for this metrics is skewed as it heavily depends on the server workload.
Discussion. The results show that the tests generated by QATCH and metrics enable detailed evaluation of the models. A clear message is that there is no single model that works best for every table and every metric. In some datasets, TAPAS has the best performance in terms of cell precision and recall, but CHATGPT is the best model in terms of tuple cardinality and tuple constraint. The

---

lower precision for CHATGPT in some cases is due to the hallucination problem with the decoder architecture.
Another take-away is that questions and tables in Spider do not tell the full story about model performance. Even a small selection of 8 tables show that the performance for established TRL models can drop dramatically on propriety data in the QA task.
Looking into the details of the failed tests, it is easy to spot the limitations of the models. Cell precision is lowered by select queries with more than one column the output. The models tend to return only one attribute even in this setting.
In some cases, Tuple cardinality is lower than Cell recall, because the latter is over set. For some queries, models return DISTINCT by default. Most of the questions in Spider return one tuple only, thus inflating the aggregate result and hiding the problem with more output attributes. For the same reason, tuple constraint scores are low because in many cases the models do not return tuple structures, but mostly separated cells. We again observe low results for proprietary data with the higher ones for Spider.
We can conclude that the qualitative performance of a TRL model depends on the target task at hand, but also on the tabular content in terms of data domain, size, schema, and data quality.
Conclusion and Future Work
We presented QATCH, testing tool for TRL models. It provides end-users with a flexible and adaptive solution for the automatic assessment of models' performance proprietary data. The key goal is to avoid trusting the results achieved on the existing benchmarks across different tasks, as they not always replicable proprictary data. By leveraging the expressive power of the SQL language, we first generate queries of varying complexity on proprictary table and then convert them in natural language. Thus, specifically assess the model capability to address complex queries rather than the linguistic properties of questions and answers. Results show that (I) existing questions in benchmarks do not capture important properties of custom tables, (2) popular metrics fail short in measuring the quality the models' output.
As future research agenda, we plan to investigate the following research directions: (1) The extension of QATCH towards the automatic assessment of additional models (e.g., Bard (Google Inc., 2023))
LLama2 (Touvron et al. 2023), Falcon (Penedo et al.) 2023)). tasks (e.g.. fact checking (Guo et al.)
2022, Nakov et al. (021). querying LLMs (Saced et al.] 2024 Urban et al., (023)), ) and more complex queries (c.g.. GROUP BY clauses, nested queries). (2) The adoption of Generative Language Models (e.g.. (OpenAll 2023)) to automatically generate templates Natural Language from the SQL declarations. (3) The use of TRL models to test reletonelational constraints, such as joins and referential constraints, among proprietary tables.
10

---

References
Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thome, Andreas Vlachos. Christos
Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1). https
//openrevies wwt..eeeoooommidd---laaaa.......
Patricia C. Arocena, Boris Glavic, Giansalvatore Mecca, Renée J. Miller, Paolo Papotti, and Donatello Santoro. 2016. Benchmarking Data Curation Systems. IEEE Data Eng. Bull. 39, (2016), 47-62.
http://sites. compupt org/ debul1/A16june/ppl
Gilbert Badaro, Mohammed Saeed, and Papotti Paolo. 2023. Transformers for Tabular Data Represen- tation: A Survey of Models and Applications. Transactions of the Association for Computational Linguistics 11 (2023), 227-249. hhtpps//doii org/ doi org/ 10. www1///1ccae0000515
Riccardo Cappuzzo, Paolo Papotti, and Saravanan Thirumuruganathan. 2020. Creating Embeddings of Heterogeneous Relational Datasets for Data Integration Tasks. In SIGMOD (Portland, OR, USA) (SIGMOD '20). Association for Computing Machinery, New York, NY, USA, 1335-1349.
https://doi. org/10. 10. 1145/3318464 338774
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. TabFact: A Large-scale Dataset for Table-based Fact Verification. In International Conference on Learning Representations. https://opente:
id=rkeJRhNYDH
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu, 2020. TURL: Table understanding through representation learning. Proceedings of the VLDB Endowment 14, 3 (2020), 307-319.
Haoyu Dong, Zhoujun Cheng, Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dongmei Zhang. 2022. Table Pre-training: A Survey on Model Architectures, Pre- training Objectives, and Downstream Tasks. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, Lud De Raedt (Ed.). International Joint Conferences on Artificial Intelligence Organization, 1426-5435. ijcai. ijjci.2022//T Survey Track.
https://doi. http:://doi.org/10.222 wtgg10.2.919.
Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. 2021. TabularNet: A Neural Network Architecture for Understanding Semantic Structures of Tabular Data. In Proceedings of the 27th ACM SIGKDD Conference Knowledge Discovery & Data Mining. 322-331.
Orest Gkini, Theofilos Belmpas, Georgia Koutrika, and Yannis loannidis. 2021. An In-Depth Benchmarking of Text-to-SQL. Systemstssss........ In Proceedings of the 2021 International Conference on Management of Data (SIGMOD '2/). Association for Computing Machinery, 632-644. https
104.11145/344116 3452836
Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avirup Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing Row and Column Semanties in Transformer Based Question Answering over Tables. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 1212-1224.
Google Inc. 2023. BARD. https://bbrdd hww.bbbrrr..gggg....ooo wwwgge.oooooooooooo
Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A Survey on Automated
Fact-Checking. Transactions of the Association for Computationnal Linguistics 10 (02 2022), 178-206. hutpp:://dot org/10 10. 1e/////talaaaa/// a_ a_00454 pdf/doi/10. 16.1/// a 00454/1987018/aaa a 00454.pdf
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisensch- los. 2020. TaPas: Weakly Supervised Table Parsing via Pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguisties, Online, 4320-4333 https:///oi. org 11tp//11.11163.61 102000 acl mell-mmann...mm 398
11

---

Hiroshi lida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: Pretrained Representa- tions of Tabular Data. arXiv:2105.0255 [cs.CL]
Zhengbao Jiang, Yi Mao, Pengcheng He, Graham Neubig, and Weizhu Chen. 2022. OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compu- lational Linguisties: Human Language Technologies. Association for Computational Linguistics, Seattle, United States, 932-942. hittps://dos wtt.:/....rr111118//1111 18653/111222... saacclmaiin 68
Tena Ipsilantis Katsaounis. 2003, Methods of Multivariate Statistics. Technometrics 45, (2003),
100-101. https://doi.orr//00 http///..119//eee www./ttchtthhheem http://eecc.......eeee s15
George Katsogiamiis-Meimarakk and Georgia Koutrika. 2021. A Deep Dive into Deep Learning Approaches for Text-to-SQL Systems. In Proceedings of the 2021 International Conference on Management of Data. 2846-2851.
Bogdan Kostié, Julian Risch, and Timo Möller. 2021. Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering. Association for Computational Linguistics, Punta Cana, Dominican Republic, 82-91.
org/2021 www.mmraaaaa. 1.8
Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. RESDSQL: Decoupling Schema Linking and Skeleton Parsing for Text-to-SOL. In AAAI.
Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian -lan-guang Lou. 2021. TAPEX: Table pre-training via learning a neural SQL executor. arXiv preprint orXiv:2107.0765) (2021).
Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeñee Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated Fact-Checken for Assisting Human Fact- Checkerss In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, Zui-Hua Zhou (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4551 4551-4558.
https://doi. org/10g. 10. 24963/ijcai.... 2021/
619 Survey Track.
OpenAl. 2023. ChatGPT 3.5. http::///penaa..co//lllg/ccheee
OpenAI. 2023. GPT-4 Technical Report. arXiv:2303.08774 [cs.CI.]
Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and Peter Fox. 2021. CLTR: An End-to- End, Transformer-Basee System for Cell-Level Table Retrieval and Table Question Answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the Ith International Joint Conference on Natural Language Processing: System Demonstrations, Association for Computational Linguistics, Online, 209-209. https://doi... org/10.18553
v1/2021. acl demo 24
Panupong Pasupat and Percy Liang. 2015. Compositional Semantic Parsing on Semi-Structurel Tables. In ACL. ACL, 1480. https://doi. org/10. 10. 3115/v1/p15-1142
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023. The Refined- Web dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01 16 (2023).
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 4902-4912. https://doi. org/ 100.//////./ 18653/ acl wiel-mainn... 442
Mohammed Saeed, Nicola De Cao, and Paolo Papotti. 2024. Querying Large Language Models with SQL. In EDBT. https://openppoceeiiggsss ora.eigg...or///// 2024/ conf / heat////bppppppeeeeeee pdff
Johannes Schneider, Stefan Seidel, Marcus Basalla, and Jan vom Brocke. 2023. Reuse, Reduce, Support: Design Principles for Green Data Mining. Bus. Inf. Syst. Eng. 65, (2023), 65-83. httt:p:///oaa.oorr//00.00//////2
12

---

Peng Shi, Patrick Ng. Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, and Bing Xiang. 2021. Learning contextual representations for semantic parsing with generation-augmentee pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 13806-13814.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çagatay Demiralp, Chen Chen, and Wang- Chiew Tan. 2021. Annotating Columns with Pre-trained Language Models. arXiv preprint arXiv:2104.01788 (2021).
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09286 (2023).
Matthias Urban, Due Dat Nguyen, and Carsten Binnig. 2023. OmniscientDB: A Large Language Mogee-Augeented DBMS That Knows What Other DBMSs Do Not Know. In aiDM @ SIGMOD. ACM, 4:1-4:7,
https://do1 org/10 /145/3593074 3593933
Enzo Veltri, Gilbert Badaro, Mohammed Saced, and Paolo Papotti. 2023. Data Ambiguity Profiling for the Generation of Training Examples. In ICDE. IEEE, 450-463, https://doi.org/
1109/ICDE55515 2023. 2.023.00041
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien- Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.0596 (2022).
Xiaoyu Yang and Xiaodan Zhu. 2021. Exploring Decomposition for Table-based Fact Verification. In EMNLP. Association for Computational Linguistics, Punta Cana, Dominican Republic, 1045-1052.
attps://aclanthologe org. 2021. findings-emn.] 90
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 8413-8426. patps://doi: org/10. 10. 18653/v1/2020. acl -main. maain.455
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang. Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. 2021. GraPPa: Grammar Augmented Pre-Training for Table Semantic Parsing. In International Conference on Learning Representations. https
//openreview net.
Tao Yu, Rui Zhang. Kai Yang. Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Pross-Domane Semantic Parsing and Text-to-SQL Task. In EMNLP. ACL, 391 1-3921. https:///oo 19t.3/00....63111111 1425
13

---

A Authors' statements
A.1 Code license and legal compliance
The source code of project is released under the Apache-2.0 license. All the authors are aware of the legal implications and gave their explicit consent to publish the paper and the code.
A.2 Reproducibility
The code and the related documentation is available at: https://githuu com/spapicchioo/ QATCH The source code is annotated in compliance with the ML reproducibility checklist pro- vided by NeurlPS 2023 organizers and available at hhtps: bstps://githube com/ http//pppptthhooeooooo selleasing-research-comm research-code We also adhere the official guidelines for the specification of the code dependencies. Notice that TRL models are referenced but not directly included in the main repository we exclusively rely on third-party models.
A.3 Long-term preservation
We guarantee that the repository, including the project code and annotations, will be maintained in the long term. Any potential issues that will be raised by end-users will be taken into consideration.
A.4 Privacy
In this work, we relcased a new testing toolbox for proprictary data. We neither released new publicly available datasets nor involved humans in the evaluation loop.
While using QATCH the input data, the tests, and the expected and achieved outcomes always remain under the full end-user's control as long TRL models QATCH allows a secure testing of proprietary tables by running models and tests on a local machine.
premises (no external APIs). Thus,
For the sake of completeness, QATCH also supports an external QA model, i.e., OpenAl ChatGPT 3.5. While using it, end-users must aware of the fact that their own data are shared with private subjects in compliance with OpenAl terms and policies /tttps://oppna... com htt//pooiiii
A.5 Ethics
There no particular ethical implications of our work. Every ethical issue related to the tested data under the liability the data owner.
A.6 Bias and fairness
The QATCH test generation module randomly picks existing content and/or metadata from the input tables. While running the tests, recommend to also vary the seed of the random generator to maximize the reliability of the tests' outcomes. Mitigating the eventual presence of bias in the input data and testing TRL model fairness are both out of scope of the present work.
A.7 Impact the environment
Testing TRL models proprictary data prevents the use of expensive and resource demanding cloud services. Since QATCH allows end-users to optimize the computational efforts by choosing the right model to use. Concerning that point, its use can be deemed example of green data mining Schneider et al. (203).
B Additional details on data, models, and tests
Tables' description. Here we describe the tables used in the experimental evaluation. We col- lected results on eight proprietary tables and on one benchmark dataset established for TRL model evaluation Yu et al. (2018)
14

---

We considered proprictary tables of various domains (i.e., E-Commerce. Finance, Medicine, and Miscellaneous) and characteristics (e.g.. tuple cardinality from few hundreds more than I million, table dimensionality from 8 to 30) to ensure the robustness of the empirical outcomes. The detailed statistics are given in Table of the main paper.
In Table [ we reported similar statistics for the Spider benchmark dataset. These statistics were not included the main paper due the lack of space.
Proprietary data preprocessing. QATCH supports standard text pre-processing procedures for the table content and metadata. For example, the Mushrooms table contains only alphabetical letters thus conveying limited information. We applied a text reconciliation step, where, for instance, in the Heart-attack table the attribute value cp is replaced by the (more meaningful) expression chest pain.
Spider preprocessing. The primary objective of QATCH is assess the impact of proprietary data on model performance. To achieve this goal, QATCH tests are deliberately designed to explore model behavior in simplified scenarios, avoiding intricacies in both the generated questions and queries. In contrast, Spider, having a different scope, inherently contains a higher level of complexity. To ensure fair comparison between the proprietary data-driven QATCH tests and the Spider dataset, preprocessing step is applied for the Spider dataset. Specifically, we excluded queries involving the LIMIT, INTERSECT, UNION, JOIN and EXCEPT operators, well those with inner queries.
Benchmark vs. proprictary data. The tables included in the Spider benchmark have quite different characteristics compared to the proprietary ones (fewer tuples and dimensions, fewer numerical attributes).
For example, in Table 5) we report the comparison between the Spider tables and the selected proprietary tables. The number of cells per table in Spider is significantly lower in 75% of the cases. This confirms the limited representativeness of the currently used benchmark data for TRL. model evaluation and fosters the need of TRL testing tools proprietary data.
% percentile
50 % percentile
75% percentile
minimum
maximum
tuples # attributes
0
7
12
15
16049
Spider tables
2
4
7
24
cells
0
30
100
112343
# tuples
303
656
5283
149k
IM
Proprietary tables
attributes
8
11
12
17
30
# cells
3636
7213
108k
1.4M
30M
Table 5: Comparison between the Spider and proprietary tables' statistics. It reports the minimum and maximum values, and the 25%, 50%, and 75% percentile over all the Spider tables and the considered proprictary tables.
Tackling TRL model limitations. To meet the TRL models' constraints on the maximum number of processed cells, whenever strictly necessary QATCH reduces the input data size by sampling tables attributes and rows using the same random seed for all tables. To avoid introducing bias in the data, we tested multiple dataset variants consisting of different attribute and tuple selections. More specifically, to address Question Answering we generated all the tests by exclusively considering the table schema, except for the Select queries where the tuple-level conditions are generated by considering representatives the maximum, minimum, or mean attribute values computed separately for each table. The main goal here is to test whether the TRL model can accurately answer queries including arbitrary selection conditions.
The inherent limitations of TRL models to handle larger tables do not influence Semantic Parsing tests, as they directly work at the schema level. Notice that such limitations are independent of the QATCH implementation, but can be conveniently highlighted while running the tests on proprietary data.
15

---

Table
Selected
f of Tapas-wiq lapex-wiq ChatGPT Omnitab
# of rows # of rows # of rows
name
columns
Sales-transactions
TransactionNo, ProductNo, Product, Price Quantity, CustomerNo, Date, Country
60
20
40
20
link
Brand Name, Device Type. Original Price,
trackers
Selling Price, Color, Model Name, Average Battery Life (in days), Rating (Out o) Strap Material, Display
50
255
30
20
link
income, payment_lyyp has mass_other___e cards,
Account-fraud link
email _i____ffre free, employment status, housing. status, date of birth distinct distinct_emmaii eemailsee_____eeeeeaae 4w, device os credit risk score, session length in minutes
50
25
30
20
CustomerID, PaperlessDate, InvniceNumber, Invoice Date, Duelate, InvoiceAmount,
Late-payment link
Disputed. PaperlessBill. DaysToSettle, DaysLate
50
25
20
Heart-attack
age, sex, сp, trtbps, chol, fbs, restoeg, caa, thall, output
45
30
30
20
link
Breast-cancer
pid, age, meno, size, grade, nodes, pgr, er, hormon, status
45
30
30
20
link
workelass, education, marital.status
Adult censuus
occupation, relationship, race, sex, hours.per. week, native.country. income
50
20
30
20
link
class, cap cap-shape, cap-surface, cap-color, bruises, odor, gill-atrachment, gill-spacing, gill-size, gill-color
Mushrooms
50
25
30
20
link
Table 6: Datasets information with maximum number of rows accepted for each model in the QA Scenario.
In Table we report the dataset, the list of table attributes, and the maximum number of tuples that each TRL model (and the corresponding tokenizer) manages to process. It is worth noticing that, even though Tapex is based on BART (maximum token limit: 1024) and Tapas is based on BERT (token limit: 512), the number of tuples that Tapas is able to process is much larger. The link to each dataset along with the pre-processing strategies are available in the github repository.
Performance metrics. Our evaluation metries rely the predicted cell values, making them essential in the SP scenario. To begin with, we employ models to predict SQL queries, which are subsequently executed on tables to retrieve the corresponding predicted cell values. This approach allows us to derive valuable insights from the metric scores associated with the SQL predictions. For instance, a low Tuple constraint metric indicates that the SQL prediction likely projecting incorrect columns. Instead, low tuple cardinality metric suggests that the SQL prediction may be selecting incorrect rows or utilizing a distinct operator inappropriately. Table shows three instances of unsuccessful predictions.
Table linearization. Using ChatGPT, which processes text sequentially from left to right, necessi- tates converting tabular data into a linear format for seamless integration. This process, known as linearization, entails transforming structured tabular data into a sequential layout that aligns with ChatGPT's processing method. A common technique involves concatenating cell contents within row, using specific delimiters for separation. However, more promising approach is to enhance this by appending column headers to cell content, incorporating structure-aware indicators as described in Badaro et al. (023). The approach used in this work, illustrated in Figure 4 represents the table as a list of lists, using Python-like syntax. In this representation, each inner list corresponds to distinct row in the table, with elements indicating values associated with respective columns marked by the
16

---

Cell
Cell
Tuple
Tuple Tuple
precision recall cardinality constraint order
SELECT DISTINCT emailisfree FROM fraud Prediction SELECT emailisfree, income FROM fraud
Target
0.5
1.0
0.2
0.0
Target SELECT emailisfree FROM fraud ORDER emailisfree ASC
1.0
1.0
1.0
1.0
0.0
Prediction SELECT emailisfree FROM fraud ORDER BY emailisfree DESC
Target
SELECT FROM fraud
1.0
0.10
1.0
0.0
Prediction SELECT emailisfree FROM fraud
Table 7: Examples of predictions and performance metrics.
B
[al, [H] AJ. [61, [H] BJ. [el, [H] C). [a2, [H] AJ, [b2, [H] BJ. [c2, [H]
b1
c1
a2
b2
Figure 4: Example of the linearization technique used in this study to process structured data with ChatGPT.
structure-awart aware indicator [H]. This format ensures the preservation of the table's structural integrity and facilitates seamless processing within the ChatGPT framework.
ChatGPT reproducibility. In this study we use ChatGPT 3.5 4.5-turo--0613 with the "in-context" task adaptation technique. In-context learning refers to the ability of models like GPT-3 consider the immediate context of conversation or text when generating responses. This means the model does not just look the current question or prompt, but also the preceding conversation or text. For instance, in-context learning can be seen as form of few-shot learning where the "shots' are the previous turns in the conversation. the attributes (ii) the projection of one column (iii) one aggregate condition.
our study, we pass three examples] (i) the projection of all
C Additional results for Question Answering
Table g compares the performance of the QA models on proprietary data.
By examining the achieved results, we identified the following patterns and trends in the models' performance:
Tapas and Tapex consistently exhibit higher cell precision but lower cell recall. This indicates that their predictions are often correct yet incomplete.
For the Projection queries, the models understand which columns include but only return a portion of the expected values.
All the models fail to return values the expected tuple format. This indicates that the models struggle comprehend the structure of the table.
Although ChatGPT generally achieves high overall score, is not always the best per- forming model. Specifically, struggles to cope with the Sales-tae cillsstransaction and pateepaymentttto tables. Tuple cardinality and Tuple constraint metrics were consistently below 0.50, indicating that the returned outcomes are still far from being optimal.
Table 99 provides additional insights by showcasing the SQL patterns in which ChatGPT per- formed poorly. It performs fairly good when the number of attributes in the output is limited (c.g.. AESTNCT-SSNGLAS Conversely, it struggles when larger number of attributes and values need to be considered, especially they mainly include numerical attributes. For example, Sales-ta wwwessstranactiiiim has 7 out of 8 columns numerical, while Late paateepaymnnt has 7 out of 10 columns as numerical.
More details can be found in the GitHub http::::/gghhuu..om/ppppaaaaaaaa
17

---

The same trend is also evident in the improved performances of ChatGPT when dealing with tables the same category, such as Fitness- Fitness-trackers trackers, which has categorical attributes of 10, and Fraud, which has 7 categorical attributes out of 10 even more noticeable, in the Mushrooms table where all the attributes are categorical.
Table category
Table
Model
Cell precision
Cell recall
Tuple cardinality
Tuple constraint
Tuple
name
order
TAPAS-WTQ
0.75
0.13
0.67
0.05
0,50
TAPEX-WTQ
0.40
0.07
0.21
0.02
0.28
Sales taaesttransactions
OMNITAB
0.24
0.02
0.17
0.0
0.50
CHATGPT
0.15
0.03
0.05
0.02
0.0
ECOMMERCE
TAPAS-WTQ
0.60
0.08
0.32
0.04
0,20
TAPEX-WTQ
0.45
0.06
0.16
0.0
0.50
Fitness- trackers
OMNITAB
0.16
0.01
0.12
0.01
0,50
CHATGPT
0.67
0.46
0.29
0.27
0.44
PAPAS-TTQ
0.64
0.11
0.38
0.06
0.35
TAPEX-WEL
0.51
0.07
0.14
0.01
0.50
Account
OMNITAB
0.26
0.02
0.11
0.00
0.50
CHATGPT
0.68
0.61
0.42
0.17
0.48
FINANCE
TAPAS-WTQ
0.77
0.12
0.55
0.05
0.40
TAPEX-WTQ
0.48
0.05
0.18
0.0
0.45
Late sate--
OMNITAB
0.35
0.02
0.16
0.0
0.50
CHAIGPT
0.13
0.05
0.04
0.03
0.0
TAPAS-W WTQ
0.60
0.13
0.45
0.08
0.23
TAPEX-WTQ
0.31
0.05
0.16
0.01
0.38
Heart-attack
OMNITAB
0.19
0.01
0.11
0.0
0.50
CHATGPT
0.87
0.74
0.41
0.35
0.58
MEDICINE
TAPAS-WTQ
0.83
0.18
0.67
0.10
0.45
TAPEX-WTQ
0.42
0.04
0.15
0.0
0.50
Breast-cancer
OMNITAB
0.37
0.02
0.13
0.0
0.50
CHATGPT
0.72
0.31
0.14
0.10
0.75
TAPAS-WTQ
0.59
0.12
0.34
0.06
0.25
TAPEX-WTQ
0.59
0.12
0.16
0.0
0.45
Adult-census
OMNITAB
0.04
0.01
0.13
0.01
0.50
CHATGPT
0.71
0.68
0.47
0.35
0.58
MISCELLANEOUS
TAPAS-WTQ
0.68
0.12
0.27
0.03
0.33
TAPEX-WTQ
0.44
0.10
0.33
0.02
0.43
Mushrooms
OMNITAB
0.17
0.03
0.13
0.02
0.50
CHATGPT
0.77
0.75
0.49
0.37
0,56
Table 8: Results for QA-TRL for the different proprietary tables.
D Additional results for Semantic Parsing
Table 10 compares the performance of the SP models proprietary data. Based the achieved results, RESDSQL performed worst the MEDICINE category. To delve deeper into the results, we fo- cused on the table displaying the lower metric score, which can be found in Table II Heart-attack exhibits remarkable low values cell recall across all where SQL categories. These inaccuracies be attributed column the heart-attach. table appears to confuse the model when attempts to project all columns. our hypothesis is confirmed by the presence of the same error in the SELECT- ALLL category.
misprojection made by the model during prediction. The presence of the output
18

---

Table
SQL
Cell precision
Cell recall
Tuple cardinality
Tuple constraint
Tuple
category
category
order
SELECT-ALL
0.00
0.00
0.00
0.00
SELECT- COL
0.43
0.03
0.03
0.03
SELECT-RANDOM- COL
0.38
0.07
0.02
0.02
ORDERBY-SINGLE
0.00
0.00
0.00
0.00
0.00
DISTINCT-MULT
0.40
0.10
0.01
0.01
DISTINCT-SINGLE
0.28
0.10
0.03
0.20
0.00
WHERE-CAT-OE CIN--MATMNNNN MIN-VALUES
0.05
0.01
0.10
0.00
WHERE-NUM. NUM-MAX -VAUUSS
0.00
0.00
0.00
0.00
WHERE- NUM-MEAN MEAN VALUES
0.00
0.00
0.00
WHERE-NUM -UMAE-NMMMMIN -VAUES
0.00
0.00
0.00
0.00
SELECT-ALL
0.00
0.00
0.00
0.00
SELECT-ADE ADD-COD
0.33
0.04
0.03
0.03
0.30
0.12
0.04
0.03
ORDERBY-SINGLE
0.00
0.00
0.00
0.00
0.00
DISTINCT-MUR
0.33
0.18
0.18
0.18
sate-payment
SEPINCT-SINGGLA
0.97
0.45
0.46
0.45
WHERE THERE-CAT-O CAT-MAM-- -VAUUES
0.08
0.02
0.01
0.00
WHERE-CAT CATRE-CAT-ME MAATMIN-VVALLI
0.08
0.02
0.01
0.00
WHERE-NUM --MMMMUMMMA----- MAX-VALUES
0.00
0.00
0.00
0.00
WHERE-NUM. KUM-MEAN MEN--E-N-VVLUES
0.00
0.00
WHERE-NUM. NUNG--UMMMMMN WWN-MIN-VAALAAAAAAAAAAAAAAAAAAA- VALUES
0.01
0.00
0.01
0.00
Table 9: Results for QA chatGPT on Sales transactions and Late-payments
19

---

Table category
Table name
Model
Cell precision
Cell recall
Tuple
Tuple constraint
Tuple
cardinality
order
RESDSQL.
0.91
0.90
0.90
0.83
1.00
GAP
0.85
0.80
0.82
0.71
0,94
Sales slles-transactons
UNIFIDDKG
0.79
0.77
0.76
0.74
1.00
CHATGPT
1.00
1.00
0.99
1.00
1.00
ECOMMERCE
RESDSQL.
0.90
0.88
0,94
0.79
1.00
GAP
0.84
0.80
0,80
0.74
1.00
Fitness-trackers
UNIFIEDSGG
0.65
0.65
0.65
0.65
1.00
CHATGPT
1.00
1.00
1.00
0.90
1.00
RESDSQL
0.89
0.86
0.95
0.77
1.00
GAP
0.79
0.77
0.75
0.73
1.00
account-frate
UNIFIEDKGG
0.80
0.75
0.68
1.00
CHATGPT
0.99
0.99
0.98
0.90
1.00
FINANCE
RESDSQL
0.90
0.87
0.95
0.77
1.00
GAP
0.80
0.79
0.78
0.74
1.00
Late- payment
UNIFEDSKG
0.77
0.76
0.71
0.65
0.95
CHATGPT
1.00
1.00
1.00
0.90
1.00
RESDSQL
0.88
0.67
0.93
0.57
0.90
GAP
0.92
0.85
0.88
0.79
0.93
Heart-attack
UNIFIEDSKG
0.72
0.66
0.71
0.61
0.90
CHATGPT
0.92
0.92
0.99
0.92
1.00
MEDICINE
RESDSQL
0.84
0.83
0.95
0.77
1.00
GAP
0.64
0.63
0.60
0.56
0,25
Breast-cancer
UNIFIEDSKG
0.72
0.71
0.70
0.70
1.00
CHATGPT
0.95
0.95
0.99
0.95
1.00
RESDSQL
0.92
0.88
0.90
0.76
1.00
GAP
0.71
0.67
0.58
0.60
1.00
Adult- census
UNIFIEDSKG
0.69
0.64
0.63
0.58
0.95
CHATGPT
0.99
0.99
0.98
0.83
1.00
MISCELLANEOUS
RESDSQL.
0.96
0.91
0.91
0.78
1.00
GAP
0.94
0.89
0.89
0.80
1.00
Mushro ooms
UNIFIDSKG
0.80
0.74
0.72
0.61
1.00
CHATGPT
0.99
0.99
0.99
0.99
1.00
Table 10: Results for SP-TRL for the different proprietary tables.
SQL
Cell precision
Cell recall
Tuple cardinality
Tuple constraint
Tuple order
category
SELECT-ALL
0.00
0.00
0.00
0.00
SELECT-ADD-COL
1.00
0.79
1.00
0.56
DELECT-RANDOX,
1.00
0.72
1.00
0.40
ORDERBY-SINGLA
1.00
0.99
0.91
0.91
1.00
DISTINCT-SINGLE
1.00
1.00
1.00
1.00
DISTINCT-MULT
1.00
0.79
0.62
0.51
VALUES
1.00
0.28
1.00
0.20
CCTTMIIN-AALO VALUES
1.00
0.28
1.00
0.20
WHERE-NUM- VAL
1,00
0.86
0.98
0.83
MANN-MMENN-VLUESS
1.00
0.78
1.00
0.75
WHERE-NUN-KIN VVLUES
0.80
0.71
0.80
0.70
Table 11: RESDSQL results for the heart-attack table.
20


Reviews:
[{"review_id": "B3hObWUkoK", "reviewer": "Reviewer_GkAC", "summary": "", "questions": "", "limitations": "There appears to be no discussions of limitations specifically called out. In section 5, there is mention of augmenting the framework with more complex queries containing GROUP BY clauses and nested queries, which would be nice to call out explicitly.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The contribution of generating test cases from various tables is clever and immediately useful to the problem. The templated approach to focus on structural question complexity rather than linguistic complexity is appreciated.\n\nI appreciate the discussion of details of failed tests near the end of Section 4 and what they say about the limitations of the model.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "09KVsg9c1e", "reviewer": "Reviewer_U5Jd", "summary": "", "questions": "", "limitations": "I don't see limitations in terms of potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A benchmark of table representation learning is proposed. \n\n2. An evaluation framework is proposed, avoiding trusting the results achieved on the existing benchmarks across different tasks, as they are not always replicable on proprietary data. \n\n3. The experiments reveal some interesting findings.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "HxWkeSh4m9", "reviewer": "Reviewer_aVca", "summary": "", "questions": "", "limitations": "While the main contribution of this paper is to validate the performance of the TRL model on a proprietary dataset, partial experiments on a benchmark dataset should also be conducted to demonstrate the validity of the methodology and model. The lack of comparison with the SOTA makes the experiments less credible. We appreciate the motivation of this paper, and it is a good work to evaluate the performance of TRL models on proprietary datasets, but based on the experiments in this paper alone, we still do not know which TRL model is advantageous on which proprietary dataset.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "This manuscript presents the QATCH toolbox for automatic generation and evaluation of test check-lists customized for proprietary data, which enriches research on the specificity and adaptability of TRL models. The authors experimentally evaluate the performance of a number of TRL models in question-answering tasks.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "knzAzicwwC", "reviewer": "Reviewer_dxvp", "summary": "", "questions": "", "limitations": "Some limitations are discussed under discussions. Limitation of the approach for tasks other than QA and SP are not discussed.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The benchmarking approach is well motivated \n- Some key insights are provided that highlight a number of interesting future directions \n- Open source and well documented code base", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper describes a framework for evaluating table representation learning models on unseen data for the tasks of question answering and semantic parsing.  It does so by generating queries against the unseen (proprietary data) and then using a set of newly introduced metrics. The test generator was highlighted as an important and useful contribution to the paper. Overall, I think this question of how to evaluate on unseen data and providing the framework to do so would be helpful for furthering the exploration of figuring out how to learn good table representations. In addition, for a benchmark and datasets I find the quality of the resource itself to be important. Here I would commend the repository itself for being clear and also guiding on how to use the framework.

I would commend the reviewers and the authors for a very informative back and forth. 

I'm recommending acceptance under the assumption that the authors will make all the changes that they identified based on the reviews comments. In particular, I would call out:
- Make sure the title and abstract correctly convey the contribution (e.g. Benchmarking SQL-centric tasks with Table Representation Learning Models on Your Data)
- Add explications about the difficulty of the tasks and also the additional supporting evidence thereof.
- Add the additional clarifications about the metric definitions.
Decision: Accept (Poster)