SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Simone Papicchio
Paolo Papotti
EURECOM
Luca Cagliero
Table Representation Learning (TRL) models are commonly pre-trained on large open-domain datasets comprising millions of tables and then used to address downstream tasks. Choosing the right TRL model to use on proprietary data can be challenging, as the best results depend on the content domain, schema, and data quality. Our purpose is to support end-users in testing TRL models on proprietary data in two established SQL-centric tasks, i.e., Question Answering (QA) and Semantic Parsing (SP). We present QATCH (Query-Aided TRL Checklist), a toolbox to highlight TRL models' strengths and weaknesses on relational tables unseen at training time. For an input table, QATCH automatically generates a testing checklist tailored to QA and SP. Checklist generation is driven by a SQL query engine that crafts tests of different complexity. This design facilitates inherent portability, allowing the checks to be used by alternative models. We also introduce a set of cross-task performance metrics evaluating the TRL model's performance over its output. Finally, we show how QATCH automatically generates tests for proprietary datasets to evaluate various state-of-the-art models including TAPAS, TAPEX, and CHATGPT.
Table Representation Learning (TRL) models are getting increasing attention for their ability to support various NLP downstream tasks involving tabular data (Badaro et al., 2023; Dong et al., 2022). Such models are built using large open-domain datasets during pre-training and can then be fine-tuned with labelled examples for the target task.
Motivation. In most settings, companies aim at adopting a pre-trained model to reduce costs. However, despite the abundance of such methods, it is still challenging to select and manage a TRL model for proprietary data. Models use different pre-training tasks and datasets, work under different assumptions, and the top performing model for an existing benchmark is not necessarily the best one when tested on proprietary data. Indeed, the performance of models fine-tuned on benchmark data is not necessarily replicable on proprietary data.
In a corporate setting, there are at least three scenarios where a given TRL model needs to be evaluated against proprietary datasets:
- Comparison: Compare TRL models fine-tuned on private examples to see which one performs best.
- Validation: As crafting examples is expensive, verify when the quality meets the requirements.
- Maintenance: Fine-tuned models need to be re-calibrated to avoid data and conceptual shifting, continuous evaluation helps the identification of this issue.
Challenges. Supporting these scenarios with accurate evaluation is a not obvious task. For the first scenario, consider an engineer who is selecting the TRL model for a Question Answering (QA) task
over their enterprise tabular data. Several options are available and they start assessing the models for a zero-shot setting, where the model is used as-is. As labelled examples are not available, the engineer has to craft some test data to assess how the model is performing on their proprietary tables, e.g., write pairs of questions in natural language (NL) and the expected data results.
The task above has a cost that increases with the number of tables and tests crafted by the engineer. Moreover, the process above applies for all the target tasks that the engineer wants to support. For example, they would have to repeat the exercise for the same model and datasets to evaluate its suitability for Semantic Parsing (SP, aka Text2Sql), where, given a question in NL and a table schema, the model returns a SQL query.
Creating tests is only half of the story. Test results must be evaluated, i.e., the data in the QA model's output  $D_{M}$  should "match" the expected output manually crafted in the test  $D_{T}$ . A simple equality test fails short, as the tuples (or values) in  $D_{M}$  can be in different order w.r.t. those in  $D_{T}$ . Different order for records and attributes does not matter in the relational model, so the test should be independent from this difference. While existing systems are evaluated with a naive accuracy metric that handles these issues, many problems that make this matching hard are ignored. First, they do not test results for relational data integrity, i.e., if tuple and attribute relationships are satisfied. Second, consider tuples in  $D_{M}$  that are similar to the expected ones in  $D_{T}$  except null values, or that differ in the order when the question requires that results are sorted - these issues are ignored by existing metrics. The same observations apply for the Semantic Parsing task, as it is a common practice to evaluate the produced SQL scripts on their results once executed (Li et al., 2023).
Automatic Assessment. QATCH (Query-Aided TRL Checklist) addresses the challenges in evaluating TRL models for proprietary data over SQL-centric tasks. QATCH tests TRL models by automatically generating and evaluating testing checklists of varying complexity for QA and SP.
Figure 1: QATCH takes a table as input and returns metrics for TRL models M1 and M2.
Given a proprietary table, such as the one in Figure 1 (step ①), QATCH produces questions in NL such as the one in the figure (step ②). The questions are passed, together with the input table, to the TRL models to evaluate (step ③). The models' output is then consumed by our tool (step ④) to produce an assessment of the results (step ⑤).
At the heart of QATCH, a query generation algorithm crafts tests covering a comprehensive range of features enabled by the expressive power of SQL. Our objective is to assess the capability of the models
concerning question complexity. To achieve this, we generate the corresponding NL questions using templates designed to maintain clarity and precision. This approach ensures that we do not inadvertently test the models' capacity to handle complex textual content, as our primary focus is on their performance in relation to question complexity.
QATCH also introduces cross-task performance metrics that evaluate the models' results while accounting for the nuances in the comparison of tabular outputs. To address the complexities in evaluating model outputs, our metrics consider numerous factors when comparing databases, including the cardinality of the results, presence of null values, missing or extra cell values, and sorting when required by the NL questions. By taking these aspects into account, QATCH provides a robust and insightful evaluation of model performance.
Our work differs from existing rigid benchmarks, such as Spider (Yu et al., 2018), which cannot capture how well a model performs on proprietary data. Instead, QATCH supports end-users in corporate settings by facilitating decisions on TRL model comparison, validation and maintenance.
- We present QATCH, a toolbox for automatically generating and evaluating test checklists tailored to proprietary data and SQL-centric tasks. Code, data, and results are available at https://github.com/spapicchio/QATCH.
- We exploit a SQL query generator to craft a testing checklist for multiple TRL models and two established tasks, i.e., Question Answering and Semantic Parsing.
- We introduce robust performance metrics that capture the nuances in comparing model outputs, such as cardinality, null values, missing or extra cell values, and sorting requirements.
- By executing tests of increasing complexity over four TRL models and 8 proprietary datasets, we report insights into the models' capacity to handle two SQL-centric tasks on tabular data.
Table Representation Learning. Table Representation Learning (TRL) refers to the process of developing and training neural models to capture the underlying structure, semantics, and relationships in tabular data. TRL is used for tasks both in natural language processing (NLP) and data management. Such models support data-driven systems that surpass the limitations of traditional declarative specifications based on first-order logic and SQL.
Examples of tasks that use these models include answering questions in natural language (Katsogiannis-Meimarakis and Koutrika, 2021; Herzig et al., 2020; Liu et al., 2021), fact-checking (Chen et al., 2020; Yang and Zhu, 2021; Aly et al., 2021), semantic parsing (Yin et al., 2020; Yu et al., 2021), table retrieval (Pan et al., 2021; Kostic et al., 2021; Glass et al., 2021), table comprehension (Suhara et al., 2021; Du et al., 2021), and table content prediction (Deng et al., 2020; Iida et al., 2021). These models are built with different architectures (encoder only, decoder only, encoder+decoder), but our approach is agnostic to this aspect and evaluates any model that can satisfy the input/output requirements of a SQL-centric task, i.e., the output may be obtained with a SQL query. In this work, we focus on two tasks that satisfy these requirements: Question Answering (QA) and Semantic Parsing (SP).
Question Answering. In the context of free text, QA aims to retrieve passages containing the answer to a given question. In the tabular data setting, QA involves returning the cells that answer a given query, with the input consisting of a question and a table (Herzig et al., 2020; Liu et al., 2021). There are two levels of complexity in tabular QA tasks. Simple QA focuses on lookup queries on tables, while more complex QA tasks require aggregation operations and numerical reasoning.
Semantic Parsing. SP in the tabular data setting involves generating a declarative query in SQL over the table's schema, given a question and a table as input (Yin et al., 2020; Liu et al., 2021; Yu et al., 2021; Li et al., 2023). The purpose of SP is to retrieve the answer to the question by producing an interpretable query rather than directly obtaining the answer. Unlike QA, where the focus is on finding the answer cells, SP emphasizes the generation of a structured query.
Other tasks, such as Table Metadata Prediction (TMP) and Table Content Population (TCP), are also widely covered by TRL models, but they do not follow the task requirements. TMP focuses on predicting inter-table metadata, such as column types, headers, cell types, and table types, as well as intra-table relationships, such as equivalence between columns and entity linking/resolution (Deng et al., 2020; Cappuzzo et al., 2020). Meanwhile, TCP addresses the recovery of corrupted cell content and aims to impute missing cell values in an input table (Iida et al., 2021). This serves to enhance and improve the consistency of table information. However, those are traditional DB problems over tabular data and there exist some relevant benchmark solutions relying on metadata creation (Arocena et al., 2016).
Previous Benchmarking Approaches. Several benchmarks have been developed for QA and SP models for measuring model performance on fixed datasets. Examples include those closer to QA (Pasupat and Liang, 2015; Chen et al., 2020) and those for SP (Yu et al., 2018; Gkini et al., 2021). WikiTableQuestions is designed for evaluating table comprehension tasks, with a dataset consisting of questions about tables from Wikipedia. TabFact focuses on fact-checking and comprises statements about tables that can be labeled as "true" or "false". Spider, and similar efforts, are corpora of databases, each with a set of SQL queries to test SP. For each query, they provide the paraphrases of the query as a NL question. These datasets have played a role in advancing TRL models across various tasks. However, they are inherently limited due to their fixed set of crafted examples and lack of test generation for different TRL models given only the proprietary datasets.
Moreover, previous approaches have relied on simple accuracy metrics, which test if the produced values are included in the ground truth. This "execution accuracy" is adopted also for SP as it is more precise than just comparing SQL scripts (Li et al., 2023; Yu et al., 2018). However, this metric fails to
Figure 2: Workflow of QATCH: given proprietary relational tables as input, it generates tests for Question Answering and Semantic Parsing; tests are executed on a given TRL model; the evaluation metrics are computed between predictions (model output) and ground-truth results.
capture nuances in comparing data instances. For example, it does not measure whether values from the same tuple in the ground truth output appear in the same manner in the model output. As a result, this simplistic metric overlooks aspects of TRL models' performance and motivates our proposal.
In terms of test generation, our work took inspiration from an existing effort in measuring the quality of traditional text language models (Ribeiro et al., 2020). Unlike QATCH, Ribeiro et al. (2020) focuses on behavioral testing of NLP models across multiple tasks. Conversely, we leverage the expressive power of SQL to evaluate the test model outcomes directly on tabular data.
3 The QATCH Toolbox
QATCH's automatically generates and evaluates test checklists on TRL models based on the three-step process depicted in Figure 2.
1. QATCH-Generate. It generates a set of queries tailored to proprietary data. For each query it formulates both the SQL declaration, its free-text version, and the expected ground truth consisting of table instances. The SQL declaration expresses the logical complexity of the query and reflects the presence/absence of specific features peculiar to relational data model.
2. TRL Model Prediction. It processes the tests for various TRL models and tasks. The toolbox supports alternative TRL models for both the QA and the SP tasks.
3. QATCH-Evaluate. It evaluates the models outputs according to a set of cross-task performance metrics. Separately for each model, it provides end-users with the results for a testing checklist to gain insights into TRL models' strengths and weaknesses.
We describe the tasks covered in the toolbox next. We then explain the test generation. Finally, we discuss the performance metrics.
3.1 Question Answering and Semantic Parsing based on TRL models
Question Answering aims at retrieving the answer A to a given question Q, where both Q and A are expressed in natural language. Here we focus on retrieving the content necessary to formulate the answer from a proprietary relational table  $T$  encoded by a TRL model M.
We specifically address two sub-tasks tailored to Table Representation Learning Models, i.e., Question Answering from TRLs (QA-TRL, in short) and Semantic Parsing based on TRLs (SP-TRLs).
Question Answering from TRLs. Let  $T$  be a relational table and let  $S_{T}$  and  $I_{T}$  be  $T$ 's schema and instance, respectively. Executing a SQL declaration SQL on  $T$  entails retrieving a (ground truth) result  $R_{SQL}$  derived from  $I_{T}$ :
$$
R _ {S Q L} = \operatorname {Q U E R Y} (S Q L, S _ {T}, I _ {T}) \tag {1}
$$
In QA-TRL we do not query the original table  $T$ , but rather the TRL model M that encodes  $T$ . This allows us to pose the query in natural language and obtain the result directly from the model.
Let  $\mathrm{SQL}_Q$  be a SQL declaration and let  $Q$  be one of its free-text reformulations, i.e., a question expressed in natural language that is semantically equivalent to  $\mathrm{SQL}_Q$ . The goal of QA-TRL is to ask question  $Q$  on  $M$  (which encodes  $T$ ) to retrieve a result  $R_Q$
$$
R _ {\mathrm {Q}} = \mathrm {Q A} - \operatorname {T R L} (\mathrm {M}, \mathrm {Q}) \tag {2}
$$
Category	SQL declaration	Free-Text question
Project	SELECT {c1,...,cn} FROM {T}	Show {c1,...,cn} in table {T}
Distinct	SELECT DISTINCT {c1,...,cn} FROM {T}	Show the different {c1,...,cn} in table {T}
Select	SELECT * FROM {T} WHERE {ci} {op} {val}	Show data of table {t} where {ci} {op} {val}
Order by	SELECT * FROM {T} ORDER BY {ci} {ord}	Show data for table {T} in {ord} order by {ci}
Table 1: Templates for queries in SQL and natural language.  $T$  is the target relational table.  $c_{i} \in \mathbf{S}_{T}$ $(1 \leq i \leq n)$  is an attribute of the  $T$ 's schema,  $op (=, !=, >, <, \ge, \le)$  is a logical operator,  $\mathrm{VAL}_i$  is an arbitrary value for attribute  $c_{i}$  occurring in  $\mathrm{I}_{T}$ .  $ord$  is the order of visualization of the tuples in the output (i.e., ascending or descending).
s.t.  $R_{\mathbf{Q}}$  is equivalent (or mostly similar, in the worst case) to the expected outcome  $R_{SQL}$  from  $T$ .
Semantic Parsing using TRLs. Given a free-text question  $Q$  and a relational table  $T$ , the goal of SP is to map  $Q$  to the corresponding SQL declaration  $\mathrm{SQL}_{Q}$ .
In SP-TRL we leverage the TRL model  $M$ , which encodes the table schema  $S_{T}$ , to perform the Text2SQL transformation.
$$
\operatorname {S Q L} _ {\mathbf {Q}} = \operatorname {S P - T R L} (\mathbf {Q}, \mathbf {M}) \tag {3}
$$
Performance Evaluation. Testing TRL models for QA-TRL and SP-TRL provides complementary information about their ability to encode complex instance- and schema-level relations holding in tabular data. Instead of verifying the syntactic overlap between free-text questions and SQL declarations, we evaluate results' consistency at the tuple level. Specifically, in QA-TRL we measure how similar the result  $R_{Q}$  returned by the query  $Q$  and the expected outcome  $R_{SQL}$  are in terms of tuple characteristics and cell values. Similarly, in SP-TRL we compare the tuples retrieved by the execution of the output query  $\mathrm{SQL}_{\mathrm{Q}}$  on  $T$  with those contained in the expected outcome  $R_{SQL}$ .
QATCH Key Steps. QATCH automatically generates SQL declarations  $\mathrm{SQL}_Q$  and free-text questions  $Q$  from a proprietary table  $T$ . Based on the table schema and instances, a query generator first returns a set  $S$  of SQL declarations of varying complexity.
$$
\mathcal {S} = \operatorname {Q u e r y G E N E R A T O R} (T) \tag {4}
$$
Then, for every query  $\mathrm{SQL}_{\mathbf{Q}} \in \mathcal{S}$ , QATCH generates a template corresponding to one of its free-text versions.
$$
Q = \text {G E N E R A T E T E M P L A T E} \left(S Q L _ {Q}\right) \tag {5}
$$
The separation of queries and questions allows us to disentangle the characteristics of the output from the linguistic capabilities of the models. Free-text questions and SQL declarations are used to test the TRL models on the QA-TRL an SP-TRL tasks.
3.2 Automatic generation of testing checklists with SQL queries
Given a relational table  $T$ , QATCH relies on a SQL query generator to produce queries of varying logical complexity for a TRL model's assessment. Each test query consists of a triple  $\langle \mathrm{Q}, \mathrm{SQL}_{\mathrm{Q}}, \mathrm{I}_{\mathrm{Q}}^{gt} \rangle$ , where  $\mathrm{SQL}_{\mathrm{Q}}$  is the input query formulated as a SQL declaration,  $\mathrm{Q}$  is one of the possible free-text reformulation of the question, and  $\mathrm{I}_{\mathrm{Q}}^{gt}$  is the corresponding ground truth.
Table 1 enumerates the templates used to generate  $\mathrm{SQL}_{\mathbf{Q}}$  first and then Q based on proprietary data. They encompass SQL projections on the table schema (Project), selections on the table instance (Select), and more complex cardinality-based and sorting criteria at the tuple level (Distinct and Order by)<sup>1</sup>. The variable in the templates are automatically filled up by the tool according to the values in the schema and active domain on the given  $T$ .
The SQL generator automatically crafts multiple versions of the input tests, each one incorporating a different feature. For example, for the Project template, QATCH explores the following three cases: (1) SELECT ALL, which selects all the attributes in the table schema; (2) SELECT-RANDOM-COL,
<sup>1</sup> Since existing QA models work on single tables, we currently disregard Join operations.
which selects  $n$  random subsets of attributes, each subset of cardinality  $1, \ldots, n$ , where  $n$  is the number of attributes in the table schema; (3) SELECT-ADD-COL, which starts from a fixed partition  $\mathbf{P}_S \in \mathbf{S}_T$  of the table schema and incrementally adds one random attribute at a time. The SQL generator also mixes categories to create more complex queries, e.g., using projection and selection in the same test.
The motivation behind the proposed templates is to examine the TRL model's capability to handle various algebraic operations that may be directly or indirectly applied to the relational schema (based on the targeted task). Our goal is not to stress the complexity of the linguistic expression, such as ambiguity (Veltri et al., 2023), but rather to analyze the logical complexity of the question according to the relational data model.
3.3 Cross-task performance metrics
QATCH summarizes the outcomes of the tests with a set of cross-task performance metrics defined on the comparison of the model output against the expected ground truth. The comparison reflects the similarity between the corresponding instances and the ability of TRL models to handle specific data issues. Specifically, given  $\mathrm{I}_{\mathrm{Q}}^{cell}$  and  $\mathrm{I}_{\mathrm{Q}}^{cell,gt}$  as the sets of distinct pairs respectively occurring in  $\mathrm{I}_{\mathrm{Q}}$  and  $\mathrm{I}_{\mathrm{Q}}^{gt}$ , it supports the following metrics:
Cell Precision. The fraction of table cells  $cell_{i}$  in the output instances that are relevant to the input query. The higher is the score, the more predicted elements are in the target. However, it does not measure how many target cells are in the prediction (measured by cell recall).
$$
\mathrm {C} - \mathrm {P R} = \frac {\left| \left\{\text {c e l l} _ {i} \mid \text {c e l l} _ {i} \in \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l}} \cap \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l} , g t} \right\} \right|}{\left| \left\{\text {c e l l} _ {i} \mid \text {c e l l} _ {i} \in \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l}} \right\} \right|}
$$
Cell Recall. The fraction of table cells that are relevant to the input query that are successfully retrieved. The higher is the score, the more target cells are in the prediction. It does not measure how many prediction cells are in the target (measured by cell precision).
$$
\mathrm {C} - \mathrm {R E C} = \frac {\left| \left\{\text {c e l l} _ {i} \mid \text {c e l l} _ {i} \in \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l}} \cap \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l} , g t} \right\} \right|}{\left| \left\{\text {c e l l} _ {i} \mid \text {c e l l} _ {i} \in \mathrm {I} _ {\mathrm {Q}} ^ {\text {c e l l} , g t} \right\} \right|}
$$
Tuple constraint. The fraction of ground truth tuples in the query output. It is one if the expected and produced outputs have both the same schema, the same cardinality and the same cell values, zero otherwise. However this hard constraint does not capture all the cases (see example below).
$$
T - C o n s = \frac {| I _ {Q} \cap I _ {Q} ^ {g t} |}{| I _ {Q} ^ {g t} |}
$$
Tuple cardinality. The ratio of output and ground truth cardinality. It is a "softer" constraint w.r.t. the tuple constraint since it does not consider neither the schema nor the cell values. However, it has to be analysed with cell precision and cell recall to be meaningful.
$$
\mathrm {T} - \mathrm {C A R D} = \frac {\left| I _ {Q} \right|}{\left| I _ {Q} ^ {g t} \right|}
$$
Tuple Order: the Spearman rank correlation coefficient (Katsaounis, 2003) between the vector representations of the ranked lists of returned and ground truth tuples. This non-parametric test is computed only for queries with the Order-By clause.
Figure 3 shows examples of the performance metrics on a toy dataset. Given the ground truth result (target) with three tuples over two attributes, we report the metric values for five predictions, coming either from a QA or from the execution of a query in SP. Notice that:
- Tuple constraint/cardinality and cell-precision/recall range between 0 (no matches) and 1 (all matches), while the tuple order ranges between 0 (opposite rank) and 1 (same rank).
Figure 3: QATCH's metrics are computed between the model output (prediction) and expected ground-truth results (target). The target is the answer of the NL question "Show me all the data" over a table with three tuples and two attributes.