SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- The tuple constraint ignores partial tuple matches (e.g., [APPLE, RED] is different from [APPLE] in Prediction 2), which is captured by cell recall.
- When the number of expected tuples is zero, tuple cardinality takes either value zero, if the number of predicted tuples is greater than zero, or one, if no tuples are predicted.
To explain the rationale behind the proposed metrics, let us consider the following target values (a1, b1, c1). The first prediction, denoted by "output 1", is composed of two tuples (a1, b1, c1), (a1, b1, c1). Instead, the second prediction, denoted by "output 2", has the following two tuples: (a1, b1), (c1). "Output 1" has cardinality two instead of one whereas "output 2" has an incorrect schema. In both cases the tuple constraint returns zero even if part of the output cells match. Notice also that even if the outputs' cardinality is the same cell precision and recall show relevant differences in the returned values.
Our Cell Precision is equivalent to the Execution Accuracy reported in most QA and SP papers. We remark that while informative, this single metric does not capture cases where the output is incorrect, such as Prediction 5 in Figure 3.
TRL models. We test six TRL models and one Large Language Model, i.e., CHATGPT (OpenAI, 2023). For QA-TRL, we report for TAPAS (Herzig et al., 2020), TAPEX (Liu et al., 2021), and OMNITAB (Jiang et al., 2022). TAPAS leverages the transformer architecture to pre-train on large-scale datasets and fine-tune to specific tasks using labeled examples. TAPEX exploits the idea of learning in the pre-training a neural SQL executor over a synthetic corpus of SQL queries and their execution outputs. OMNITAB exploits a pretraining approach that uses both natural and synthetic data to learn reasoning over multiple table elements. We use TAPAS and TAPEX fine-tuned on the WTQ dataset for QA.
For SP-TRL, we report for RESDSQL (Li et al., 2023), GAP (Shi et al., 2021), UNIFIEDSKG (Xie et al., 2022). RESDSQL is based on a seq2seq architecture with a ranking-enhanced encoding and skeleton-aware decoding framework. UNIFIEDSKG implements the encoder-decoder (text-to-text) model based on T5. We use UNIFIEDSKG and RESDSQL with the T5 large setting. GAP employs generative models to create pre-training data, enabling the simultaneous learning of natural language utterances and table schemas.
CHATGPT is a language model employing a decoder-only transformer setup; by leveraging a chat interface, it allows to specify a range of tasks (including QA and SP) with the in-context adaptation technique. This technique leverages the model's ability to understand and consider the immediate context of a conversation or text. Further details to reproduce the results can be found in the appendix.
Category	Table Name	# rows	# categorical cols	# numerical cols	Example cols
ECOMMERCE	Sales-transactions	500k	5	3	ProductNo, Date
Fitness-trackers	565	8	3	Brand Name, Display
FINANCE	Account-fraud	1M	4	26	DaysSinceRequest, Velocity6h
Late-payment	2466	6	6	InvoiceDate, Disputed
MEDICINE	Heart-attack	303	1	11	# trtps, # oldpeak
Breast-cancer	686	5	6	pgr, rfstime
MISC	Adult-census	32.6k	9	6	education, fnlwgt
Mushrooms	8.1k	23	0	cap-shape, ring-type
Table 2: Information for the proprietary tables used in the experiments.
Datasets. We generate tests for 8 proprietary tables that we have selected from Kaggle to maximize variety both in terms of content (four categories) and in terms of data properties (different sizes and arity), as detailed in Table 2. These datasets are available online, thus possibly "seen" by the models, but are not available for question answering and semantic parsing tasks, i.e. they do not come with natural language questions or queries. The 8 tables are used in this paper as a sample to show the benefit of automatic test generation with QATCH, we are not suggesting to use them as a new corpus with a fixed set of questions and tables. We also report results for the widely used Spider benchmark, as this is the standard for QA and SP tasks evaluation, using its tables and its questions. This comparison aims to emphasize the discrepancies between the quality results obtained using Spider and the practical application of models on proprietary tables. More details on the data processing phase can be found in the appendix.
As models are limited in the input, for every table we sample a subset of rows and attributes that can be executed within the model context (e.g., about 4k tokens for CHATGPT).
Category	Model	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order	Avg
ECOMMERCE	PROPRIETARY DATA
TAPAS-LARGE-WTQ	0.71	0.12	0.53	0.05	0.33	0.35
TAPEX-LARGE-WTQ	0.40	0.06	0.18	0.01	0.40	0.21
OMNITAB	0.20	0.01	0.14	0.00	0.50	0.17
CHATGPT 3.5	0.44	0.24	0.20	0.10	0.42	0.28
FINANCE	TAPAS-LARGE-WTQ	0.72	0.12	0.48	0.05	0.38	0.35
TAPEX-LARGE-WTQ	0.52	0.06	0.16	0.01	0.48	0.25
OMNITAB	0.30	0.02	0.13	0.00	0.50	0.19
CHATGPT 3.5	0.71	0.52	0.38	0.21	0.48	0.46
MEDICINE	TAPAS-LARGE-WTQ	0.72	0.16	0.57	0.09	0.34	0.38
TAPEX-LARGE-WTQ	0.37	0.04	0.15	0.0	0.44	0.20
OMNITAB	0.29	0.01	0.12	0.0	0.50	0.18
CHATGPT 3.5	0.77	0.46	0.22	0.12	0.70	0.45
MISCELLANEOUS	TAPAS-LARGE-WTQ	0.67	0.12	0.34	0.04	0.29	0.29
TAPEX-LARGE-WTQ	0.48	0.10	0.25	0.01	0.44	0.26
OMNITAB	0.12	0.02	0.13	0.01	0.50	0.17
CHATGPT 3.5	0.76	0.67	0.36	0.16	0.50	0.49
Spider	EXISTING BENCHMARK DATA
TAPAS-LARGE-WTQ	0.64	0.42	0.53	0.30	0.64	0.51
TAPEX-LARGE-WTQ	0.62	0.45	0.54	0.21	0.51	0.47
OMNITAB	0.30	0.24	0.53	0.23	0.52	0.36
CHATGPT 3.5	0.74	0.77	0.86	0.66	0.75	0.76
Table 3: Results for QA-TRL models: average of all tests on multiple tables. ChatGPT version: ChatGPT 3.5-turbo-0613
Category	Model	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order	Avg
ECOMMERCE	PROPRIETARY DATA
RESDNSQL	0.91	0.89	0.92	0.81	1.00	0.90
GAP	0.84	0.80	0.81	0.73	0.97	0.83
UNIFIEDSKG	0.71	0.71	0.69	0.69	1.00	0.76
CHATGPT 3.5	0.98	0.98	0.99	0.95	1.00	0.98
FINANCE	RESDNSQL	0.90	0.87	0.95	0.77	1.00	0.90
GAP	0.79	0.78	0.76	0.74	1.00	0.81
UNIFIEDSKG	0.79	0.76	0.74	0.67	0.98	0.79
CHATGPT 3.5	0.96	0.96	0.99	0.90	1.00	0.96
MEDICINE	RESDNSQL	0.86	0.75	0.94	0.67	0.95	0.83
GAP	0.77	0.73	0.73	0.67	0.59	0.70
UNIFIEDSKG	0.72	0.69	0.70	0.66	0.95	0.74
CHATGPT 3.5	1.00	1.00	0.98	0.99	1.00	0.99
MISCELLANEOUS	RESDNSQL	0.94	0.90	0.90	0.77	1.00	0.90
GAP	0.82	0.78	0.73	0.69	1.00	0.80
UNIFIEDSKG	0.74	0.69	0.68	0.59	0.98	0.73
CHATGPT 3.5	0.98	0.98	0.98	0.91	1.00	0.97
EXISTING BENCHMARK DATA
Spider DEV	RESDNSQL	0.93	0.93	0.97	0.84	0.99	0.93
GAP	0.95	0.95	0.96	0.91	0.96	0.95
UNIFIEDSKG	0.81	0.82	0.82	0.80	1.00	0.85
CHATGPT 3.5	0.93	0.96	0.97	0.92	0.90	0.94
Spider TRAIN	CHATGPT 3.5	0.90	0.92	0.92	0.88	0.97	0.92
Table 4: Results for SP-TRL models: average of all tests on multiple tables. CHATGPT version: CHATGPT 3.5-turbo-0613. UNIFIEDISKG and RESDSQL evaluated with the T5 large.
Results. Tables 3 and 4 show the results for QA-TRL and SP-TRL, respectively. All metric results are averaged over all tests and over tables grouped by category; we report detailed results for every dataset and every test category in the appendix.
For the QA-TRL task, Table 3 reports the results achieved on the proprietary tables and on the Spider benchmark to enable a comparison between the two groups. All models show promising results in terms of Cell precision, which is the metric closest to the one used in previous papers. However, the other metrics show lower values, with all models struggling in preserving the intra-tuple value relations in their output (i.e., low Tuple constraints). Also, in almost all experiments the models tend to remove duplicates, even when the Distinct clause is not present. This is evident with the low scores for Cell recall and Tuple Cardinality. Finally, none of the models is able to return results according to the Order by requirements (i.e., low Tuple order scores).
Results for the SP task in Table 4 show that, in general, the TRL models performs much better in this task. The most problematic aspect is the preservation of the tuple structure for the returned values (Tuple constraint).
Tests can be generated for a specific dataset using QATCH in just a few seconds. The testing times varies based on the data, particularly the size of the table. For instance, comparing a prediction and a ground truth containing 1000 tuples and 20 attributes takes less than a second.
As we do not control the infrastructure of OpenAI, we do not report exact execution times for CHATGPT. On average, it takes a few seconds to execute one test, but the distribution for this metrics is skewed as it heavily depends on the server workload.
Discussion. The results show that the tests generated by QATCH and its metrics enable detailed evaluation of the models. A clear message is that there is no single model that works best for every table and every metric. In some datasets, TAPAS has the best performance in terms of cell precision and recall, but CHATGPT is the best model in terms of tuple cardinality and tuple constraint. The
lower precision for CHATGPT in some cases is due to the hallucination problem with the decoder architecture.
Another take-away is that questions and tables in Spider do not tell the full story about model performance. Even a small selection of 8 tables show that the performance for established TRL models can drop dramatically on propriety data in the QA task.
Looking into the details of the failed tests, it is easy to spot the limitations of the models. Cell precision is lowered by select queries with more than one column in the output. The models tend to return only one attribute even in this setting.
In some cases, Tuple cardinality is lower than Cell recall, because the latter is over a set. For some queries, models return DISTINCT by default. Most of the questions in Spider return one tuple only, thus inflating the aggregate result and hiding the problem with more output attributes. For the same reason, tuple constraint scores are low because in many cases the models do not return tuple structures, but mostly separated cells. We again observe low results for proprietary data with the higher ones for Spider.
We can conclude that the qualitative performance of a TRL model depends on the target task at hand, but also on the tabular content in terms of data domain, size, schema, and data quality.
We presented QATCH, a testing tool for TRL models. It provides end-users with a flexible and adaptive solution for the automatic assessment of models' performance on proprietary data. The key goal is to avoid trusting the results achieved on the existing benchmarks across different tasks, as they are not always replicable on proprietary data. By leveraging the expressive power of the SQL language, we first generate queries of varying complexity on proprietary table and then convert them in natural language. Thus, we specifically assess the model capability to address complex queries rather than the linguistic properties of questions and answers. Results show that (1) existing questions in benchmarks do not capture important properties of custom tables, (2) popular metrics fail short in measuring the quality of the models' output.
As future research agenda, we plan to investigate the following research directions: (1) The extension of QATCH towards the automatic assessment of additional models (e.g., Bard (Google Inc., 2023), LLama2 (Touvron et al., 2023), Falcon (Penedo et al., 2023)), tasks (e.g., fact checking (Guo et al., 2022; Nakov et al., 2021), querying LLMs (Saeed et al., 2024; Urban et al., 2023)), and more complex queries (e.g., GROUP BY clauses, nested queries). (2) The adoption of Generative Language Models (e.g., (OpenAI, 2023)) to automatically generate templates in Natural Language from the SQL declarations. (3) The use of TRL models to test inter-relational constraints, such as joins and referential constraints, among proprietary tables.
Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, and Bing Xiang. 2021. Learning contextual representations for semantic parsing with generation-augmented pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 13806-13814.
Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Cagatay Demiralp, Chen Chen, and Wang-Chiew Tan. 2021. Annotating Columns with Pre-trained Language Models. arXiv preprint arXiv:2104.01785 (2021).
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
Matthias Urban, Duc Dat Nguyen, and Carsten Binnig. 2023. OmniscientDB: A Large Language Model-Augmented DBMS That Knows What Other DBMSs Do Not Know. In aiDM@SIGMOD. ACM, 4:1-4:7. https://doi.org/10.1145/3593078.3593933
Enzo Veltri, Gilbert Badaro, Mohammed Saeed, and Paolo Papotti. 2023. Data Ambiguity Profiling for the Generation of Training Examples. In ICDE. IEEE, 450-463. https://doi.org/10.1109/ICDE55515.2023.00041
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966 (2022).
Xiaoyu Yang and Xiaodan Zhu. 2021. Exploring Decomposition for Table-based Fact Verification. In EMNLP. Association for Computational Linguistics, Punta Cana, Dominican Republic, 1045-1052. https://aclanthology.org/2021-findings-emnlp.90
Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 8413-8426. https://doi.org/10.18653/v1/2020.acl-main.745
Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, bailin wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, richard socher, and Caiming Xiong. 2021. GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing. In International Conference on Learning Representations. https://openreview.net/forum?id=kyaIeYj4zZ
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In EMNLP. ACL, 3911-3921. https://doi.org/10.18653/v1/D18-1425
A Authors' statements
A.1 Code license and legal compliance
The source code of our project is released under the Apache-2.0 license. All the authors are aware of the legal implications and gave their explicit consent to publish the paper and the code.
The code and the related documentation is available at: https://github.com/spapicchio/QATCH. The source code is annotated in compliance with the ML reproducibility checklist provided by NeurIPS 2023 organizers and available at https://github.com/paperswithcode/releasing-research-code. We also adhere to the official guidelines for the specification of the code dependencies. Notice that TRL models are referenced but not directly included in the main repository as we exclusively rely on third-party models.
A.3 Long-term preservation
We guarantee that the repository, including the project code and annotations, will be maintained in the long term. Any potential issues that will be raised by end-users will be taken into consideration.
A.4 Privacy
In this work, we released a new testing toolbox for proprietary data. We neither released a new publicly available datasets nor involved humans in the evaluation loop.
While using QATCH the input data, the tests, and the expected and achieved outcomes always remain under the full end-user's control as long as TRL models are run on premises (no external APIs). Thus, QATCH allows a secure testing of proprietary tables by running models and tests on a local machine.
For the sake of completeness, QATCH also supports an external QA model, i.e., OpenAI ChatGPT 3.5. While using it, end-users must be aware of the fact that their own data are shared with private subjects in compliance with OpenAI terms and policies (https://openai.com/policies).
A.5 Ethics
There are no particular ethical implications of our work. Every ethical issue related to the tested data is under the liability of the data owner.
A.6 Bias and fairness
The QATCH test generation module randomly picks existing content and/or metadata from the input tables. While running the tests, we recommend to also vary the seed of the random generator to maximize the reliability of the tests' outcomes. Mitigating the eventual presence of bias in the input data and testing TRL model fairness are both out of scope of the present work.
A.7 Impact of the environment
Testing TRL models on proprietary data prevents the use of expensive and resource demanding cloud services. Since QATCH allows end-users to optimize the computational efforts by choosing the right model to use. Concerning that point, its use can be deemed as an example of green data mining Schneider et al. (2023).
B Additional details on data, models, and tests
Tables' description. Here we describe the tables used in the experimental evaluation. We collected results on eight proprietary tables and on one benchmark dataset established for TRL model evaluation Yu et al. (2018).
We considered proprietary tables of various domains (i.e., E-Commerce, Finance, Medicine, and Miscellaneous) and characteristics (e.g., tuple cardinality from few hundreds to more than 1 million, table dimensionality from 8 to 30) to ensure the robustness of the empirical outcomes. The detailed statistics are given in Table 2 of the main paper.
In Table 5 we reported similar statistics for the Spider benchmark dataset. These statistics were not included in the main paper due to the lack of space.
Proprietary data preprocessing. QATCH supports standard text pre-processing procedures for the table content and metadata. For example, the Mushrooms table contains only alphabetical letters thus conveying limited information. We applied a text reconciliation step, where, for instance, in the Heart-attack table the attribute value  $cp$  is replaced by the (more meaningful) expression chest pain.
Spider preprocessing. The primary objective of QATCH is to assess the impact of proprietary data on model performance. To achieve this goal, QATCH tests are deliberately designed to explore model behavior in simplified scenarios, avoiding intricacies in both the generated questions and queries. In contrast, Spider, having a different scope, inherently contains a higher level of complexity. To ensure a fair comparison between the proprietary data-driven QATCH tests and the Spider dataset, a preprocessing step is applied for the Spider dataset. Specifically, we excluded queries involving the LIMIT, INTERSECT, UNION, JOIN and EXCEPT operators, as well as those with inner queries.
Benchmark vs. proprietary data. The tables included in the Spider benchmark have quite different characteristics compared to the proprietary ones (fewer tuples and dimensions, fewer numerical attributes).
For example, in Table 5) we report the comparison between the Spider tables and the selected proprietary tables. The number of cells per table in Spider is significantly lower in  $75\%$  of the cases. This confirms the limited representativeness of the currently used benchmark data for TRL model evaluation and fosters the need of TRL testing tools on proprietary data.
minimum	25% percentile	50% percentile	75% percentile	maximum
Spider tables	# tuples	0	7	12	15	16049
# attributes	2	4	5	7	24
# cells	0	30	55	100	112343
Proprietary tables	# tuples	303	656	5283	149k	1M
# attributes	8	11	12	17	30
# cells	3636	7213	108k	1.4M	30M
Table 5: Comparison between the Spider and proprietary tables' statistics. It reports the minimum and maximum values, and the  $25\%$ ,  $50\%$ , and  $75\%$  percentile over all the Spider tables and the considered proprietary tables.
Tackling TRL model limitations. To meet the TRL models' constraints on the maximum number of processed cells, whenever strictly necessary QATCH reduces the input data size by sampling the tables attributes and rows using the same random seed for all tables. To avoid introducing bias in the data, we tested multiple dataset variants consisting of different attribute and tuple selections. More specifically, to address Question Answering we generated all the tests by exclusively considering the table schema, except for the Select queries where the tuple-level conditions are generated by considering as representatives the maximum, minimum, or mean attribute values computed separately for each table. The main goal here is to test whether the TRL model can accurately answer queries including arbitrary selection conditions.
The inherent limitations of TRL models to handle larger tables do not influence Semantic Parsing tests, as they directly work at the schema level. Notice that such limitations are independent of the QATCH implementation, but can be conveniently highlighted while running the tests on proprietary data.
Table name	Selected columns	# of rows	# of rows	# of rows	# of rows
Tapas-wtq	Tapex-wtq	ChatGPT	Omnitab
Sales-transactions link	TransactionNo, ProductNo, Product, Price Quantity, CustomerNo, Date, Country	60	20	40	20
Fitness-trackers link	Brand Name, Device Type, Original Price, Selling Price, Color, Model Name, Average Battery Life (in days), Rating (Out of 5) Strip Material, Display	50	25	30	20
Account-fraud link	income, payment_type, has_other Cards, email_is_free, employment_status, housing_status, date_of_birth_distinct Emails_4w, device_os credit_risk_score, session_length_in Minutes	50	25	30	20
Late-payment link	CustomerID, PaperlessDate, InvoiceNumber, Invoice Date, DueDate, InvoiceAmount, Disputed, PaperlessBill, DaysToSettle, DaysLate	50	20	25	20
Heart-attack link	age, sex, cp, trbps, chol, fbs, restecg, caa, thall, output	45	30	30	20
Breast-cancer link	pid, age,edo, size, grade, nodes, pgr, er, hormon, status	45	30	30	20
Adult-census link	workclass, education, marital.status, occupation, relationship, race, sex, hours.per.week, native country, income	50	20	30	20
Mushrooms link	class, cap-shape, cap-surface, cap-color, bruises, odor, gill-attachment, gill-spacing, gill-size, gill-color	50	25	30	20
Table 6: Datasets information with maximum number of rows accepted for each model in the QA Scenario.
In Table 6 we report the dataset, the list of table attributes, and the maximum number of tuples that each TRL model (and the corresponding tokenizer) manages to process. It is worth noticing that, even though Tapex is based on BART (maximum token limit: 1024) and Tapas is based on BERT (token limit: 512), the number of tuples that Tapas is able to process is much larger. The link to each dataset along with the pre-processing strategies are available in the github repository.
Performance metrics. Our evaluation metrics rely on the predicted cell values, making them essential in the SP scenario. To begin with, we employ models to predict SQL queries, which are subsequently executed on tables to retrieve the corresponding predicted cell values. This approach allows us to derive valuable insights from the metric scores associated with the SQL predictions. For instance, a low Tuple constraint metric indicates that the SQL prediction is likely projecting incorrect columns. Instead, a low tuple cardinality metric suggests that the SQL prediction may be selecting incorrect rows or utilizing a distinct operator inappropriately. Table 7 shows three instances of unsuccessful predictions.