SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table linearization. Using ChatGPT, which processes text sequentially from left to right, necessitates converting tabular data into a linear format for seamless integration. This process, known as linearization, entails transforming structured tabular data into a sequential layout that aligns with ChatGPT's processing method. A common technique involves concatenating cell contents within a row, using specific delimiters for separation. However, a more promising approach is to enhance this by appending column headers to cell content, incorporating structure-aware indicators as described in Badaro et al. (2023). The approach used in this work, illustrated in Figure 4, represents the table as a list of lists, using Python-like syntax. In this representation, each inner list corresponds to a distinct row in the table, with elements indicating values associated with respective columns marked by the
Cell precision recall	Tuple cardinality	Tuple constraint order
Target Prediction	SELECT DISTINCT emailisfree FROM fraud	0.5	1.0	0.2	0.0	-
SELECT emailisfree, income FROM fraud
Target Prediction	SELECT emailisfree FROM fraud ORDER BY emailisfree ASC	1.0	1.0	1.0	1.0	0.0
SELECT emailisfree FROM fraud ORDER BY emailisfree DESC
Target Prediction	SELECT * FROM fraud	1.0	0.10	1.0	0.0	-
SELECT emailisfree FROM fraud
Table 7: Examples of predictions and performance metrics.
Figure 4: Example of the linearization technique used in this study to process structured data with ChatGPT.
structure-aware indicator [H]. This format ensures the preservation of the table's structural integrity and facilitates seamless processing within the ChatGPT framework.
ChatGPT reproducibility. In this study we use ChatGPT 3.5-turbo-0613 with the "in-context" task adaptation technique. In-context learning refers to the ability of models like GPT-3 to consider the immediate context of a conversation or text when generating responses. This means the model does not just look at the current question or prompt, but also the preceding conversation or text. For instance, in-context learning can be seen as a form of few-shot learning where the 'shots' are the previous turns in the conversation. For our study, we pass three examples<sup>2</sup>: (i) the projection of all the attributes (ii) the projection of one column (iii) one aggregate condition.
C Additional results for Question Answering
Table 8 compares the performance of the QA models on proprietary data.
By examining the achieved results, we identified the following patterns and trends in the models' performance:
- Tapas and Tapex consistently exhibit higher cell precision but lower cell recall. This indicates that their predictions are often correct yet incomplete.
- For the Projection queries, the models understand which columns to include but only return a portion of the expected values.
- All the models fail to return the values in the expected tuple format. This indicates that the models struggle to comprehend the structure of the table.
- Although ChatGPT generally achieves a high overall score, it is not always the best performing model. Specifically, it struggles to cope with the Sales-transactions and Late-payment tables. Tuple cardinality and Tuple constraint metrics were consistently below 0.50, indicating that the returned outcomes are still far from being optimal.
Table 9 provides additional insights by showcasing the SQL patterns in which ChatGPT performed poorly. It performs fairly good when the number of attributes in the output is limited (e.g., DISTINCT-SINGLE). Conversely, it struggles when a larger number of attributes and values need to be considered, especially they mainly include numerical attributes. For example, Sales-transactions has 7 out of 8 columns as numerical, while Late-payment has 7 out of 10 columns as numerical.
2More details can be found in the GitHub https://github.com/spapicchio/QATCH/
The same trend is also evident in the improved performances of ChatGPT when dealing with tables in the same category, such as Fitness-trackers, which has 6 categorical attributes out of 10, and Fraud, which has 7 categorical attributes out of 10 or, even more noticeable, in the Mushrooms table where all the 10 attributes are categorical.
Table category	Table name	Model	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order
ECOMMERCE	Sales-transactions	TAPAS-WTQ	0.75	0.13	0.67	0.05	0.50
TAPEX-WTQ	0.40	0.07	0.21	0.02	0.28
OMNITAB	0.24	0.02	0.17	0.0	0.50
CHATGPT	0.15	0.03	0.05	0.02	0.0
Fitness-trackers	TAPAS-WTQ	0.60	0.08	0.32	0.04	0.20
TAPEX-WTQ	0.45	0.06	0.16	0.0	0.50
OMNITAB	0.16	0.01	0.12	0.01	0.50
CHATGPT	0.67	0.46	0.29	0.27	0.44
FINANCE	Account-fraud	TAPAS-WTQ	0.64	0.11	0.38	0.06	0.35
TAPEX-WTQ	0.51	0.07	0.14	0.01	0.50
OMNITAB	0.26	0.02	0.11	0.00	0.50
CHATGPT	0.68	0.61	0.42	0.17	0.48
Late-payment	TAPAS-WTQ	0.77	0.12	0.55	0.05	0.40
TAPEX-WTQ	0.48	0.05	0.18	0.0	0.45
OMNITAB	0.35	0.02	0.16	0.0	0.50
CHATGPT	0.13	0.05	0.04	0.03	0.0
MEDICINE	Heart-attack	TAPAS-WTQ	0.60	0.13	0.45	0.08	0.23
TAPEX-WTQ	0.31	0.05	0.16	0.01	0.38
OMNITAB	0.19	0.01	0.11	0.0	0.50
CHATGPT	0.87	0.74	0.41	0.35	0.58
Breast-cancer	TAPAS-WTQ	0.83	0.18	0.67	0.10	0.45
TAPEX-WTQ	0.42	0.04	0.15	0.0	0.50
OMNITAB	0.37	0.02	0.13	0.0	0.50
CHATGPT	0.72	0.31	0.14	0.10	0.75
MISCELLANEOUS	Adult-census	TAPAS-WTQ	0.59	0.12	0.34	0.06	0.25
TAPEX-WTQ	0.59	0.12	0.16	0.0	0.45
OMNITAB	0.04	0.01	0.13	0.01	0.50
CHATGPT	0.71	0.68	0.47	0.35	0.58
Mushrooms	TAPAS-WTQ	0.68	0.12	0.27	0.03	0.33
TAPEX-WTQ	0.44	0.10	0.33	0.02	0.43
OMNITAB	0.17	0.03	0.13	0.02	0.50
CHATGPT	0.77	0.75	0.49	0.37	0.56
Table 8: Results for QA-TRL for the different proprietary tables.
D Additional results for Semantic Parsing
Table 10 compares the performance of the SP models on proprietary data. Based on the achieved results, RESDNSQL performed worst in the MEDICINE category. To delve deeper into the results, we focused on the table displaying the lower metric score, which can be found in Table 11. Heart-attack exhibits remarkable low values for cell recall across all where SQL categories. These inaccuracies can be attributed to a misprojection made by the model during prediction. The presence of the output column in the heart-attack table appears to confuse the model when it attempts to project all columns. Our hypothesis is confirmed by the presence of the same error in the SELECT-ALL category.
Table category	SQL category	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order
Sales-transactions	SELECT-ALL	0.00	0.00	0.00	0.00
SELECT-ADD-COL	0.43	0.03	0.03	0.03
SELECT-RANDOM-COL	0.38	0.07	0.02	0.02
ORDERBY-SINGLE	0.00	0.00	0.00	0.00	0.00
DISTINCT-MULT	0.40	0.10	0.01	0.01
DISTINCT-SINGLE	1.00	0.28	0.28	0.28
WHERE-CAT-MAX-VALUES	0.10	0.03	0.20	0.00
WHERE-CAT-MIN-VALUES	0.05	0.01	0.10	0.00
WHERE-NUM-MAX-VALUES	0.00	0.00	0.00	0.00
WHERE-NUM-MEAN-VALUES	0.00	0.00	0.00	0.00
WHERE-NUM-MIN-VALUES	0.00	0.00	0.00	0.00
Late-payment	SELECT-ALL	0.00	0.00	0.00	0.00
SELECT-ADD-COL	0.33	0.04	0.03	0.03
SELECT-RANDOM-COL	0.30	0.12	0.04	0.03
ORDERBY-SINGLE	0.00	0.00	0.00	0.00	0.00
DISTINCT-MULT	0.33	0.18	0.18	0.18
DISTINCT-SINGLE	0.97	0.45	0.46	0.45
WHERE-CAT-MAX-VALUES	0.08	0.02	0.01	0.00
WHERE-CAT-MIN-VALUES	0.08	0.02	0.01	0.00
WHERE-NUM-MAX-VALUES	0.00	0.00	0.00	0.00
WHERE-NUM-MEAN-VALUES	0.00	0.00	0.00	0.00
WHERE-NUM-MIN-VALUES	0.01	0.00	0.01	0.00
Table 9: Results for QA chatGPT on Sales-transactions and Late-payment.
Table category	Table name	Model	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order
ECOMMERCE	Sales-transactions	RESDSQL	0.91	0.90	0.90	0.83	1.00
GAP	0.85	0.80	0.82	0.71	0.94
UNIFIEDSKG	0.79	0.77	0.76	0.74	1.00
CHATGPT	1.00	1.00	0.99	1.00	1.00
Fitness-trackers	RESDSQL	0.90	0.88	0.94	0.79	1.00
GAP	0.84	0.80	0.80	0.74	1.00
UNIFIEDSKG	0.65	0.65	0.65	0.65	1.00
CHATGPT	1.00	1.00	1.00	0.90	1.00
FINANCE	Account-fraud	RESDSQL	0.89	0.86	0.95	0.77	1.00
GAP	0.79	0.77	0.75	0.73	1.00
UNIFIEDSKG	0.80	0.76	0.75	0.68	1.00
CHATGPT	0.99	0.99	0.98	0.90	1.00
Late-payment	RESDSQL	0.90	0.87	0.95	0.77	1.00
GAP	0.80	0.79	0.78	0.74	1.00
UNIFIEDSKG	0.77	0.76	0.71	0.65	0.95
CHATGPT	1.00	1.00	1.00	0.90	1.00
MEDICINE	Heart-attack	RESDSQL	0.88	0.67	0.93	0.57	0.90
GAP	0.92	0.85	0.88	0.79	0.93
UNIFIEDSKG	0.72	0.66	0.71	0.61	0.90
CHATGPT	0.92	0.92	0.99	0.92	1.00
Breast-cancer	RESDSQL	0.84	0.83	0.95	0.77	1.00
GAP	0.64	0.63	0.60	0.56	0.25
UNIFIEDSKG	0.72	0.71	0.70	0.70	1.00
CHATGPT	0.95	0.95	0.99	0.95	1.00
MISCELLANEOUS	Adult-census	RESDSQL	0.92	0.88	0.90	0.76	1.00
GAP	0.71	0.67	0.58	0.60	1.00
UNIFIEDSKG	0.69	0.64	0.63	0.58	0.95
CHATGPT	0.99	0.99	0.98	0.83	1.00
Mushrooms	RESDSQL	0.96	0.91	0.91	0.78	1.00
GAP	0.94	0.89	0.89	0.80	1.00
UNIFIEDSKG	0.80	0.74	0.72	0.61	1.00
CHATGPT	0.99	0.99	0.99	0.99	1.00
Table 10: Results for SP-TRL for the different proprietary tables.
SQL category	Cell precision	Cell recall	Tuple cardinality	Tuple constraint	Tuple order
SELECT-ALL	0.00	0.00	0.00	0.00
SELECT-ADD-COL	1.00	0.79	1.00	0.56
SELECT-RANDOM-COL	1.00	0.72	1.00	0.40
ORDERBY-SINGLE	1.00	0.99	0.91	0.91	1.00
DISTINCT-SINGLE	1.00	1.00	1.00	1.00
DISTINCT-MULT	1.00	0.79	0.62	0.51
WHERE-CAT-MAX-VALUES	1.00	0.28	1.00	0.20
WHERE-CAT-MIN-VALUES	1.00	0.28	1.00	0.20
WHERE-NUM-MAX-VALUES	1.00	0.86	0.98	0.83
WHERE-NUM-MEAN-VALUES	1.00	0.78	1.00	0.75
WHERE-NUM-MIN-VALUES	0.80	0.71	0.80	0.70
Table 11: RESDNSL results for the heart-attack table.