LLM-Symbolic Integration for Robust Temporal Tabular Reasoning
Authors: Atharv Kulkarni, Kushagra Dixit, Vivek Srikumar, Dan Roth, Vivek Gupta
Accepted at: acl25

LLM-Symbolic Integration for Robust Temporal Tabular Reasoning

Atharv Kulkarni1*, Kushagra Dixit1,
Vivek Srikumar1, Dan Roth2, Vivek Gupta3†
1University of Utah, 2University of Pennsylvania, 3Arizona State University
(atharvk, danroth)@seas.upenn.edu
kushagra.dixit@utah.edu svivek@cs.utah.edu
vgupt140@asu.edu

Abstract

Temporal tabular question answering presents a significant challenge for Large Language Models (LLMs), requiring robust reasoning over structured data—a task where traditional prompting methods often fall short. These methods face challenges such as memorization, sensitivity to table size, and reduced performance on complex queries. To overcome these limitations, we introduce TEMP TABQA-C, a synthetic dataset designed for systematic and controlled evaluations, alongside a symbolic intermediate representation that transforms tables into database schemas. This structured approach allows LLMs to generate and execute SQL queries, enhancing generalization and mitigating biases. By incorporating adaptive few-shot prompting with contextually tailored examples, our method achieves superior robustness, scalability, and performance. Experimental results consistently highlight improvements across key challenges, setting a new benchmark for robust temporal reasoning with LLMs. Code and TEMP TABQA-C dataset: https://coral-lab-asu.github.io/llm_symbolic.

1 Introduction

Large Language Models (LLMs) have revolutionized natural language processing, achieving remarkable success across a variety of tasks. However, answering questions about temporal tabular data task that requires precise reasoning over structured information with time-based attributes remains a significant challenge. This capability is crucial in fields such as finance, healthcare, and policymaking, where actionable insights often depend on understanding and analyzing evolving datasets. Yet, existing methods often fall short, struggling with complex queries, large datasets, and scenarios that require nuanced reasoning. An example of such task, with a long table and corresponding query and it’s answer is shown in Figure 1.

<table>
  <tr>
    <th colspan="2">Áron Szilágyi</th>
  </tr>
  <tr>
    <th colspan="2">Personal information</th>
  </tr>
  <tr>
    <td>Born</td>
    <td>14 January 1990 (age 34)</td>
  </tr>
  <tr>
    <th colspan="2">Olympic Games</th>
  </tr>
  <tr>
    <td>2012 London</td>
    <td>Individual</td>
  </tr>
  <tr>
    <td>2016 Rio de Janeiro</td>
    <td>Individual</td>
  </tr>
  <tr>
    <td>2020 Tokyo</td>
    <td>Individual</td>
  </tr>
  <tr>
    <td>2024 Paris</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2020 Tokyo</td>
    <td>Team</td>
  </tr>
  <tr>
    <th colspan="2">World Championships</th>
  </tr>
  <tr>
    <td>2007 Saint Petersburg</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2022 Cairo</td>
    <td>Individual</td>
  </tr>
  <tr>
    <td>2023 Milan</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2016 Rio de Janeiro</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2023 Milan</td>
    <td>Individual</td>
  </tr>
  <tr>
    <th colspan="2">European Games</th>
  </tr>
  <tr>
    <td>2023 Kraków–Malopolska</td>
    <td>Individual</td>
  </tr>
  <tr>
    <th colspan="2">European Championships</th>
  </tr>
  <tr>
    <td>2015 Montreux</td>
    <td>Individual</td>
  </tr>
  <tr>
    <td>2018 Novi Sad</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2022 Antalya</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2024 Basel</td>
    <td>Team</td>
  </tr>
  <tr>
    <td>2013 Zagreb</td>
    <td>Team</td>
  </tr>
</table>

Q. In which year did Áron Szilágyi achieve his personal highest number of gold medal wins? Gold Answer. 2022

Direct prompt (CoT): In 2020, Áron Szilágyi won a Gold Medal in the Individual format at the Tokyo Olympics. This was his most successful year in terms of highest Gold Medal wins since he did not win multiple Gold Medals in any other format or competition. Answer: 2020.

Symbolic Intermediate Representation (SQL):

WITH gold_medal_counts AS (
  SELECT m.year, COUNT(m.medal_id) AS gold_medals
  FROM Medal m JOIN Format f ON m.format_id = f.format_id
  JOIN Tournament t ON f.tournament_id = t.tournament_id
  JOIN Athlete a ON t.athlete_id = a.athlete_id
  WHERE a.name = 'Áron Szilágyi' AND m.type = 'MedalGold'
  GROUP BY m.year
)
SELECT year FROM gold_medal_counts
WHERE gold_medals = (SELECT MAX(gold_medals) FROM gold_medal_counts);

Answer: 2022

Figure 1: Structured table of Áron Szilágyi’s achievements with question and answers. Direct prompting fails, whereas, Symbolic Intermediate Representation give correct answer.
These limitations underscore the need for robust, scalable, and interpretable solutions. A key obstacle lies in the lack of benchmarks that adequately capture the complexity and diversity of temporal reasoning tasks. Existing benchmarks, typically created manually, are inconsistent and fail to provide the variability needed to thoroughly evaluate models. Without rigorous evaluation frameworks, it becomes difficult to diagnose weaknesses or ensure models can handle real-world scenarios. This raises an essential question: How can we design benchmarks that effectively evaluate temporal reasoning across a range of challenging contexts?

Equally important is the need for robust methods. Many existing approaches rely on direct prompting, which often depends on heuristics and memorized patterns rather than true reasoning. This results in semantic biases and poor performance in demanding scenarios, such as counterfactual reasoning, large table contexts, or multi-step queries. This leads to a second critical question: How can we develop methods that remain robust across diverse table structures, dynamic data, and complex queries?

To address these challenges, we propose a comprehensive framework that reimagines how LLMs approach temporal tabular data. At its core is TempTabQA-C, a synthetic dataset generation method designed to fill the gaps in existing benchmarks. TempTabQA-C provides precise control over data characteristics, enabling consistent and systematic evaluation across a wide range of scenarios, including counterfactual reasoning and intricate temporal queries. Building on this foundation, we introduce a symbolic intermediate representation approach that transforms unstructured tables into structured database schemas. LLMs are guided to generate SQL queries based on these schemas, which are executed to produce accurate answers. E.g. in Figure 1, the SQL query serves as a symbolic representation and provides the correct answer, whereas direct prompting fails on given query. This structured pipeline reduces semantic biases, enhances interpretability, and significantly improves the generalization of models across different table configurations. Additionally, we incorporate adaptive few-shot prompting, a dynamic approach that selects contextually relevant examples tailored to each query. This method overcomes the limitations of static examples, further improving the robustness of the system in complex scenarios.

Our experiments demonstrate that this framework delivers substantial improvements over direct prompting methods. It excels in critical areas such as counterfactual reasoning, scalability to larger datasets, and the handling of complex queries. Beyond these technical advancements, our work establishes a new benchmark for temporal tabular question answering by addressing fundamental weaknesses in existing approaches and introducing innovative tools for evaluation and reasoning. These contributions pave the way for building more interpretable, scalable, and robust AI systems with implications for critical real-world applications. Our contributions are as follows:

1. We introduce TempTabQA-C, a synthetic dataset designed for precise and robust evaluation of temporal tabular reasoning across diverse and challenging scenarios.

2. We analyze the limitations of direct prompting, including reliance on memorization, sensitivity to table size, and struggles with complex multi-step or counterfactual reasoning.

3. We propose a symbolic intermediate representation approach that enhances interpretability, reduces biases, and improves generalization by guiding LLMs to generate and execute SQL queries on structured schemas.

4. We enhance this approach with adaptive few-shot prompting, enabling context-specific example selection for improved flexibility and performance in diverse scenarios.

2 The TempTabQA-C Dataset

The TempTabQA-C dataset is a large-scale, semi-automatically generated resource designed for evaluating temporal reasoning in Large Language Models (LLMs). It provides a benchmark for analyzing the tabular temporal abilities of LLMs by enabling controlled variations in data characteristics, making it superior to traditional human-curated datasets. This section describes the dataset creation process, its schema, and key characteristics.

2.1 TempTabQA-C Creation Pipeline

The creation of TempTabQA-C follows a systematic pipeline to extract, structure, and store temporal information from Wikipedia infoboxes. Below, we describe the steps involved in detail.

Extracting Temporal Information. Temporal information about athletes, tournaments, events,
and achievements is extracted from Wikipedia infoboxes. These tables contain attributes such as "Name," "Date of Birth," "Tournaments Played," and "Medals Won," which are programmatically extracted and input into a relational database using a predefined schema. This step ensures that the raw tabular data is converted into a structured format for efficient querying and storage.

Relational Database Creation. The structured temporal data is converted into a relational database schema to enable efficient storage and querying. The schema is designed to represent key entities and their relationships comprehensively:

• Athlete Table: Contains a unique athlete_id and the corresponding athlete’s name.

• Personal Information Table: Captures birth year, month, and day for each athlete, linked to the athlete_id.

• Tournament Table: Stores tournament details, such as the name (e.g., "Olympic Games") and the athlete_id.

• Format Table: Represents event formats (e.g., "100m Freestyle"), linked to tournaments through tournament_id.

• Medal Table: Documents medals, including type (e.g., "Gold"), year (e.g., "2016"), and location (e.g., "Rio de Janeiro"), linked to formats through format_id.

This schema ensures all entities are interconnected via primary and foreign keys, enabling complex queries like calculating an athlete’s age at the time of their first medal or comparing performance across tournaments.

Question and Answer Generation Questions are generated using predefined templates filled with key attributes from the relational database. Templates capture a wide range of temporal reasoning scenarios, such as:
• At what age did [Athlete] win his most recent [Tournament] [Medal Type]?
• At what age did Michael Phelps win his most recent Pan Pacific Championships Silver Medal?

To generate answers, the relational database is queried using SQL-based logic, which systematically retrieves the necessary information. For instance, answering a question about the age of an athlete during a specific tournament involves retrieving the athlete’s birth year and the tournament year from the database and calculating the difference. Similarly, questions about medal counts or locations are answered by aggregating or filtering data from the tables.

The SQL-based logic is generalized across various question types, allowing the generation of thousands of unique question-answer pairs. Examples include:
• At what age did Michael Phelps win his most recent Olympic Games Silver Medal? Answer: 29
• In which city did Caeleb Dressel win his most recent Olympic Games Silver Medal? Answer: Tokyo

This approach ensures the dataset is both scalable and robust for evaluating temporal reasoning in LLMs.

2.2 TEMP TABQA-C Composition and Splits
The TEMP TABQA-C dataset is divided into Original and CounterFact questions, with each category further subdivided based on table size and question reasoning difficulty. This structure enables fine-grained and comprehensive evaluations.

Original Questions. Original questions are derived directly from the structured database and are categorized as follows:

1. Table size: we make questions on the table with varied sizes: (a.) Small Tables: Contain concise data, typically representing athletes with fewer medals, (b.) Large Tables: Contain extensive data, often representing athletes with a larger number of medals.

2. Question Complexity: we answer questions on varied difficulty some requiring complex multi-hop reasoning: (a.) Easy: Require basic facts retrieval or single-step reasoning. E.g.: "How many formats has Michael Phelps played?", (b.) Medium: Involve multi-step reasoning, such as calculations or comparisons. E.g.: "At what age did Michael Phelps win his most recent Olympic Silver Medal?", and (c.) Hard: Demand complex reasoning, temporal analysis, and synthesis of multiple facts. E.g.: "What is the shortest time span (in years) within which Michael Phelps won gold, silver, and bronze medals in the same format across any tournament?"

Counterfactual Questions. Counterfactual questions modify specific facts in the original dataset while maintaining the same categorization based on table size and difficulty of the reasoning of the questions. This design challenges models to reason effectively under hypothetical scenarios.
Significance of TempTabQA-C. The dataset offers several unique advantages: (a.) Controlled Evaluation: Provides a framework for systematically testing LLMs across diverse data characteristics., (b.) Scalability: Comprises over 200,000 questions spanning a wide range of contexts and complexities., and (c.) Fine-Grained Analysis: Facilitates benchmarking of model biases and limitations, particularly for temporal reasoning.

By providing a controlled, scalable, and diverse dataset, TempTabQA-C establishes a robust foundation for advancing research on temporal reasoning in LLMs.

2.3 TempTabQA-C Test Statistics.

In order to evaluate the LLM’s we created a subset of the TempTabQA-C dataset having the following number of question per category:

<table>
  <tr>
    <th>Category</th>
    <th>#Examples</th>
    <th>Category</th>
    <th>#Examples</th>
  </tr>
  <tr>
    <td>Original</td>
    <td>578</td>
    <td>Easy</td>
    <td>732</td>
  </tr>
  <tr>
    <td>Counterfactual</td>
    <td>699</td>
    <td>Medium</td>
    <td>507</td>
  </tr>
  <tr>
    <td>Small Table</td>
    <td>855</td>
    <td>Hard</td>
    <td>719</td>
  </tr>
  <tr>
    <td>Large Table</td>
    <td>538</td>
    <td><b>Total</b></td>
    <td><b>5067</b></td>
  </tr>
</table>

Table 1: Dataset Splits and Their Number of Examples.

In our test set, the average context length for the Small Table Split is **53.80** words when using the infobox as the context, while for the Large Table Split, it increases significantly to **348.85** words.

3 Answering Questions about Tables

In this section, we describe two high level approaches for answering questions about structured data.

3.1 Approach 1: Direct Prompting

Direct prompting is the standard approach today, where the LLM is presented with the table contents alongside a natural language question and is expected to return an answer. To improve reasoning, prompting methods have been proposed, such as Chain of Thought (CoT), Program of Thought (PoT), Faithful Chain of Thought (F-CoT), and Plan and Solve, etc.

While these methods introduce symbolic elements and explicit reasoning, they still rely on raw table data and suffer from limitations. The model may rely on memorized patterns rather than reasoning about the content, and be sensitive to table size and inconsequential perturbations in the table. Moreover, although CoT, PoT and F-CoT provide explicit reasoning, the reasoning remains loosely structured, making verification and consistency challenging.

Despite these issues, direct prompting remains widely used due to its ease of implementation.

3.2 Approach 2: Symbolic Intermediate Representation

We propose an alternative approach: **symbolic intermediate representation**. We hypothesize that it can mitigate the issues of direct prompting by shifting from raw table data processing to structured query generation.

Instead of providing raw table contents, we transform the table into a **structured schema**, which consists of metadata such as column names, data types, and relational links, without exposing actual values. The LLM then builds its reasoning upon this schema by generating a **structured query** (e.g., SQL) to retrieve the answer.

By passing only the schema, the LLM is guided to reason in a structured manner, reducing reliance on spurious table patterns. The model’s generated query is executed on the database to retrieve the final answer, ensuring explicit and verifiable reasoning.

With this approach, since the raw data is masked, the model is less likely to hallucinate due to table noise, and less sensitive to minor table modifications. Moreover, the time to answer a question depends less on the size of the table, because query execution in database engines is efficient even for large tables. Finally, the generated query provides a highly structured reasoning path, ensuring systematic verification and consistency.

4 Experimental Setup

We designed experiments to address the following research questions:

1. **Robustness to Counterfactual Data**: How robust are direct LLM prompts to counterfactual data, and can symbolic intermediate representations improve this?

2. **Handling Large Tables**: Can a symbolic intermediate representation outperform direct prompting when applied to larger tables?

3. **Impact of Question Complexity**: How does increasing question complexity impact the performance of these two approaches?
To answer these questions, we conducted experiments to evaluate the two core approaches: Direct Prompting and Symbolic Intermediate Representation.

4.1 Direct Prompting
In this approach, models are provided with the table and question in natural language, and they generate answers as free text or structured programs based on the raw table. We evaluate multiple configurations. In the static few-shot setup, the model is presented with a fixed set of examples, whereas in the adaptive few-shot setup, examples are dynamically selected based on their relevance to the given question.

Beyond few-shot prompting, we evaluate C.L.E.A.R, a structured method that first extracts relevant rows from the table, decomposes the question into sub-questions, solves each sub-question individually, and synthesizes the final answer. We also consider Program of Thought (PoT), where the model generates structured Python programs to extract the necessary table contents and store them as variables before computing the answer. Faithful Chain of Thought (FCoT) extends PoT by requiring the model to decompose the reasoning process into explicit steps before generating the corresponding program. We also evaluate Chain of Thought (CoT), where the model explicitly generates intermediate logical steps before producing the final answer, and Plan and Solve, a two-stage approach where the model first formulates a reasoning plan before executing step-by-step calculations.

4.2 Symbolic Intermediate Representation
In contrast to direct prompting, this approach does not expose the raw table contents to the model. Instead, the model is provided with only the table schema and must generate an SQL query, which is executed to retrieve the answer. We evaluate two variations: static few-shot SQL, where a fixed set of natural language-to-SQL mappings is included in the prompt alongside the schema, and adaptive few-shot SQL, where SQL examples are dynamically selected based on their relevance to the given question. In both cases, the model receives the schema as context and must generate an appropriate SQL query to obtain the answer.

We analyze the performance of these methods in the Results and Analysis section.

We used the TempTabQA-C dataset, which includes Original, counterfactual, and question difficulty (Easy, Medium, Hard) splits, along with small and large table contexts. Models were evaluated using Exact Match Score (EMS) \( ^1 \), focusing on the following key splits:

• Original vs. Counterfactual: We examined whether the gap between Original and Counterfactual data reduces as we move toward symbolic intermediate reasoning.
• Large Table vs. Small Table: We evaluated if the gap between large and small tables decreases with symbolic intermediate reasoning.
• Performance by Question Complexity: We analyzed performance trends across Easy, Medium, and Hard questions, particularly the improvement brought by symbolic intermediate reasoning.

Through these experiments, we aim to demonstrate that symbolic intermediate reasoning reduces sensitivity to counterfactual data, scales better with table size, and handles increasing question complexity more effectively than direct prompting.

5 Results and Analysis
In this section, we present the results for GPT-4o and Gemini 1.5 Pro. Additionally, we evaluated Gemini 1.5 Flash, GPT-4o Mini, Mixtral, Llama 3.1 70B, Code Llama, and SQL Coder, which demonstrated similar trends. The results of these additional experiments are included in the Appendix.

5.1 Robustness on Counterfactual Data.
To evaluate counterfactual robustness, we compare model performance on original and counterfactual datasets. Table 2 and 3 summarizes these results, including the performance gap (\( \Delta \)) between the original and counterfactual performance for GPT 4o and Gemini-1.5-Pro.

<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Original</th>
    <th>CounterF.</th>
    <th>\( \Delta \)</th>
  </tr>
  <tr>
    <td rowspan="6">Direct</td>
    <td>Static</td>
    <td>56.57</td>
    <td>42.35</td>
    <td>14.22</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>58.13</td>
    <td>40.92</td>
    <td>17.21</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>65.57</td>
    <td>48.21</td>
    <td>17.36</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>69.90</td>
    <td>48.50</td>
    <td>21.40</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>69.55</td>
    <td>46.50</td>
    <td>23.05</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>56.40</td>
    <td>47.07</td>
    <td>9.33</td>
  </tr>
  <tr>
    <td></td>
    <td>Faithful CoT</td>
    <td>57.44</td>
    <td>47.78</td>
    <td>9.66</td>
  </tr>
  <tr>
    <td rowspan="2">SQL</td>
    <td>Static</td>
    <td>65.22</td>
    <td>60.94</td>
    <td>4.28</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>71.63</td>
    <td>68.67</td>
    <td>2.96</td>
  </tr>
</table>

Table 2: Original vs Counterfactual for GPT-4o.

\( ^1 \)We also used a relaxed version of EMS (REMS), with similar results detailed in the appendix.
<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Original</th>
    <th>CounterF.</th>
    <th>△</th>
  </tr>
  <tr>
    <td rowspan="7">Direct</td>
    <td>Static</td>
    <td>52.91</td>
    <td>39.92</td>
    <td>12.99</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>53.48</td>
    <td>44.19</td>
    <td>9.29</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>49.33</td>
    <td>40.66</td>
    <td>8.67</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>66.46</td>
    <td>55.75</td>
    <td>10.71</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>60.75</td>
    <td>54.31</td>
    <td>6.44</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>53.83</td>
    <td>46.20</td>
    <td>7.63</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>53.95</td>
    <td>47.64</td>
    <td>6.31</td>
  </tr>
  <tr>
    <td rowspan="2">SQL</td>
    <td>Static</td>
    <td><b>59.08</b></td>
    <td><b>55.52</b></td>
    <td><b>3.56</b></td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td><b>65.29</b></td>
    <td><b>65.13</b></td>
    <td><b>0.16</b></td>
  </tr>
</table>

Table 3: Original vs Counterfactual for Gemini 1.5 Pro.

Analysis: Comparing the performance of GPT-4o and Gemini 1.5 Pro across original and counterfactual datasets provides valuable insights into the robustness of Direct Prompting and SQL-based reasoning methods. A model that truly reasons about data should not be affected by the origin of the data.

However, for Direct Prompting, both models exhibit significant performance gaps between the original and counterfactual datasets, indicating a heavy reliance on memorized knowledge rather than robust reasoning capabilities. For GPT-4o, the performance gaps are 14.22 (non-adaptive) and 17.21 (adaptive), while Gemini 1.5 Pro shows similar gaps of 12.99 and 9.29, respectively. Notably, the adaptive approach improves the performance on original data but increases sensitivity to counterfactuals for GPT-4o. In contrast, Gemini 1.5 Pro’s adaptive Direct Prompting reduces the gap but still fails to address the core issue of data sensitivity.

On the other hand, SQL-based methods, demonstrate superior robustness in both models, with performance gaps significantly smaller than those in Direct Prompting. For GPT-4o, the non-adaptive SQL gap is just 4.28, and the adaptive SQL gap is 2.96. Similarly, for Gemini 1.5 Pro, the non-adaptive SQL gap is 3.56, and the adaptive SQL gap reduces further to 0.16, showcasing its capability to reason effectively across the original and counterfactual datasets. The use of symbolic intermediate representations in SQL methods explains this robustness, as these approaches operate independently of the data origin, focusing instead on schema-driven reasoning.

Finally, the adaptive approach enhances performance across both methods and models, particularly for SQL. For example, in GPT-4o, adaptive SQL improves counterfactual performance by 7.73 points compared to non-adaptive SQL, while in Gemini 1.5 Pro, it further narrows the performance gap to an almost negligible 0.16 points, while improving counterfactual performance by 9.61 points. This highlights the role of adaptive few-shot examples in enhancing model reasoning capabilities and robustness across diverse datasets.

5.2 Impact of Table Size.

To evaluate the impact of data size, we compare model performance on small and large datasets. Table 4 and 5 presents the results, including the gap between small and large datasets for GPT 4o and Gemini 1.5 Pro.

<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Small</th>
    <th>Large</th>
    <th>△</th>
  </tr>
  <tr>
    <td rowspan="7">Direct</td>
    <td>Static</td>
    <td>71.11</td>
    <td>46.84</td>
    <td>24.27</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>73.92</td>
    <td>48.88</td>
    <td>25.04</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>76.49</td>
    <td>53.53</td>
    <td>22.96</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>74.15</td>
    <td>56.13</td>
    <td>18.02</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>75.44</td>
    <td>55.20</td>
    <td>20.24</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>62.22</td>
    <td>50.19</td>
    <td>12.03</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>62.69</td>
    <td>50.19</td>
    <td>12.50</td>
  </tr>
  <tr>
    <td rowspan="2">SQL</td>
    <td>Static</td>
    <td><b>73.57</b></td>
    <td><b>70.63</b></td>
    <td><b>2.94</b></td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td><b>73.92</b></td>
    <td><b>72.86</b></td>
    <td><b>1.06</b></td>
  </tr>
</table>

Table 4: Small Table vs Large Table for GPT-4o

<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Small</th>
    <th>Large</th>
    <th>△</th>
  </tr>
  <tr>
    <td rowspan="7">Direct</td>
    <td>Static</td>
    <td>65.02</td>
    <td>43.86</td>
    <td>21.16</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>67.27</td>
    <td>41.86</td>
    <td>25.41</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>56.85</td>
    <td>40.95</td>
    <td>15.90</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>77.55</td>
    <td>57.67</td>
    <td>19.88</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>73.43</td>
    <td>56.72</td>
    <td>16.71</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>64.68</td>
    <td>49.84</td>
    <td>14.83</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>63.85</td>
    <td>48.47</td>
    <td>15.39</td>
  </tr>
  <tr>
    <td rowspan="2">SQL</td>
    <td>Static</td>
    <td><b>77.41</b></td>
    <td><b>71.32</b></td>
    <td><b>6.09</b></td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td><b>75.31</b></td>
    <td><b>72.43</b></td>
    <td><b>2.88</b></td>
  </tr>
</table>

Table 5: Small Table vs Large Table for Gemini 1.5 Pro

Analysis: A model capable of genuine reasoning should operate independently of data size. For example, the correctness of an SQL query’s result is unaffected by the size of the tables—it impacts only the computation time, not the quality of the outcome. However, the trends for small vs. large tables closely mirror those observed in the original vs. counterfactual analysis. Direct Prompting shows significant performance drops with larger tables, with GPT-4o and Gemini 1.5 Pro exhibiting gaps of 24.27 and 21.16 in non-adaptive settings, respectively. This underscores the method’s sensitivity to data complexity and dependence on memory.

In contrast, SQL-based methods demonstrate remarkable robustness, maintaining minimal performance gaps across table sizes (e.g., 1.06 for adaptive SQL in GPT-4o and 2.88 in Gemini 1.5
Pro). This resilience stems from schema-driven reasoning, which abstracts away from the data’s size or origin 2. Adaptive few-shot examples further enhance performance, particularly for SQL-based methods, allowing them to consistently deliver high accuracy even with larger tables.

These findings emphasize that Direct Prompting struggles with data complexity and scale, mirroring its limitations in counterfactual settings. SQL-based methods, on the other hand, exemplify robustness and scalability by leveraging schema-driven symbolic representations that are agnostic to data size or source. The dynamic selection of adaptive examples further strengthens their reliability, making them a superior choice for reasoning over complex and evolving datasets.

5.3 Effect of question complexity.

To evaluate question complexity effects, we compare model performance on Easy, Medium, and Hard questions. Table 6 and 7 summarizes the results for GPT-4o and Gemini-1.5-Pro respectively.

<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="7">Direct</td>
    <td>Static</td>
    <td>71.18</td>
    <td>63.12</td>
    <td>53.35</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>74.38</td>
    <td>63.91</td>
    <td>54.17</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>76.40</td>
    <td>71.99</td>
    <td>62.62</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>78.43</td>
    <td>75.35</td>
    <td>64.06</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>77.96</td>
    <td>70.81</td>
    <td>60.25</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>69.94</td>
    <td>57.20</td>
    <td>48.81</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>69.24</td>
    <td>56.02</td>
    <td>47.17</td>
  </tr>
  <tr>
    <td>SQL</td>
    <td>Static</td>
    <td>78.89</td>
    <td>75.15</td>
    <td>62.31</td>
  </tr>
  <tr>
    <td></td>
    <td>Adaptive</td>
    <td>80.06</td>
    <td>73.37</td>
    <td>66.74</td>
  </tr>
</table>

Table 6: Easy, Medium, and Hard results for GPT-4o

<table>
  <tr>
    <th>Approach</th>
    <th>Method</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="7">Direct</td>
    <td>Static</td>
    <td>65.79</td>
    <td>58.53</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>66.26</td>
    <td>56.47</td>
    <td>46.74</td>
  </tr>
  <tr>
    <td>C.L.E.A.R</td>
    <td>58.38</td>
    <td>59.77</td>
    <td>51.14</td>
  </tr>
  <tr>
    <td>CoT</td>
    <td>83.29</td>
    <td>72.69</td>
    <td>65.87</td>
  </tr>
  <tr>
    <td>Plan & Solve</td>
    <td>79.42</td>
    <td>72.25</td>
    <td>63.60</td>
  </tr>
  <tr>
    <td>PoT</td>
    <td>73.97</td>
    <td>57.12</td>
    <td>47.57</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>74.52</td>
    <td>58.15</td>
    <td>46.85</td>
  </tr>
  <tr>
    <td>SQL</td>
    <td>Static</td>
    <td>80.86</td>
    <td>70.33</td>
    <td>59.59</td>
  </tr>
  <tr>
    <td></td>
    <td>Adaptive</td>
    <td>75.86</td>
    <td>71.47</td>
    <td>59.24</td>
  </tr>
</table>

Table 7: Easy, Medium, and Hard results on Gemini 1.5 Pro

Analysis: Performance consistently declines across all models and settings as question complexity increases from Easy to Hard, aligning with previous findings on the influence of data size and complexity. While such a drop is expected for both models and humans (though less severe for the latter), the key question is: can we do better and reduce this decline? Direct Prompting struggles as question complexity increases, with significant drops in accuracy (e.g., from 71.18 to 53.35 for non-adaptive GPT-4o). Direct adaptive prompting also struggles to mitigate this decline and remains limited in handling complex queries effectively.

SQL-based methods demonstrate greater resilience to complexity, maintaining higher accuracy across all levels. For example, non-adaptive SQL in GPT-4o drops moderately from 78.89 (Easy) to 62.31 (Hard), while adaptive SQL, achieves the best performance of 66.74 on the Hard data split. Similarly, Gemini 1.5 Pro exhibits stable performance with SQL based reasoning.

These results reinforce SQL’s robustness through schema-driven reasoning, which abstracts complexity and reduces reliance on memorization. Introducing adaptive examples lifts accuracy across prompting strategies, with the most pronounced gains in the SQL-based variants, which remain the strongest on the hardest queries. This underscores the importance of structured reasoning and adaptive techniques for tackling increasing data and query complexity effectively.

6 What Did We Learn?

1. Impact of Symbolic Representations. Parsing data into symbolic queries consistently boosts model performance. Symbolic representations bridge counterfactual gaps, reduce dependence on data size, and enhance the handling of complex questions. By structuring data more clearly, symbolic queries improve robustness and address challenges like noise and memorization.

2. Benefits of Schema-Based Reasoning. Schemas provide a clean, data-agnostic abstraction of database structures, removing irrelevant noise and simplifying reasoning. By presenting only the schema without any underlying data, we ensure there is no room for memorization. Unlike raw tables, which mix useful and irrelevant data, schemas provide a stable framework that ensures consistent performance, especially in counterfactual scenarios where structured reasoning is critical.

3. Effect of Data Size. Data size significantly affects model performance. Larger tables often introduce noise, increasing the risk of hallucinations.

2We tested counterfactual versions, showing similar findings to section 4.1.
Schemas mitigate this by segmenting data into key components, reducing cognitive overload and clarifying the reasoning process, allowing models to perform more reliably on large, complex datasets.

4. Handling Complex Questions. Schema-based reasoning excels in answering complex questions by supporting logical, step-by-step reasoning. SQL query generation fosters clarity and reduces ambiguity. In contrast, raw text tables, especially those with counterfactual data, often lack structure, leading to errors or incomplete reasoning. By offering a predefined framework, schemas reduce cognitive demands, enabling models to handle nuanced queries more effectively.

7 Discussion of Model Failures

7.1 Inadequacy of Direct Complex Strategies
Several techniques, such as Program of Thought (PoT) Chen et al. (2023), Chain of Table(Wang et al., 2024a), Binder(Cheng et al.), Dater Ye et al. (2023b), and Plan and Solve Wang et al. (2023), aim to handle complex queries. However, these methods fall short when detailed query plans are needed. The complexity of tasks involving multiple steps, conditional logic, and dependencies cannot be captured by direct prompting alone. Each query introduces unique variables, making strategies like PoT fails for complex reasoning.

For example, a query requiring the join of three large tables with specific conditions cannot be effectively handled by PoT, which may only generate simple steps like "select from Table A" or "filter Table B." These methods fail to capture the necessary logic for combining tables or handling multiple joins and nested queries. Such complexity requires a carefully constructed query plan, which direct prompting cannot produce.

The core issue is the complexity of the underlying query plans. PoT may generate query plans, but they struggle with complex operations like joins, aggregations, and nested subqueries, which demand precise sequencing and optimization. Research, particularly by Akioyamen et al. (2024), argues that query planning requires structured approaches like SQL to manage these complexities, reinforcing that simpler prompting strategies are insufficient for intricate query reasoning.

7.2 Challenges with Symbolic Approach
Despite advancements in symbolic representation, several challenges remain in improving model reliability and performance:

1. SQL Query Inconsistencies  The model often misuses SQL constructs, such as over-relying on LIMIT 1 when multiple answers are needed "List all the formats in which Carolina Marín has won medals?" or adding redundant joins that slow execution "How many tournaments did Michael Phelps win in 2008?". It also misaligns query objectives, failing to handle aggregates or GROUP BY clauses properly "What are the medal counts for each athlete in the Olympics?".

2. Temporal and Positional Reasoning Errors
The model struggles with temporal and positional reasoning, often hallucinating columns or misinterpreting data "At what age did Michael Phelps win his most recent Olympic Gold Medal?". It also misaligns aggregations over time "Which athlete had the most consistent medal wins over the last decade?" and hierarchical relationships "Which was Michael Phelps’ most recent tournament medal?".

3. Nested and Conditional Logic Challenges
Errors occur in nested and conditional logic, such as incorrect use of WITH clauses "Which event had the shortest duration between P. V. Sindhu’s medal wins?" or failing to respect conditions "List all tournaments where Carolina Marín won a medal after 2015?". The model also mishandles multi-field responses "List the medal type, location, and year for Hugo Calderano’s wins.".

4. Aggregates, Joins, and Dependencies
The model struggles with nested aggregates, non-standard joins, and dependency tracking. It fails to construct valid joins "Which format had the highest number of gold medals in 2020?" or align dependencies in complex queries "Which medal did Michael Phelps win in the same tournament as his fastest recorded swim?". It also ignores group-level constraints, leading to overgeneralized results "Which city hosted the most gold-medal-winning tournaments for P. V. Sindhu?".

5. Inconsistencies and Robustness Issues
Inconsistent query structures lead to variable results for similar tasks "How old was Hugo Calderano when he won his first medal?" vs. "At what age did Michael Phelps win his most recent Olympic Gold Medal?". The model struggles with entity disambiguation "List all the medals won by Michael Phelps in the Olympic Games?" and overlooks edge cases "How many medals has an athlete
with no wins received?". Ranking logic is often mishandled, such as ignoring ordering requirements "Which city hosted the most tournaments in 2019?".

8 Comparison with Related Work

Temporal reasoning in LLMs is an evolving field intersecting with advancements in tabular reasoning, logic, and symbolic methods. Our work advances this area by introducing the TEMP TABQA-C dataset for detailed evaluation of temporal reasoning in tabular contexts. Key advancements in related areas are discussed below.

Tabular Reasoning. Applying LLMs to semi-structured tables has been studied for question answering, semantic parsing, and table-to-text generation (Chen et al., 2020; Gupta et al., 2020; Zhang et al., 2020; Zhang and Balog, 2020). Approaches like TAPAS (Herzig et al., 2020), TaBERT (Yin et al., 2020), and TABBIE (Iida et al., 2021) enhance table understanding through joint tabular-textual embeddings, while Table2vec (Zhang et al., 2019) and TabGCN (Pramanick and Bhattacharya, 2021) investigate alternative representations.

Recent work also explores symbolic reasoning for structured tables with fixed schemas (Cheng et al., 2023; Ye et al., 2023a; Wang et al., 2024b). Building on these advances, we use SQL-based symbolic methods for temporal queries in semi-structured data. Our TEMP TABQA-C dataset further enables fine-grained evaluation of temporal reasoning across diverse tabular characteristics.

Temporal Reasoning. Temporal reasoning is central to question answering and event-centric tasks, with datasets like TIME-SENSITIVEQA (Chen et al., 2021) and TORQUE (Ning et al., 2020) addressing time-sensitive comprehension in text, and TEMPOQA-WD (Neelam et al., 2022) and CRONQUESTIONS (Saxena et al., 2021) focusing on temporal links in knowledge graphs. Models like CRONKBQA (Saxena et al., 2021) further enhance performance by incorporating temporal reasoning during training.

Our work extends these efforts to structured tabular datasets. While datasets such as TempTabQA (Gupta et al., 2023) and TRAM (Wang and Zhao, 2024) tackle similar challenges, TEMP TABQA-C advances the field by introducing counterfactual reasoning, scalable table sizes, and diverse question difficulties, offering a broader framework for evaluating temporal reasoning.

Logical Reasoning and Symbolic Approaches
Frameworks like LOGIC-LM (et al., 2023b) and neurosymbolic methods such as LINC (et al., 2023c) show that integrating symbolic reasoning and external tools enhances logical inference in LLMs. Auto-formalization (e.g., NL2FOL (et al., 2023a)) further boosts reasoning accuracy by mapping natural language to structured forms.

Building on these advances, our SQL-based symbolic approach enables precise temporal reasoning over tables by translating queries into executable SQL. The TEMP TABQA-C dataset offers a comprehensive benchmark at the intersection of tabular, temporal, and symbolic reasoning, featuring original and counterfactual splits, scalable table sizes, and varied question difficulties—all aligned with SQL-based structured reasoning.

9 Conclusion

This work investigates temporal tabular question answering with LLMs, tackling key challenges in counterfactual robustness, data sensitivity, and question complexity. We introduced TEMP TABQA-C, a controlled benchmark designed for systematic evaluations. By combining symbolic intermediate representations with adaptive few-shot prompting, our approach leverages database schemas and SQL query generation to address the limitations of direct prompting.

Our experiments demonstrate that symbolic representations improve generalization, counterfactual robustness, and scalability, especially when handling larger tables. Additionally, adaptive prompting enhances reasoning for complex queries. These results highlight the necessity for the model to be data-blind, reasoning exclusively on metadata—such as the table schema—to ensure robust reasoning rather than relying on mere memorization. Immediate future work can focus on conducting detailed error analysis and exploring fine-tuning techniques. A deeper analysis of the results will further illuminate the strengths and limitations of the approach. TEMP TABQA-C lays a strong foundation for advancing structured temporal reasoning in LLMs and encourages future efforts to develop interpretable and robust temporal reasoning in AI systems.
Limitations

We demonstrated the effectiveness of our approach through extensive experiments in English. However, extending the study to a multilingual context could reveal its applicability across diverse languages. While our work focuses on simple, entity-centric tables, real-world datasets are often more complex, such as hierarchical or multi-relational tables. Future research should explore these more intricate structures to expand the method’s utility.

Our experiments assume static tables, yet many real-world scenarios involve dynamic data, such as streaming or frequently updated tables. Adapting the method to handle evolving datasets would enhance its practical relevance. Additionally, the approach does not leverage external domain knowledge, which could complement symbolic reasoning and broaden its applications.

The dataset may also exhibit inherent biases, such as domain-specific or entity-centric constraints, limiting generalizability. Future datasets should aim for greater diversity to better reflect real-world scenarios. Finally, due to computational constraints, we did not fine-tune models on the TEMP TABQA-C dataset. Future work should address this limitation by exploring fine-tuning on larger datasets and evaluating the approach in more resource-intensive and dynamic settings for a comprehensive assessment.

Ethics Statement

We are deeply committed to upholding the highest ethical standards in research and publication. To ensure transparency and reproducibility, we will publicly release our code, enhanced evaluation set, and detailed documentation, enabling the research community to validate, reproduce, and build upon our work. By sharing our resources, we aim to foster collaboration and accountability within the computational linguistics field.

Our methodology reflects a commitment to the responsible and fair use of tools and techniques, with all claims grounded in rigorously validated experimental results. To address the stochastic nature of black-box models, we maintained a fixed temperature throughout our experiments, ensuring consistent outcomes. AI tools were employed responsibly during the writing process, with careful oversight to prevent bias or inaccuracies. We provide comprehensive details about annotations, dataset splits, models, and prompting methods to ensure full reproducibility and empower researchers to evaluate our work rigorously.

Recognizing the importance of inclusivity and fairness, we acknowledge that our dataset may carry inherent biases, such as domain-specific or entity-centric limitations. While we strive for broad applicability, future iterations will prioritize greater diversity to enhance fairness and generalizability. By adhering to these principles, we aim to advance knowledge in computational linguistics while promoting ethical and responsible research practices that emphasize transparency, equity, and reproducibility.

Acknowledgments

This work was primary sponsored by the Army Research Office under Grant Number W911NF-20-1-0080. The views and conclusions expressed herein are those of the authors and do not necessarily reflect the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright notice herein. This research was also partially supported by ONR Contract N00014-19-1-2620. V.S. acknowledges support in part from NSF awards #2217154 and #2411319.

We are grateful to the Utah NLP group (University of Utah) and the CogComp group (University of Pennsylvania) for their valuable insights. We also thank Dibyakanti Kumar and Harsh Kumar for their significant contributions to the development of the dataset. The author is also grateful to Prof. Ana Marasović, University of Utah for their support. We appreciate the thoughtful feedback from our anonymous reviewers and the computational support provided by the Complex Data Analysis and Reasoning Lab at Arizona State University. Lastly, we thank our CoRAL lab cat, Coco, for her unwavering dedication to ensuring that professor, V.G., remained just the right amount of creatively unhinged during deadline season.

References

Peter Akioyamen, Zixuan Yi, and Ryan Marcus. 2024. The unreasonable effectiveness of llms for query optimization. arXiv preprint arXiv:2411.02862.

Wenhui Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2023. Program of thoughts
prompting: Disentangling computation from reasoning for numerical reasoning tasks. Preprint, arXiv:2211.12588.

Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations.

Wenhu Chen, Xinyi Wang, and William Yang Wang. 2021. A dataset for answering time-sensitive questions. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2023. Binding language models in symbolic languages. Preprint, arXiv:2210.02875.

Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, et al. Binding language models in symbolic languages. In The Eleventh International Conference on Learning Representations.

Chen et al. 2023a. Nl2fol: Natural language to first-order logic. arXiv preprint arXiv:2304.09102.

Liangming Pan et al. 2023b. Logic-lm: Empowering large language models with symbolic solvers for faithful logical reasoning. Findings of the Association for Computational Linguistics.

Theo X. Olausson et al. 2023c. Linc: Logical inference via neurosymbolic computation. arXiv preprint arXiv:2310.15164.

Vivek Gupta, Pranshu Kandoi, Mahek Vora, Shuo Zhang, Yujie He, Ridho Reinanda, and Vivek Srikumar. 2023. TempTabQA: Temporal question answering for semi-structured tables. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2431–2453, Singapore. Association for Computational Linguistics.

Vivek Gupta, Maitrey Mehta, Pegah Nokhiz, and Vivek Srikumar. 2020. INFOTABS: Inference on tables as semi-structured data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2309–2324, Online. Association for Computational Linguistics.

Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.

Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. Tabbie: Pretrained representations of tabular data. Preprint, arXiv:2105.02584.

Sumit Neelam, Udit Sharma, Hima Karanam, Shahjith Ibkal, Pavan Kapanipathi, Ibrahim Abdelaziz, Nandana Mihindukulasooriya, Young-Suk Lee, Santosh Srivastava, Cezar Pendus, Saswati Dana, Dinesh Garg, Achille Fokoue, G P Shrivatsa Bhargav, Dinesh Khandelwal, Srinivas Ravishankar, Sairam Gurajada, Maria Chang, Rosario Uceda-Sosa, Salim Roukos, Alexander Gray, Guilherme Lima, Ryan Riegel, Francois Luus, and L Venkata Subramaniam. 2022. A benchmark for generalizable and interpretable temporal question answering over knowledge bases. Preprint, arXiv:2201.05793.

Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, and Dan Roth. 2020. TORQUE: A reading comprehension dataset of temporal ordering questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1158–1172, Online. Association for Computational Linguistics.

Aniket Pramanick and Indrajit Bhattacharya. 2021. Joint learning of representations for web-tables, entities and types using graph convolutional network. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1197–1206, Online. Association for Computational Linguistics.

Apoorv Saxena, Soumen Chakrabarti, and Partha Talukdar. 2021. Question answering over temporal knowledge graphs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6663–6676, Online. Association for Computational Linguistics.

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL).

Yuqing Wang and Yun Zhao. 2024. Tram: Benchmarking temporal reasoning for large language models. Preprint, arXiv:2310.00835.

Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Milicicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024a. Chain-of-table: Evolving tables in the reasoning chain for table understanding. In International Conference on Learning Representations (ICLR).

Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Milicicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024b. Chain-of-table: Evolving tables in the reasoning chain for table understanding. Preprint, arXiv:2401.04398.
Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023a. Large language models are versatile decomposers: Decompose evidence and questions for table-based reasoning. Preprint, arXiv:2301.13808.

Yunhu Ye, Binyuan Hui, Min Yang, Binhua Li, Fei Huang, and Yongbin Li. 2023b. Large language models are versatile decomposers: Decomposing evidence and questions for table-based reasoning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 174–184, New York, NY, USA. Association for Computing Machinery.

Pengcheng Yin, Graham Neubig, Wen tau Yih, and Sebastian Riedel. 2020. Tabert: Pretraining for joint understanding of textual and tabular data. Preprint, arXiv:2005.08314.

Li Zhang, Shuo Zhang, and Krisztian Balog. 2019. Table2vec: Neural word and entity embeddings for table population and retrieval. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’19. ACM.

Shuo Zhang and Krisztian Balog. 2020. Web table extraction, retrieval, and augmentation: A survey. ACM Trans. Intell. Syst. Technol., 11(2):13:1–13:35.

Shuo Zhang, Zhuyun Dai, Krisztian Balog, and Jamie Callan. 2020. Summarizing and exploring tabular data in conversational search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, pages 1537–1540, New York, NY, USA. Association for Computing Machinery.

10 Appendix

10.1 Examples:

10.1.1 Example 1:
Q. Which Olympic year marked Michael Phelps’ record for the most gold medals won?

Steps for SQL Reasoning

Step 1: Start with the infobox table of Michael Phelps’ medals

<table>
  <tr>
    <th>Medal</th>
    <th>Year</th>
    <th>Event</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2008 Beijing</td>
    <td>100 m butterfly</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2008 Beijing</td>
    <td>200 m medley</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2004 Indianapolis</td>
    <td>200 m freestyle</td>
  </tr>
  <tr>
    <td>Silver</td>
    <td>2002 Yokohama</td>
    <td>4×200 m freestyle</td>
  </tr>
</table>

Step 2: Transform the data (all swimmer infoxes) into a relational schema and organize it into structured database tables for efficient querying.

Database Schema:

Athlete Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Athlete Name</td>
  </tr>
</table>

Tournament Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>tournament_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Foreign Key (Athlete)</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Tournament Name</td>
  </tr>
</table>

Format Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>format_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>tournament_id</td>
    <td>Foreign Key (Tournament)</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Event Name</td>
  </tr>
</table>

Medal Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>medal_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>format_id</td>
    <td>Foreign Key (Format)</td>
  </tr>
  <tr>
    <td>type</td>
    <td>Medal Type</td>
  </tr>
  <tr>
    <td>year</td>
    <td>Year of Achievement</td>
  </tr>
  <tr>
    <td>location</td>
    <td>Medal Location</td>
  </tr>
</table>

PersonalInformation Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>info_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Foreign Key (Athlete)</td>
  </tr>
  <tr>
    <td>birth_year</td>
    <td>Birth Year</td>
  </tr>
  <tr>
    <td>birth_month</td>
    <td>Birth Month</td>
  </tr>
  <tr>
    <td>birth_day</td>
    <td>Birth Day</td>
  </tr>
</table>

Step 3: Write the SQL Query
The following query retrieves the year with the most gold medals:

WITH gold_medal_counts AS (
  SELECT m.year, COUNT(m.medal_id) AS gold_medals
  FROM Medal m
  JOIN Format f ON m.format_id = f.format_id
  JOIN Tournament t ON f.tournament_id = t.tournament_id
  JOIN Athlete a ON t.athlete_id = a.athlete_id
  WHERE a.name = 'Michael Phelps'
    AND m.type = 'MedalGold'
  GROUP BY m.year
)
SELECT year
FROM gold_medal_counts
WHERE gold_medals = (
  SELECT MAX(gold_medals)
  FROM gold_medal_counts
);

Step 4: Execute the Query
The query outputs the year with the highest number of gold medals.

Final Result: 2008

Direct Reasoning with Chain-of-Thought (CoT):
To perform direct reasoning using Chain-of-Thought (CoT), LLM arrange the medal in year and count the number of gold medals per year from the table:

Year 2008:
- 100 m butterfly (Gold)
Total: 1 gold medals

Year 2004:
- 200 m freestyle (Gold)
- 200 m medley (Gold)
Total: 2 gold medal

Year 2002:
- 4x200 m freestyle (Silver)
Total: 0 gold medals

Answer (CoT Reasoning): 2004 has the most gold medals with a count of 2.

However, due to direct reasoning errors or omissions, it misinterpret the complex table, and hence CoT fails whereas Symbolic succeed.

10.1.2 Example 2:
Q. Does Emma Weyant have more Bronze Medals than Gold Medals?

Steps for SQL Reasoning

Step 1: Start with the infobox table of Emma Weyant’s medals.

<table>
  <tr>
    <th colspan="2">Emma Weyant</th>
  </tr>
  <tr>
    <th colspan="2">Olympic Games</th>
  </tr>
  <tr>
    <td>S</td>
    <td>2020 Tokyo 400 m medley</td>
  </tr>
  <tr>
    <td>B</td>
    <td>2024 Paris 400 m medley</td>
  </tr>
  <tr>
    <td colspan="2">World Championships (LC)</td>
  </tr>
  <tr>
    <td>B</td>
    <td>2022 Budapest 400 m medley</td>
  </tr>
  <tr>
    <td colspan="2">World Championships (SC)</td>
  </tr>
  <tr>
    <td>S</td>
    <td>2021 Abu Dhabi 4x200 m freestyle</td>
  </tr>
  <tr>
    <td colspan="2">Junior Pan Pacific Championships</td>
  </tr>
  <tr>
    <td>G</td>
    <td>2018 Suva 400 m medley</td>
  </tr>
  <tr>
    <td>B</td>
    <td>2018 Suva 800 m freestyle</td>
  </tr>
</table>

Figure 2: Emma Weyant’s Medal Infobox

Step 2: Transform the data (all swimmer infoboxes) into a relational schema and organize it into structured database tables for efficient querying.

Database Schema:

Athlete Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Athlete Name</td>
  </tr>
</table>

Tournament Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>tournament_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Foreign Key (Athlete)</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Tournament Name</td>
  </tr>
</table>

Format Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>format_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>tournament_id</td>
    <td>Foreign Key (Tournament)</td>
  </tr>
  <tr>
    <td>name</td>
    <td>Event Name</td>
  </tr>
</table>

Medal Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>medal_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>format_id</td>
    <td>Foreign Key (Format)</td>
  </tr>
  <tr>
    <td>type</td>
    <td>Medal Type</td>
  </tr>
  <tr>
    <td>year</td>
    <td>Year of Achievement</td>
  </tr>
  <tr>
    <td>location</td>
    <td>Medal Location</td>
  </tr>
</table>

PersonalInformation Table:
<table>
  <tr>
    <th>Column</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>info_id</td>
    <td>Primary Key</td>
  </tr>
  <tr>
    <td>athlete_id</td>
    <td>Foreign Key (Athlete)</td>
  </tr>
  <tr>
    <td>birth_year</td>
    <td>Birth Year</td>
  </tr>
  <tr>
    <td>birth_month</td>
    <td>Birth Month</td>
  </tr>
  <tr>
    <td>birth_day</td>
    <td>Birth Day</td>
  </tr>
</table>

Step 3: Write the SQL Query
The following query checks whether Emma Weyant has more Bronze medals than Gold medals:

SELECT CASE
    WHEN SUM(CASE WHEN m.type = 'MedalBronze' THEN 1 ELSE 0 END) > SUM(CASE WHEN m.type = 'MedalGold' THEN 1 ELSE 0 END)
    THEN 'Yes'
    ELSE 'No'
END AS has_more_bronze_than_gold
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Emma Weyant';
Step 4: Execute the Query
The query outputs whether Emma Weyant has more Bronze medals than Gold medals.

Final Result: Yes

Direct Reasoning with Chain-of-Thought (COT):
Using manual reasoning, the LLM counts the medals directly from the table:

Gold Medals:
- 2018 Suva: 400 m medley
Total: 1 Gold Medal

Bronze Medals:
- 2018 Suva: 800 m freestyle
- 2022 Budapest: 400 m medley
- 2024 Paris: 400 m medley
Total: 3 Bronze Medals

Final Count:
Gold: 1
Bronze: 3

LLM’s Answer: No  Emma Weyant has one Gold Medal and three Bronze Medals.

Why the LLM’s Answer is Incorrect and Symbolic Reasoning Succeeds:

• Direct Reasoning Errors: The LLM correctly identifies the count but fails in its logical comparison, leading to an incorrect conclusion.

• Symbolic Reasoning Accuracy: SQL-based reasoning explicitly performs the correct comparison and produces an unambiguous result.

• Scalability and Consistency: SQL-based methods remain reliable as data size and complexity grow, unlike manual reasoning.

Conclusion: Symbolic SQL reasoning eliminates errors inherent in manual reasoning methods like Chain-of-Thought, ensuring precise and reliable results.

10.1.3 Example 3:
Q. In which city did Yohan Blake win his first medal?

Step 1: Start with the infobox table of Yohan Blake’s medals.

Yohan Blake’s Medal Record:

<table>
  <tr>
    <th>Medal</th>
    <th>Year</th>
    <th>Format</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2012 London</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2016 Rio de Janeiro</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Silver</td>
    <td>2012 London</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Silver</td>
    <td>2012 London</td>
    <td>200 m</td>
  </tr>
  <tr>
    <th colspan="3">World Championships</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2011 Daegu</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2011 Daegu</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <th colspan="3">Commonwealth Games</th>
  </tr>
  <tr>
    <td>Bronze</td>
    <td>2018 Gold Coast</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Bronze</td>
    <td>2018 Gold Coast</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <th colspan="3">World Relays</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2014 Bahamas</td>
    <td>4×100 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2014 Bahamas</td>
    <td>4×200 m</td>
  </tr>
  <tr>
    <td>Bronze</td>
    <td>2017 Bahamas</td>
    <td>4×200 m</td>
  </tr>
  <tr>
    <th colspan="3">World Junior Championships</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Beijing</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Silver</td>
    <td>2008 Bydgoszcz</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Bronze</td>
    <td>2006 Beijing</td>
    <td>100 m</td>
  </tr>
  <tr>
    <th colspan="3">Pan American Junior Championships</th>
  </tr>
  <tr>
    <td>Silver</td>
    <td>2007 São Paulo</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Bronze</td>
    <td>2007 São Paulo</td>
    <td>4×400 m relay</td>
  </tr>
  <tr>
    <th colspan="3">CAC Junior Championships (U20)</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Port of Spain</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Port of Spain</td>
    <td>200 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Port of Spain</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <th colspan="3">CARIFTA Games</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Les Abymes</td>
    <td>200 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2006 Les Abymes</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2007 Providenciales</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2007 Providenciales</td>
    <td>4×100 m relay</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2008 Basseterre</td>
    <td>100 m</td>
  </tr>
  <tr>
    <th colspan="3">CARIFTA Games</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2005 Bacolet</td>
    <td>100 m</td>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2005 Bacolet</td>
    <td>200 m</td>
  </tr>
  <tr>
    <th colspan="3">Continental Cup</th>
  </tr>
  <tr>
    <td>Gold</td>
    <td>2018 Ostrava</td>
    <td>4×100 m</td>
  </tr>
</table>

Step 2: Transform the data into a relational schema and organize it into structured database tables for efficient querying (similar to Step 2 in previous examples).

Step 3: Write the SQL Query
The following query retrieves the location where Yohan Blake won his first medal:

SELECT DISTINCT m.location
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Yohan Blake'
AND m.year = (
  SELECT MIN(m2.year)
  FROM Medal m2
  JOIN Format f2 ON m2.format_id = f2.format_id
  JOIN Tournament t2 ON f2.tournament_id = t2.tournament_id
  JOIN Athlete a2 ON t2.athlete_id = a2.athlete_id
  WHERE a2.name = 'Yohan Blake'
);

Step 4: Execute the Query
The query outputs the location where Yohan Blake won his first medal.
Final Result: Bacolet

Direct Reasoning with Chain-of-Thought (COT):
Using manual reasoning, the LLM incorrectly identifies the location as Beijing:

Year 2006:
- Gold: 4x100 m relay (World Junior Championships, Beijing)
- Bronze: 100 m (World Junior Championships, Beijing)
Conclusion: First medal location is Beijing.

LLM’s Answer: Beijing. In 2006, Yohan Blake won his first medal at the World Junior Championships in Beijing, where he secured a Gold in the 4x100 m relay and a Bronze in the 100 m.

Why the LLM’s Answer is Incorrect and Symbolic Reasoning Succeeds:

• Direct Reasoning Errors: The LLM overlooks earlier results from 2005 in the CARIFTA Games held in Bacolet, where Yohan Blake won two Gold medals.

• Symbolic Reasoning Accuracy: SQL explicitly finds the minimum year and correctly identifies the location associated with the first medal.

• Consistency and Scalability: Symbolic SQL reasoning reliably handles large, complex medal records without omission or error.

Conclusion: Symbolic SQL reasoning eliminates the errors inherent in Chain-of-Thought reasoning, ensuring accurate and reliable results.

10.1.4 Example 4:
Q. How many medals did Mayu Matsumoto win in her twenties?

Step 1: Start with the infobox table of Mayu Matsumoto’s medals.
Step 2: Transform the data into a relational schema and organize it into structured database tables for efficient querying. (similar to Step 2 in previous examples).

Step 3: Write the SQL Query
The following query calculates how many medals Mayu Matsumoto won between the ages of 20 and 29:

SELECT COUNT(m.medal_id) AS total_medals
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
JOIN PersonalInformation pi ON a.athlete_id = pi.athlete_id
WHERE a.name = 'Mayu Matsumoto'
AND (m.year - pi.birth_year) BETWEEN 20 AND 29;

Step 4: Execute the Query
The query calculates medals for Mayu Matsumoto won during her twenties (from 2015 to 2024, since she was born in 1995).

Final Result: 12

Direct Reasoning with Chain-of-Thought (COT):
Using manual reasoning, the LLM incorrectly calculates the total as follows:
Mayu Matsumoto, born in 1995, was in her twenties from 2015 to 2024. During this period:
World Championships: 2 Gold (2018, 2019), 1 Silver (2021), 2 Bronze (2021, 2022).
Sudirman Cup: 2 Silver (2019, 2021), 1 Bronze (2023).
Uber Cup: 1 Silver (2020), 1 Bronze (2022).
Asian Championships: 1 Silver (2019), 1 Bronze (2023).
Total (COT Reasoning): 10 medals.

Trends in the LLM’s Reasoning:

• Incorrect Medal Count for World Champi-
onships: The LLM claims 2 Gold, 2 Silver, and 1 Bronze, but the infobox shows 2 Gold (2018, 2019), 1 Bronze (2022), and only 1 Silver (2021).

• Overcounted/Undercounted Totals: The total medals, when carefully counted, sum to 12, not 10:

– World Championships: 2 Gold, 1 Silver, 1 Bronze (Total = 4).
– Sudirman Cup: 2 Silver, 1 Bronze (Total = 3).
– Uber Cup: 1 Silver, 1 Bronze (Total = 2).
– Asian Championships: 1 Silver, 1 Bronze (Total = 2).
– Incorrectly Excluded 2020 Medal: Asian Team Championships (2020, age 25) is excluded incorrectly.
– Correctly Excluded 2013 Medal: Asian Junior Championships (2013, age 18) is excluded correctly.

• Temporal Misinterpretation: The LLM fails to count some of the medals in the 20-29 age range and fails to sum them accurately.

Symbolic Reasoning Accuracy:

• SQL precisely filters years between 2015 and 2024, ensuring only valid medals are counted.

• Symbolic reasoning eliminates human counting errors and temporal miscalculations.

• The result is accurate: **12 medals**.

Conclusion: The LLM’s Chain-of-Thought reasoning undercounts Mayu Matsumoto’s medals, providing an incorrect total of 10 due to miscounting and temporal errors. Symbolic SQL reasoning accurately identifies the correct total as **12 medals** won during her twenties.

10.1.5 Example 5:
Q. How many times did Sandra Sánchez win a medal in the World Championships before 2021?

Step 1: Start with the infobox table of Sandra Sánchez’s medals.
Step 2: Transform the data into a relational schema and organize it into structured database tables for efficient querying (similar to Step 2 in previous examples).

Step 3: Write the SQL Query
The following query calculates how many medals Sandra Sánchez won in the World Championships before the year 2021:

SELECT COUNT(m.medal_id) AS total_medals
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Sandra Sánchez'
AND t.name = 'World Championships'
AND m.year < '2021';

Step 4: Execute the Query
The query outputs the total number of medals Sandra Sánchez won in the World Championships before 2021.

Final Result: **2**

Direct Reasoning with Chain-of-Thought (COT):
The LLM incorrectly provides the following reasoning:

Sandra Sánchez won a Bronze medal in the World Championships in 2016, which is before 2021. Therefore, the answer is 1.

Errors in the LLM’s Reasoning:

<table>
  <tr>
    <th colspan="2">Sandra Sánchez</th>
  </tr>
  <tr>
    <th colspan="2">Summer Olympics</th>
  </tr>
  <tr>
    <td>2020 Tokyo</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <th colspan="2">World Championships</th>
  </tr>
  <tr>
    <td>2018 Madrid</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2021 Dubai</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2016 Linz</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <th colspan="2">European Championships</th>
  </tr>
  <tr>
    <td>2015 Istanbul</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2016 Montpellier</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2017 Kocaeli</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2018 Novi Sad</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2019 Guadalajara</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2021 Poreč</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2022 Gaziantep</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <th colspan="2">European Games</th>
  </tr>
  <tr>
    <td>2015 Baku</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2019 Minsk</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <th colspan="2">World Beach Games</th>
  </tr>
  <tr>
    <td>2019 Doha</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <th colspan="2">World Games</th>
  </tr>
  <tr>
    <td>2022 Birmingham</td>
    <td>Individual kata</td>
  </tr>
  <tr>
    <td>2017 Wroclaw</td>
    <td>Individual kata</td>
  </tr>
</table>

Figure 4: Sandra Sánchez’s Medal Infobox
• Missed Medal in 2018: While the LLM identifies the 2016 Bronze medal, it fails to recognize the 2018 Gold medal in Madrid, which also occurred before 2021.

• Incomplete Temporal Analysis: The LLM does not account for all relevant years when performing temporal reasoning, leading to an undercount of medals.

Symbolic Reasoning Accuracy:

• SQL explicitly filters medals in the World Championships where the year is less than 2021.

• The query correctly identifies both the 2016 Bronze medal and the 2018 Gold medal, producing the accurate total of 2 medals.

• Symbolic reasoning eliminates human oversight by systematically querying all relevant data within the temporal range.

Conclusion: The LLM’s Chain-of-Thought reasoning incorrectly counts only 1 medal due to missed temporal filtering. Symbolic SQL reasoning, by explicitly querying for medals before 2021, produces the correct result: 2 medals.

10.2 Result Analysis for all models:
10.2.1 Analysis for GPT-4o:
From Table 8, comparing SQL Adaptive with Direct Adaptive across key aspects, we observe:

• Counterfactual Gap: For Table - Adaptive (EMS), the gap between Original (58.13) and CounterFact (40.92) is 17.21. For SQL Schema - Adaptive (EMS), the gap reduces to 2.96, indicating improved robustness to counterfactual data.

• Scalability to Table Size: For Table - Adaptive (EMS), the gap between Large (48.88) and Small (73.92) tables is 25.04. For SQL Schema - Adaptive (EMS), the gap decreases significantly to 1.06, demonstrating better scalability to large table sizes.

• Question Complexity: For Table - Adaptive (EMS), the performance on Easy, Medium, and Hard questions is 74.38, 63.91, and 54.17, respectively. For SQL Schema - Adaptive (EMS), the performance improves to 80.06 (Easy), 73.37 (Medium), and 66.74 (Hard), showing better handling of increasing question complexity.

• Adaptive Few-Shot Effectiveness: For Original data, SQL Schema - Adaptive (EMS: 71.63) outperforms Table - Adaptive (EMS: 58.13), highlighting the benefit of adaptive prompting in achieving higher accuracy.

These results demonstrate that SQL Adaptive consistently outperforms Direct Adaptive by reducing the counterfactual gap, improving scalability to large tables, and enhancing performance across question complexities.

10.2.2 Analysis for GPT-4o Mini:
From Table 9, comparing SQL Adaptive with Table Adaptive across key aspects, we observe:

• Counterfactual Gap: For Table - Adaptive (EMS), the gap between Original (48.79) and CounterFact (35.48) is 13.31. For SQL Schema - Adaptive (EMS), the gap reduces significantly to 3.45 (Original: 68.69, CounterFact: 65.24), indicating improved robustness to counterfactual data.

• Scalability to Table Size: For Table - Adaptive (EMS), the gap between Large (39.96) and Small (64.80) tables is 24.84. For SQL Schema - Adaptive (EMS), the gap reduces to 3.34 (Large: 65.43, Small: 68.77), demonstrating better scalability to large table sizes.

• Question Complexity: For Table - Adaptive (EMS), the scores for Easy, Medium, and Hard questions are 63.16, 52.07, and 44.49, respectively. For SQL Schema - Adaptive (EMS), the performance improves to 76.87 (Easy), 70.41 (Medium), and 63.13 (Hard), showcasing better handling of increasing question complexity.

• Adaptive Few-Shot Effectiveness: For Original data, SQL Schema - Adaptive (EMS: 68.69) significantly outperforms Table - Adaptive (EMS: 48.79), highlighting the superior accuracy achieved with symbolic reasoning and adaptive prompting.

These results clearly show that SQL Adaptive consistently outperforms Table Adaptive, with smaller counterfactual and table size gaps, and better performance across question complexity levels.
10.2.3 Analysis for Gemini 1.5 Flash:
From Table 10, comparing SQL Adaptive with Table Adaptive, we observe:

• Counterfactual Gap: For Table - Adaptive (EMS), the gap between Original (52.90) and CounterFact (42.91) is 9.99. In comparison, for SQL Schema - Adaptive (EMS), the gap is reduced to 2.78 (Original: 65.49, CounterFact: 62.71), indicating significantly improved robustness to counterfactual data.

• Scalability to Table Size: For Table - Adaptive (EMS), the gap between Large (41.02) and Small (66.25) tables is 25.23. For SQL Schema - Adaptive (EMS), the gap reduces to 4.23 (Large: 69.30, Small: 73.53), showcasing SQL’s superior handling of larger tables.

• Question Complexity: For Table - Adaptive (EMS), the scores for Easy, Medium, and Hard questions are 65.76, 55.95, and 45.92, respectively. For SQL Schema - Adaptive (EMS), the scores improve to 76.26 (Easy), 73.72 (Medium), and 63.12 (Hard), highlighting better performance as question complexity increases.

• Adaptive Few-Shot Effectiveness: For Original data, SQL Schema - Adaptive (EMS: 65.49) outperforms Table - Adaptive (EMS: 52.90), demonstrating the effectiveness of adaptive few-shot prompting with symbolic reasoning.

These observations show that SQL Adaptive significantly reduces the counterfactual gap, scales better with large tables, and consistently achieves higher accuracy across all question complexities compared to Table Adaptive.

10.2.4 Analysis for Gemini 1.5 Pro:
From Table 11, comparing SQL Adaptive with Table Adaptive, we observe:

• Counterfactual Gap: For Table - Adaptive (EMS), the gap between Original (53.48) and CounterFact (44.19) is 9.29. For SQL Schema - Adaptive (EMS), the gap is reduced to 0.16 (Original: 65.29, CounterFact: 65.13), showcasing excellent robustness to counterfactual data.

• Scalability to Table Size: For Table - Adaptive (EMS), the gap between Large (41.86) and Small (67.27) tables is 25.41. For SQL Schema - Adaptive (EMS), the gap reduces significantly to 2.88 (Large: 72.43, Small: 75.31), demonstrating better scalability with large tables.

• Question Complexity: For Table - Adaptive (EMS), the scores for Easy, Medium, and Hard questions are 66.26, 56.47, and 46.74, respectively. For SQL Schema - Adaptive (EMS), the scores improve to 75.86 (Easy), 71.47 (Medium), and 59.24 (Hard), highlighting superior handling of increasing question complexity.

• Adaptive Few-Shot Effectiveness: For Original data, SQL Schema - Adaptive (EMS: 65.29) significantly outperforms Table - Adaptive (EMS: 53.48), demonstrating the clear benefits of symbolic reasoning combined with adaptive few-shot prompting.

These results clearly highlight that SQL Adaptive consistently reduces counterfactual gaps, scales better with table size, and improves performance across question complexities compared to Table Adaptive.

10.2.5 Analysis for Llama 3.1 70B:
From Table 12, comparing SQL Adaptive with Table Adaptive, we observe:

• Counterfactual Gap: For Table - Adaptive (EMS), the gap between Original (53.63) and CounterFact (39.20) is 14.43. For SQL Schema - Adaptive (EMS), the gap reduces to 0.55 (Original: 64.36, CounterFact: 63.81). SQL Adaptive is clearly more robust to counterfactual data.

• Scalability to Table Size: For Table - Adaptive (EMS), the gap between Large (46.65) and Small (68.07) tables is 21.42. For SQL Schema - Adaptive (EMS), the gap remains smaller at 1.98 (Large: 66.91, Small: 68.89), showing better performance scalability with table size.

• Question Complexity: For Table - Adaptive (EMS), the scores for Easy, Medium, and Hard questions are 69.40, 59.76, and 52.85, respectively. For SQL Schema - Adaptive
(EMS), the scores are **77.19** (Easy), **67.65** (Medium), and **60.92** (Hard), showing a consistent improvement across complexities.

• **Adaptive Few-Shot Effectiveness**: For Original data, **SQL Schema - Adaptive (EMS: 64.36)** performs significantly better than **Table - Adaptive (EMS: 53.63)**, confirming the benefit of symbolic reasoning with adaptive few-shot prompting.

Overall, **SQL Adaptive** demonstrates clear improvements over **Table Adaptive** in counterfactual robustness, scalability to table size, and performance across question complexity levels. The observed gaps in Table Adaptive remain substantial, especially for counterfactual and large table scenarios.

**10.2.6 Analysis for Mixtral 8x7B:**
From Table 13, comparing **SQL** and **Table** under both *Adaptive* and *Static* few-shot settings, we observe:

• **Counterfactual Gap**: For **Table – Adaptive** (EMS), the gap between Original (**37.54**) and CounterFact (**30.62**) is **6.92**. For **SQL Schema – Adaptive** (EMS), the gap is **4.20** (Original: **25.09**, CounterFact: **20.89**). For **Table Text – Static** (EMS), the gap is **8.59** (Original: **48.79**, CounterFact: **40.20**). For **SQL Schema – Static** (EMS), the gap is **5.82** (Original: **46.02**, CounterFact: **40.20**). SQL settings therefore show smaller counterfactual gaps than their Table counterparts in both few-shot modes.

• **Scalability to Table Size**: For **Table – Adaptive** (EMS), the gap between Large (**34.94**) and Small (**47.72**) tables is **12.78**. For **SQL Schema – Adaptive** (EMS), the gap is **9.03** (Large: **24.54**, Small: **33.57**). For **Table Text – Static** (EMS), the gap is **24.14** (Large: **40.89**, Small: **65.03**). For **SQL Schema – Static** (EMS), the gap is **5.02** (Large: **47.96**, Small: **52.98**). SQL configurations exhibit smaller size-related drops than Table configurations, particularly in the Static setting.

• **Question Complexity**: For **Table – Adaptive** (EMS), the scores are **50.96** (Easy), **38.46** (Medium), and **35.74** (Hard). For **SQL Schema – Adaptive** (EMS), the scores are **26.78** (Easy), **46.55** (Medium), and **21.56** (Hard). For **Table Text – Static** (EMS), the scores are **62.30** (Easy), **53.25** (Medium), and **45.62** (Hard). For **SQL Schema – Static** (EMS), the scores are **67.35** (Easy), **45.36** (Medium), and **40.61** (Hard). SQL Adaptive surpasses Table Adaptive on *Medium* questions only; SQL Static exceeds Table Text Static on *Easy* questions but trails on Medium and Hard ones.

• **Few-Shot Results on Original Subset**: **Table – Adaptive** achieves **37.54** EMS, while **SQL Schema – Adaptive** reaches **25.09**. **Table Text – Static** records **48.79** EMS, and **SQL Schema – Static** attains **46.02**. Table maintains higher absolute EMS in both few-shot modes, though the SQL–Table gap is narrower in the Static setting.

**10.2.7 Analysis for SQL Coder 70B:**
Since this is a code-based model, we only evaluate baselines related to code generation and exclude text generation baselines. From Table 14, we observe the following for **SQL Schema**:

• **Counterfactual Gap**: For **SQL Static** (EMS), the gap between Original (**52.08**) and CounterFact (**47.93**) is **4.15**. For **SQL Adaptive** (EMS), the gap reduces to **2.38** (Original: **55.88**, CounterFact: **53.50**), demonstrating improved robustness with adaptive few-shot prompting.

• **Scalability to Table Size**: For **SQL Static** (EMS), the gap between Large (**62.28**) and Small (**59.53**) tables is **2.75**. For **SQL Adaptive** (EMS), the gap is slightly larger at **4.22** (Large: **63.17**, Small: **58.95**), showing minor regression in scalability.

• **Question Complexity**: For **SQL Static** (EMS), the scores for Easy, Medium, and Hard questions are **77.39**, **51.19**, and **28.91**, respectively. For **SQL Adaptive** (EMS), the scores improve for Medium (**58.38**) and Hard (**51.74**) questions but decrease for Easy (**63.93**), indicating uneven performance gains.

• **Overall Accuracy**: For Original data, **SQL Adaptive (EMS: 55.88)** outperforms **SQL Static (EMS: 52.08)**, highlighting the effectiveness of adaptive few-shot prompting for code-specific tasks.
Overall, SQL Adaptive demonstrates improved robustness and accuracy compared to SQL Static, particularly on counterfactual and medium-complexity queries.

10.2.8 Analysis for Code Llama 70B:
Since this is a code-based model, we only evaluate baselines related to code generation and exclude text generation baselines. From Table 15, we observe the following for SQL Schema:

• Counterfactual Gap: For SQL Static (EMS), the gap between Original (15.84) and CounterFact (32.62) is substantial at 16.78, indicating performance degradation. For SQL Adaptive (EMS), the gap reduces to 16.53 (Original: 23.53, CounterFact: 40.06). While there is slight improvement, the gap remains significant.

• Scalability to Table Size: For SQL Static (EMS), the gap between Large (29.82) and Small (41.64) tables is 11.82. For SQL Adaptive (EMS), the gap decreases to 10.53 (Large: 37.61, Small: 48.14), indicating modest improvements in handling table size.

• Question Complexity: For SQL Static (EMS), the scores for Easy, Medium, and Hard questions are 53.42, 41.62, and 38.94, respectively. For SQL Adaptive (EMS), the scores improve across all complexities to 65.16 (Easy), 50.89 (Medium), and 40.61 (Hard), showing clear improvements, particularly for Easy and Medium questions.

• Overall Accuracy: For Original data, SQL Adaptive (EMS: 23.53) outperforms SQL Static (EMS: 15.84), demonstrating the benefits of adaptive few-shot prompting for overall accuracy.

Overall, SQL Adaptive shows moderate improvements over SQL Static, particularly in handling table size and question complexities, though counterfactual robustness remains a challenge.
<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Metric</th>
    <th colspan="3">Gap (Original - CounterFact)</th>
    <th colspan="4">Gap (Small - Large)</th>
  </tr>
  <tr>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="15">Text Table</td>
    <td rowspan="15">None</td>
    <td>-</td>
    <td>REMS</td>
    <td>25.91</td>
    <td>13.45</td>
    <td>12.46</td>
    <td>23.84</td>
    <td>25.79</td>
    <td>1.95</td>
    <td>28.67</td>
    <td>25.55</td>
    <td>23.83</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>25.09</td>
    <td>12.16</td>
    <td>12.93</td>
    <td>23.05</td>
    <td>24.68</td>
    <td>1.63</td>
    <td>27.26</td>
    <td>24.06</td>
    <td>22.35</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>55.94</td>
    <td>42.26</td>
    <td>13.68</td>
    <td>45.24</td>
    <td>67.81</td>
    <td>22.57</td>
    <td>72.34</td>
    <td>59.43</td>
    <td>52.90</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>53.46</td>
    <td>40.06</td>
    <td>13.40</td>
    <td>43.31</td>
    <td>64.56</td>
    <td>21.25</td>
    <td>69.86</td>
    <td>57.20</td>
    <td>49.02</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>58.69</td>
    <td>44.69</td>
    <td>14.00</td>
    <td>48.96</td>
    <td>73.57</td>
    <td>24.61</td>
    <td>73.45</td>
    <td>65.39</td>
    <td>56.97</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>56.57</td>
    <td>42.35</td>
    <td>14.22</td>
    <td>46.84</td>
    <td>71.11</td>
    <td>24.27</td>
    <td>71.18</td>
    <td>63.12</td>
    <td>53.35</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>60.71</td>
    <td>43.10</td>
    <td>17.61</td>
    <td>51.41</td>
    <td>76.97</td>
    <td>25.56</td>
    <td>76.89</td>
    <td>66.00</td>
    <td>58.00</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>58.13</td>
    <td>40.92</td>
    <td>17.21</td>
    <td>48.88</td>
    <td>73.92</td>
    <td>25.04</td>
    <td>74.38</td>
    <td>63.91</td>
    <td>54.17</td>
  </tr>
  <tr>
    <td>Clear</td>
    <td>REMS</td>
    <td>66.84</td>
    <td>50.17</td>
    <td>16.67</td>
    <td>55.54</td>
    <td>77.86</td>
    <td>22.32</td>
    <td>78.97</td>
    <td>72.70</td>
    <td>64.72</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>65.57</td>
    <td>48.21</td>
    <td>17.36</td>
    <td>53.53</td>
    <td>76.49</td>
    <td>22.96</td>
    <td>76.40</td>
    <td>71.99</td>
    <td>62.62</td>
  </tr>
  <tr>
    <td>Chain Of Thought</td>
    <td>REMS</td>
    <td>70.96</td>
    <td>50.43</td>
    <td>20.53</td>
    <td>58.36</td>
    <td>76.02</td>
    <td>17.66</td>
    <td>81.02</td>
    <td>75.52</td>
    <td>66.51</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>69.90</td>
    <td>48.50</td>
    <td>21.40</td>
    <td>56.13</td>
    <td>74.15</td>
    <td>18.02</td>
    <td>78.43</td>
    <td>75.35</td>
    <td>64.06</td>
  </tr>
  <tr>
    <td>Plan And Solve</td>
    <td>REMS</td>
    <td>70.36</td>
    <td>48.62</td>
    <td>21.74</td>
    <td>57.14</td>
    <td>76.91</td>
    <td>19.77</td>
    <td>80.30</td>
    <td>72.30</td>
    <td>62.64</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>69.55</td>
    <td>46.50</td>
    <td>23.05</td>
    <td>55.20</td>
    <td>75.44</td>
    <td>20.24</td>
    <td>77.96</td>
    <td>70.81</td>
    <td>60.25</td>
  </tr>
  <tr>
    <td>Program of Thought</td>
    <td>REMS</td>
    <td>59.05</td>
    <td>50.28</td>
    <td>8.77</td>
    <td>51.66</td>
    <td>64.35</td>
    <td>12.69</td>
    <td>72.65</td>
    <td>60.85</td>
    <td>52.89</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>56.40</td>
    <td>47.07</td>
    <td>9.33</td>
    <td>50.19</td>
    <td>62.22</td>
    <td>12.03</td>
    <td>69.94</td>
    <td>57.20</td>
    <td>48.81</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>REMS</td>
    <td>60.02</td>
    <td>50.90</td>
    <td>9.12</td>
    <td>51.76</td>
    <td>64.85</td>
    <td>13.09</td>
    <td>72.02</td>
    <td>59.78</td>
    <td>51.25</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>57.44</td>
    <td>47.78</td>
    <td>9.66</td>
    <td>50.19</td>
    <td>62.69</td>
    <td>12.50</td>
    <td>69.24</td>
    <td>56.02</td>
    <td>47.17</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>51.44</td>
    <td>48.94</td>
    <td>2.50</td>
    <td>55.81</td>
    <td>59.92</td>
    <td>4.11</td>
    <td>63.53</td>
    <td>65.20</td>
    <td>45.16</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>49.31</td>
    <td>46.64</td>
    <td>2.67</td>
    <td>53.90</td>
    <td>57.43</td>
    <td>3.53</td>
    <td>61.22</td>
    <td>63.12</td>
    <td>42.02</td>
  </tr>
  <tr>
    <td>SQL Schema</td>
    <td>Static</td>
    <td>REMS</td>
    <td>66.76</td>
    <td>62.41</td>
    <td>4.35</td>
    <td>71.82</td>
    <td>75.37</td>
    <td>3.55</td>
    <td>81.27</td>
    <td>75.88</td>
    <td>64.38</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>65.22</td>
    <td>60.94</td>
    <td>4.28</td>
    <td>70.63</td>
    <td>73.57</td>
    <td>2.94</td>
    <td>78.89</td>
    <td>75.15</td>
    <td>62.31</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>71.87</td>
    <td>69.04</td>
    <td>2.83</td>
    <td>73.05</td>
    <td>74.55</td>
    <td>1.50</td>
    <td>80.17</td>
    <td>74.35</td>
    <td>67.38</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>71.63</td>
    <td>68.67</td>
    <td>2.96</td>
    <td>72.86</td>
    <td>73.92</td>
    <td>1.06</td>
    <td>80.06</td>
    <td>73.37</td>
    <td>66.74</td>
  </tr>
</table>

Table 8: Test Set Results for GPT-4o.

<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Few Shots</th>
    <th rowspan="2">Metric</th>
    <th colspan="6">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="15">Text Table</td>
    <td rowspan="15">None</td>
    <td>-</td>
    <td>REMS</td>
    <td>21.48</td>
    <td>13.90</td>
    <td>20.21</td>
    <td>22.75</td>
    <td>26.08</td>
    <td>22.30</td>
    <td>19.03</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>20.24</td>
    <td>13.02</td>
    <td>19.70</td>
    <td>21.87</td>
    <td>24.84</td>
    <td>20.51</td>
    <td>17.40</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>49.59</td>
    <td>33.52</td>
    <td>40.18</td>
    <td>63.60</td>
    <td>62.07</td>
    <td>47.53</td>
    <td>43.62</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>47.23</td>
    <td>31.04</td>
    <td>37.73</td>
    <td>60.47</td>
    <td>59.19</td>
    <td>44.58</td>
    <td>39.55</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>49.94</td>
    <td>36.94</td>
    <td>41.33</td>
    <td>66.45</td>
    <td>64.23</td>
    <td>51.13</td>
    <td>47.13</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>47.58</td>
    <td>34.91</td>
    <td>39.03</td>
    <td>63.63</td>
    <td>61.84</td>
    <td>48.32</td>
    <td>43.36</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>51.13</td>
    <td>38.37</td>
    <td>42.43</td>
    <td>67.66</td>
    <td>66.10</td>
    <td>54.81</td>
    <td>48.63</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>48.79</td>
    <td>35.48</td>
    <td>39.96</td>
    <td>64.80</td>
    <td>63.16</td>
    <td>52.07</td>
    <td>44.49</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>39.93</td>
    <td>41.28</td>
    <td>38.76</td>
    <td>48.50</td>
    <td>57.45</td>
    <td>50.88</td>
    <td>34.21</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>38.24</td>
    <td>39.63</td>
    <td>37.36</td>
    <td>46.67</td>
    <td>55.30</td>
    <td>49.51</td>
    <td>31.41</td>
  </tr>
  <tr>
    <td>SQL Schema</td>
    <td>Static</td>
    <td>REMS</td>
    <td>57.44</td>
    <td>51.18</td>
    <td>53.94</td>
    <td>66.52</td>
    <td>77.57</td>
    <td>65.29</td>
    <td>57.04</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>56.57</td>
    <td>50.36</td>
    <td>53.16</td>
    <td>65.73</td>
    <td>75.93</td>
    <td>65.29</td>
    <td>56.33</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>68.97</td>
    <td>65.40</td>
    <td>65.70</td>
    <td>69.20</td>
    <td>76.98</td>
    <td>71.07</td>
    <td>63.93</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>68.69</td>
    <td>65.24</td>
    <td>65.43</td>
    <td>68.77</td>
    <td>76.87</td>
    <td>70.41</td>
    <td>63.13</td>
  </tr>
</table>

Table 9: Test Set Results for GPT-4o-mini
<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Metric</th>
    <th colspan="6">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Gap (Original - CounterFact)</th>
    <th>Large</th>
    <th>Small</th>
    <th>Gap (Small - Large)</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="13">Text</td>
    <td rowspan="13">Table</td>
    <td rowspan="2">None</td>
    <td>REMS</td>
    <td>22.11</td>
    <td>15.02</td>
    <td>7.09</td>
    <td>23.88</td>
    <td>21.74</td>
    <td>-2.14</td>
    <td>27.81</td>
    <td>23.23</td>
    <td>21.59</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>18.69</td>
    <td>11.76</td>
    <td>6.93</td>
    <td>19.49</td>
    <td>18.93</td>
    <td>-0.56</td>
    <td>24.03</td>
    <td>19.41</td>
    <td>17.30</td>
  </tr>
  <tr>
    <td rowspan="2">zero shots</td>
    <td>REMS</td>
    <td>55.17</td>
    <td>45.42</td>
    <td>9.75</td>
    <td>42.71</td>
    <td>69.88</td>
    <td>27.17</td>
    <td>68.78</td>
    <td>56.61</td>
    <td>51.18</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>48.79</td>
    <td>37.61</td>
    <td>11.18</td>
    <td>34.96</td>
    <td>61.11</td>
    <td>26.15</td>
    <td>59.15</td>
    <td>48.82</td>
    <td>39.00</td>
  </tr>
  <tr>
    <td rowspan="2">Static</td>
    <td>REMS</td>
    <td>57.46</td>
    <td>47.20</td>
    <td>10.26</td>
    <td>46.44</td>
    <td>71.89</td>
    <td>25.45</td>
    <td>73.76</td>
    <td>61.94</td>
    <td>54.99</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>50.00</td>
    <td>39.08</td>
    <td>10.92</td>
    <td>38.35</td>
    <td>62.35</td>
    <td>24.00</td>
    <td>63.16</td>
    <td>53.53</td>
    <td>42.96</td>
  </tr>
  <tr>
    <td rowspan="2">Adaptive</td>
    <td>REMS</td>
    <td>59.09</td>
    <td>48.25</td>
    <td>10.84</td>
    <td>47.93</td>
    <td>73.17</td>
    <td>25.24</td>
    <td>76.28</td>
    <td>64.94</td>
    <td>57.83</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>52.90</td>
    <td>42.91</td>
    <td>9.99</td>
    <td>41.02</td>
    <td>66.25</td>
    <td>25.23</td>
    <td>65.76</td>
    <td>55.95</td>
    <td>45.92</td>
  </tr>
  <tr>
    <td rowspan="2">Clear</td>
    <td>REMS</td>
    <td>62.01</td>
    <td>50.46</td>
    <td>11.55</td>
    <td>53.38</td>
    <td>73.07</td>
    <td>19.69</td>
    <td>79.65</td>
    <td>68.02</td>
    <td>59.37</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>60.63</td>
    <td>48.95</td>
    <td>11.68</td>
    <td>51.11</td>
    <td>72.19</td>
    <td>21.08</td>
    <td>77.92</td>
    <td>66.52</td>
    <td>56.67</td>
  </tr>
  <tr>
    <td rowspan="2">Chain Of Thought</td>
    <td>REMS</td>
    <td>60.97</td>
    <td>47.28</td>
    <td>13.69</td>
    <td>51.61</td>
    <td>73.45</td>
    <td>21.84</td>
    <td>77.28</td>
    <td>65.60</td>
    <td>56.62</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>59.54</td>
    <td>45.48</td>
    <td>14.06</td>
    <td>49.52</td>
    <td>72.09</td>
    <td>22.57</td>
    <td>75.29</td>
    <td>64.17</td>
    <td>54.55</td>
  </tr>
  <tr>
    <td rowspan="2">Plan And Solve</td>
    <td>REMS</td>
    <td>53.58</td>
    <td>42.58</td>
    <td>11.00</td>
    <td>48.40</td>
    <td>65.07</td>
    <td>16.67</td>
    <td>70.61</td>
    <td>59.35</td>
    <td>51.43</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>52.00</td>
    <td>40.34</td>
    <td>11.66</td>
    <td>46.14</td>
    <td>63.85</td>
    <td>17.71</td>
    <td>68.54</td>
    <td>57.12</td>
    <td>48.53</td>
  </tr>
  <tr>
    <td rowspan="2">Program of Thought</td>
    <td>REMS</td>
    <td>49.26</td>
    <td>42.85</td>
    <td>6.41</td>
    <td>44.31</td>
    <td>57.29</td>
    <td>12.98</td>
    <td>67.10</td>
    <td>48.61</td>
    <td>44.13</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>46.90</td>
    <td>40.55</td>
    <td>6.35</td>
    <td>42.33</td>
    <td>55.30</td>
    <td>12.97</td>
    <td>64.53</td>
    <td>45.52</td>
    <td>40.47</td>
  </tr>
  <tr>
    <td rowspan="2">Faithful CoT</td>
    <td>REMS</td>
    <td>48.98</td>
    <td>40.90</td>
    <td>8.08</td>
    <td>43.46</td>
    <td>58.09</td>
    <td>14.63</td>
    <td>64.12</td>
    <td>53.26</td>
    <td>45.68</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>46.78</td>
    <td>38.66</td>
    <td>8.12</td>
    <td>41.38</td>
    <td>56.23</td>
    <td>14.85</td>
    <td>61.50</td>
    <td>50.66</td>
    <td>42.67</td>
  </tr>
  <tr>
    <td rowspan="2">zero shots</td>
    <td>REMS</td>
    <td>47.27</td>
    <td>42.56</td>
    <td>4.71</td>
    <td>46.23</td>
    <td>56.45</td>
    <td>10.22</td>
    <td>63.49</td>
    <td>59.74</td>
    <td>42.09</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>39.34</td>
    <td>33.45</td>
    <td>5.89</td>
    <td>42.29</td>
    <td>53.65</td>
    <td>9.36</td>
    <td>52.96</td>
    <td>52.48</td>
    <td>34.59</td>
  </tr>
  <tr>
    <td rowspan="3">SQL</td>
    <td rowspan="3">Schema</td>
    <td rowspan="2">Static</td>
    <td>REMS</td>
    <td>66.43</td>
    <td>62.38</td>
    <td>4.05</td>
    <td>71.39</td>
    <td>77.20</td>
    <td>5.81</td>
    <td>88.17</td>
    <td>79.22</td>
    <td>65.04</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>57.93</td>
    <td>54.43</td>
    <td>3.50</td>
    <td>63.13</td>
    <td>70.89</td>
    <td>7.76</td>
    <td>79.54</td>
    <td>68.96</td>
    <td>58.42</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>72.91</td>
    <td>71.67</td>
    <td>1.24</td>
    <td>78.18</td>
    <td>81.71</td>
    <td>3.53</td>
    <td>87.06</td>
    <td>80.80</td>
    <td>74.04</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>65.49</td>
    <td>62.71</td>
    <td>2.78</td>
    <td>69.30</td>
    <td>73.53</td>
    <td>4.23</td>
    <td>76.26</td>
    <td>73.72</td>
    <td>63.12</td>
  </tr>
</table>

Table 10: Test Set Results for Gemini 1.5 Flash.

<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Metric</th>
    <th colspan="4">Gap (Original - CounterFact)</th>
    <th colspan="4">Gap (Small - Large)</th>
  </tr>
  <tr>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="13">Text</td>
    <td rowspan="13">Table</td>
    <td rowspan="2">None</td>
    <td>REMS</td>
    <td>23.37</td>
    <td>15.88</td>
    <td>7.49</td>
    <td>24.20</td>
    <td>24.08</td>
    <td>0.12</td>
    <td>29.15</td>
    <td>26.45</td>
    <td>23.38</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>19.90</td>
    <td>13.45</td>
    <td>6.45</td>
    <td>21.19</td>
    <td>20.99</td>
    <td>0.2</td>
    <td>24.94</td>
    <td>21.76</td>
    <td>18.77</td>
  </tr>
  <tr>
    <td rowspan="2">zero shots</td>
    <td>REMS</td>
    <td>54.94</td>
    <td>45.40</td>
    <td>9.54</td>
    <td>46.34</td>
    <td>69.74</td>
    <td>23.4</td>
    <td>75.72</td>
    <td>65.10</td>
    <td>57.43</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>48.06</td>
    <td>37.39</td>
    <td>10.67</td>
    <td>38.98</td>
    <td>61.11</td>
    <td>21.13</td>
    <td>62.59</td>
    <td>53.24</td>
    <td>45.60</td>
  </tr>
  <tr>
    <td rowspan="2">Static</td>
    <td>REMS</td>
    <td>59.01</td>
    <td>46.87</td>
    <td>12.14</td>
    <td>52.42</td>
    <td>72.93</td>
    <td>20.51</td>
    <td>75.72</td>
    <td>65.10</td>
    <td>60.28</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>52.91</td>
    <td>39.92</td>
    <td>12.99</td>
    <td>43.86</td>
    <td>65.02</td>
    <td>21.16</td>
    <td>65.79</td>
    <td>58.53</td>
    <td>50.00</td>
  </tr>
  <tr>
    <td rowspan="2">Adaptive</td>
    <td>REMS</td>
    <td>60.23</td>
    <td>49.04</td>
    <td>11.19</td>
    <td>48.79</td>
    <td>73.98</td>
    <td>25.19</td>
    <td>78.04</td>
    <td>66.43</td>
    <td>59.28</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>53.48</td>
    <td>44.19</td>
    <td>9.29</td>
    <td>41.86</td>
    <td>67.27</td>
    <td>25.41</td>
    <td>66.26</td>
    <td>56.47</td>
    <td>46.74</td>
  </tr>
  <tr>
    <td rowspan="2">Clear</td>
    <td>REMS</td>
    <td>50.29</td>
    <td>42.82</td>
    <td>7.47</td>
    <td>42.83</td>
    <td>58.04</td>
    <td>15.21</td>
    <td>59.91</td>
    <td>60.79</td>
    <td>52.82</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>49.33</td>
    <td>40.66</td>
    <td>8.67</td>
    <td>40.95</td>
    <td>56.85</td>
    <td>15.90</td>
    <td>58.38</td>
    <td>59.77</td>
    <td>51.14</td>
  </tr>
  <tr>
    <td rowspan="2">Chain Of Thought</td>
    <td>REMS</td>
    <td>67.34</td>
    <td>57.16</td>
    <td>10.18</td>
    <td>59.51</td>
    <td>78.47</td>
    <td>18.96</td>
    <td>84.67</td>
    <td>73.36</td>
    <td>67.31</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>66.46</td>
    <td>55.75</td>
    <td>10.71</td>
    <td>57.67</td>
    <td>77.55</td>
    <td>19.88</td>
    <td>83.29</td>
    <td>72.69</td>
    <td>65.87</td>
  </tr>
  <tr>
    <td rowspan="2">Plan And Solve</td>
    <td>REMS</td>
    <td>62.11</td>
    <td>55.68</td>
    <td>6.43</td>
    <td>58.38</td>
    <td>74.88</td>
    <td>16.50</td>
    <td>81.19</td>
    <td>73.28</td>
    <td>65.56</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>60.75</td>
    <td>54.31</td>
    <td>6.44</td>
    <td>56.72</td>
    <td>73.43</td>
    <td>16.71</td>
    <td>79.42</td>
    <td>72.25</td>
    <td>63.60</td>
  </tr>
  <tr>
    <td rowspan="2">Program of Thought</td>
    <td>REMS</td>
    <td>56.31</td>
    <td>49.35</td>
    <td>6.97</td>
    <td>53.02</td>
    <td>66.46</td>
    <td>13.44</td>
    <td>76.94</td>
    <td>60.62</td>
    <td>51.75</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>53.83</td>
    <td>46.20</td>
    <td>7.63</td>
    <td>49.84</td>
    <td>64.68</td>
    <td>14.83</td>
    <td>73.97</td>
    <td>57.12</td>
    <td>47.57</td>
  </tr>
  <tr>
    <td rowspan="2">Faithful CoT</td>
    <td>REMS</td>
    <td>56.49</td>
    <td>50.15</td>
    <td>6.34</td>
    <td>50.63</td>
    <td>65.84</td>
    <td>15.21</td>
    <td>77.28</td>
    <td>61.53</td>
    <td>51.03</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>53.95</td>
    <td>47.64</td>
    <td>6.31</td>
    <td>48.47</td>
    <td>63.85</td>
    <td>15.39</td>
    <td>74.32</td>
    <td>58.15</td>
    <td>46.85</td>
  </tr>
  <tr>
    <td rowspan="2">zero shots</td>
    <td>REMS</td>
    <td>49.24</td>
    <td>43.56</td>
    <td>5.68</td>
    <td>48.21</td>
    <td>57.62</td>
    <td>9.41</td>
    <td>64.42</td>
    <td>61.10</td>
    <td>43.00</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>41.32</td>
    <td>35.31</td>
    <td>6.01</td>
    <td>45.87</td>
    <td>55.11</td>
    <td>9.24</td>
    <td>54.78</td>
    <td>53.79</td>
    <td>35.96</td>
  </tr>
  <tr>
    <td rowspan="3">SQL</td>
    <td rowspan="3">Schema</td>
    <td rowspan="2">Static</td>
    <td>REMS</td>
    <td>67.76</td>
    <td>63.63</td>
    <td>4.13</td>
    <td>76.80</td>
    <td>82.82</td>
    <td>6.02</td>
    <td>89.33</td>
    <td>80.81</td>
    <td>66.42</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>59.08</td>
    <td>55.52</td>
    <td>3.56</td>
    <td>71.32</td>
    <td>77.41</td>
    <td>6.09</td>
    <td>80.86</td>
    <td>70.33</td>
    <td>59.59</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>73.04</td>
    <td>73.58</td>
    <td>0.54</td>
    <td>81.94</td>
    <td>84.38</td>
    <td>2.44</td>
    <td>87.31</td>
    <td>80.91</td>
    <td>71.43</td>
  </tr>
  <tr>
    <td>EMS</td>
    <td>65.29</td>
    <td>65.13</td>
    <td>0.16</td>
    <td>72.43</td>
    <td>75.31</td>
    <td>2.88</td>
    <td>75.86</td>
    <td>71.47</td>
    <td>59.24</td>
  </tr>
</table>

Table 11: Test Set Results for Gemini 1.5 Pro.
<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Method</th>
    <th rowspan="2">Metric</th>
    <th colspan="7">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Gap (Original - CounterFact)</th>
    <th>Large</th>
    <th>Small</th>
    <th>Gap (Small - Large)</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="6">Text</td>
    <td rowspan="6">Table</td>
    <td>None</td>
    <td>REMS</td>
    <td>17.46</td>
    <td>12.74</td>
    <td>4.72</td>
    <td>20.89</td>
    <td>18.09</td>
    <td>-2.80</td>
    <td>22.31</td>
    <td>17.93</td>
    <td>17.18</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>16.61</td>
    <td>12.16</td>
    <td>4.45</td>
    <td>19.89</td>
    <td>17.31</td>
    <td>-2.58</td>
    <td>21.31</td>
    <td>16.96</td>
    <td>16.27</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>54.63</td>
    <td>39.93</td>
    <td>14.70</td>
    <td>46.10</td>
    <td>64.62</td>
    <td>18.52</td>
    <td>70.10</td>
    <td>58.31</td>
    <td>53.37</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>52.60</td>
    <td>37.48</td>
    <td>15.12</td>
    <td>44.05</td>
    <td>62.57</td>
    <td>18.52</td>
    <td>67.49</td>
    <td>56.21</td>
    <td>50.35</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>64.33</td>
    <td>48.44</td>
    <td>15.89</td>
    <td>57.27</td>
    <td>75.40</td>
    <td>18.13</td>
    <td>79.04</td>
    <td>70.23</td>
    <td>60.98</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>62.46</td>
    <td>46.21</td>
    <td>16.25</td>
    <td>55.58</td>
    <td>73.68</td>
    <td>18.10</td>
    <td>76.91</td>
    <td>67.46</td>
    <td>57.44</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>55.73</td>
    <td>41.29</td>
    <td>14.44</td>
    <td>48.48</td>
    <td>70.56</td>
    <td>22.08</td>
    <td>71.35</td>
    <td>62.54</td>
    <td>56.43</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>53.63</td>
    <td>39.20</td>
    <td>14.43</td>
    <td>46.65</td>
    <td>68.07</td>
    <td>21.42</td>
    <td>69.40</td>
    <td>59.76</td>
    <td>52.85</td>
  </tr>
  <tr>
    <td>Clear</td>
    <td>REMS</td>
    <td>54.37</td>
    <td>39.43</td>
    <td>14.94</td>
    <td>37.63</td>
    <td>68.64</td>
    <td>31.01</td>
    <td>68.86</td>
    <td>62.91</td>
    <td>50.87</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>53.46</td>
    <td>38.34</td>
    <td>15.12</td>
    <td>36.43</td>
    <td>67.02</td>
    <td>30.59</td>
    <td>67.49</td>
    <td>62.33</td>
    <td>49.51</td>
  </tr>
  <tr>
    <td>Chain Of Thought</td>
    <td>REMS</td>
    <td>61.87</td>
    <td>45.29</td>
    <td>16.58</td>
    <td>49.87</td>
    <td>73.88</td>
    <td>24.01</td>
    <td>77.68</td>
    <td>70.40</td>
    <td>59.48</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>59.86</td>
    <td>43.06</td>
    <td>16.80</td>
    <td>48.51</td>
    <td>72.04</td>
    <td>23.53</td>
    <td>75.13</td>
    <td>69.23</td>
    <td>57.16</td>
  </tr>
  <tr>
    <td>Plan And Solve</td>
    <td>REMS</td>
    <td>59.31</td>
    <td>45.13</td>
    <td>14.18</td>
    <td>50.55</td>
    <td>71.82</td>
    <td>21.27</td>
    <td>75.65</td>
    <td>67.06</td>
    <td>57.32</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>57.79</td>
    <td>43.49</td>
    <td>14.30</td>
    <td>49.07</td>
    <td>70.29</td>
    <td>21.22</td>
    <td>73.63</td>
    <td>66.08</td>
    <td>55.21</td>
  </tr>
  <tr>
    <td>Program of Thought</td>
    <td>REMS</td>
    <td>46.63</td>
    <td>31.83</td>
    <td>14.80</td>
    <td>28.30</td>
    <td>63.03</td>
    <td>34.73</td>
    <td>61.16</td>
    <td>51.98</td>
    <td>41.83</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>44.64</td>
    <td>29.90</td>
    <td>14.74</td>
    <td>27.69</td>
    <td>61.40</td>
    <td>33.71</td>
    <td>59.15</td>
    <td>48.91</td>
    <td>39.36</td>
  </tr>
  <tr>
    <td>Faithful CoT</td>
    <td>REMS</td>
    <td>40.19</td>
    <td>29.44</td>
    <td>10.75</td>
    <td>25.11</td>
    <td>60.30</td>
    <td>35.19</td>
    <td>56.52</td>
    <td>46.22</td>
    <td>37.86</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>38.75</td>
    <td>27.75</td>
    <td>11.00</td>
    <td>24.54</td>
    <td>58.83</td>
    <td>34.29</td>
    <td>55.19</td>
    <td>43.79</td>
    <td>35.74</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>34.77</td>
    <td>31.52</td>
    <td>3.25</td>
    <td>37.08</td>
    <td>41.36</td>
    <td>4.28</td>
    <td>45.83</td>
    <td>40.81</td>
    <td>29.74</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>33.56</td>
    <td>30.47</td>
    <td>3.09</td>
    <td>36.06</td>
    <td>39.77</td>
    <td>3.71</td>
    <td>44.26</td>
    <td>39.65</td>
    <td>27.54</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>45.49</td>
    <td>47.31</td>
    <td>7.28</td>
    <td>55.38</td>
    <td>63.12</td>
    <td>7.74</td>
    <td>68.51</td>
    <td>60.83</td>
    <td>56.15</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>53.81</td>
    <td>46.64</td>
    <td>7.17</td>
    <td>54.65</td>
    <td>62.22</td>
    <td>7.57</td>
    <td>67.21</td>
    <td>60.55</td>
    <td>55.63</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>64.61</td>
    <td>64.04</td>
    <td>0.57</td>
    <td>67.40</td>
    <td>69.54</td>
    <td>2.14</td>
    <td>77.42</td>
    <td>68.01</td>
    <td>61.78</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>64.36</td>
    <td>63.81</td>
    <td>0.55</td>
    <td>66.91</td>
    <td>68.89</td>
    <td>1.98</td>
    <td>77.19</td>
    <td>67.65</td>
    <td>60.92</td>
  </tr>
</table>

Table 12: Test Set Results for Llama 3.1 70B.

<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Few Shots</th>
    <th rowspan="2">Metric</th>
    <th colspan="7">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="6">Text</td>
    <td rowspan="6">Table</td>
    <td>None</td>
    <td>REMS</td>
    <td>17.53</td>
    <td>13.52</td>
    <td>21.71</td>
    <td>20.31</td>
    <td>20.38</td>
    <td>20.06</td>
    <td>19.95</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>15.92</td>
    <td>12.73</td>
    <td>20.82</td>
    <td>19.53</td>
    <td>19.26</td>
    <td>17.75</td>
    <td>18.08</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>40.26</td>
    <td>32.62</td>
    <td>37.58</td>
    <td>56.55</td>
    <td>54.09</td>
    <td>46.27</td>
    <td>38.65</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>38.06</td>
    <td>30.47</td>
    <td>35.50</td>
    <td>53.33</td>
    <td>51.78</td>
    <td>43.79</td>
    <td>35.05</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>50.92</td>
    <td>42.31</td>
    <td>43.17</td>
    <td>68.13</td>
    <td>64.75</td>
    <td>55.80</td>
    <td>49.06</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>48.79</td>
    <td>40.20</td>
    <td>40.89</td>
    <td>65.03</td>
    <td>62.30</td>
    <td>53.25</td>
    <td>45.62</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>39.98</td>
    <td>33.23</td>
    <td>37.68</td>
    <td>50.69</td>
    <td>53.84</td>
    <td>41.02</td>
    <td>40.33</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>37.54</td>
    <td>30.62</td>
    <td>34.94</td>
    <td>47.72</td>
    <td>50.96</td>
    <td>38.46</td>
    <td>35.74</td>
  </tr>
  <tr>
    <td>zero shots</td>
    <td>REMS</td>
    <td>20.35</td>
    <td>15.07</td>
    <td>19.30</td>
    <td>27.74</td>
    <td>21.65</td>
    <td>27.19</td>
    <td>22.43</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>19.90</td>
    <td>14.74</td>
    <td>18.96</td>
    <td>26.55</td>
    <td>20.63</td>
    <td>27.02</td>
    <td>21.56</td>
  </tr>
  <tr>
    <td>SQL Schema</td>
    <td>Static</td>
    <td>REMS</td>
    <td>47.42</td>
    <td>41.38</td>
    <td>48.78</td>
    <td>54.63</td>
    <td>69.46</td>
    <td>46.22</td>
    <td>42.45</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>EMS</td>
    <td>46.02</td>
    <td>40.20</td>
    <td>47.96</td>
    <td>52.98</td>
    <td>67.35</td>
    <td>45.36</td>
    <td>40.61</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>25.45</td>
    <td>21.26</td>
    <td>24.72</td>
    <td>33.96</td>
    <td>26.91</td>
    <td>47.24</td>
    <td>22.20</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>EMS</td>
    <td>25.09</td>
    <td>20.89</td>
    <td>24.54</td>
    <td>33.57</td>
    <td>26.78</td>
    <td>46.55</td>
    <td>21.56</td>
  </tr>
</table>

Table 13: Test Set Results for Mixtral 8x7B.

<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Few Shots</th>
    <th rowspan="2">Metric</th>
    <th colspan="7">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="4">SQL</td>
    <td rowspan="4">Schema</td>
    <td>zero shots</td>
    <td>REMS</td>
    <td>19.59</td>
    <td>18.32</td>
    <td>20.56</td>
    <td>27.60</td>
    <td>29.05</td>
    <td>20.28</td>
    <td>17.29</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>18.17</td>
    <td>17.02</td>
    <td>19.46</td>
    <td>26.08</td>
    <td>26.78</td>
    <td>19.72</td>
    <td>15.72</td>
  </tr>
  <tr>
    <td>Static</td>
    <td>REMS</td>
    <td>53.37</td>
    <td>49.32</td>
    <td>63.43</td>
    <td>61.01</td>
    <td>77.39</td>
    <td>54.05</td>
    <td>30.19</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>52.08</td>
    <td>47.93</td>
    <td>62.28</td>
    <td>59.53</td>
    <td>77.39</td>
    <td>51.19</td>
    <td>28.91</td>
  </tr>
  <tr>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>57.10</td>
    <td>55.03</td>
    <td>64.00</td>
    <td>60.55</td>
    <td>65.02</td>
    <td>60.13</td>
    <td>54.07</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>55.88</td>
    <td>53.50</td>
    <td>63.17</td>
    <td>58.95</td>
    <td>63.93</td>
    <td>58.38</td>
    <td>51.74</td>
  </tr>
</table>

Table 14: Test Set Results for SQL Coder 70B.
<table>
  <tr>
    <th rowspan="2">Output</th>
    <th rowspan="2">Context</th>
    <th rowspan="2">Few Shots</th>
    <th rowspan="2">Metric</th>
    <th colspan="7">Results Across Categories</th>
  </tr>
  <tr>
    <th>Original</th>
    <th>CounterFact</th>
    <th>Large</th>
    <th>Small</th>
    <th>Easy</th>
    <th>Medium</th>
    <th>Hard</th>
  </tr>
  <tr>
    <td rowspan="4">SQL</td>
    <td rowspan="2">Schema</td>
    <td>zero shot</td>
    <td>REMS</td>
    <td>12.37</td>
    <td>19.77</td>
    <td>13.76</td>
    <td>21.31</td>
    <td>33.81</td>
    <td>25.98</td>
    <td>19.19</td>
  </tr>
  <tr>
    <td></td>
    <td>EMS</td>
    <td>12.22</td>
    <td>19.46</td>
    <td>13.76</td>
    <td>21.00</td>
    <td>33.47</td>
    <td>25.64</td>
    <td>18.64</td>
  </tr>
  <tr>
    <td></td>
    <td>Static</td>
    <td>REMS</td>
    <td>16.53</td>
    <td>33.29</td>
    <td>29.82</td>
    <td>42.04</td>
    <td>54.13</td>
    <td>41.62</td>
    <td>39.09</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>EMS</td>
    <td>15.84</td>
    <td>32.62</td>
    <td>29.82</td>
    <td>41.64</td>
    <td>53.42</td>
    <td>41.62</td>
    <td>38.94</td>
  </tr>
  <tr>
    <td></td>
    <td>Adaptive</td>
    <td>REMS</td>
    <td>24.02</td>
    <td>40.73</td>
    <td>38.00</td>
    <td>48.56</td>
    <td>65.56</td>
    <td>51.78</td>
    <td>41.63</td>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td>EMS</td>
    <td>23.53</td>
    <td>40.06</td>
    <td>37.61</td>
    <td>48.14</td>
    <td>65.16</td>
    <td>50.89</td>
    <td>40.61</td>
  </tr>
</table>

Table 15: Test Set Results for Code Llama.
11  SQL Code Generation Prompt

# Task Instruction:
You will be given a question and your task is to provide the SQL logic to answer a natural language question based on the provided schema. Few Examples of the task will be provided below. Assume that all the data is already inserted into the database.

1. Table Schemas:

CREATE TABLE Athlete (
    athlete_id INT AUTO_INCREMENT PRIMARY KEY,
    name VARCHAR(100) NOT NULL
);
CREATE TABLE Tournament (
    tournament_id INT AUTO_INCREMENT PRIMARY KEY,
    athlete_id INT,
    name VARCHAR(100) NOT NULL,
    FOREIGN KEY (athlete_id) REFERENCES Athlete(athlete_id)
);
CREATE TABLE Format (
    format_id INT AUTO_INCREMENT PRIMARY KEY,
    tournament_id INT,
    name VARCHAR(100) NOT NULL,
    FOREIGN KEY (tournament_id) REFERENCES Tournament(tournament_id)
);
CREATE TABLE Medal (
    medal_id INT AUTO_INCREMENT PRIMARY KEY,
    format_id INT,
    type VARCHAR(50) NOT NULL,
    year INT,
    location VARCHAR(100) NOT NULL,
    FOREIGN KEY (format_id) REFERENCES Format(format_id)
);
CREATE TABLE PersonalInformation (
    info_id INT AUTO_INCREMENT PRIMARY KEY,
    athlete_id INT,
    birth_year INT,
    birth_month INT,
    birth_day INT,
    FOREIGN KEY (athlete_id) REFERENCES Athlete(athlete_id)
);

2. Table Descriptions:

describe athlete;
+----------+---------+------+-----+---------+----------------+
| Field    | Type    | Null | Key | Default | Extra          |
+----------+---------+------+-----+---------+----------------+
| athlete_id | int(11) | NO   | PRI | NULL    | auto_increment |
| name      | varchar(100)| NO  |     | NULL    |                |
+----------+---------+------+-----+---------+----------------+

describe personalinformation;
+----------+---------+------+-----+---------+----------------+
| Field    | Type    | Null | Key | Default | Extra          |
+----------+---------+------+-----+---------+----------------+
| info_id  | int(11) | NO   | PRI | NULL    | auto_increment |
| athlete_id | int(11) | YES  | MUL | NULL    |                |
| birth_year | int(11) | YES  |     | NULL    |                |
| birth_month | int(11) | YES  |     | NULL    |                |
| birth_day | int(11) | YES  |     | NULL    |                |
+----------+---------+------+-----+---------+----------------+

describe tournament;
+----------+---------+------+-----+---------+----------------+
| Field    | Type    | Null | Key | Default | Extra          |
+----------+---------+------+-----+---------+----------------+
| tournament_id | int(11) | NO   | PRI | NULL    | auto_increment |
| athlete_id | int(11) | YES  | MUL | NULL    |                |
| name      | varchar(100)| NO  |     | NULL    |                |
+----------+---------+------+-----+---------+----------------+

describe format;
+----------+---------+------+-----+---------+----------------+
| Field    | Type    | Null | Key | Default | Extra          |
+----------+---------+------+-----+---------+----------------+
| format_id | int(11) | NO   | PRI | NULL    | auto_increment |
| tournament_id | int(11) | YES  | MUL | NULL    |                |
| name      | varchar(100)| NO  |     | NULL    |                |
+----------+---------+------+-----+---------+----------------+

describe medal;
<table>
  <tr>
    <th>Field</th>
    <th>Type</th>
    <th>Null</th>
    <th>Key</th>
    <th>Default</th>
    <th>Extra</th>
  </tr>
  <tr>
    <td>medal_id</td>
    <td>int(11)</td>
    <td>NO</td>
    <td>PRI</td>
    <td>NULL</td>
    <td>auto_increment</td>
  </tr>
  <tr>
    <td>format_id</td>
    <td>int(11)</td>
    <td>YES</td>
    <td>MUL</td>
    <td>NULL</td>
    <td></td>
  </tr>
  <tr>
    <td>type</td>
    <td>varchar(50)</td>
    <td>NO</td>
    <td></td>
    <td>NULL</td>
    <td></td>
  </tr>
  <tr>
    <td>year</td>
    <td>int(11)</td>
    <td>YES</td>
    <td></td>
    <td>NULL</td>
    <td></td>
  </tr>
  <tr>
    <td>location</td>
    <td>varchar(100)</td>
    <td>NO</td>
    <td></td>
    <td>NULL</td>
    <td></td>
  </tr>
</table>

3. Example Data:
Athlete Table
<table>
  <tr>
    <th>athlete_id</th>
    <th>name</th>
  </tr>
  <tr>
    <td>50</td>
    <td>Carolina Marin</td>
  </tr>
</table>
PersonalInformation Table
<table>
  <tr>
    <th>info_id</th>
    <th>athlete_id</th>
    <th>birth_year</th>
    <th>birth_month</th>
    <th>birth_day</th>
  </tr>
  <tr>
    <td>40</td>
    <td>50</td>
    <td>1993</td>
    <td>6</td>
    <td>15</td>
  </tr>
</table>
Tournament Table
<table>
  <tr>
    <th>tournament_id</th>
    <th>athlete_id</th>
    <th>name</th>
  </tr>
  <tr>
    <td>281</td>
    <td>50</td>
    <td>Olympic Games</td>
  </tr>
  <tr>
    <td>282</td>
    <td>50</td>
    <td>World Championships</td>
  </tr>
  <tr>
    <td>285</td>
    <td>50</td>
    <td>European Women</td>
  </tr>
</table>
Format Table
<table>
  <tr>
    <th>format_id</th>
    <th>tournament_id</th>
    <th>name</th>
  </tr>
  <tr>
    <td>392</td>
    <td>281</td>
    <td>Women's singles</td>
  </tr>
  <tr>
    <td>393</td>
    <td>282</td>
    <td>Women's singles</td>
  </tr>
  <tr>
    <td>396</td>
    <td>285</td>
    <td>Women's team</td>
  </tr>
</table>
Medal Table
<table>
  <tr>
    <th>medal_id</th>
    <th>format_id</th>
    <th>type</th>
    <th>year</th>
    <th>location</th>
  </tr>
  <tr>
    <td>692</td>
    <td>392</td>
    <td>MedalGold</td>
    <td>2016</td>
    <td>Rio de Janeiro</td>
  </tr>
  <tr>
    <td>696</td>
    <td>393</td>
    <td>MedalSilver</td>
    <td>2023</td>
    <td>Copenhagen</td>
  </tr>
  <tr>
    <td>706</td>
    <td>396</td>
    <td>MedalBronze</td>
    <td>2016</td>
    <td>Kazan</td>
  </tr>
</table>

Example 1:
Question: Which tournament(s) has Zhang Jike won the most Medals in?

WITH medal_counts AS (
  SELECT t.name AS tournament_name, m.year, COUNT(m.medal_id) AS total_medal_count
  FROM Medal m
  JOIN Format f ON m.format_id = f.format_id
  JOIN Tournament t ON f.tournament_id = t.tournament_id
  JOIN Athlete a ON t.athlete_id = a.athlete_id
  WHERE a.name = 'Zhang Jike'
  GROUP BY t.name, m.year
)
SELECT tournament_name, year
FROM medal_counts
WHERE total_medal_count = (
  SELECT MAX(total_medal_count)
  FROM medal_counts
);

Example 2:
Question: In which year(s) did Seo Seung-jae win medals in the Asian Junior Championships?

SELECT DISTINCT m.year
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Seo Seung-jae'
    AND t.name = 'Asian Junior Championships';

Example 3:
Question: Which was the most current medal win for Dola Banerjee?

SELECT m.type, m.year, m.location, f.name AS format_name, t.name AS tournament_name
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Dola Banerjee'
    AND m.year =
        SELECT MAX(m2.year)
        FROM Medal m2
        JOIN Format f2 ON m2.format_id = f2.format_id
        JOIN Tournament t2 ON f2.tournament_id = t2.tournament_id
        JOIN Athlete a2 ON t2.athlete_id = a2.athlete_id
        WHERE a2.name = 'Dola Banerjee'
;

Example 4:
Question: How many international medals did Rawinda Prajongjai win in 2023?

SELECT COUNT(m.medal_id) AS total_medals
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Rawinda Prajongjai'
    AND m.year = 2023;

Example 5:
Question: In which year(s) did Huang Dongping win the highest number of medals during their career?

SELECT m.year
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Huang Dongping'
GROUP BY m.year
ORDER BY COUNT(m.medal_id) DESC
LIMIT 1;

Example 6:
Question: In which year(s) did Tomokazu Harimoto win the lowest number of medals during their career?

SELECT m.year
FROM Medal m
JOIN Format f ON m.format_id = f.format_id
JOIN Tournament t ON f.tournament_id = t.tournament_id
JOIN Athlete a ON t.athlete_id = a.athlete_id
WHERE a.name = 'Tomokazu Harimoto'
GROUP BY m.year
HAVING COUNT(m.medal_id) =
    (SELECT MIN(medal_count)
    FROM (
        SELECT COUNT(m2.medal_id) AS medal_count
        FROM Medal m2
        JOIN Format f2 ON m2.format_id = f2.format_id
        JOIN Tournament t2 ON f2.tournament_id = t2.tournament_id
        JOIN Athlete a2 ON t2.athlete_id = a2.athlete_id
        WHERE a2.name = 'Tomokazu Harimoto'
        GROUP BY m2.year
    ) AS yearly_medal_counts
    );

Instructions for Writing Queries:

1. If a question can have multiple answers, do not limit the response to only one. Instead, output all possible answers.

2. Use the column names as specified in the schema to find the necessary parameters for the query.

3. An event is a combination of Tournament, Format, and the corresponding year.

4. There are three types of medals in the Medal Table: MedalGold, MedalSilver, MedalBronze.

Reviewer paper summaries: This paper proposes extracting information from Wikipedia infoboxes to synthesize a dataset called TEMPTABQA-C for temporal tabular question answering. The authors conduct experiments to analyze the limitations of the current direct prompt-based approach (CoT) in solving this task. To improve the performance, this paper designs a symbolic intermediate representation approach, in which a large language model (LLM) is prompted to generate SQL queries to obtain answers. The paper also explores adaptive few-shot prompting to enhance performance.

The paper introduces the TEMPTABQA-C dataset, a synthetic benchmark intended for evaluation of temporal reasoning in tabular data. To address challenges like handling large tables, counterfactual reasoning, and complex queries, the authors propose using a symbolic intermediate representation. They also explore adaptive few-shot prompting, whereby in-context examples are chosen dynamically. Experimental results show that SQL-based reasoning consistently outperforms direct prompting.
Summary of strengths: 1.The constructed TEMPTABQA-C dataset provides a controlled benchmark for evaluating models’ temporal tabular reasoning ability, which is a contribution to the field.


2.The designed symbolic-integration pipeline demonstrates better performance compared to the standard direct prompting approach.


3.The authors have conducted thorough experiments to compare the performance of the existing direct prompting approach (CoT) and their proposed method across different subsets of their benchmark. These experiments reveal key challenges and insights into temporal reasoning capabilities.

1.	The authors present a well-designed synthetic dataset.
2.	The authors methodically evaluate multiple axes (original vs. counterfactual, small vs. large tables, easy vs. hard questions), offering a thorough performance comparison.
3.	The experiments clearly demonstrate the benefits of the symbolic approach, with quantitative results showing smaller performance gaps under difficult conditions.
Summary of weaknesses: 1.The TEMPTABQA-C creation pipeline is heavily reliant on templates, resulting in less diverse datasets. This may compromise the benchmark's value and applicability to broader contexts.

2.While the symbolic intermediate representation approach is effective, its novelty is limited since such techniques have been extensively explored in prior research.

3.Generating SQL queries via LLMs often requires including predefined information in the system prompt. Additionally, syntax errors are common when executing these queries over databases, which raises scalability concerns.

1.	The proposed SQL generation method in this paper appears to be based on established techniques already widely discussed in the text-to-SQL field. It does not seem to present significant advancements or novel insights compared to existing approaches. 
2.	The dynamic contextual prompting technique mentioned in the paper has already been utilized in many prior studies. The authors need to provide a clear explanation of how their version of dynamic prompting offers a distinctive contribution or improvement.
3.	The paper outlines the general pipeline for building TEMPTABQA-C but does not give enough technical detail about the data generation rules or how the templates are formed.
4.	Although the paper references several prior works in symbolic reasoning, it could provide more direct, quantitative comparisons against existing neuro-symbolic or logic-based frameworks on the same dataset.
Comments, suggestions and typos: Question 1: The questions and creation pipeline in TEMPTABQA-C seem focused solely on sports (e.g., athletes). Why not diversify the dataset? Were there specific challenges in doing so?

Question 2: In Table 4, it appears that the direct prompting approach (CoT) with adaptive few-shot prompting achieves better performance than the SQL-based approach with adaptive prompting on smaller tables. Could you clarify this observation?

Question 3: Given that LLMs like GPT-4 can solve problems effectively (as suggested by the example in Figure 1), why not evaluate the proposed approach on open-source or smaller LLMs to assess its broader applicability?

Question 4:  From Tables 4 and 5, SQL-based methods show significantly better performance on larger tables. Could you explain the reasons behind this? Additionally, since you have not provided the code or dataset, could you share a concrete example (including the complete input and output) of a failure case unsolved by GPT-4 but successfully handled by your approach? If the example is too lengthy, an anonymized version would be acceptable?

None
Overall assessment scores: 3

2.5