PoTable: Programming Standardly on Table-based Reasoning Like a Human Analyst

Authors: Qingyang Mao, Qi Liu, Zhi Li, Mingyue Cheng, Zheng Zhang, Rui Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Table-based Reasoning, Large Language Model, Symbolic Tools, Real-time Program Execution, Human Cognitive Behavior

Abstract:
Table-based reasoning has garnered substantial research interest, particularly in its integration with Large Language Model (LLM) which has revolutionized the general reasoning paradigm. Numerous LLM-based studies introduce symbolic tools (e.g., databases, Python) as assistants in complex information understanding and arithmetic computations. However, they emphasize extensive and flexible utilization of symbolic tools, without fully considering the intrinsic logic of the reasoning process. In this study, we propose PoTable as a simple yet effective table-based reasoning method. Specifically, PoTable features a planning phase and an executing phase, implemented with an LLM-based operation planner and code generator and a Python interpreter as the real-time executor. To incorporate logical top-level guidance, we split the entire reasoning process into several distinct analysis stages with macroscopic instruction injection. As the reasoning process is structured suitably under the top-level guidance with precise and specific goals, PoTable produces superior reasoning results with highly accurate, steply commented and completely executable code. To summarize, PoTable enjoys the advantages of accuracy and explainability that make it a distinguished tabular data analyst. Extensive experiments over three evaluation datasets from two public benchmarks on two backbones demonstrate the outstanding performance of PoTable. In particular, GPT-based PoTable achieves over 4% higher absolute accuracy than runner-ups on all evaluation datasets. Our code is available at https://anonymous.4open.science/r/PoTable-6788.

Decision: reject
Reviewer scores: 5, 3, 3, 6

Reviews and discussion:
<thread>
creation date:
1729667114485

thread_type:
official_review

<thread_object>
creator: reviewer_wujl

<content>
summary:
The study introduces PoTable, a novel method for table-based reasoning that effectively simulates human cognitive behavior by integrating a Python interpreter with a Large Language Model (LLM). This approach addresses limitations in previous methods by employing a human-like logical stage split and expanding the operation pool into an open-world space. PoTable uses an LLM-based operation planner and code generator to produce highly accurate, step-by-step commented, and fully executable programs. The method demonstrates strong effectiveness and explainability, achieving over 4% higher accuracy than competitors across multiple datasets.

strengths:
- Tabular analysis is crucial for both research and practical applications, highlighting the relevance of this study.
- The proposed method, POTABLE, achieves state-of-the-art performance on selected evaluation datasets, outperforming existing approaches.

weaknesses:
1. I don't think Initialization and Final Answering are important enough to be count as stages.
2. The claim that your analysis stage split is human-like is too arbitrary. How do you know that human do tabular analysis in this way? Have you ever done any research about this? if so, please cite them in the paper.
3.  The evaluation datasets are not extensive enough. Evaluation on just two datasets cannot really support your concusion. For evaluation datasets, why did you choose to use both validation and test sets of WiKiTQ? is there any inherent difference between these two sets? The evaluation results in Table 1 actually show almost similar results for both sets.
4. The results analysis in the seconde paragraph in Sec 3.2 could be further improved. I think it is important to discuss the phenomenon that the difference between GPT and LLAMA is quite different across approaches, i.e., for WiKiTQ, GPT > LLAMA for Binder, Dater, GPT < LLAMA for Chain-of-Table, GPT = LLAMA for TabSQLify and PoTable. The same for TabFact.
5. In Figure 3, it is suggested to discuss why "removing data type cleaning (w/o Dty. Cle.)" gets the lowest scores. It would be weird if you ignore this result because it is so obvious in the figure.
6. I found that the max tokens of the backbone models used in the study is 2048 (Appendix A). Thus i am particular interested in that, if you put the full table in the prompt, how do handle the table with large content exceeding 2048 tokens?  Also, the code_base would also increase as the analysis stage goes ahead. 
7. In figure 11, the prompt contains several suggested operations when generating the reasoning plan. Would this be contradicted with the claim that the PoTable "extends the operation pool into an open-world space without any constraints."? If not, please provide some examples to prove.

questions:
See questions in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730037293444

thread_type:
official_review

<thread_object>
creator: reviewer_xpqj

<content>
summary:
This paper proposes a method, PoTable, to solving table-based reasoning tasks with two contributions: (i) logically standard reasoning and (ii) unrestricted operation tools.

strengths:
1. The proposed method appears to be effective on the three datasets experimented. 

2. The analysis of PoTable performance on varied task difficulty and table size is interesting. It would be further informative if the result breakdown on baseline methods are also provided, so it tells more clearly on which category the proposed PoTable method provides the most increases.

weaknesses:
1. Unclear unique contribution of the proposed method: particularly, it is unclear to me where two main component highlighted are unique contributions of this paper.

(i) non-standard logical splits: (first of all, this term is somewhat vague so suggest changing it.) This paper proposed “standard” stages including (initialization, row selection, data type cleaning, reasoning, and final answering), where previous works such as Chain-of-Table (https://arxiv.org/abs/2401.04398) and Datar (https://arxiv.org/abs/2301.13808) have proposed similar decompositions, yet it is unclear why the split proposed in this paper is “more standard” than existing ones. Adding some clarifications on this could be helpful.

(ii) constrained operation pools: many table-related works do not have a restricted operation pool, for example, they can use arbitrary Python programs (ReAcTable: https://arxiv.org/abs/2310.00815, Chameleon: https://arxiv.org/abs/2304.09842) or even allowing model self-designed new operations (TroVE: https://arxiv.org/abs/2401.12869) 

Overall, both dimensions claimed as novel in this work seem to be already explored by existing works. It would be helpful to highlight the differences of this work further to clarify the novelty aspect.
Further, it is not entirely clear to me why (i) and (ii) leads to human-like analysis, and what “human-like analysis” is defined/characterized in this work.

questions:
1. How did you decide on the five steps as proposed in the PoTable method? And why do you believe that is the most "standard" split of the task? Can this split be generally applied to all table-reasoning tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730323788975

thread_type:
official_review

<thread_object>
creator: reviewer_hb7z

<content>
summary:
This work proposes a pure prompt-based method to solve reasoning that involves table, namely PoTable. The author proposes to split the process explicitly into several stages: initialization, row selection, data type cleaning, reasoning and final answering. These stages include prompting API LLMs for planning, and for using pandas package to manipulate the table. The experiments show the proposed method performs better on certain types of table reasoning tasks.

strengths:
See below

weaknesses:
## Novelty

PoTable is a pure prompt-based method with hardwired reasoning stages designed specifically for solving a certain type of table reasoning tasks. On top of API LLMs, PoTable is also equipped with tool usage and self-correction, but both of these techniques are widely studied in prior work on various types of reasoning. The highlighted human-like reasoning stage design lacks justification and is somewhat overfitted to a certain type of table reasoning, leading to poor generalization over other types of questions. That said, this work lacks novelty.


## Quality
Several methodological concerns undermine the paper's quality:

Hardwired reasoning stages:
- The authors claim that the 5-stage split represents "standard logic" (L85, L90), but there no direct evidence is provided to justify why these particular stages should be considered standard.
- Scalability issue: How flexible is this setup? Though it is claimed to be standard, can this approach handle general tabular reasoning scenarios? How does it perform with complex queries requiring merging tables? What about multi-step reasoning problems?  

Pipeline design choices not well-motivated or -justified:
- Why only use few-shot examples at the final stage?


## Clarity

This paper is generally easy to follow.


## Significance

As mentioned above, this work targets a specific type of table reasoning tasks that involve single-step, single-table reasoning. While being able to beat the baseline methods, the performance gain is moderate. More importantly, the introduced 5-stage reasoning significantly limits its application to other types of reasoning, further limits its impact on the field. That said, the significance is minor.

questions:
See above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715634765

thread_type:
official_review

<thread_object>
creator: reviewer_npzy

<content>
summary:
This paper introduces POTABLE, a table-based reasoning method that simulates human analyst behavior through a staged analytical approach. The method combines a Python interpreter for execution with an LLM-based operation planner and code generator. The method mimics human-like logical stage split (initialization, row selection, data cleaning, reasoning, and answering) and open-world operation space. The authors demonstrate POTABLE's effectiveness through experiments on WikiTQ and TabFact datasets, showing performance improvements.

strengths:
1. The paper provides a systematic refinement of existing table-based agents. By breaking down the process into well-defined stages and implementing careful error handling, POTABLE addresses common failure modes of general-purpose agents. The empirical results demonstrate that these refinements lead to meaningful performance improvements, showing the value of structured approaches to agent design.

2. The authors test their method across different datasets (WikiTQ and TabFact), model backbones (GPT and LLAMA), and conduct detailed ablation studies. The performance improvement over baselines (>4% on multiple datasets) suggests that the structured approach does offer tangible benefits over more generic agent implementations.

weaknesses:
1. The idea of using LLMs with pandas for table reasoning has already been previously explored in various works. For instance, LangChain has implemented pandas agents [1] at least a year ago, and at least 2 paper [2][3] (and I think there are more as this is a paper one year ago) has discussed the relationship between agent-based and direct reasoning approaches for tabular understanding. Under this situation, the main contribution will be in providing a more refined (and maybe fixed) workflow for table reasoning, which might have diminishing returns with more capable models. 
2. The paper lacks a crucial analysis of token efficiency and alternative approaches. A key consideration for any agent-based system is whether the increased token consumption from multiple LLM calls is justified by the performance gains. For instance, an alternative approach using self-consistency with multiple single-pass attempts might achieve similar accuracy while being more token-efficient. Without this comparison, it's difficult to determine if the improved accuracy (>4%) justifies what could be a significant increase in token usage. The paper should have included a detailed analysis comparing token consumption with simpler approaches and demonstrating why their multi-stage workflow is more effective than alternative methods like self-consistency.
3. There are several fundamental limitations in current (especially programming-based) table agents. Specifically, these agents (including POTABLE) struggle with three key scenarios: (1) multi-table reasoning that requires joining or comparing information across tables, (2) hybrid reasoning that combines tabular data with textual context, and (3) handling irregular table structures with multiple headers or nested hierarchies. While these challenges may be outside the paper's intended scope, they represent critical real-world use cases that limit the practical applicability of such methods.

---
[1] https://python.langchain.com/docs/integrations/tools/pandas/

[2] "Rethinking Tabular Data Understanding with Large Language Models", NAACL 2024, https://arxiv.org/abs/2312.16702

[3] "API-Assisted Code Generation for Question Answering on Varied Table Structures", EMNLP 2023, https://arxiv.org/pdf/2310.14687

questions:
- See weakness and:
- How do you envision POTABLE could be extended to handle multi-table scenarios or hybrid reasoning with both tabular and textual data or irregular table structures ?  
- As LLM capabilities continue to advance, would you agree or maybe disagree that the benefits of structured workflows might need to be reconsidered? (One could argue that giving more autonomy to stronger models - perhaps with fewer constraints on their reasoning process - might actually lead to better performance, but i would be interested in your thoughts on this trade-off between structure and autonomy.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732384885254

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Overall Revision of the Submitted Manuscript

comment:
We would like to thank all the reviewers for their efforts in reviewing our manuscript. 
Your valuable suggestions inspire us to improve our study and revise our manuscript in a graceful manner. 
Specifically, we have resubmitted a newly revised manuscript following your suggestions.
The main revision parts can be summarized as below.

**1. Motivation and Contribution**

We apologize for our previous expression that has impacted your understanding of our study. 
This prompted us to rethink and refine our motivation and contributions. 
Specifically, we overemphasized that our stage split was **standard** rather than conveying a more essential concept of **integrating logical top-level guidance**, as some existing reasoning approaches emphasize extensive and flexible utilization of symbolic tools without fully considering the intrinsic logic of the overall reasoning process. 
Therefore, we have revised the corresponding expressions about motivation and contribution in the abstract, introduction and contribution. 
Some related expressions (*e.g.*, advantages) have also been improved, without modifying our implemented approach. 

**2. Experiments**

We have conducted efficiency experiments and added a new related subsection into the experimental part with results and analysis. 
In addition, more details in the comparison experiments and ablation studies have also been improved. 

Apart from the above, we have improved the overall presentation of our new manuscript. 

**3. Title Revision on System**

According to the author guide in https://iclr.cc/Conferences/2025/AuthorGuide, we are allowed to revise the title of our manuscript, yet the revision system does not provide a block for title selection. Therefore, please check our new title in the revised manuscript as **POTABLE: PROGRAMMING ON TABLES TO REASON LIKE A DISTINGUISHED HUMAN DATA ANALYST**.

Anyway, thank you again for your valuable suggestions. If you have any other concerns, feel free to contact us.

Sincerely,

Paper 10382 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper concerns table-based reasoning with LLMs and proposes PoTable, which combines an LLM-based operation planner and code generator with a python interpreter for real-time execution. PoTable mimics a human-like logical stage split and expanding the operation pool into an open-world space. Experimental evaluations show that PoTable outperforms existing approaches on two public benchmarks (i.e., WikiTQ and TabFact) by over 4% absolute accuracy improvement. The idea of using LLMs for table reasoning has been extensively explored by many recent works (references shared by reviewers, NPzY and Xpqj) and the proposed method follows a similar (or refined) workflow, making the proposed method incremental. Furthermore, the new method relies on multiple LLM calls and has a low token efficiency compared to existing approaches. Comparing the performance without analyzing the computation costs (in terms of LLM calls)  making the 4% improvement sound less promising.