Triples as the Key: Structuring Makes Decomposition and Verification Easier in LLM-based TableQA

Authors: Zhen Yang, Ziwei Du, Minghan Zhang, Wei Du, Jie Chen, Zhen Duan, Shu Zhao
Primary area: generative models
Keywords: TableQA, Triples

Abstract:
As the mainstream approach, LLMs have been widely applied and researched in TableQA tasks. Currently, the core of LLM-based TableQA methods typically include three phases: question decomposition, sub-question TableQA reasoning, and answer verification. However, several challenges remain in this process: i) Sub-questions generated by these methods often exhibit significant gaps with the original question due to critical information overlooked during the LLM's direct decomposition; ii) Verification of answers is typically challenging because LLMs tend to generate optimal responses during self-correct. To address these challenges, we propose a Triple-Inspired Decomposition and vErification (TIDE) strategy, which leverages the structural properties of triples to assist in decomposition and verification in TableQA. The inherent structure of triples (head entity, relation, tail entity) requires the LLM to extract as many entities and relations from the question as possible. Unlike direct decomposition methods that may overlook key information, our transformed sub-questions using triples encompass more critical details. Additionally, this explicit structure facilitates verification. By comparing the triples derived from the answers with those from the question decomposition, we can achieve easier and more straightforward validation than when relying on the LLM's self-correct tendencies. By employing triples alongside established LLM modes, Direct Prompting and Agent modes, TIDE achieves state-of-the-art performance across multiple TableQA datasets, demonstrating the effectiveness of our method.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730226783793

thread_type:
official_review

<thread_object>
creator: reviewer_uglh

<content>
summary:
The paper proposed an inference time approach, TIDE, to address table question answering. Their method builds on a Decomposition-Reasoning-Verification approach for Table QA using LLM, and they proposed to use semantic triples as intermediate representations instead of natural text. First, during question decomposition, they extract triples from the question then generate subquestions based on those triples. Then, they apply CoT (for DP mode)/generating python code (for agent mode) to answer each subquestions. Finally, in the answer verification stage, they compare the triples from stage 1 with triples from stage 2 to filter correctly answered triples. 

Their contribution is using triples as intermediate representations in order to facilitates decomposition and verification, and showed that it leads to improvements over SOTA.

strengths:
Incorporating triples as intermediate steps is an interesting approach, and it intuitively makes sense as a natural intermediate representation to tackle the verification of subquestions. The authors effectively combine existing ideas from different problems and apply it in the settings of table QA, showing SOTA results on challenging table qa datasets.

weaknesses:
My main concern is clarity and contribution.
For clarity
1.  Details of how each module is implemented is confusing. The formulation really only has one equation (eq 1), and it is about the Table QA problem. Formal explanation (instead of examples only) for each submodule would help improve clarity. Furthermore, I think the prompt for extracting triples are not provided?

2. It’s unclear to me how the final answer is synthesized after the verification stage. Again, providing formulation for this stage would be more helpful. 

3. In Table 3 of your ablation studies, you have TIDE vs TIDE-decomposition vs TIDE-verification. It is confusing to me how the ablated versions are implemented. Again, I think having formulation would help improve clarity.

In terms of contribution, since this is combining different existing ideas from other tasks (see below for examples), I would appreciate more in-depth analysis of the proposed method.

1.  Would other language models be equally effective with generating triples? When does this assumption breakdown? How does model size affect the result and error propagation through each stage ... etc. Have you tested this with open source large/small sized language models or other information extraction approaches for triple extraction? Is it possible to substitute some LLM API calls with other non-LLM methods without hindering performance?

2. The strong performance could potentially come from data contamination. How do you attribute the success of TIDE vs GPT having seen those tables in pretraining? 

3. In verification stage, you mentioned the LLM is prompted to regenerate if verification failed. I would appreciate more analysis on this part such as showing how often verification fails or how many times max you limit the LLM to regenerate.


PS by combining existing ideas I mean
1. Question decomposition is not new. See [1] [2] for example
2. using LLM for information extraction is not new. See [3] for example

I think there are some merits in combining these insights and seeing that triples could apply to this problem, so I think more analysis could help improve this paper.

[1] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop Reading Comprehension through Question Decomposition and Rescoring. ACL 2019.

[2] Xiaoyu Yang and Xiaodan Zhu.  Exploring Decomposition for Table-based Fact Verification. EMNLP Findings 2021

[3] Zhiyuan Fan and Shizhu He. 2023. Efficient Data Learning for Open Information Extraction with Pre-trained Language Models.  EMNLP Findings 2023

questions:
This goes back to the clarity of the verification stage.The example authors provided seems to be a ‘COUNT’ type questions. How does the system handle other types of numeric aggregation? For questions that are not numeric in nature (e.g. compose-style, multi-hop questions), how does the final stage work after verification? Does this approach only work on aggregation type questions? What algorithms are you using to unify the triples for each subquestion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638020157

thread_type:
official_review

<thread_object>
creator: reviewer_bvqy

<content>
summary:
This paper proposed a decomposition and verification strategy based on triple to address TableQA tasks. Specifically, it decomposes and transforms questions using triples to capture more key information, while utilizes explicit structure for better verification. By combining LLM-based reasoning and symbolic reasoning based on structured representations, TIDE improves the performance in the TableQA task

strengths:
1. The motivation of the work is clear. It leverages triple extraction as a tool to enhance understanding and validation of TableQA questions.
  
2. The experiment is sufficient, which compares TIDE with many TableQA baselines. Further study can reveal the properties of the proposed method.

weaknesses:
1. The contribution is limited; it only proposes a framework based on LLMs with triple extraction. It would be better if it could be demonstrated on some open-source models like Llama3.
  
2. The conclusion drawn from the experimental results that TIDE improves the effectiveness of TableQA problems is questionable. The results of DP and Agent baseline for both datasets are not SOTA. So, can it be considered that the gain in this result comes from additional reasoning integration of DP and Agen rather than the introduction of triplet?

questions:
The appendix lacks cases, including successful cases compared to other methods, error cases analysis, etc. I think showing some cases can better explain the working mechanism of the entire pipeline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700529074

thread_type:
official_review

<thread_object>
creator: reviewer_ymyg

<content>
summary:
The authors use Triples in different phases of TableQA. The generation of triples during Question decomposition phase helps in extracting key subquestions and conversion of answers during answer verification phase helps remove LLM bias in verification and enables better answer verification.

strengths:
1. Incorporating triples in multiple stages of TableQA is a novel idea and as shown by the evidence in the paper, helps improve the overall QA performance. 
2. The authors demonstrate the effectiveness of TIDE by performing ablation study which is critical in identifying the impact of this method.
3. This method works alongside DP and adds an additional step of generating triples, so this keeps all the benefits of DP as well.

weaknesses:
1. Inorder to demonstrate the impact of TIDE, it could have been useful to see TIDE with other standard LLMs as well. The ablation study shows the impact of TIDE but is this method tailored to work with GPT 3.5 in any specific way?

questions:
1. Given that this is an additional step along the pipeline of TableQA steps, what does this step do to the average response time and number of tokens used? 
2. Does this method of generating triples work for mathematical questions such as average age/salary etc? Does this mean that the subquestions would require triples for every entry in the table?
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings:
The authors propose TIDE, a method that uses triples to structure question decomposition and verification in TableQA tasks. The decomposition phase generates triples to reduce semantic gaps in sub-questions, and the verification phase uses these triples for explicit answer validation. Experiments demonstrate improvements over existing baselines on datasets like WTQ and TabFact, with state-of-the-art (SOTA) results.

(b) Strengths:

- Novel incorporation of triples into TableQA pipelines.
- Comprehensive experiments, including ablation studies, demonstrating the effectiveness of TIDE.
- Applicability to both direct prompting (DP) and agent-based reasoning modes.
- Improved performance across multiple LLMs and significant gains over non-LLM methods.

(c) Weaknesses and Missing Aspects:

- Limited exploration of computational overhead and the impact of model size on performance in the main text.
- Lack of discussion on potential data contamination in pretrained LLMs affecting results.
- Some concerns about whether TIDE’s gains stem from triple integration or added reasoning layers.

(d) Decision Factors:
The strong empirical results, comprehensive rebuttal, and clarifications provided during the review period outweigh concerns about initial clarity and limited novelty. The inclusion of additional experiments and detailed analyses on other LLMs strengthens the case for acceptance.