TAPERA: Enhancing Faithfulness and Interpretability in Long-Form Table QA by Content Planning and Execution-based Reasoning (2024)

Abstract: Long-form Table Question Answering (LFTQA) requires systems to generate paragraph long and complex answers to questions over tabular data. While Large language models based systems have made significant progress, it often hallucinates, especially when the task involves complex reasoning over tables. To tackle this issue, we propose a new LLM-based framework, TAPERA, for LFTQA tasks. Our framework uses a modular approach that decomposes the whole process into three sub-modules: 1) QA-based Content Planner that iteratively decomposes the input question into sub-questions; 2) Execution-based Table Reasoner that produces executable Python program for each sub-question; and 3) Answer Generator that generates long-form answer grounded on the program output. Human evaluation results on the FETAQA and QTSUMM datasets indicate that our framework significantly improves strong baselines on both accuracy and truthfulness, as our modular framework is better at table reasoning, and the long-form answer is always consistent with the program output. Our modular design further provides transparency as users are able to interact with our framework by manually changing the content plans. § https://github.com/yilunzhao/TaPERA

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12824–12840 August 11-16, 2024 ©2024 Association for Computational Linguistics
§ https://github.com/yilunzhao/TaPERA

heading: 1 Introduction
text: Long-form question answering (LFQA) presents a unique challenge in natural language processing (NLP), demanding models that not only comprehend the long context but also maintain factual accuracy during answer generation (Fan et al., 2019; Xu et al., 2023a). A prevalent and concerning issue is hallucination, where models generate answers that are coherent yet factually incorrect or irrelevant to the input context. This problem undermines the trustworthiness of NLP systems, posing a barrier
to their real-world application in domains where reliability is paramount, such as finance and healthcare. The hallucination issues become even more pronounced when the task involves reasoning over tables, e.g., long-form table question answering (LFTQA) (Nan et al., 2022b; Zhao et al., 2023d).
12824
As illustrated in Figure 1, given the complex question and numerous data points in a table, the system must be capable of understanding the relationships within the data and perform human-like reasoning over the tabular content to compose the paragraphlong answer.
This paper introduces TAPERA, aiming to enhance the trustworthiness of LLM-based methods in long-form TAble QA. As depicted in Figure 2, TAPERA employs a modular approach comprising three components: (1) QA-based Content Planner: Given a complex question, this module generates and iteratively updates a QA-based content plan. It controls the step-by-step reasoning process to shape the final answer content. (2) Execution-based Table Reasoner: For each subquestion in the QA-based plan, this module generates executable programs (i.e. Python programs) to extract and process question-relevant information from tables. (3) Answer Generator: Given the sub-question and the executed output from the table reasoner, this module composes a natural language sub-answer, ensuring consistency with the reasoner’s output. Moreover, once the entire QAbased plan is finalized, it generates the final answer based on the plan and all sub-answers.
TAPERA features several core advancements: (1) Modularity. The complex task is decomposed into multiple easier sub-tasks, which can be solved with more tailored solutions (e.g., reasoning enhanced modules). (2) Faithfulness The table reasoner first generates an executable Python program as external tools to output question-relevant facts, then the answer generator is applied to ensure the consistency between the generated answer and the program output, which improves the faithfulness of generated answer. (3) Interpretability. The system allows users for easy identification of which sub-plan or reasoning step leads to an undesired final answer. Additionally, our user study demonstrates the potential of applying user interaction to modify the generated plan to control and improve the final answer.
We experiment TAPERA framework on the FETAQA (Nan et al., 2022b) and QTSUMM (Zhao et al., 2023d) datasets. Human evaluation results demonstrate that TAPERA achieves significant improvements in terms of both accuracy and faithfulness. Specifically, TAPERA surpasses 1) the second-best system in comprehensiveness, i.e. Blueprint (Narayan et al., 2023), by 5.4%; and 2) the second-best system in faithfulness, i.e.
Dater (Ye et al., 2023), by 5.0%. Our human evaluation results further indicate the inherent limitations in aligning automated metrics and human preference, consistent with previous work (Xu et al., 2023b). In addition, we We also conduct comprehensive human studies, where users are allowed to interact with the system by modifying and improving the QA-based plans generated by the content planner. The results reveal that using such user-controlled plans can further improve model performance on those challenging questions.
Our contributions are summarized below:
• We propose TAPERA, a novel modular approach that combines a QA-based content planner and an execution-augmented reasoning module, aiming to enhance the faithfulness and interpretability of LLM-based systems in LFTQA.
• Human evaluation results on the FETAQA and QTSUMM datasets demonstrate that TAPERA achieves significant improvements. Moreover, we conduct detailed analysis on each module.
• Our user study reveals that user interaction with the plans generated by the QA-based content planner can further refine and improve model’s performance, highlighting the potential of userdriven customization in LLM applications.

heading: 2 Related Work
text: Reasoning over Tabular Data Table reasoning helps non-expert users interact with complex tabular data and has received significant attention. It has several tasks, such as table question answering (Pasupat and Liang, 2015; Iyyer et al., 2017; Zhong et al., 2017; Zhao et al., 2022a; Nan et al., 2022b; Zhao et al., 2023a,b,f), table fact verification (Chen et al., 2020b; Gupta et al., 2020), and table-to-text generation (Chen et al., 2020a; Cheng et al., 2022b; Nan et al., 2022a; Zhao et al., 2023c). Prior work on table reasoning mainly pre-trains models with joint table-text data and then finetunes over specific table-relevant tasks (Herzig et al., 2020; Liu et al., 2022b; Zhao et al., 2022b; Liu et al., 2022a; Jiang et al., 2022; Cheng et al., 2022a; Xie et al., 2022), and recent work shows LLMs also achieves competitive results (Touvron et al., 2023; OpenAI, 2023; Chen, 2023; Zhao et al., 2023e; Tang et al., 2024). To further advance the capabilities of LLMs in table QA tasks, research has primarily focused on two avenues: (1) Enhancing LLMs with code execution, which utilize code-optimized LLMs to generate executable programs to answer questions over tabular
data (Cheng et al., 2023; Nan et al., 2023; Kong et al., 2024) or to locate relevant table regions conditioned on a given question (Ye et al., 2023; Patnaik et al., 2024). While this approach is effective for deriving a single fact from tables for short-form answers, it struggles in LFTQA tasks that requires a more complex information synthesis. (2) Question decomposition, which breaks down questions into more fine-grained components, thereby simplifying the table reasoning required (Ye et al., 2023; Nan et al., 2023). However, existing methods rely on a single-round question decomposition, which might not always provide sufficient evidence to address the main question in LFTQA.
Content Planning for Text Generation Content planning is effective in capturing and controlling both the content and structure of generated text, particularly for longer outputs (You et al., 2023; Narayan et al., 2023; Huot et al., 2023). The process of plan-based text generation typically involves two distinct stages: planning an outline, and subsequently filling in the specific details. Previous work on table-to-text generation typically represents plans as a sequence of phrase keywords,
events, and their interactions (Puduppully et al., 2018; Su et al., 2021; Puduppully and Lapata, 2021; Li et al., 2023), aiming to improve the coherence and narrative flow of generated text. However, such plan representations are not ideally suited for LFTQA tasks, especially in the era of LLMs. This is because the primary objective of LFTQA is to extract and present question-relevant information accurately, rather than creating a narrative flow – an aspect where LLMs already excel (Goyal et al., 2023). This work adapts the idea of QA-based plan (Narayan et al., 2023; Huot et al., 2023; Liu et al., 2023a), utilizing question-answer pairs as intermediate plans for LFTQA tasks. Our work extends beyond previous work on QA-based plans to table-relevant tasks and uses LLMs to iteratively generate and refine the plans based on the feedback (i.e., answer) from table reasoning module.
Automatic Evaluation of LFQA One primary challenge for LFQA lies in accurately evaluating the long-form generation. Recent work (Krishna et al., 2021; Xu et al., 2023b; Krishna et al., 2023; Liu et al., 2023b) indicates that there is no existing automatic text generation metrics are predictive of
human preference judgments. We find the same issue when evaluating TAPERA in LFTQA, therefore, we mainly use human evaluation for results.

heading: 3 Long-form Table QA Task
text: In this section, we first formulate the LFTQA task, and then discuss the datasets used in our work.

heading: 3.1 Problem Formulation
text: The LFTQA task can be formulated as follows. The input is a user query Q, and a table T . The table T = W ∪ {Ti,j |i ≤ RT , j ≤ CT } has RT rows and CT columns, with W being the table title and Ti,j being the content in the (i, j)-th cell. The task is to generate a paragraph-long answer1 Y = (y1, y2, . . . , yn) given the query Q and source table T :
Y = argmax n∏
i=1
P (yi|y<i,Q,T ; θ), (1)
where θ denotes the parameters of a neural text generation model, and yi denotes the i-th tokens in the generated answer.

heading: 3.2 Evaluated Datasets
text: We conduct experiments and analysis on the test sets of FETAQA (Nan et al., 2022b) and QTSUMM (Zhao et al., 2023d). Table 1 illustrates the basic data statistics of these two datasets.
• FETAQA is a challenging dataset designed for long-form question answering over tables, with answers averaging 18.9 words. It requires models to fetch multiple entities from the given Wikipedia table, aggregate and reason over these
1In the query-focused summarization task (Dang, 2006), the answer Y is also named as summary of the user query.
Algorithm 1: QA-based Plan Generation and Refinement using the TAPERA Framework
1: Input: Complex question Q, table T 2: Output: Final answer A 3: 4: P0 = {q0,0, ..., q0,n} ▷ Content planner initializes a plan
(Section 4.1) 5: i← 0 ▷ Initialize iteration counter 6: 7: while Pi is not finalized do ▷ Iterative Refinement 8: for Every sub-question qi,j in Pi do 9: eoi,j ← Table reasoner generates program over
T to obtain qi,j-relevant execution output (Section 4.2) 10: 11: ai,j ← Answer generator generates the sub-
answer using (qi,j , eoi,j) (Section 4.3) 12: end for 13: 14: Pi ← Content planner updates a new plan based on
Pi (Section 4.1) 15: i← i+ 1 16: end while 17: 18: A ← Answer generator produces a final answer (Sec-
tion 4.3) 19: 20: return A
entities, and structure the inferred information to produce a coherent long-form answer.
• QTSUMM challenges text generation models to perform human-like reasoning and analysis on Wikipedia-sourced tables to produce tailored paragraph-length answers (i.e. summaries). In contrast to the FETAQA dataset, QTSUMM has longer output lengths, with answers averaging 68.0 words.

heading: 4 Long-form Table QA with TAPERA
text: In this section, we discuss our proposed framework TAPERA for LFTQA (Figure 2). Solving LFTQA requires complex table reasoning and consistent generation, both are challenging for existing LLM-based systems, and these challenges are more pronounced when they are entangled. Therefore, we propose a modular framework (Andreas et al., 2016) that decomposes the complex task into multiple sub-tasks, and adopt multiple sub-modules to tackle each sub-task. More specifically, our framework contains three main components: a QA-based content planner, a execution-based table reasoner, and an answer generator (Algorithm 1). Given a complex question Q, we generate a QA-based plan that includes a sequence of sub-questions (Section 4.1). Then for each subquestion, we first apply table reasoner to generate an executable program to obtain question-
relevant facts as program output (Section 4.2). Subsequently, the answer generator derives a subanswer from the program output (Section 4.3). However, due to the complexity of the task, we expect our framework might fail with a single attempt (Narayan et al., 2023). Thus, we adopt an iterative refinement approach that repeats the above process until the content planner finalizes a confident QA-based plan. The answer generator generates the final answer based on the finalized plan and sub-answers.

heading: 4.1 QA-based Content Planner
text: Previous work on QA-based content generation (You et al., 2023; Narayan et al., 2023; Huot et al., 2023) mainly relies on task-specific models. However, they often struggle to generalize to complex reasoning tasks. We instead adopt LLMs (OpenAI, 2023; Touvron et al., 2023) as a content planner for QA-based content plan generation and introduce an iterative refinement pipeline to enhance plan generation. We detail the workflow of content planner as follows:
Plan Generation Given a complex question Q over table T , the content planner decomposes the question into a series of step-by-step subquestions. This results into an initial plan P0 (with empty answers). The finalized plans for the FETAQA and QTSUMM contain an average of 2.6 and 4.7 QA pairs, respectively.
Iterative Plan Refinement The LLMs might not faithfully generate a consistent plan in a single attempt. Additionally, a sub-question might depend on the incorrect answer (from table reasoner and answer generator) to the previous subquestion, causing cascading errors. Therefore iterative refinement is necessary for consistency and accuracy. Specifically, in each iteration, after getting all sub-answers, we use CoT prompting to instruct LLMs to 1) determine whether the plan is finalized, and 2) refine and generate a revised plan Pi+1. If the LLMs determine that the current plan Pi is comprehensive enough to answer the given complex question, we terminate the iteration.

heading: 4.2 Execution-Based Table Reasoner
text: Given a sub-question in the current plan, the LLMs might not generate an accurate answer end-to-end. We therefore propose execution-based table reasoner that generates executable programs as an intermediate reasoning step to enhance table
reasoning. Unlike recent work in table-relevant tasks (Cheng et al., 2023; Nan et al., 2023; Kong et al., 2024) that uses SQL queries as programs, we use Python programs instead. This is mainly because 1) SQL queries are primarily designed to interact with structured data in relational databases but not most tables in the wild; 2) while SQL is efficient for straightforward data retrieval and basic operations, it becomes less intuitive and user-friendly when dealing with complex table reasoning tasks.
Executable Program Generation Given a subquestion and a table, we prompt execution-based table reasoner to generate a Python program. However, in practice, LLMs are likely to generate programs that are not executable. In such cases, we apply SELF-DEBUG prompting methods (Chen et al., 2023), providing LLMs with execution error message to instruct it to regenerate a program.

heading: 4.3 Answer Generator
text: The objective of LLM-based answer generator is two-fold: 1) to provide a sub-answer corresponding to the execution output from table reasoner in each plan-refinement iteration, and 2) to generate the final answer based on the finalized QA-based plan. The design principle of answer generator is not to perform new reasoning over tables, which may introduce hallucinations. Instead, we focus on transforming the output from table reasoner into an long-form answer that presents coherent narratives, ensuring the content consistency between the program and the long-form answer.
Generating Sub-Answer from Execution Output We first execute the generated Python program over the table to produce short-form answers, then we prompt answer generator to generate a sentencelong answer.
Generating Final Answer Based on Plan Once the QA-based plan is finalized (Section 4.1), the answer generator is applied to generate the final answer based on the plan and all generated answers for corresponding sub-questions.

heading: 5 Experiment
text: We next discuss the baselines, human and automated evaluations, implementation details and main experimental results.

heading: 5.1 Baseline Systems
text: We implement the following baseline systems:
• Zero-/One-shot Direct Answering. We prompt LLMs to directly output the final answer based on the given table and question.
• Chain-of-Thought (Wei et al., 2022) prompts LLMs to generate the reasoning chain in textual format before answering the question.
• Blueprint (Narayan et al., 2023) employs a finetuned model to generate a QA-based plan for long-form text generation. In our experiment, we adapt the Blueprint method by prompting LLMs to first generate the QA-based plan in a singleround, and then prompt them again to generate the final answer based on the plan.
• REFACTOR (Zhao et al., 2023d) employs rule-based methods to extract several questionrelevant facts from the table. These facts are then concatenated into the input context for the LLMs.
• Dater (Ye et al., 2023) prompts LLMs to break down the table into smaller, question-relevant segments, and then to decompose complex questions into sub-question-answer lists using intermediate SQL queries. Finally, it assembles these elements to compose the final answer.
We also develop three variants of TAPERA to research the effectiveness of each component:
• TAPERA without Table Reasoner. In this setting, we remove the component of execution-based table reasoner from TAPERA, while keeping the rest the same.
• TAPERA without Iterative Plan Refinement. In this setting, content planner is only applied to generate the initial plan, with each pending sub-answer fulfilled by table reasoner and answer generator in each iteration.
• TAPERA without Sub-Answer Generation. In this setting, we directly use the execution output from table reasoner as the sub-answer.

heading: 5.2 Human Evaluation
text: For human evaluation, we assess the following two dimensions:
• Faithfulness: A good answer should correctly answer the given question. It should not contain any unfaithful or hallucinated text.
• Comprehensiveness: A good answer should provide all the necessary information to answer the question. Moreover, it should avoid details that are consistent with tabular data yet irrelevant to the given question (Potluri et al., 2023).
Each generated answer was scored from 1 (worst) to 5 (best) for each criteria, with the final score averaged across different evaluators. It is worth noting that we do not evaluate aspects of fluency and coherence. This is because our preliminary study has found that all the evaluated GPTbased systems are capable of generating coherent and grammatically correct answers. This finding is also corroborated in the QTSUMM paper.

heading: 5.3 Automated Evaluation
text: Following QTSUMM, we adopt following popular automated evaluation metrics:
• BLEU (Papineni et al., 2002) computes the geometric average of the precision over output text’s n-grams. We use SacreBLEU (Post, 2018) for BLEU score calculation.
• ROUGE (Lin and Hovy, 2003) measures the word overlap between the candidate and reference text. We reported F1 score for ROUGE-L.
• METEOR (Banerjee and Lavie, 2005) is based on a generalized concept of unigram matching between the generated text and reference.
• BERTScore (Zhang et al., 2020) measures the similarity between the reference and generated text using contextual word embeddings.
• TAPAS-Acc (Liu et al., 2022a) is a referencefree metric that uses TAPAS (Herzig et al., 2020) fine-tuned on the TabFact dataset (Chen et al., 2020b) as backbone to evaluate the faithfulness of generation. However, since TabFact only contains single-sentence statements, the reliability of using TAPEX-Acc for evaluating long-form generated text remains questionable.
• AutoACU (Liu et al., 2023c) is a referencebased evaluation system. The A2CU first extracts atomic content units (ACUs) from the generation and then evaluates them against reference. A3CU is an accelerated version of A2CU that directly computes the similarity between two text without extracting ACUs, but with the similar evaluation target. We use F1 score of A3CU for evaluation.

heading: 5.4 Implementation Details
text: We use gpt-3.5-turbo-1106 as the backbone for all the evaluated systems. For LLM hyperparameter settings, we set temperature as 0.7, Top P as 1.0, and maximum output length as 512. Following QTSUMM, we represent tabular data in Markdown format. In practice, we have found that GPT-*
models can effectively understand this table format. To ensure fair comparison, for all modules in the baseline systems and TAPERA, we use a one-shot sample if in-context samples are required in the prompts (the first one provided in the official implementation when multiple samples are available). The prompt for each module of TAPERA can be found in the Appendix.

heading: 5.5 Main Experimental Results
text: Table 2 illustrates the results of human evaluation; and Table 3 and Table 7 show the automated evaluation results for QTSUMM and FETAQA, respec-
tively. We draw the following conclusions:
Effectiveness of TAPERA framework Among the evaluated systems, TAPERA achieves the best performance in both faithfulness- and comprehensiveness-level human evaluation, demonstrating the effectiveness of the entire framework. Specifically, TAPERA improves the best baseline in comprehensiveness, Blueprint (which is the worst among baselines in faithfulness) by 5.4%; Similarly, TAPERA improves the best baseline in faithfulness, Dater (one of the worst among baselines in comprehensiveness) by 5.0%. Moreover, the removal of any component from TAPERA leads to a degradation in both comprehensiveness and faithfulness, which underscores the necessity of each module in the system.
Mismatch between automated evaluation and human evaluation Despite receiving the best scores in human evaluation, TAPERA achieves relatively low scores under all automated evaluation metrics. This significant disparity between human evaluation and automated evaluation is consistent with prior work (Xu et al., 2023b), suggesting that future work should focus on developing more reliable automatic evaluation methods.

heading: 6 Human Analysis of TAPERA
text: To obtain a deeper insight into the effectiveness and potential of TAPERA, we conduct a human analy-
sis on the QTSUMM dataset. Specifically, we collect examples where TAPERA’s output received a human evaluation score of 3 or lower in either faithfulness or comprehensiveness (234 out of 1,078 examples). For these erroneous examples, we provide human evaluators with the question, table, and finalized QA-based plan. The evaluators are asked to identify which module (i.e., content planner, table reasoner, answer generator, or a combination thereof) is responsible for the errors. Among the 234 filtered examples, 180 and 77 are marked as errors attributable to the table reasoner and content planner modules, respectively. Meanwhile, only 12 are marked as errors attributable to answer generator, demonstrating that GPT-3.5 performs well in generating answers that are consistent with the execution results or plans. Table 4 and Table 5 present a comprehensive error analysis of table reasoner and content planner, respectively. In the following subsections, we discuss our analysis of these two modules.

heading: 6.1 Analysis of Table Reasoner
text: The modularity feature of TAPERA allows us to update the backbone of each component with more powerful LLMs (i.e., gpt-4-1106-preview). We investigate whether using more powerful LLMs as the backbone of table reasoner can effectively
handle those erroneous examples made by a GPT3.5. Specifically, for 180 erroneous examples, we repeat the same procedure with the GPT-4-based table reasoner, providing it with the same table and sub-questions that led to errors with GPT-3.5. We then manually check the execution results of the GPT-4-based table reasoner. We find that 82 out of 180 cases are successfully handled by GPT-4. This significant improvement showcases the vast potential of TAPERA with the continuous advancements in LLMs.

heading: 6.2 User Study of Content Planner
text: The plan-conditioned answer generation offers transparency and interpretability, as it enables users to identify which sub-plan or reasoning step leads to an undesired final answer. In real-world scenarios, it would be beneficial to develop systems that can satisfy users’ customized information needs through user-model interaction. Therefore, we conduct a user study where human evaluators are provided with content planner-caused erroneous examples. They are then asked to improve the plan by modifying the sub-questions (but not the subanswers). Then we use the user-modified plan as the initial plan (with all sub-answers set as pending) and apply the same iterative plan-refinement procedure to obtain the answer. This setting ensures that the quality of new answer is based solely on the plan itself, rather than table reasoner. We then manually check the quality of the newly generated answers by comparing them with the original ones. The human evaluation results show that user interaction with the content plan can improve answer generation in both comprehensiveness and faithfulness, as illustrated in Table 6.

heading: 6.3 Automated Evaluation Case Analysis
text: To gain a deeper insight into the failure cases of automated evaluation systems for LFTQA, we conducted detailed human analyses by exploring the
scenarios where automated evaluations fail. Specifically, we randomly sampled 100 model output pairs from TAPERA and 1-shot Direct on QTSUMM, where TAPERA received lower scores from at least 4 out of 6 metrics but achieved better results in human evaluations. We chose the 1-shot Direct system for comparison because it demonstrated the best performance in automated evaluations, as shown in Table 3. We meticulously analyzed these failure cases and identified four common scenarios where existing automated metrics tend to fall short: (1) system reveals additional reasoning-intensive information; (2) ground-truth answer is redundant; (3) original question is ambiguous; and (4) ground-truth answer is low-quality and misaligned the question. Detailed explanations for each failure scenarios are provided in Table 8 in Appendix.

heading: 7 Conclusion
text: This work proposes a new modular based framework TAPERA for LFTQA, which decomposes the complex QA task into three sub-tasks: QAbased content plan generation, execution-based table reasoning and consistent answer generation. Human evaluation on two benchmark datasets indicate that TAPERA outperforms strong baselines in both comprehensiveness and faithfulness. This is because TAPERA has stronger table reasoning capacities, and our final answer is consistent with the table reasoning module outputs. We believe this work provides insights for future work in advancing controllable text generation in complex tasks that are often entangled with reasoning challenges.
Limitations and Future Work
There are still some limitations in our work: (1) Due to computational resource constraints, we mainly test our framework on GPT series LLMs. This is because our preliminary analysis indicates that few-shot learning with open-source LLMs (e.g., Llama-2 7B) has low performance on LFTQA and struggles to generate executable programs. Future work could explore on fine-tuning these opensource LLMs on LFTQA (Zha et al., 2023; Zhang et al., 2024; Zhuang et al., 2024). (2) Our human study indicates that TAPERA is significantly more faithful over all baselines, and the final answer is always consistent with the program output. However, our answer generator module is based on prompting LLMs and cannot avoid hallucination
in theory. Replacing answer generator with toolaugmented LLMs (Schick et al., 2023) is a promising direction to further tackle hallucination. (3) The findings and conclusions of this paper are mainly based on human evaluation, as none of the current automatic evaluation metrics are predictive of human preference judgments. A possible future direction is to explore fine-grained evaluation metrics that focus on different aspects of generated long-form answers. Moreover, we believe that LLM-based evaluation can also be explored for the LFTQA task (Liu et al., 2023d; Min et al., 2023; Jiang et al., 2024).

heading: Acknowledgement
text: Chen Zhao is supported by Shanghai Frontiers Science Center of Artificial Intelligence and Deep Learning, NYU Shanghai. We are grateful to the Google TRC program for their support. We would also like to thank the anonymous reviewers and area chairs for constructive feedback.

heading: Future Work
text: We believe this work provides insights for future work in advancing controllable text generation in complex tasks that are often entangled with reasoning challenges. Limitations and Future Work
There are still some limitations in our work: (1) Due to computational resource constraints, we mainly test our framework on GPT series LLMs. This is because our preliminary analysis indicates that few-shot learning with open-source LLMs (e. g. , Llama-2 7B) has low performance on LFTQA and struggles to generate executable programs. Future work could explore on fine-tuning these opensource LLMs on LFTQA (Zha et al. , 2023; Zhang et al. , 2024; Zhuang et al. , 2024). (2) Our human study indicates that TAPERA is significantly more faithful over all baselines, and the final answer is always consistent with the program output. However, our answer generator module is based on prompting LLMs and cannot avoid hallucination
in theory. Replacing answer generator with toolaugmented LLMs (Schick et al. , 2023) is a promising direction to further tackle hallucination