Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Authors: Jinyang Li
Nan Huo
Yan Gao
Jiayi Shi
Yingxiu Zhao
Ge Qu
Bowen Qin
Yurong Wu
Xiaodong Li
Chenhao Ma
Jian-Guang Lou
Reynold Cheng
Venue: ICML 2025 poster (2025)
Keywords: Tabular Data Analysis
Large Language Models
Conversation
Primary area: applications->language_speech_and_dialog
TL;DR: We propose a method of construction of multi-turn tabular data analysis datasets which are scalable and efficient along with two implementation of two agent modes for comprehensive LLMs.

Abstract: Conversational Tabular Data Analysis, a collaboration between humans and machines, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic conversational logs for tabular data analysis hinder comprehensive quantitative evaluation of Large Language Models (LLMs) in this task. To mitigate this issue, we introduce **CoTA**, a new benchmark to evaluate LLMs on conversational tabular data analysis. **CoTA** contains 1013 conversations, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, **CoTA** is constructed by an economical multi-agent environment, Decision Company, with few human efforts. This environment ensures efficiency and scalability of generating new conversational data. Our comprehensive study, conducted by data analysis experts, demonstrates that Decision Company is capable of producing diverse and high-quality data, laying the groundwork for efficient data annotation. We evaluate popular and advanced LLMs in **CoTA**, which highlights the challenges of conversational tabular data analysis. Furthermore, we propose Adaptive Conversation Reflection (ACR), a self-generated reflection strategy that guides LLMs to learn from successful histories. Experiments demonstrate that ACR can evolve LLMs into effective conversational data analysis agents, achieving a relative performance improvement of up to 35.14%.

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Jinyang Li Nan Huo
Yan G Jiayi Shi
Bowen Qin Yurong Wu 2 Jian-Guang Lou? Reynold Cheng'
Yingxiu Zhao Ge Qu
Xiaodong Li Chenhao Ma
Abstract
1. Introduction
The exponential growth of big data calls for accessible data analysis techniques that cater to wide range of applications, such as healthcare, games, and entertainment (Khanbabaci et al., 2018; Han et al., 201 1; Fayyad et al., 1996). Recently, the development of LLM agents (Liu et al., 2024b; Xu et al., 2024; Zeng et al., 2024; Xu et al., 2023; Deng et al., 2024; Si et al., 2023) has attracted lot of attention. They are capable of understanding natural language queries, as well as generating codes for data manipulation and visualization, through reasoning (Huang & Chang, 2023; Wei et al., 2022; Wang et al., 2024a) and tool calls (Li et al., 2023b; Huang et al., 2024b; Qin et al., 2024). Among the vast types of data available, tabular data stands out as one of the most prevalent and interpretable formats organized by rows and columns (Hu et al., 2024; Wu et al., 2024; Liu et al., 2024a).
Conversational Tabular Data Analysis, collab- oration between humans and machines, enables real-time data exploration for informed decision- making. The challenges and costs of collecting realistic conversational logs for tabular data anal- ysis hinder comprehensive quantitative evaluation of Large Language Models (LLMs) in this task. To mitigate this issue, we introduce COTA, a new benchmark to evaluate LLMs on conversa- tional data analysis. CoTA contains 1013 conver- sations, covering 4 practical scenarios: NORMAL, ACTION, PRIVATE, and PRIVATE ACTION. No- tably, COTA constructed by multi-agent en- vironment, DECISION COMPANY. This environ- ment ensures efficiency and scalability of generat- ing new conversational data. Our comprehensive study, conducted by data analysis experts, demon- strates that DECISION COMPANY is capable of producing diverse and high-quality data, laying the groundwork for efficient data annotation. We evaluate popular and advanced LLMs in COTA, which highlights the challenges of conversational tabular data analysis. Furthermore, propose Adaptive Conversation Reflection (ACR), a self- generated reflection strategy that guides LLMs to learn from successful histories. Experiments demonstrate that ACR can evolve LLMs into ef- fective conversational tabular data analysis agents, achieving relative performance improvement of up to 35.14%. Code can be found at https: //tappilleee crossingg github.ion io/
Tabular data analysis agents (Li al., 2024b; Zha al., 2023; Zhang al., 2023a) provide automatic workflow based on user queries. However, the dynamic and uncertain nature of real-world analysis hinders effective human-agent conversation (De Vries et al., 2020; Yan et al., 2023; Wang et al., 2024b), thus users may need to adjust their analysis strategies based on intermediate results (Yan et al., 2023; Yao et al., 2020). For example, in Figure 1, the notable opponents s could refer to a variety of interpretations, such as the opponents with the highest wins, the most frequent opponents. Towards this end, a comprehensive benchmark is indispensable for gauging their capability in conversational user engagement within data analysis scenarios.
In this paper, we introduce Conversational Tabular data Analysis (CoTA), new benchmark for evaluating LLM agents in conversational tabular data analysis tasks. COTA is designed to simulate real-world data analysis scenarios, where users converse with LLM agents to generate codes for data exploration and decision making. It includes 1013 user- machine conversations with 1162 user intents, spanning four practical scenarios, as shown in Figure 1) Normal mode refers the scenario where all questions and user require- ments are explicit, agents can answer questions by referring only to table contents and dialog histories. This would eval- uate fundamental capabilities of agents in handling data analysis tasks; 2) Action mode represents that agents must
"Equal contribution 'School of Computing and Data Science, The University of Hong Kong Microsoff "The Chinese University of Hong Kong. Shenzhen Allibaba Group Beijing Academy of Artificial Intelligence 'Xiamen University. Correspondence to: Reynold Cheng <ekcheng wee.eeess.hhhhkh. Chenhao Ma <machen- hao www.ccuk.oooo edu.c..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Are Large Language Models Ready for Multi-Ture Tabular Data Analysis?
any patreens
Are there any patterns or trends in the top 10 player's performance grass courts, oaponents they strugglc against? Return me list of their names.
Are there any patieern or frends in the top 10 playar's per foromennce on arass courts, such as: notable opoonents they struggle against? Upponencs who have won at least 3 matches are notabie opponents. Return me a litt of their rssmess
Ar there any patterns or trends the
top 10 player's performance on grass opporents strugalo againsi? Opporamns have notable opponents. Rieuum
top 10 player's performente mentaa on grans courts. such as notable osponents they seruggle againet! Return me list their names.
as: notable
won
Under SpeciFit
More Importantlyy can you what opponents notable??
if Find the opponents who have vion agaiist the cop 10 players opponents wonegrrss matchesigrans (enrche//Whneere iinttop 10 playe in omest Falss [Winner]] couutuc
To answer these questions, we'll analyze the dataset to find notable opponents the top 10 players struggle against, identify areas of improvement for discuss other factors that may impact their performance on grass courts. More importantly, can you make sure what kind opporents are notable?
from decision, company import read ctv. file, count unique, values, check-olement eloments, Inn Jist, reset, Indox, convertt tor wwr.paaraayy Lonvert_ng, lot
Clartflerttes
Opponenti nés who have won at least matches are nosable opporentts.
player,
# Find the opoonents who have won against the top 10 players appanents, wwnneooun Luique valuesdg rass, maadc es check elmenkss, In InJisttar ass matche/ (Winner) top, 10, player, pames] -- Fahal] Winner)
V Find opponents who wen against the top 10 plaerss
4 Fillor the opporenti who have son kest matches against the top 10 players
copooenns wonecount uniquee vslinigg ass, matchesfchack alammnts in, Feilgy mottches WWinner]
notable ocponents
ooponents_ wan coppoopnnntts,.eee won >n 3]
top 10 player namess Falsel] D...
Coastratior
Action (Claritication)
Private
Privete Action (clarificationl
Figure 1. An overview of the four conversation modes in COTA, illustrated by relevant aspects of the associated codes or actions.
infer diverse user intents first to deliver satisfactory results. For example, they need to interpret ambiguous terms such as notable opponents by asking questions and gener- ate appropriate responses based on user clarification. This tests their ability respond to complex and dynamic user queries during conversations; 3) Private mode is designed to examine the true semantic parsing capability of agents when encountering unseen packages provided by users (Zan et al., 2022); and 4) Private Action mode unifies the chal- lenges of Private and Action modes, more closely reflecting real-world data analysis. Answer types can be summarized into two categories: 1) Code Generation, which can test whether the agent can correctly interpret the user query and generate the corresponding code for data analysis, and 2) Multiple-Choice questions, which can evaluate the ability of agents to understand the returned results being executed codes and provide users with appropriate insights.
proach are rigorously evaluated through 10 real data analysis experts outside annotations. The result shows that DECI- SION COMPANY can scale and maintain data quality.
We evaluate the popular advanced LLMs and LLM agents CoTA. The results underscore the challenges of con- versational data analysis and fuel the need for more ad- vanced LLM agents that can handle diverse user intents and feedback. To further evolve the LLMs towards effective conversational data analysis agents, we propose Adaptive Conversation Reflection (ACR), which guides LLM agents to learn from successful history via self-generated pseudo logic reflection. Our experiments demonstrate that ACR can significantly enhance the performance of LLMs, in which Slaude-3.5-Sonnel can gain relative improvement of 35.14% % compared to its model base, offering an insight into how to improve L.L. agents in conversational tabular data analysis.
Constructing high-quality conversational data analysis datasets poses significant challenges. First, obtaining re- conversational logs for data analysis can be difficult due to proprietary restrictions and privacy concerns (Sun et al., 2022; Choi et al., 2019). Second, the conventional approach of using crowdsourcing for dataset construction, particularly for complex. high-quality conver- sational tasks, demands substantial human expertise, mak- ing both time-consuming and costly (Yu et al., 201 9a; Li et al., 2023a; Guo et al., 2021; Li al., 2024d; Zhang al., 2023c). Furthermore, such methods are prone to data leak- age risks. To address these challenges, we introduce a novel multi-agent sandbox, DECISION COMPANY, enabling the efficient creation of CoTA through human- sandbox collab- oration under stringent execution- based evaluation script comprising test cases. In this simulated sandbox, 4 GPT-4 agents interact with each other under continuous human expert supervision to simulate realistic data analysis tasks while maintaining high data quality. The reliability and potential biases of such
2. Preliminaries
alistic
Task Formulation. Conversational tabular data analy- sis with LLM agents involves sequence of user-agent turns, [(u,, a1), (2),,,))... (un, an)], where each turn (u, a) consists of a user query u and an agent response a. Queries can be instructions or feedback, while responses can be code snippets or selected answers. Dialogs start with Wi and end with a.,. Given the current user query u, all previous user-agent history H from turn I to t 1, and sam- pled table contents T, the agent should act, such as asking for clarification, and generate an answer a, fo (u,. H, T), where fo refers to the agent built based on LLMSs with model weights 0. This setup allows CoTA to evaluate conversa- tional agent performance in a static and systematic manner.
Action Types. We identify 6 common actions during conversations, each serving as a specific evaluation mode in COTA. The actions in CoTA include Update_ __ode, which addresses user requests for bug fixes or refinements;
collaborative ap-
2

---

Are Large Language Models Ready for Multi- Turn Tabular Data Analysis?
Fast_Fail, which alerts users insufficient data fac- tual errors; and Cl air f cation, where agents seek ad- ditional information for under-specified queries. To reduce user impatience and long dialogue issues, Best_Guess allows agents to make assumptions based on data, domain knowledge, and commonsense, though it risks incorrect guesses. The Plot_QA action helps users understand plot- derived insights, while Insight_Mining involves sum- marizing executed results to aid in decision-making, evolv- ing agents into comprehensive data analysis tools. Detailed examples and interpretations are in Appendix J, and evalua- tion methods for each mode are in Appendix L.
of client into series of specific data analysis questions with well-defined conditions. Each question is provided by an expected result type, such dataframes, lists, or various plot types, which helps reduce question ambiguity and ease the pressure on evaluation metries (Yin et al., 2023; He et al., 2024; Zhang et al., 2023b). The dialogue between the agents further refines the questions with specific conditions. For example, as depicted in Figure 2 B.4, the client Garcia's question could be further elaborated on the basis of his following responses, making all questions more answerable. In particular, Agent Garcia, fully cognizant of his persona created in B.2, adds the condition grass, reflecting his London location. This implies that the role-playing aspect of the agent can be instrumental in generating a wider range of questions that are both diverse and reasonable (Li et al., 2024a; Park et al., 2023).
3. The COTA Construction
3.1. Sandbox Construction
The construction of CoTA is mainly based on the AI Agent Sandbox, DECISION COMPANY, as depicted in Figure 2.
Conversation Log Generation. Following the plan discus- sion, the conversation simulation phase begins. Here, the AI Chatbot Agent takes the lead, executing the data analysis plan agreed on during the previous stage. The AI Chatbot Agent converses with the Data Scientist Agent to answer a series of questions defined in the plan by generating codes and analyzing returned results.
The first step in the construction of COTA is the acquisition and pre- processing of data. We collect open-source tables from Kaggle, a popular data science platform. These tables cover 18 analysis topics under 5 common domains, namely ATP Tennis, Credit Card, Fast Food, Laptop Price, and Melbourne Housing, as detailed in Ap- pendix B.3. Given any of the tables, Administrator Agent will generate column meanings and value illustrations.
Data Acquisition & Preprocessing.
3.2. Human-Sandbor Annotation
While the DECISION COMPANY can generate a wealth of data analysis conversations in a zero-shot prompting manner, human supervision is indispensable ensure the quality of the data set annotation (Lu et al., 2023; Zhuo et al., 2025). Therefore we engaged two groups of experts: 1) 6 annotators who are also authors, each with over 10 years of experience in data analysis. 2) a team of 3 more experienced expert data scientists for final data collection decision.
Client Persona Generation. The construction of COTA proceeds to the generation of client personas. These per- sonas with specific tasks and topies related to the data are created by the Administrator Agent, Each persona is defined by Name, Location, Job, and Background with diverse range of interests and backgrounds.
Annotation Phase. Each annotator begins by Simulation of Analysis Scenarios in DECISION COMPANY to select scenarios and continue process until arriving Conversation Log Generation for annotation. An illustrative starting prompt is provided in Figure 17, Appendix Q. Beginning with this prompt. the AI Chatbot generates code in response to various analysis questions. Each annotator then executes the generated code, verifies its outputs for correctness, fixes any bugs, and confirms that the results align with the anticipated output types, ensuring that the code accurately addresses the posed questions. This finalized code referenced as the ground-truth code for subsequent benchmarking. Additionally, the annotator generate evaluation scripts (eval py) consisting of a series test cases for testing model-generated code. After running these tests, the annotator executes the code and feed the results back to the AI Chatbot, which generates free-form text analysis statements (e.g.. "The credit card application rate of people with 4 4-year employment is 73.5% higher than
Simulation of Analysis Scenarios. Then, the Ad- ministrator Agent "interviews" each Client Agent to ob- tain their Scenario description, including Scenario Name and the Goal. In preliminary trials, we observe that Client Agents often proposed very uniform topics, which limited the diversity of the dialogue. To address this, each Client Agent is instructed to propose 3 scenar- ios, from which human annotators selected most appro- priate one scenario based on its diversity and coherence with the tabular data. For instance, in B.3 of Figure 2, the scenario Court Condition Impact was chosen because Pl Player Performance Analysis was less diverse since multiple Clients tend to generate it as the top choice, and Sponsor Attract ion required extensive information beyond the table contents, leading to many unanswerable questions.
Plan Discussion. In this process, the Data Scientist Agent engages with the Client Agent to convert the requirements
3

---

Are Large Language Models Ready for Multi- Turn Tabular Data Analysis?
Tabular Data Resource
Agent Sandbox
Interaction Log Annotation
Hoo the bep pieyetss aethom rgg cerc offert Wouide ooderor ai wurkioo ancond
Assulai Lets create ride decaltara cocsaut beaciis hasal reaches
you iippont
Sory aleas returt rasula thise players
for tna salle lens) (iftre po susilliii up. (i--
Tour
Carlos Garcia
SCEMARIO There potentaal
ryon and Faii maric bords
Horee ooes gancionn
there ana specli aathon
Spenuor Atraction
Court:
Serfice
Eor has uniemter mnaeegng plaves cfficte
Thanks sas atantion gras
Round
NAME Dount Condition in
betwven
Figure 2. The construction pipeline of CoTA by the Al Agent Sandbox DECISION COMPANY. 7 denotes human intervention
those with no employment"), Each annotator reviews and corrects these statements, converting them into multiple-choice question formats for more reliable and objective evaluation. All corrected code and free-form text analysis are stored static user-AI conversation history, for reference when annotating the following turns. Each conversation is presented as series of clean, noise-free turns. Annotators then inject meaningful action types, shown in Section 2, drawing on established dialogue research and their practical experience in data analysis. For private library code annotations, each annotator will follow three steps to annotate: 1) summarizing packages from original code; 2) refactoring and converting them to user customized functions with light human supervision; 3) regenerating codes via customized functions with human calibration. Further details are provided in Appendix Q
Table Data characteristics
STATISTIC
NUMBER
Total conversations
1013
clear conversations
280
action conversations
478
private lib. conversations
206
private act. conversations
49
# of private lib functions
137
Answer Types
# of code generation answers
590
# of multi-choice answers
423
Quality & Cost
inter-agreement
92.78
AVG # of turns
14.15
Validation Phase. Once the conversation annotations are complete, annotators swap their annotated data for cross- validation. The validation process involves three key steps: (1) adding more test cases to the evaluation scripts to check the code; (2) assuming the role of a "red team" to inten- tionally introduce errors into the code, then confirming the evaluation scripts correctly flag these errors; and (3) crit- ically judging the suitability of the chosen scenarios and action types. Annotators resolve any identified errors disagreements through discussion. If they cannot reach con- sensus, the expert team steps in to refine or, if necessary, eliminate the disputed items.
regarding question and code length, we utilize t tiktoken to compute the number of tokens for each dataset. As shown in Table 2, COTA includes comprehensive evaluation settings across private library, multi-turn, and multi-modal conversations. Besides, the complexity of this dataset, reflected by the long questions and their associated code snippets, is amplified by the inclusion of multi-inten queries. These queries, encapsulating multiple intents within a single question, require a versatile array of computational strategies for effective handling. For example, the query, "Please provide histogram plots and mean for employment status of credit card appli ixplican cants.t.......... demands both data visualization and statistical evaluation. Finally, COTA con- tains 1013 data analysis conversations. The inter-agreement of 92.78 for initial cross-validation promises the high quality of the dataset.
4. Data Statistics & Metrics
4.1. Dataset Statistics
Figure provides key statistics for our dataset, while Table 2 offers comparison between CoTA and other datasets related to data analysis. To ensure fair comparison

---

Are Large Language Models Ready for Multi-Tun Aurti-Turn Tabular Data Analysis?
Table Comparison of COTA and other data analysis datasets. The first 5 datasets are single-tur data analysis sets featuring both SQL. and Python codes. The following 3 benchmarks are multi-turn or conversational data analysis datasets. COTA represents challenging dataset data analysis with more comprehensive settings. represents that the end code Python. means the target code is SQL.
Dataset
# Q Intents
# Toks. /Q
Toks. / Code
Code Type
Analysis
Multi-Tune
Private Lib
Multi-modal
Evaluation
Huaa (Chen et al,, 2021)
164 | 164
60.9
24.4
Test Cases
MRPP (Austin al., 2021)
974
Test
Spider (Yu et 2018)
1034 1034
12.4
18.3
Acc- EM
BIRD 2023a)
1534 1534
14.5
Ace VES
DS-1000 (Lai et al., 2023)
1000 1000
282,4
42.1
Test Cases + SFC
SparC. (Yu al,, 2019h)
1203 1203
CoSQI. (Yu et al., 2019t)
1008 | 1008
13.1
31.4
Ace
ARCADE (Yin et al., 2023)
1066
Fuzzy
COTA
1013| 1162
207.5
164.7
Acc AceR
Table Acceptance ratio human evaluation on general metrics of the dataset quality, the higher the better. The table reports the percentage of samples considered qualified or being accepted for each metric.
Conversation
Eval Scripts
Annotation
Conversation Coherence
Scenarios Diversity and Reasonableness
Conversation Topic Coherence
Ethics and Bias Representation
Conversation Naturalness
Evaluation Scripts Quality
Evaluation Scripts Scalability
w/o human
0.19
0.46
0.17
0.41 1.00
0.67
w/ human
0.97
0.96
0.93
0,95
0.98
0.94
4.2. Evaluation Metrics
Fost Fail (17.5%)
Pler (14.11)
Accuracy (Acc). Acc is a metric that evaluates the ability of agents in generating codes that execute correctly swer multiple-choice questions accurately. It is defined the proportion of instances whose predicted outputs match the ground-truth output, examined by evaluation scripts. For a given dataset with N instances, where C2 is the expected outcome (either execution result or correct answer) and C₂ is the predicted outputs for the 7t instance, Ace is calculated as follows:
Clarification (15.2%)
Insight Mining (39.25)
Best (7.1%)
Update Code (6. 9%)
Acc N U
(1)
Figure 3. Distribution of ACTION Mode in COTA. It contains 6 common types conversational data analysis tasks.
where I is an indicator function that returns 1 if C.- = Cj, and 0 otherwise.
Accuracy with Private Library Recall (AccR). Recog- nizing the importance of accurately leveraging specific user- defined libraries in code generation, we extend Acc to in- clude a recall-based adjustment for instances involving pri- vate libraries. This ensures that AccR not only evaluates the direct accuracy of code execution and question answering but also the inclusion and correct usage of private library functions. AccR can be computed as follows:
where R(C, E.) quantifies the recall rate of relevant library functions in the predicted code. F(C,) and F(C;) denote the set of private library functions in the reference codes and the set actually utilized by agents in the predicted codes, respectively. The final score would be weighted sum of Acc and AccR. We conduct an in-depth analysis of the impact of AccR on Private mode evaluation in Appendix M.
N
5. Dataset Quality Evaluation
AccR
Σ I(C. = C.) -.,C,
(2)
N
To ensure the data quality of COTA and the reliability of our proposed human-sandbod data generation, we conduct a comprehensive human evaluation focusing on both gen-
R(Ci, Ōi) (P(C))F((C))
[F(Ci)
5

---

Are Large Language Models Ready for Multi-Ture Tabular Data Analysis?
eral and action-specific aspects by selecting 500 samples randomly which is approximately 50% of the full dataset. To do this, we invite 10 experts with extensive data analysis experience outside authors to review the dataset. Details of evaluation instructions can be found in Appendix O.
powered by GPT-4-Turbo, tests the models' ability to gener- ate codes after clarifying details when facing under-specific questions. The chart-to-table (Liu et al., 2023a) converter mitigates the prevalent issue of LLMs' inability to compre- hend plots by converting them into tables. Descriptions of these tool sets can be found in Appendix H.I.
General Metries. Following (Hu et al., 2024), we conduct human evaluation on more NL metrics about reasonableness and coherence across turns of conversations. The results are shown in Table 3, where w/o human refers to the data which is fully annotated by LLM agents. And w/ human means the data in the CoTA, which is annotated by human- sandbox collaboration. From the table, shows, after human involvement, the acceptance ratio rises to approximately 0.95, indicating that the involvement of human annotators who are professionals in data analysis is sufficient to ensure the quality of the auto-genert seooogeeeeatedd dataset, thus demonstrating the balance of trade-off in DECSION COMPANY between efficiency and quality of complex data annotation workflow.
6.2. Reasoning
Reasoning is a critical process for transitioning LLMs into data analysis agents (Huang & Chang, 2023). In COTA, we incorporate two primary reasoning methods for code generation and multiple-choice answers. The first is the Chain-of-Though (COT) prompting technique (Wei et al., 2022), which enhances the complex reasoning abilities of LLMs by dividing the reasoning path into multiple steps. The second method is CodeAct, which enables models to make decisions by generating reasoning traces and codes in an interleaved manner (Yao et al., 2023; Wang et al., 2024a).
6.3. Adaptive Conversation Reflection (ACR)
Action-wise Metrics. Human evaluation is also conducted with a focus on the actions. Figure 4 illustrates the con- sensus among experts that all actions in CoTA are both necessary and commonly observed in real-world data anal- ysis scenarios. All instructions and details can be found in Appendix O.
Successful conversations are important since they encapsu- late the logic necessary to meet user requirements and en- sure correct steps of analysis or code generation. Motivated by this, we propose the Aapptiv Conversation Reflection (ACR) approach to enable data analysis agents learn from successful user-code histories through two-step process.
10256
Eudie:
Far
Inic
Pseudo Code Logic Generation. First, given the last pre- vious history (u,-1; 14-1), when t > 1, we prompt the data analysis agent to reflect and generate its underlying logic m = fe(u;-1: -1). where fe refers agent based LLMs with parameter O. Also, (æ; y) represents two elements and y are concatenated in the prompt. In our work, we consider the pseudocode to be m, as it serves as an intermediate logic between natural language queries and codes.
Accept Ratio
95%
00%G
885%
80%
RessossNattee
Figure 4. Results of human evaluation action- wise metrics of the dataset quality.
Re-Org One-Shot Reasoning. Second, we re-organize them into self-generated one-shot example with the or- der: P:-1 = (u,-1 {m,- 1; a which represents the scenario where the input u;-1 is given, the agent should generate a logic m,-1 first, then generate answers a;-1- Finally, the data analysis agent can learn from P:-1 to first generate logic m, = fo (P:-1; U1) and generate an answer a, = [u;; (u;; m,) in the current turn When t = 1, we keep the same reasoning method of the original agent. Appendix G provides detailed example for further illustration.
6. Evolving LLMs Towards Conversational Data Analysis Agents
In this section, discuss our approach of equipping L.MSs as data analysis agents with tools and reasoning. We then introduce our self-generan reflection strategy, ACR, to enhance their performance in conversational settings.
6.1. Toolkit
7. Experiments
Our tool sets include an executor, a user simulator, and chart-to-table converter. The executor provides an environ- ment for models to observe real-time feedback on their inter- mediate code results (Xie et al., 2023; Wang et al., 2024b). The user simulator (Wang et al., 2024b; Yan et al., 2023),
7.1. Setup
Models. Our experiments primarily involve popular
L.MSs that are capable of generating code and following
6

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Table Overall results LLMs base, agent, and inter-age agent modes on the COTA dataset. Pri-Act refers to private library action evaluation mode. Agent refers to agents using COT in code type and CodeAct in choice type. Agent-R involves textual reflection. Multi-Agent uses multiple LLM agents collaborate. Inter-Agent refers our proposed method incorporating ACR.
Conversation Mode
Answer Type Choice
Model
Overall
Normal
Action
Private
Pri-Act
Code
Mistral-7B
5.9
16.4
1.5
1.9
4.8
16.7
9.1
Mistral-8 x 7B
17.2
23.6
3.7
1.9
11.6
23.9
16.1
CodeLJama--34B
28.2
19.7
2.6
16.1
19.8
17.4
GPT-4-Turbo
30.8
18.5
6.4
3.7
18.9
18.5
18.8
Claude-3-Opus
22.6
28.7
2.6
5.6
14.5
29.0
19.8
GPT-4-32k
31.1
25.5
7.5
0
19.7
25.6
21.9
Llamm-3.3-700
33.6
32.4
6.7
3.7
21.4
32.7
25.5
Claude-3.5-Sonnet
37.9
36.3
10.5
9.3
25.6
36.6
29.6
+ Agent
37.3
43.1
10.1
7.4
25.6
43.5
32.1
+ Agent-R
36.2
48.5
13.5
9.3
26.8
49.0
34.9
+ Multi-Agent
43.5
45,2
16.1
11.1
31.2
45.5
36.4
+ Agent
42.9
52.8
18.0
14.8
32.4
53.3
40.0
complex human instructions since this is basic requirement in data analysis. Therefore, we investigate performance of 4 families of models, covering Mistral (Jiang al., 2023),
Moda
Agost
Overall O Performance (%)
35
Inin-Agest
LLama (Roziere et al., 2023; Dubey et al., 2024), Claude, and GPT (Achiam et al., 2023) models.
20
Implementation. The implementations could be divided into 5 settings: Model Ba refers the LLM itself
10
without reasoning and tool calls. 2) Agent mode involves multiple tool usage and reasoning. (COT + CodeAct) 3) Agent-- mode involves textual reflection for Agent. 4) Multi-Agent mode implemented through the coor- dinated collaboration of multiple specialized agents, each responsible for distinct set of functions inspired by (Liu al., 2023d; Hong al.). 3) Inter-Agent mode incor- porates ACR as described in Section 6.3 beyond the AGENT. Further details can be found in Appendix I.
Codo
Opus
Figure 5. Visualization performance Code-Llama-3444 GPT-4-Turbo, Claude-3-opus, GPT-4-32k, and Claude-3.5-Soma with base, agent, and inter_agent versions.
and applies these in the following code, thereby improving readability and logic. This is particularly beneficial in tasks related to data-analysis code generation. Such tasks often require the composition of API functions, which demands a profound understanding of the context and the ability to ex- tract common patterns into reusable functions. By defining and reusing symbolic functions, CodeLlama can streamline complex contexts, making them more logical, which an advantage for resolving complex tasks (Gu et al., 2023). 4) Mlaude3.5--Somel and Llama-3.3-70B perform better than GPT-4 on base mode proving that our benchmark is not overfitting to the GPT family of models.
7.2. Experimental Results
Table 4 illustrates the comprehensive performance of popu- lar LLMs and developed agent modes based on Claude-3.5- Sonnect on the CoTA. From the results, we can deduce the following: 1) CoTA is a challenging benchmark in which SOTA method only achieves a score 40.0%, leaving large room for improvement. 2) Despite the performance of GPT-4-Turbo being nearly on par with GPT-4-32K in code generation, its overall performance still falls short of GPT-4-32K. This indicates that beyond code writing, under- standing results, and analysis are equally important. Fortu- nately, the comprehensive settings of CoTA can assist users in selecting models for data analysis tasks. 3) We observe that lama frequently defines functions automatically
LLM Agent Performance. We also implement different agent modes for several mainstream LLMs. As shown in Figure 5, most models with Agent version outperform their

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Table 5. Definitions and Examples of main error types.
Error Type
Definition
Example
Key Error (23%)
Refers the instance where the model imag- ines reasonable but non- existent column name to retrieve the data table.
to find clients who have stable employment. We can consider stable employment as those with employment duration of 4 years more.
Question:
Gold: geth__redi__leee duuaationnhhhh_ccrroo htap://hhhhhheet_ornrieett _
['employment" Error: high_ fetoh__red____tteeeeeeeee duratiion(highh Brrllighh_reii_eee cwwedttleong_ddeeee duration _ongitttttnnnnnioonnnnnnnoo ('employment duration") == 'x>=4']
'x>=4"]
Lazy Assumption (39%)
Refers to instance where the model tends to assume some middle re- sults have already been prepared.
Question: Now, need to know the potential impact of the up- dated odds on the later rounds of the tournament.
Error: Assuming 'updated_ode _odds_dddff df' already created and contains the updated odds
updated odds df pickle.load( lupdated_odds oddds odds_dff.kk')
Bad Instruction Following (49%)
Refers to the instance where the model can not follow instructions well, leading to the fallure of answering questions.
Question: Please answer the multi-choice question ... .... you will need to generate Python code which can assist yourself to answer the question in this step.
Error: 'D. None of above' Explanation: There no specific test or condition to check ...
Update__ode Code
Fast_Fas Fast_Fail queries and executing Update_Code ac- tions. However, it falls short in the Best__uest action when compared to the Agent. We note that ACR tends to make agents overly tractable in re-org one-shot example P:-1 and current generated logics m, If P:-1 and m, do not contain instructions on making assumptions, agents tend to select None of Above. This observation suggests that excessive reliance on historical data may hinder the inher- ent ability of models to conjecture based on instant user behaviors. Therefore, striking a trade-off between user-code history exploration and real-time user conversation, espe- cially in under-specific questions, is crucial for improving LLM agents' performance in conversational settings.
50
40
30
FastFall
esight_Mining
10
Clanfication
HoLoA
Test, Cuesx
Model Base
Agent
Inter_Agent
Error Analysis. We conducted an error analysis by sam- pling 200 error cases from each LLMs to gain insights into conversational data analysis as shown in Table 5. The er- rors were categorized into three main types: (1) Key Error (23 (23%), where the model incorrectly matches column names in the provided data table or references nonexistent values; (2) Lazy Assumption (39 %). where the model assumes that intermediate results or states are already available or saved on disk without verification; and (3) Poor Instruction Following (49 %),, where the model fails to strictly follow instructions, resulting in incorrect answers. Furthermore, Figure 7 demonstrates that ACR effectively reduces errors in each category. The special common errors in private li- brary mode are: (1) LLMs fail to retrieve useful packages, and usually retrieve excessive packages instead (67%); and (2) some L.MSs show conservative usage of private libraries to minimize errors, which violate the user intent (39%). A more detailed error analysis can be found in Section N.
Figure 6. Visualization of the performance of GPT-4-32k across various categories in ACTION Mode. The comparison includes base, agent, and inter agent versions.
base version, highlighting the crucial role of tools and rea- soning in enhancing the performance of LLMs under com- plex tasks (Liu et al., 2024b; Xie et al., 2023). Also, all mod- the Inter-Agee mode with ACR. This indicates that the underlying logic of successful conversation histories is instrumental in guiding LLMs to become more proficient data analysis agents in conversational settings.
els exhibit obvious improvements
Fine-Grained Results on ACTION Modes. Figure 6 pro- vides comparative evaluation of GPT-4 model across various ACTION modes detailed in Section 2. The con- versational data analysis agent, Inter-Agent, obvi- ously outperforms in most areas, especially in managing
8

---

Are Large Language Models Ready for Multi-Ture Tabular Data Analysis?
Key Enor
40
Lazy Assumptton
Ratio of Error Types
Bad Instraction Following
Key Enor (later-Agess)
30
Lazy Assumption (inter Ageut)
Bad Instruction Fellowing (ntte-Agenn)
20
10
Code-LLama- 34B
GPT-4-Turee
Claude-3 3-pus
GPT-4432k
Claude-3. 3.5de-----Soneett
Figure 7. The visualization of different error types across different models and settings.
8. Related Work
Multi-Agent Environments for Data Generation. LLMs have proven to be effective in constructing multi-agent en- vironments for automatic data generation. For instance, Lu et al. (2023) and Ding et al. (2023) simulate dialogs for QA and text generation tasks. Also Li et al. (2023b) generates data about API calls using multi aulti--gentt environments. This because LLM agents can simulate believable human actions when placed in environment with dynamically updating knowledge and memory (Park et al., 2023). Inspired by this, we created DECISION COMPANY to generate conversa- tion log data for data analysis with more behaviors. Unlike prior dataset- -eanrrrggnn studies, we pioneer a benchmark for assessing conversational data-analysis agents.
Large Language Models for Data Analysis. The use of LLMS for data analysis has been topic of interest in recent years. LLMs powered by In-Context Learning (Yang et al., 2023; Dai et al., 2023; Dong et al., 2024) have been employed in various data analysis tasks, such as SQL query generation (Pourreza & Rafeei, 2024; Gao et al., 2024; Lei et al., 2024; Zhang et al., 2024; Gu et al., 2024; Wang et al., 2025; Qu et al., 2024; Li al., 2024c), pandas or python code generation (Jain al., 2024; Chen et al., 2024; 2023a; Li al., 2024b; Zha et al., 2023; Zhang et al., 2023a; Zheng et al., 2024b), and data visualization (Chen et al., 2023b; Huang et al., 2024a). However, most of these works focus on single-turn setting, where the user query is explicit and does not require any conversation or clarification. Recently, there has been a growing interest in conversational data analysis, where the user intents may need to be clarified or refined through conversations (De Vries al., 2020; Yan et al., 2023; Wang et al., 2024b).
9. Annotation Cost
Similar to BIGCODEBENCI (Zhuo et al., 2025) and SPI- DER 2.0 (Lei et al., 2025), our annotators are recognized via authorship rather than direct payment. Working exclu- sively in a crivacy-preserving controlled interface, they logged a total of 3,677 minutes, incurring $71.39 LLM usage, and annotator cost of $1,392.66 with $0.37875 per minute for professional data-analysis experts (Kazemitabaar et al., 2024; Liu et al., 2023b), yielding overall bench- mark development cost of $1,464.03, or roughly $1.45 .55 per COTA instance, which is over four times cheaper than BIRD- SQL (Li et al., 2023a) (36.13) and TABLEBENCH (Wu et al., 2024) ($6.00) despite COTA'S 's higher complexity.
The development of bench- marks for data analysis tasks has been crucial factor in driving the progress of LLMs in data science. Existing benchmarks can be broadly categorized into single-turn and multi-turn benchmarks. Single-turn benchmarks, such as HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), Spider (Yu et al., 2018), BIRD (Li et al., 2023a), Text2Analysis (He et al., 2024), DABench (Hu et al., 2024) and DS-1000 (Lai et al., 2023), focus generating code snippets or closed-form insight summaries for data anal- ysis given a single user query. To explore conversational nature of real-world data analysis scenarios, where the user intent may need to be clarified or refined through conversa- tional communication, several multi-turn benchmarks have been proposed, including CoSQL (Yu et al., 2019a), and ARCADE (Yin et al., 2023). However, these benchmarks are primarily focused on code generation and do not cover other aspects of data analysis, such as data visualization and understanding based on intermediate results. Our work ex- tends the existing literature by introducing a new benchmark for evaluating L.M agents in conversational data analysis.
Data Analysis Benchmarks.
10. Conclusion
We introduce CoTA, a new benchmark for evaluating LLM agents in conversational data analysis tasks. COTA is con- structed via scalable and expert-reoonniee high-quality multi-agen environment, DECISION COMPANY, and covers a wide range of practical scenarios. We evaluate data analy- sis agents based on popular L.MSs on COTA, highlighting the challenges of conversational tabular data analysis. We also propose ACR, an effective reflection strategy for con- versational data analysis agent evolution. Our experiments demonstrate that ACR can significantly enhance the perfor- mance of LLM agents, paving the way for future research.

---

Are Large Language Models Ready for Multi- Turti-Turn Tabular Data Analysis?
Acknowledgement
Choi, I., Ahmadvand, A., and Agichtein, E. Offline and online satisfaction prediction in open-domain con- versational systems. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp. 122-,,,,, 2019.
Reynold Cheng, Jinyang Li, Nan Huo, and Ge Qu are supported by the Hong Kong Jockey Club Charities Trust (Project (00920140), the University of Hong Kong (Project 2409100399), the HKU Outstanding Research Student Su- pervisor Award 2022-23, and the HKU Faculty Exchange Award 2024 (Faculty of Engineering). Bowen Qin was sup- ported by National Science and Technology Major Project (Project 2022ZD01 16306). Chenhao Ma was partially sup- ported by NSFC under Grant 62302421, Basic and Applied Basic Research Fund in Guangdong Province under Grant Ant Group through CCF-Ant Research Fund, Shenzhen Research Institute of Big Data under grant SIF20240004, and the Guangdong Provincial Key Laboratory of Big Data Computing, The Chinese University of Hong Kong, Shenzhen.
Dai, D., Sun, Y., Dong. L., Hao, Y., Ma, S., Sui, Z., and Wei, F. Why can GPT learn in-context? language mod- els secretly perform gradient descent meta-optimizers, In Rogers, A., Boyd-Grabber, J., and Okazaki, N. (eds.), Findings of the Association for Computational Linguis- tics: ACL 2023, pp. 4005-4019, Toronto, Canada, July 2023. Association for Computational Linguisties.
2023A151501 1280,
De Vries, H., Bahdanau, D., and Manning, C. Towards ecologically valid research on language user interfaces. arXiv preprint arXiv:2007. 14435, 2020.
Deng, X., Gu, Y., Zheng, B., Chen, S., Stevens, S., Wang, B., Sun, H., and Su, Y. Mind2web: Towards general- ist agent for the web. Advances in Neural Information Processing Systems, 36, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work. none which we feel must be specifically highlighted here.
Ding, N., Chen, Y., Xu, B., Qin, Y., Hu, S., Liu, Z., Sun, M., and Zhou, B. Enhancing chat language models by scaling high-quality instructional conversations. Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6 10. 2023, pp. 3029-3051. Association for Computational Linguistics, 2023.
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint 0877023003.08774. 2023.
Dong, Q., Li, L., Dai, D., Zheng, C., Ma, J., Li, R., Xia, H., Xu, J., Wu, Z., Chang, B., Sun, X., Li, L.,, and Sui, 7. A survey on in-context learning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1128, Miami, Florida,
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q. et al. Program synthesis with large language models. arXiv preprint arXiv: (0f.:2008.077333 2021.
USA, November 2024, Association for Computational Linguistics.
Chen, M., Tworek, J., Jun, H., Yuan, Q.. Pinto, H. P. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv: 007.03374, 2021.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle,
A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. 2024.
Chen, X., Aksitov, R., Alon, U., Ren, J., Xiao, K., Yin, P., Prakash, S., Sutton, C., Wang, X., and Zhou, D. Univer- sal self-consistency for large language model generation. arXiv preprint ar arXiv:2311.17311 2023a.
Fayyad, U. M., Piatetsky-Shaping G., and Smyth, P. From data mining to knowledge discovery in databases. Al Mag., 17(3):37-54, 1996.
Chen, X., Lin, M., Schärli, N., and Zhou, D. Teaching large language models self-debug. In The Twelfth International Conference Learning Representations, 2024.
Gao, D., Wang. H., Li, Y., Sun, X., Qian, Y., Ding, B., and Zhou, Text-to-sql empowered by large language models: A benchmark evaluation. Proc. VLDB Endow., 17(5), 2024.
Chen, Z., Zhang, C., Wang, Q., Troidl, J., Warchol, S., Beyer, J., Gehlenborg, N., and Pfister, H. Beyond gener- ating code: Evaluating gpt on a data visualization course. arXiv preprint wxxx:23366002 029/4, 2023b.
Gu, Y., Deng. X., and Su, Y. Don't 't generate, discriminate: A proposal for grounding language models to real-world environments. In Rogers, A., Boyd-Graber, J. L., and
10

---

Are Large Language Models Ready for Multi-Ture Tabular Data Analysis?
Okazaki, N. (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics ( (Vol- ume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 9928.994.. Association for Computa- tional Linguistics, 2023.
Huang, K.-H., Zhou, M., Chan, H. P., Fung, Y., Wang. Z., Zhang, I.., Chang, S.-F., and Ji, H. Do LVLMs understand charts? analyzing and correcting factual errors in chart captioning. In Findings of the Association for Compu- tational Linguisties: ACI. 2024, pp. 730-749, Bangkok, Thailand, 2024a. Association for Computational Linguis- tics.
Gu, Y., Shu, Y., Yu, H., Liu, X., Dong, Y., Tang, J., Srini- vasa, J., Latapie, H., and Su, Y. Middleware for LLMs: Tools are instrumental for language agents in complex environments. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 7646-7663, Miami, Florida, USA, 2024. Association for Computational Linguistics.
Huang, Q. Vora, J., Liang, P., and Leskovec, J. Bench- marking large language models as ai research agents. In NeurlPS 2023 Foundation Models for Decision Making Workshop, 2023.
Huang, Y., Shi, J., Li, Y., Fan, C., Wu, S., Zhang, Q. Liu, Y., Zhou. P., Wan, Y., Gong, N. Z., and Sun, L. Metatool benchmark for large language models: Deciding whether to use tools and which to use. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024b.
Guo, J., Si, Z., Wang, Y., Liu, Q., Fan, M., Lou, J., Yang, Z., and Liu, T. Chase: A large-scale and pragmatic chi- nese dataset for cross-database context-dependent text-to-
sql. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting of the Associa- tion for Computational Linguistics and the 11th Interna- tional Joint Conference on Natural Language Processing, ACL/JJCLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pp. 2316-2331. Association for Computational Linguistics. 2021.
Jain, N., Zhang, T., Chiang, W., Gonzalez, J. E.,, Sen, K., and Stoica, I. Llm-assisted code cleaning for training accurate code generators. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 1- 2024, 2024.
Han, J., Kamber, M., and Pei, J. Data Mining: Concepts and Techniques, 3rd edition. Morgan Kaufmann, 2011. ISBN 028-0123814791
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., et al. Mistral 7b, arXiv preprint arXiv:2310.06825 2023.
He, X., Zhou, M., Xu, X., Ma, X., Ding, R., Du, L.,, Gao, Y., Jia, R., Chen, X., Han, S., Yuan. Z., and Zhang, D. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. In Thirty- Eighth AAAI Conference on Artificial Intelligence. AAAI 2024, pp. 18206-18215 AAAI Press, 2024.
Jimenez, C. E. Yang, J., Wettig, A., Yao. S., Pei, K., Press, O., and Narasimhan, K. R. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2023.
Kazemitahaar, M., Williams, J., Drosos, L., Grossman, T., Henley, A. Z., Negreanu, C., and Sarkar, A. Improving steering and verification in ai-assisted data analysis with interactive task decomposition. In Proceedings of the 37th Anmual ACM Symposium on User Interface Software and Technology, pp. 1-19, 2024.
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J.. Zhang, C., Wang, Z., Yau, S. K. S., Lin, Z., et al. Metagpt: Meta programming for a multi-agent collabora- tive framework. In The Twelfth International Conference on Learning Representations.
Hu, X., Zhao, Z., Wei, S., Chai, Z., Ma, Q., Wang, G., Wang, X., Su, J., Xu, J., Zhu, M., Cheng, Y., Yuan, J., Li, J., Kuang, K., Yang, Y., Yang, H., and Wu, F. Infiagent- dabench: Evaluating agents on data analysis tasks. In Forty-first International Conference on Machine Learn- ing, ICML. 2024, Vienna, Austria, July 21-27, 2024. Open- Review.net, 2024.
Khanbabaei, M., Sobhani, F. M., Alborzi, M., and Radfar, R. Developing an integrated framework for using data mining techniques and ontology concepts for process improvement. J. Syst. Softw., 137:78-95, 2018.
Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W., Fried, D., Wang, S. I., and Yu, T. DS-1000: natural and reliable benchmark for data science code gen- eration. In Krause, A., Brunskill, E.,, Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Con- ference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii USA, volume 202 of Proceed- ings of Machine Learning Research, pp. 88319-18345 PMLR, 2023.
Huang, J. and Chang, K. C. Towards reasoning in large language models: A survey. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.). Findings of the Associa- tion for Computational Linguistics: ACL 2023, Toronto, Canada, July 941 2023, pp. 1049-1065. Association for Computational Linguisties, 2023.

---

Are Large Language Models Ready for Multi- Turn Tabular Data Analysis?
Lei, F., Liu, Q. Huang, Y., He, S., Zhao, J., and Liu, K. S3Eval: A synthetic, scalable, systematic evaluation suite for large language model. In Proceedings of the 2024 Con- ference of the North American Chapter the Association for Computational Linguistics: Human Language Tech- nologies (Volume I: Long Papers), Mexico City, Mexico, 2024. Association for Computational Linguistics.
(Volume I: Long Papers), Bangkok, Thailand, 2024d. Association for Computational Linguistics.
Liu, F., Fisenschlos, J. M., Piccinno, F., Krichene, S., Pang, C., Lee, K., Joshi, M., Chen, W., Collier, N., and AI- tun, Y. Deplot: One-shot visual language reasoning by plot-to-table translation. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 10381-10399, Association for Computational Linguistics, 2023a.
Lei, F., Chen, J., Ye, Y., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., Zhong, V., Xiong, C., Sun, R., Liu, Q. Wang, S., and Yu, T. Spider 2.0: Evaluating lan- guage models on real-world enterprise text-to-sql work- flows. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., and Gordon, A. D. "what it wants me to say": Bridging the abstraction gap between end-user programmers and code- oode-generating large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-31, 2023b.
Li, G., Hammoud, H., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36, 2024a.
Liu, T., Xu, C., and McAuley, J. Repobench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Represen- tations, 2023c.
Li, H.. Su, J., Chen, Y., Li, Q., and ZHANG, Z.-X. Sheet- copilot: Bringing software productivity to the next level through large language models. Advances in Neural In- formation Processing Systems, 36, 2024b.
Liu, X., Wu, Z., Wu, X., Lu, P., Chang, K.-W., and Feng, Y. Are Ilms capable of data-based statistical and causal rea- soning? benchmarking advanced quantitative reasoning with data. Findings of the Association for Computational Linguisties: ACL 2024, 2024a.
Li, H., Zhang. J., Liu, H., Fan, J., Zhang, X., Zhu, J., Wei, R., Pan, H., Li, C., and Chen, H. Codes: Towards building open-source language models for text-to-sql. Proc. ACM Manag. Data, 2(3), 2024c.
Liu, X., Yu, H., Zhang, H., Xu, Y., Lei, X., Lai, H., Gu, Y., Ding, H., Men, K., Yang. K., Zhang, S., Deng, X.. Zeng, A., Du, Z., Zhang, C., Shen, S., Zhang, T., Su, Y., Sun, H., Huang, M., Dong, Y., and Tang, J. Agentbench: Evaluating llms as agents. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024b.
Li, J., Hui, B., QU, G., Yang. J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., Zhou, X., Ma, C., Li, G., Chang, K., Huang, F., Cheng, R., and Li, Y. Can LLM already serve database interface? a Blg bench for large-scale database grounded text-to-SQLa In Thirty- seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023a.
Liu, Z., Yao, W., Zhang, J., Xue, L., Heinecke, S., Murthy, R., Feng, Y., Chen, Z., Niebles, J. C., Arpit. D., et al. Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05400 2023d.
Li, M., Zhao, Y., Yu, B., Song, F., Li, H., Yu, H., Li, Z., Huang, F., and Li, Y. Api-bank: A comprehensive bench- mark for nool-augmetted llms. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Confer- ence on Empirical Methods in Natural Language Process- ing. EMNLP 2023, Singapore, December 6-10, 2023, pp. 3102-3116. Association for Computational Linguistics, 2023b.
Lu, B.RR., Haduong, N., Lee, C.-.., Wu, Z., Cheng, H., Koester, P., Utke, J., Yu, T., Smith, N. A., and Osten- dorf, M. Dialgen: collaborative human-Im generated dialogues for improved understanding of conversations. arXiv preprint arXiv:2307.07047 2023.
Li, R., Patel, T., and Du, X. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint ar arXiv:2307.02762. 2023c.
Mialon, G., Fourrier, C., Wolf, T., LeCun, Y., and Scialom, T. Gaia: benchmark for general ai assistants. In The Twelfth International Conference on Learning Represen- lations, 2023.
Li, Y., Hui, B., Xia, X., Yang, J., Yang, M., Zhang, I.,, Si, S., Chen, L.-H., Liu, J., Liu, T., Huang, F., and Li, Y. One-shot learning as instruction data prospector for large language models. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics
Park, J. S., O O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive
12

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
simulacra of human behavior. In Follmer, S., Han, J., Steimle, J., and Riche, N. H. (eds.), Proceedings of the 36th Annual ACM Symposium User Interface Software and Technology. UIST 2023, San Francisco, CA, USA, 29 October 2023- 1 November 2023, pp. 2:1-2:22. ACM, 2023.
on Computational Linguistics, Abu Dhabi, UAE, 2025. Association for Computational Linguistics.
Wang, X., Chen, Y., Yuan, L.,, Zhang, Y., Li, Y., Peng, H., and Ji, H. Executable code actions elicit better Ilm agents. ICML, 2024a.
Pourreza, M. and Rafici, D. Din-sql: Decomposed in- context learning of text-to-sql with self-correction. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang. X., Wang, Z., Liu, J., Chen, Y., Yuan, L.,, Peng, H., and Ji, H. MINT: Evaluating LLMs in multi-turn inter- action with tools and language feedback. In The Twelfth International Conference on Learning Representations, 2024b.
Qin, Y., Hu, S., Lin, Y., Chen, W., Ding, N., Cui, G., Zeng, Z. Zhou, X., Huang, Y., Xiao, C., Han, C., Fung, Y. R., Su, Y.. Wang, H., Qian, C., Tian, R., Zhu, K., Liang, S., Shen, X., Xu, B., Zhang, Z., Ye, Y., Li, B., Tang, Z., Yi, J.. Zhu, Y. Dai, Z., Yan, L.. Cong, X., Lu, Y., Zhao, W., Huang, Y., Yan, J., Han, X., Sun, X., Li, D., Phang, J., Yang, C., Wu, T., Ji, H., Li, G., Liu, Z., and Sun, M. Tool learning with foundation models. ACM Comput. Surv., 57(4), 2024. ISSN 0360-0300.
Wang. Y., Le, H., Gotmare, A., Bui, N., Li, J., and Hoi, S. CodeTT5: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, 2023, Association for Computa- tional Linguistics.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of- ofothhttht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35: 24824-24837, 2022.
Qu, G., Li, Li, B., Qin, B., Huo, Ma, C., and Cheng, R. Before generation, align it! a novel and effective strategy for mitigating hallucinations in text-to-SQI. generation. In Findings of the Association for Computational Linguis- ties ACL 2024, Bangkok, Thailand and virtual meeting, 2024. Association for Computational Linguistics.
Wu, X., Yang, J., Chai, L., Zhang, G., Liu, J., Du, X., Liang, D., Shu, D., Cheng, X., Sun, T., et al. Tablebench: A comprehensive and complex benchmark for table ques- tion answering. arXiv preprint arXiv:2408.09/ 7400099974140 2024.
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv:2308. /2950, 2023.
Xie, T., Zhou, F., Cheng, Z., Shi, P., Weng, L., Liu, Y., Hua, T. J., Zhao, J., Liu, Q., Liu, C., et al. Openagents: An open platform for language agents in the wild. arXiv preprint arXiv:2310.1063
Si, S., Ma. W. Gao, H., Wu, Y., Lin, T.-E., Dai, Y., H., Yan, R., Huang, F., and Li, Y. SpokenWOZ. A large-scale speech-text benchmark for spoken trientree dialogue agents. In shirty-seventh Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2023.
2023.
Xu, B., Liu, X., Shen, H., Han, Z., Li, Y., Yue, M., Peng, Z., Liu, Y., Yao, Z., and Xu, D. Gentopia.aä ai: A collabo- rative platform for tool- augmented Ilms. In Feng, Y. and Lefever, E. (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023 SSysem Demonstrations, Singapore, De- cember 6-10, 2023, pp. 237-245. Association for Com- putational Linguistics, 2023.
Sun, H., Xu, G., Deng, J., Cheng, J., Zheng, C., Zhou, H., Peng, N., Zhu, X., and Huang, M. On the safety of conversational models: Taxonomy, dataset, and bench- mark. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 3906-3923, 2022.
Xu, Y., Su, H., Xing, C., Mi, B., Liu, Q., Shi, W., Hui, B., Zhou, F., Liu, Y., Xie, T., Cheng, Z., Zhao, S., Kong, L., Wang, B., Xiong, C., and Yu, T. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Pan, Y., Wu, Y., Haotian, H., Weichuan, L.,, Liu, Z., and Sun, M. De- bugBench: Evaluating debugging capability of large lan- guage models. In Findings of the Association for Com- putational Linguistics: ACL 2024, Bangkok, Thailand, 2024. Association for Computational Linguistics.
Wang, B., Ren, C., Yang, J., Liang, X., Bai, J., Chai, L., Yan, Z., Zhang, Q.- Yin, D., Sun, X., and Li, Z. MAC- SQL: A multi-agent collaborative framework for text-to- SQL., In Proceedings of the 3/st st International Conference
Yan, H., Srivastava, S., Tai, Y., Wang, S. Yih, W., and Yao, Z. Learning to simulate natural language feedback for interactive semantic parsing. In Rogers, A., Graberr, J. L.,, and Okazaki, N. (eds.), Proceedings of the 6
13

---

Are Large Language Models Ready for Multi- Turn Tabular Data Analysis?
Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACI. 2023, Toronto, Canada, July 9-14, 2023, pp. 3149-3170. Association for Computational Linguisties, 2023.
China, November 3-7, 2019, pp. 1962-1979. Association for Computational Linguistics, 2019a.
Yu, T., Zhang, R., Yasumaga, M., Tan, Y. C., Lin, X. V., Li, S., Er, H., Li, I., Pang. B., Chen, T., Ji, E., Dixit, S., Proc- tor, D., Shim, S., Kraft, J., Zhang, V., Xiong, C., Socher, R., and Radev, D. R. Spare: Cross-domain semantic parsing in context. In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 451 1511-11 1-4523. Association for Computational Linguistics, 2019b.
Yang. J., Hui, B., Yang, M., Li, B., Huang, F,, and Li, Y. Iterative forward tuning boosts in-context learning in language models. arXiv preprint 23X10 2023.
Yao, S., Zhao, J.. Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Zan, D., Chen, B., Lin, Z., Guan, B., Wang, Y., and Lou, J. When language model meets private library. In Goldberg, Y.. Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pp. 277-288. Association for Computational Linguistics, 2022.
Yao, 7., Tang, Y., Yih, W., Sun, H., and Su, Y. An imitation game for learning semantic parsers from user interaction. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.), Pro- ceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 6883-6902. Association for Computational Linguisties, 2020.
Zeng, A., Liu, M., Lu, R., Wang, B., Liu, X., Dong, Y., and Tang, J. AgentTuning: Enabling generalized agent abilities for LLMSs In Findings of the Association for Computational Linguistics: ACL 2024, pp. 3053-3077, Bangkok, Thailand, 2024. Association for Computational Linguistics.
Yin, P., Li, W., Xiao, K., Rao, A., Wen, Y., Shi, K., How- land, J., Bailey, P., Catasta, M., Michalewski, H., Polozov, O., and Sutton, C. Natural language to code generation in interactive data science notebooks. In Rogers, A., Boyd-Graber, J. L.,, and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto Canada. July 9-14, 2023, pp. 126-173. Association for Computational Linguistics, 2023.
Zha, L., Zhou, J., Li, L.,, Wang, R., Huang, Q. Yang. S., Yuan, J., Su, C., Li, Li, X., Su, A., et al. Tablegpt: Towards unifying tables, nature language and commands into one gpt. arXiv preprint arXiv:2307.0800
2023.
Zhang, W., Shen, Y., Lu, W., and Zhuang, Y. Data-copilot: Bridging billions of data and humans with autonomous workflow. arXiv preprint arXiv:2306.07209 2023a.
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang. Li, Zi, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. R. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text- to-sql task. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J. (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pp- 3911-3921. Association for Computational Linguisties. 2018.
Zhang. Y., Henkel, J., Floratou, A., Cahoon, J., Deep, S., and Patel, J. M. Reactable: Enhancing react for table question answering. Proc. VLDB Endow, 17(8): 1981- 1994, 2024.
Zhang, Z., L., Gao, Y., and Lou, J. CRT-QA: A dataset of complex reasoning question answering over tabular data. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singa- pore, December 6-10, 2023, pp. 2131-2153. Association for Computational Linguistics, 2023b.
Yu, T., Zhang. R., Er, H., Li, S., Xue, E., Pang, B., Lin, X. V., Tan, C., Shi, T., Li, Z., Jiang, Y., Yasunaga, M., Shim, S., Chen, T., Fabbri, A. R., Li, Z., Chen, L., Zhang, Y. Dixit, S., Zhang, V., Xiong, C., Socher, R., Lasecki, W. S., and Radev, D. R. Cosql: conversational text- to-sql challenge towards cross doos--dmminn natural language interfaces to databases. In Inui, K., Jiang, J., Ng. V., and Wan, X. (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, E.NIIP-IJCNIP 2019, Hong Kong,
Zhang, Z., Li, X., Gao, Y., and Lou, J.-G, CRT-QA: A dataset of complex reasoning question answering over tabular data. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 2131-2153, Sin- gapore, December 2023c. Association for Computational Linguistics.
14

---

Are Large Language Models Ready for Multi-Tun Turn Tabular Data Analysis?
Zheng, L.,, Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z Zhuang, Y., Lin, Z-, Li, Z.,, Li, D., Xing, E.,, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36, 2024a.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Openroddlnterpreteer: Integrating code generation with execution and refinement. In Find- ings of the Association for Computational Linguistics: ACL 2024, pp. 12834-12859, Bangkok, Thailand, 2024b. Association for Computational Linguistics.
Zhuo, T. Y., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., Brunner, S., Gong. C., Hoang, J., Zebaze, A. R., Hong, X., Li, W., Kaddour, J., Xu, M., Zhang, Z., Yadav, P., and et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
15

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Table Comparison of CoTA popular I.I Agent-related datasets, CoTA represents challenging dataset in data analysis with comprehensive settings.
Q
eval public available
multi-modal
private lih
multi turn
conversation trajectory
data creation
output type
GAIA (Mislon et al,, 2023)
466
from- from-seratch
Text
ResearchAgent (Huang al., 2023)
semi
Hllei-Types
SWE-bench (Jimenez al., 2023)
2290
from-scratch
Patch
AgenlBench (Liu al., 2024b)
1091
semi
Multi- Types
RepoRench (Liu 2023c)
1669
Code
DebugBench (Tian et al., 2024)
4253
frem-scratch
Code
CoTA
1013
from-scratch
Code/Choice
A. Background Knowledge
Requirements of client. It naturally refers to specific data analysis tasks or questions that users want to accomplish, expressed in natural language.
Result Type. It typically refers to the format or nature of the output produced from analyzing data. Common result types
include: "dataframes, lists, or various plot types."
User intent.
It represents the underlying analytical goal or purpose behind a user's query.
B. Data Resource
B.1. COTA
Our CoTA data is available under the lisense CC BY-SA 4.0.
B.2. Kaggle Tabular Data
The tabular data that we used to create COTA are following open-source licenses: 1) Public Domain: Public Domain Mark
2) CC-BY: Creative Commons Attribution 4.0 International.
B.3. Data Distribution
The Figure 8(a) visualizes our covered topics and domains.
C. Comparison with Agent-related Benchmarks
Besides the benchmark comparisons in Table 2, we list more popular LLM Agent-related benchmarks comparisons in Table 6. Each column means: (1) #Q: This column represents the unique identifier or number assigned to each benchmark or dataset. (2) eval public available: This column specifies whether the evaluation metrics of the benchmark or dataset publicly available for use. (3) multi-modal: This column shows whether the benchmark supports multi-modal data, meaning it can handle multiple types of input data (e.g., text, images) simultaneously. (4) private lib: This column indicates whether the benchmark or dataset includes private library. (5) Multi-Turn: This column specifies whether the benchmark supports multi-tur conversations, which conversations that involve multiple exchanges steps. (6) conversation trajectory: This column indicates whether the benchmark involves conversation trajectories, which track the sequence and flow of conversations over time. (7) data creation: This column describes the method of data creation for the benchmark. Frromsscratc" means the data was created anew specifically for the benchmark, while "semi" indicates that the data was created using a mix of new and existing data. (8) output type: This column specifies the type of output produced by the benchmark or dataset. Examples include "Text," " "Multi-Types.." "Patch," "Code," and "Code/Choice," indicating the nature of the outputs generated during evaluations.
"https://creet www.eiiiwwooooooooooo..oooo httpppssr/cccttteeeeeooooooooooo baae/byyyaaaaaaaaeaa /eedd.
16

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
D. Model Descriptions
In this section, we provide an overview of the various models used in our research. These models include both widely recognized and otate-of-the-art LLMs that have been instrumental in advancing NLP tasks.
(1) 1iistral-7B-inttuutttt:1::: Mistral-7B is a powerful LLM designed to handle diverse NLP tasks. It is known for its efficiency in terms of parameter size while maintaining high performance. The 7 billion parameters enable it to process and generate human-like text effectively.
(2) Claude-3.5-Sonnet- Gaaude-..5-oome is an advanced version of the Claude series of LLMs This iteration brings improvements in both accuracy and processing speed, making a suitable choice for complex language understanding and generation tasks.
(3) Mistral-8 7B-instruct-v01: Mistral-8 7B represents a collection of 8 models, each with 7 billion parameters. This ensemble approach allows for enhanced performance through model averaging and provides robustness in generating more accurate results across different tasks.
(4) CodeLLama-3344 34B-Insttuct--ff
CodeLLama-344D is a specialized model focused on code-related tasks. With 34 billion parameters, it excels in code generation, code completion, and understanding programming languages, making it a valuable tool for software development and code-related research.
(5) GPT-4-Turbo (1pt-4-0125prreviev)) GPT-4-Turbo a highly optimized version of the GPT-4 model. It offers faster inference times and improved efficiency while maintaining the high-quality output that GPT-4 is known for. This model is particularly useful for applications requiring quick responses without compromising quality.
(6) Claude-3-Opus: Claude-3-Opus is the latest in the Claude series, bringing substantial improvements in language understanding and generation. It integrates advanced techniques to enhance its contextual comprehension and generation capabilities, making it top choice for sophisticated NLP tasks.
(7) L 3.n-stt-:: Llama-3.3-70BB is LLM with 70 billion parameters. This model is designed to tackle the most challenging NLP tasks, providing unparalleled performance in terms of accuracy and coherence in text generation.
(8) GPT-4-32k: GPT-4-32k is variant of the GPT-4 model with an extended context window of 32,000 tokens. This extended context window allows it to handle long-form content more effectively, making it ideal for applications requiring extensive context retention and understanding.
D.1. Model Shortcoming Analysis
Long-Context Challenges. The challenge of handling long-contexts is considerable in COTA, especially for models with shorter maximum input lengths. Models such as Codellama-34B. which has a maximum input length of 16k, are particularly affected. For example, it is essential for LLMs to access all private function descriptions and codes for effective code generation with retrieved functions. The statisties shows that the average number of prompt tokens for PRIVATE 15.7k, and notably. 20.9% of their prompts surpass the 16k length.
Instruction Following. Our experiments reveal that Claude-3.5-Soneee requires minimal effort in prompt design due to its exceptional ability to follow human instructions. To be specific, only 3.2% of their results deviate from the provided instructions. However, other models exhibit higher proportion of unexpected result types. For instance, extracting generated codes or answers from an LLM proves to be extremely challenging since it often embeds the answer in the middle of outputs rather than at the end as defined. We also observe that GPT-4-Turbo tends to generate longer codes in any setting. While this characteristic enhances its performance in code generation, it also results in 58.6% of the code generated during CodeAct reasoning being non-executable, thereby leading to incorrect answers. Furthermore, CodeLlama-3AA 4ama-344-Innstuuc exhibits a lack of robustness when faced with longer more complex prompts. With the addition of COT, the performance of CodeLlama significantly drops from 26.5% with simpler instructions to 19.5% in NORMAL code generation.
E. Dynamic History Combination
E. History Relational Database (H-RDB)
From all the User-Al conversation data shown in Figure 8(a), we split the User-Al conversation into several single-turn user queries and AI answers stored in a relational database, indexed by the conversational order as shown in figure 8(b). This
17

---

Are Large Language Models Ready for Multi-Tun Turt Tabular Data Analysis?
Awareness
Local
Event
Holiday
Seasion Promotions
rtant- Saved
Auto
Product
Pinarcing
Expeo
Snawcahie
Healts
tnd
Temees
Erom
pre resienationn
Food
Improved
Digital
Banking
Services
Credit Card
Buttee--frund fruunde
Consolidation
tearch
Company -aidd Laptop Upgradi adt
Decining
Laptop
Investigation
Housing
Gaming Laptop
Residential
Stssaimable
Expension
Development
Portable
erotessio prptesiiocal
ATP Tennis
Housing
Affordaable
Laptop
Laurcht
Tournemen Perfommnn Arciyas
Upatahing
Inittative
Colc Windrraml
Condition Impact
(a) Visualization of 18 topics and 5 data sources of COТA.
(b) The screenshot of History Relational Databasc (H-RDB).
Figure 8. Visualization of 18 topics and data sources and constructed H-RDB of CoTA.
storage subject dynamic combinations for different scenarios.
E.2. History Retrieval Queries
When retrieving the stored history information, we use sqlite3? python package. The search query is provided in sqlite3
format, for example: SELECT (Prompt) FROM (table) WHERE AND Domain
AND
CoTA-Alpha
CoTA-Beta
Overall Performance o%)
CoTA-Gamma
25
20
15
10
5
CurbooTurbo
Claude-3-Opus
GPT-4-32k
Claude-3, Souude-35-Sonneee Sonnet
Figure 9, Visualization of the performance of CoTA-Alpha, Beta and Gamma.
wte.. htm1
18

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Last Successful History
(u,.
Pseudo Code Logic
m.
Great, I will generate the code to identify categorical variables in the credit customers dataset and suggest encoding methods for converting them numerical variables.
import pandas
X Imparr necessary ilbrarkis
IMPORT pandas library pd
it Load the dotaset
credit, credit_customes customers covt".?
i/ Load the cataset from CSV hile
LOAD DATA FROM "credit, customers.co" INTO credit customers
a identily categorical variables
categonical collumns
credit, customers.weeed. wtwpee/esseeeeeeeddddddieeeee dippediicddd)) object Bioouumes
cateporical variables SET categorical, cellmns COLUMNS credit. OF
Re-Org One Shot: P.1 ( U.1
(m.s a.1))
necessary IMPORT pandss library pd
Great, ...I generate the code to identify categorical variables the credit customers dataset and suggest encading methods converting them to numerical variables.
Load dataset from CSV file LOAD DATA DATA FROM credi_customersccs" INTO credit rustomers
customers poumad pou
dataset:
Ideettify categorical variables categorticll collmnss
Jf iddeniffy categorical variables in dotoset SET categorical.com columns TD COLUMNS credit customers OF TYPE "shject
credit, cus.omerssseeee wwwpmeeeeeeeeseooooeeeooeeeeeeee dtypes(nncu de= labjers SebjerrDDcollli "Dcollamss
Current Question u,
Generate Logic: m,
Generate Ans: a,
Alright, Could you write some code to figure out if we need to normaliie the credit customers dataset? Just check if the value over 1, and if it is, we should normalize it and generate the top normalized dataframe...
// Fit transfons the numerical columns StandordScaler
and transform the numerical
FOR EACH
MAX VALUE OF a.merical_columer custamers CREATER
columns StandardScaler in numerital columnee if credit customers] (asammessssc[mm]]]]] 1
credit customens(col] scalerdit, ttt.trrsfoom/rrrel.ouuuuoee [col]]
TRANSFORM column credit, customers
customers
Figure 10. This is an overview of our proposed method, ACR. The areas highlighted in purple represent results generated hy the agents.
F. Dialog Types
CoTA can be categorized into Statement- (longer) and Colloquial- (shorter) dialogs. The statement-dialbl are more formal, resulting in more complex user instructions and code generations, which are commonly found in computational notebooks (Yin et al., 2023). On the other hand, colloquial dialogs involve shorter and simpler user questions, but exhibit more colloquial and conversational characteristics. This category of dialogs is primarily constructed through the process of prompting GPT-4 to segment and reinterpret the existing statement-dialogs.
G. ACR Implementation
Figure 10 presents the detailed steps ACR.
H. AGENT Implementation
H.I. Toolkit
Executor. To get the execution results of code generated by LLMs, we adopt Python Executor exec () which is implemented in Python 3, within isolated Python environment. The output the code execution, whether it be any return values, print statements, or error messages, then captured by the Executor. This output subsequently returned to LMs, providing them with feedback on the results of their code generation to make a better next-step action or decision.
User Simulator. In addressing the clarification action type, LLMs are permitted to request clarification when they feel ambigous about conditions from user queires. Therefore, we employ GPT-4 Turbo (with fixed version) to emulate the behavior of users, considering that GPT-4 has been demonstrated to provide feedback of equivalent quality human responses (Wang et al., 2024b).
Phttps: http://///ec..yhoooooorr//////aa
html texec
19

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Chart-to-Table. We employ deplot (Liu et al., 2023a) convert images into table. Given the table, then LLMs can reason and answer the questions.
H.2. Reasoning
COT. To evaluate the pure code generalization capability of data analysis, we restrict LLMs from executing code during generation. Therefore, employ zero-shot COT for the reasoning of the Agent mode. The key prompt implement such COT is:
write a step-by-step outline and then write the code:
CodeAct. To evaluate analytical capabilities beyond mere code generation, we employ CodeAct for multiple-choice questions. Specifically, we set the MAX STEP for CodeAct reasoning to 5, with the Executor serving as the primary tool. Data analysis agents are tasked to generate, analyze, and draw conclusions about their results. If the result contains bugs, the corresponding message is returned to the agent for rectification, although this process may consume additional reasoning steps. We also manually provide a one-shot example to guide agents on how to CodeAct in CoTA. To prevent data leakage, we cross-reference examples across different tabular data. For instance, an example curated from ATP_Teenn _Tennis could be used guide LLMs in the Laptop Pricing dataset.
H.3. Multi-Agent Implementation.
As a valuable and interesting agent type, Multi-Agent recognized to have the potential to enhance performance in many reasoning tasks including BOLAA (Liu et al., 2023d) and MetaGPT (Hong et al.). However, MetaGPT was designed specifically for software development requirements without mechanisms for handling structured data and conversation histories, making it less applicable our problem setting. Therefore implemented the Multi-Agent reasoning type as introduced in (Liu et al., 2023d), which is a more general framework and can be implemented more flexibly in different settings. To be specific, except central CONTROLLEER, we also create TOOL. AGENT, CODE AGENT, DECISTON-MAKING AGENT, and PRIVATE-LIB AGENT. Our results in Table 4 clearly indicate that the Multi-Agent configuration obviously outperforms the original Agent setting and model base setting, underscoring its potential. Notably, it achieves performance on par with our Agent configuration, partieularly showing improvement in Multi-Choice tasks due to the important role played by the DECISION-MAKING AGENT. This supports one of our motivations: beyond code generation, providing insightful analysis for users based on results is crucial in data analysis tasks. However, requires higher costs in our dataset and more sophisticated prompt design for each agent. Moreover, its performance begins to decay with increasing turns since each agent must be provided with not only conversations with users but also conversations between agents. This results in a prompt token consumption that is approximately 5.3 times higher than that of the Aeent. Therefore, this observation reinforces the necessity of our design of ACR in the Inter-Agent. It is not only effective but also efficient, so it's more suitable in conversational settings.
note that the Multi-Agent
I. Implementation Details
L.. General Implementation
The temperature parameter is set 0.0 for Claude, GPT-4, and GPT-4-Turbo and top_p to 1.0.
J. Action Description
In this section, we categorize and formalize the action types in COTA, identifying five distinct sub-categories that correspond to different types of user queries.
J.1. Update_Coom
The Update_Coom _ode action refers to instances where the user requests corrections for bugs or refinements to the conditions of previous queries.
J.2. Fast Fail
Fast_ Fail or when user queries contain factual errors.
1 is an action that alerts users when the current data contents or resources are insufficient to meet their requests,
20

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
CORRECTION:
Agent Action:
User:
Agent:
Update Code
Woops, it reports the error: ('Original Odd_1", 'Original Odd 2') not in index.
Sure, seems we didn't define I'Original Odd OOriginal Odd_2] let me refine the code:....
UNANSERABLE:
User:
Agent:
Fast Fail
there any significant differences digital service preferences between...
Sorry, we don't have the digital service preferences data, please input or refine.....
UNDER_SPEILE SPEICIFC:
User:
Agent:
Clarification
Do the top-performing gaming laptops have the desired screen size and screen resolution?
What the desired screen size and screen resolution...
Do the pop-performing gaming laptops have the desired screen size and screen resolution?
The desired sizes and resalutions would be: "15.6\" "17.3V"" and 4k
Best Best_Guest Guess
VISUALIZATION:
User:
Agent:
Plot QA
Could the suburb be viable long-ter long-term investment opportunity according to the scatter plot?
Recognize that scatter plots show relationships between two variables... Therefore, yes, it's...
ANALYSIS:
User:
Agent:
Insight Mi.iiinnn
Which brands indicated have highest number of laptops with powerful CPUs and at least 8GB of RAM
First, filter and count the data to only include laptops at least 8GB RAM. ... Based on the analysis, Dell Lenovo stands out.
Figure 11. This figure provides overview of action types in CoTA, illustrated examples. We emphasize the keywords specific each category, and demonstrate the relevant sections of the associated queries, as well as the agent actions. The number of t symbols represents the relative difficulty of each action. Please note that all free-text examples presented in this figure are only used for illustration purpose. In COTA, each answer format limited to either code generation or multiple-choice questions.
J.3. Clarification
Clarif fication is common action in response to under-specified questions, which are frequent in data- data-analysis queries. In this action, agents make the conditions of the question more specific and clear by seeking additional information from users.
J.4. Best Guess
While Clari cation is an effective action to reduce the uncertainty, it can lead to issues such user impatience due to unsteadily asking, and long dialog histories that result in attention distraction and long-context problems. Therefore, the Best_ action can address these issues by making appropriate assumptions based on data contents, domain knowledge, and commonsense knowledge for under-specific questions. However, there is also a risk that incorrect guesses can lead to hallucinations.
J.S. Plot_Qu. QA
In real data analysis settings, agents are also expected to answer user questions about insights derived from plots. The Plot_0 QA action can assist users in better understanding the contents of plots for decision making.
21

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
J.6. Insight_Mining
Beyond generating codes for users retrieve expected results, code agents are also tasked with summarizing executed results from the environment assist users in making informed decisions. This process, known as Insight_Minn plays important role in data analysis since it contributes to the evolution of code agents into comprchensive data analysis agents.
K. Evaluation Metric Details
We introduce evaluation metric details in Section K.1, and implementations for each result type. And the distribution of result typs is presented in Figure 12(b)
K.1. Evaluation Metrics
Accuracy (Acc). Ace is a common metric that evaluates ability of agents to generate code that executes correctly or to accurately answer multi-choice questions. It defined as the proportion of instances where the predicted outputs match the expected reference output, across all evaluated tasks. For given dataset with N instances, where C2 is the expected outcome (either execution result or correct answer) and C, is the predicted outputs for the ith instance, Acc is calculated as follows:
Acc IIG=
(4)
where I is an indicator function that returns 1 if C; = C₁ and 0 otherwise.
Ace with Private Lib Recall (AccR). Recognizing the importance of accurately leveraging specific user-defined libraries in code generation, we extend Ace to include recall-based adjustment for instances involving private libraries. This ensures that AccR not only evaluates the direct accuracy of code execution and question answering but also evaluates the inclusion and correct usage of private library functions. AccR can be computed as follows:
N AccR
N Σ ICC. = C.) -R(C,,
1
(5)
R(Ci, Ci) F(Ci) ((CI))F(C)) [F(Ci))
(6)
where R(C, C.) quantifies the recall rate of relevant library functions in the predicted code. F(C;) and F(C:) denote the set of private library functions in the reference codes and the set actually utilized by agents in the predicted codes, respectively. The final score would be weighted sum of Acc and AccR.
K.2. DataFrame Comparison
The function compares two dataframes (df_1 and _f__2) by checking their indices, column presence, and column data. uses np.al a aalle c loseee () for numeric data and direct comparison for non-numeric data. If a column df_ is absent in the original dataframe, it searches for a matching column df _f__. The function returns True if df_ and of_2 are equivalent, otherwise F 1 se. Please note, the column names will not be computed since different LLMs may have their only preference names. For example, the win ratio generated by GPT-4 could be called winning ratio by Claude.
K.3. Visualization Comparison
We note that it is hard compare the closed-form results for cewpaliizztion-aassee code generation since parameters of plots may be varied significant across models. For instance, GPT-4 generated plots may be the same with CodeLlama while their title names may be different, which leads to false negatives. Therefore we utilize P PTL package to compute similarity between plots. To be specific, the function resizes them to 800x600 pixels using the LANCZOS method, and saves them. The images are then read in grayscale mode to avoid the difference brought by colors. The function computes and returns the Structural Similarity Index (SSIM), a measure of image similarity, between the two images. This function can be used to compare an
compare_plotee takes two image file paths as inputs (ai_output and
re learence_outpoe
22

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Al model's output with reference output. Finally, the code generated will be considered correct if the similarity is larger than 0.6.
K.4. Multi-Intent Evaluation
In this work, we evaluate the code generation performance on intent manner, which means if one user query contains multiple intents, then the total scores of this query will be the number of intents. We evaluate each intent separately and sum up the scores of all intents as the denominator when caleulate the performance of each model in percentage.
K.S. Private Function Recall
We notice that some LLMs tend to import many possible private functions while not using all of them. Thus, to extract all indeed used private functions in the customized function library, we utilize AST package. After extracting the used private functions, we calculate the recall coefficient according to Equation 6.
K.6. Code Similarity Equivalance (CSE)
In the context of CoTA, the complexity of code generation lasks-many of which yield score of zero-preeents huge challenges in evaluating performance through Acc or AccR only. This is particularly evident when distinguishing between codes that differ by merely a single line of етгоr or output, both of which would result in an Acc or AccR of zero, despite their obvious differences in code generation capabilities. To overcome this limitation, we propose the introduction of Code Similarity Equivalence (CSE), nuanced evaluation metric designed to assess the similarity between generated codes and reference codes. Given that these codes originate based on identical user instructions, high degree of similarity is expected. Our approach leverages hybrid combination of models to reduce the bias, incorporating CodeTT+ and OpenAl models, which are affordable and available for most institutes. This combination
Ada (t (text-embed www.neebbeddingg-eeeeeeenen--n-e has demonstrated strong correlation with human evaluative preferences, offering more refined and accurate measure of code generation performance.
Details. We introduce here about how to conduct more nuanced evaluation of Acc or AccR with CSE. 1) We collect 180 instances of code generation including both NORMAL and PRIVATE. To evaluate the quality of these codes, we enlist experts who are proficient in data science and Python as evaluation committee.
2) They evaluate code generated by several models, including GPT-4-32k. GPT-4-Turbo,
Claude- B.5-Sonnet. Each evaluator provided with comprehensive user code histories, tabular contents, the current query, access the decision_ cesssinn__coanyyyceeeeam private library. Please note that evaluations are conducted only based on their expertise and experience, without any predefined guidelines and discussion, to avoid bias.
3) We ask for a relative ranking of generated codes among models over absolute scoring to avoid potential variability in scoring preferences among the evaluators.
4) In cases of parts of divergent rankings, the evaluators engage in discussions regarding the specific code samples until a consensus was reached. This step ensures more reliable and agreed-upon evaluation outcome.
5) The evaluation committee then examine various open-source and readily available embedding models to measure code similarity, aiming to closely match their ranking preferences. Our exploration identifies that the score system consisting of CodeT5+ (Wang et al., 2023) and Ada (rext-embedding adaa ada-002) most closely aligned with human evaluative preferences.
Introduction of a Mixed Evaluation Metric (AceSE & AccSER). To accurately reflect the nuanced capabilities of code generation models, we propose a composite metric that integrates Code Similarity Evaluation (CSE) with Accuracy (Acc), termed Accuracy for Similarity Evaluation (AccSE). This metric is concisely defined as:
1.0,
ifC = Ô Sf > 0.85 S2 > 0.9, if if 0.85 S2 0.9) v(SI S 0.85 A S 0.9), otherwise.
0.5,
AccSE
0.25,
7
0.
Where:
23

---

Are Large Language Models Ready for Multi-Tune Turt Tabular Data Analysis?
C and C' represent the reference and generated code execution outcomes, respectively.
G denotes the CSE score based C5
S2 denotes the CSE score based Ada.
This formulation succinctly captures the evaluation criteria for AceSE, with symbols S₁ and S2 representing the CSE scores based on CodeTS+ and Ada, respectively. The logical operators and V are used for "and" and "or" conditions, respectively, to further compact the notation. AccSER is computed in the similar way just times recall score for each value Eq. 6.
We hold this for future evaluation system of CoTA when we conduct more extensive cases with involved with more expert volunteers,
Rationale Against SPTT-4-Based and Multi-Agent Evaluation Methods. While existing research suggests that GPT-4- based soft evaluation could enhance the assessment of complex generative tasks, such approaches are deemed unsuitable for COTA due several critical reasons:
1) Bias Concerns: The prototype annotations and questions in our study originate from GPT-4-based agent environment. Employing GPT-4 for evaluation purposes could inadvertently introduce seff-enhancemmet bias (Zheng et al., 2024a), compromising fairness across model evaluations.
2) Cost Concerns: Although multi-agent evaluation frameworks, incorporating diverse families of LLMs, is to mitigate bias (Li et al., 2023c), the economical and computational overhead obvious. Specifically, evaluations such settings require at least twice the token consumption than that used in generation alone, rendering it impractically expensive in CoTA.
Given these considerations, our research proposes an alternative evaluation methodology that both cost-effective and reliable for evaluating the accuracy of complex data science code generation this time. We demonstrate that CodeT5+, remarkably efficient code embedding model, can obviously distinguish between varying performance levels and accurately identify correct code logic. Crucially, this model offers pragmatic balance between evaluation thoroughness and resource efficiency.
K.7. Other Value Types
For other result types, such as dictionry, set, list, we directly compute the exectued results and determine whether they are equal or not.
K.8. Case-by-Case Evaluation
While we categorize instances according to result types and provide evaluation codes for each type, some scenarios requires a case-by-case evaluation script. For instance, in most dataframe or matrix comparisons, we employ np. close () and stringg match for result comparison. However, in some cases, such as using a dataframe or matrix to display a classifier's Confusion Matrix, the predicted code deemed correct if its f1 fi-score surpasses that of the referenced code, even if their fl-scores are not similar, For the evaluation script of CoTA, we manually review and adjust the scripts to accommodate each case.
K.9. Evaluation Script Caching
Each example will be provided by a specific evaluation script to ensure the precision of our assessments (Lai et al., 2023). To manage the extensive effort required design scripts for each example, we introduce cache-based evaluation binding approach. Initially, we classify mainstream result types, which collected through steps introduced in Section 3.1, and develop highly specialized scripts for each type, such as dataframes and dictionaries. When new data is generated, an evaluation script is automatically assigned based on the result type. Annotators then review the assigned script to ensure its accuracy; if necessary, they adapt and generate a new script tailored to the specific case. This method allows us to streamline the evaluation process, making more efficient with minimal human intervention. The details of the evaluation script in each result type can be found in Section K.
24

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
L. ACTION Evaluation Mode
L.1. Correction
Update_ Code. This could be evaluated within a static setting where the bug feedback is embedded into user-code history. Agents are requried to update the previous code via user feedback.
Upde
L.2. Unawserable
Fast Fail. In DECISION COMPANY, we keep the original unanswerable questions and categorize them as multi-choice questions. This is done to evaluate if agents can identify these questions based on their analysis of table contents and commonsense knowledge. To prevent any biased setting, such specially designed prompts that might mislead agents into determining question unanswerable, We then reformulate their choices, enabling the model to decide whether question is answerable with clarification or assumption, or to directly classify it as unanswerable.
sample equal number of under-specified problems and answerable questions.
L.3. Under SSecific
Clarification. To evaluate the performance of agents clarification action, we employ dynamic setting that incorporates a User Simulator. This simulator mimics user feedback based on the ground truth code or answer. Initially, conversational data analysis agents are expected to pose questions for clarification, simulator will answer it according to the ground truth answers. Subsequently, these agents are tasked with generating the final code, understanding both the original history and the history of clarifications. This setup provides a robust assessment of the agents' ability to converse with human, clarify ambiguities, and generate accurate code.
Best Guess. We aim to evaluate the ability of conversational data analysis agents to make accurate assumptions when faced with ambiguous questions, without resorting to constant clarification, which could potentially frustrate users. We believe that the best guess of an agent should not impact the final decision and this evaluation metric should be somehow flexible. For instance, in a credit card application scenario, the term young people could refer to individuals aged 20-40 or 25-45, making it challenging to be evaluated by fixed metries. Therefore, we opt to use multiple-choice questions to assess the agents' cocuuption-maknnnni capabilities. We posit that an assumption is appropriate only if does not influence the final decision-making process.
L.4. Visualization
Plot_QA. choices.
We evaluate the analysis capability of agents around plot CoTA. The end format of answer would be multiple
L.5. Analysis
Insight_Mingg Mining. to evaluate it.
We evaluate the analysis capability of agents generally in COTA. We opt to use multi-choice questions
M. Private Mode Analysis
Overall Results. Table 4 and Figure 12(a) indicates that the PRIVATE setting presents a considerable obstacle, with the best performing Claude-3.5-Somee Inter-Agent only achieving 18%. This demonstrates that understanding and implementing user-specific functions is a critical and urgent skill for L.M agents real-world data analysis tasks (Zan al., 2022).
The Critical Role of Function Relative Recall. Notably, CodeLlama outperforms GPT-4-Turbo in Acc within the PRIVATE setting. However, its performance declines significantly relative to GPT-4-Turbo upon the consideration of private library relative recall in the generated codes, as measured by the AccR metric. This observation suggests that CodeLlama tends to reply less on user-defined private functions, aiming to reduce risk of code errors. Therefore, AccR metric can spotlight the balance required between proficient code generation and the meticulous integration of user-specified private libraries to foster safer and more satisfying code production.
25

---

Are Large Language Models Ready for Multi-Tune Tabular Data Analysis?
dictionary (1.5%)
unanswerable (9.3%)
value (8.7%)
model (0.8%)
30
GPT-4-3X GPT- Turbo
tuple (2.2%)
list (6.0%)
25
Codellama
index (1.0%)
20
Claude 2.1
series (5.1%)
Scores
class (0.1%)
plot (8.8%)
10
pd.df (25.0%)
umy (0.5%
5-
string (1.3%)
0-
Nommal
Private
Private Recall
Code Generation Modes
matrix (1.0%)
(a) Visualization of the performance of LLMs on NORMAL. PRIVATE, PRIVATE w/ RECALL. The Scores of first two modes are Acc. The last score is AccR.
bool (0.3%)
multi choice (26.4%)
function (2.0%)
(b) Visualization of the results types distribution of CoTA.
Figure 12. Visualization of the performance of LLMs with PRIVATE mode and results types distribution.
N. Detailed Error Analysis
In this study, the error patterns exhibited by tested LLMs are critically examined to provide insight into the predominant challenges faced during their operation. A detailed discussion is provided in Table 5. We analyze 200 randomly sampled instances, categorizing errors into three main types as follows: (1) Key Error (23%) occurs when the model incorrectly assumes the existence of a column name in the provided data table which does not exist. This error reflects fundamental misinterpretation of the table information and hallucination where the model uses non-existent fields for data retrieval. An example is the model's 's incorrect use of correct attribute '[" ['employment''' This error type suggests that the model may overly rely on its trained patterns rather than accurately assessing the real structure of the data, leading to 'hallucinated" column references. (2) Lazy Assumption (39%) refers to instances where the model tends to assume that intermediate results or states are already available or saved on disk. This often leads to erroneous or incomplete code execution paths, such the premature use of "updated sedd___dd____ df' without ensuring its prior creation and calculation. This type of error may arise because models often seek shortcuts in their processing, opting to retrieve and manipulate existing objects rather than generating solutions from scratch. This lendency can reduce the reliability and flexibility of the model, as it may fail under conditions where dependencies are not treeessabliseed. (3) Bad Instruction Following (49 %) describes the model's failure to adhere strictly to given instructions, resulting in an inability to properly answer the posed questions. This is exemplified by the response of the model with 'D. None of above' when asked to generate Python code to help solve question, showing lack of direct engagement with the query requirements. This type of error often becomes more pronounced in later conversation turns, suggesting a compounding of misunderstandings or degradation of context over the course of a session. These error types are critical in understanding the limitations of the current model, guiding future improvements in LLM agents' evolving abilities. Understanding these patterns helps in pinpointing specific areas where training data, model architecture, or conversation protocols need enhancement to improve overall performance and reliability.
peredit_looe weadiitt__o______e_e i on 'employment "implyyment_duratioon]]] instead of the
Error Type Analysis Across Different LLMs and Conversation Settings. Figure 7 showcases the distribution of error types across various LLMs and settings. This figure provides a comparative insight into the frequency of three primary еттоr types: Key Error, Lazy Assumption, and Bad Instruction Following, both in Model-Base and Inter-Agent setting, Key Error rates vary significantly across models. For instance, Code-LLLama-3pp exhibits a notably higher rate of Key Errors compared to other models, which might suggest a less effective understanding or integration of database schema information in this model. Lazy Assumption errors are consistently high across all models, indicating a common model behavior where assumptions are made about the state of computations or data availability. This could reflect an inherent model optimization minimize computational expense by reusing existing data states structures, which, while efficient, can lead to inaccuracies when those states are not correctly initialized or updated. Bad Instruction Following shows a general high trend across models, particularly noticeable in settings involving Agent conversations. This suggests that as models engage in more complex dialogues or tasks requiring cooperative problem solving, their ability to follow detailed instructions without deviation diminishes. This could be due to accumulating contextual misunderstandings or the increasing complexity of managing multiple instruction streams. Inter-Agent Variations are particularly interesting; while Key Errors
26

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
and Lazy Assumptions increase slightly, Bad Instruction Following emrors show a marked increase. This may be due the added complexity of coordinating and maintaining consistent task strategies between agents, highlighting critical area for further research and model training refinement. These insights are crucial for understanding specific weaknesses in current LLM implementations and point towards necessary areas for improvement in model training protocols and architecture adjustments. Enhanced training methods focusing on schema understanding and multi-agent coordination could mitigate some these prevalent errors.
Private Library Mode Common Errors. Given the private-lib based code generation such as (Zan al., 2022; Li et al., 2023b) usually follows a Retrieval-Agmenned Generation (RAG) workflow. The common types of errors can be divided into the following categories:
Retrieval Phase Errors: LLM--ased models prioritize high recall over precision, retrieving excessive packages, Even the best inter-agent performance only achieves 0.73 Fl score.
Error patterns vary significantly by models. CodeLlama shows conservative usage of private libraries to minimize errors. Instead, GPT-4-Turbo demonstrates higher private library utilization but increased error rates. We introduced this in Appendix N.
Generation Phase Errors:
Also, the general error types depicted above in Section N also exist. Therefore, developing a more advanced RAG system would be helpful.
O. Dataset Quality Evaluation Details
0.1. Human Evaluation
To evaluate the quality of the dataset annotation, we also conduct a thorough human evaluation. We select 500 samples and invite 10 experts with extensive data analysis experience review the dataset. The evaluation metrics listed below are carried out using binary scoring system, with scores of 0 or (Reject Accept). We divide the measurement metrics into two levels: General Metrics and Action-wise Metrics, which are elaborated below.
General Metrics. To ensure the overall quality of the dataset, we apply a set of comprehensive general metrics. These metrics are designed to evaluate the ability of dataset to capture meaningful, diverse, and coherent multi-turn conversations in the data analysis domain. Here a brief introduction:
Conversation Coherence: Evaluate whether the dataset contains logically consistent conversations including generated codes that flow naturally across multiple turns and lead to expected answers. If yes, score 1; if no, score 0.
Scenarios Diversity and Reasonableness Assess whether the dataset contains wide range of scenarios without duplication, tasks, and user intents. And whether they are reasonable and can be fully supported by the given tabular data. This metric is satisfied if both sub-metrics are satisfied. If yes, score 1; if no, score 0.
Conversation Topic Coherence: Measure the overall relevance of the conversation to the given topic, ensuring that the conversation stays on track and go off the data analysis questions. This metric is satisfied if both sub-metrics are satisfied:
Conversation Goal Relevance: Evaluate whether each conversation turn contributes meaningfully achieving the final goal. Turns must remain focused on the final conversation goal. If yes, score no, score 0.
- -Table Relevance: Measure whether at least 40% of the conversation turns involve conversation with the provided tabular data for specific conditions. This ensures the conversation sufficiently relevant to the dataset being analyzed. If yes, score I; no, score O.
Ethics and Bias Representation: Assess whether the dataset avoids biased, harmful, or unethical content. If yes, score
1; if no, score O.
Conversation Naturalness: Measure whether the conversation in the dataset reflect natural, conversational language, avoiding overly robotic or AI-like responses. Conversations should resemble real human conversations in tone and flow. If yes, score 1; if no, score O.
27

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Evaluation Scripts Quality: This metric is satisfied if all sub-metrics are satisfied.
Tool Reliability: Measure whether tools usage logs such as the logs of user simulator tool are reasonable and trustworthy, and whether the Python executor tool utilizes common, reliable packages to ensure consistent and accurate results. If yes, score I; if no, score 0.
Evaluation Script Flexibility and Comprehensiveness:
Flexibility: Ensure that the evaluation scripts are not overly rigid and can accept multiple valid, reasonable outputs correct, allowing for variations in agent responses. If yes, score 1; if no, score 0.
2. Comprehensiveness: Measure whether the scripts are robust enough to handle a wide range of scenarios, including corner cases, ensuring they effectively evaluate all possible outcomes. If yes, score 1; if no, score O.
3. For multi-choice questions, the provided options should be valuable and challenging enough, where can't 't be easily figured out from merely question and options. If yes, score 1; if no, score 0.
Evaluation Script Scalability: Measure how easily the evaluation scripts can be extended or adapted to accommodate new data. This metric evaluates whether the framework allows for seamless integration of new evaluation scripts without requiring significant modifications, ensuring efficient scalability as the dataset grows. If yes, score 1; if no, score O.
Action-wise Metries In addition to the general metrics, we apply specific metrics to evaluate the detailed actions captured in the dataset. These action types include Update_ Fast_ Fail, and Insight_ Insight_Mining
Best Guess, Plot_QA
Each action type is evaluated using core metries ensure its relevance, accuracy, and contextual consistency within the dataset. These metrics guarantee that the dataset reflects realistic and valid scenarios in the data analysis domain. Here is an overview:
Action Commonness in Data Analysis: Evaluate whether the action cases commonly seen in real-world data analysis tasks. This ensures that the dataset captures authentic, practical scenarios that analysts frequently encounter. If yes, score 1; if no, score 0.
Correctness of Reference Answers: Assess whether the ground-truth reference answers provided in the dataset are accurate. The dataset should provide correct, verifiable solutions to the user query represented in each action. If yes, score I; if no, score 0.
Contextual Reasonableness: Measure whether the context surrounding the action is comprehensive and logical. This ensures that the action occurs within a reasonable, coherent dialogue flow, taking into account all relevant factors from previous turns. If yes, score 1; if no, score O.
Dataset Quality Statistics. We compare the quality of the 500 sampled annotated data before and after human calibration, where the pre-calibration data was fully annotated by large language models (LLMs). As shown in Table 3, there is an obvious improvement in dataset quality following human calibration performed by annotators. The low acceptance ratio of the data prior to calibration underscores the necessity of this process. After calibration, the acceptance ratio rises to approximately 0.95, indicating that the involvement of annotators who are experienced in data analysis is sufficient to ensure the quality of the dataset, thus demonstrating the trade-off between efficiency and quality of our annotation workflow.
Specifically, the metrics for Scenario Diversity and Reasonableness improved from 0.46 to 0.96, reflecting enhanced coverage and variety of scenarios as shown in Section 3.1. Moreover, Conversation Topic Coherence increased from 0. 0.17 to 0.93, indicating better alignment with the conversation topics. The Ethics and Bias Representation metric achieved a perfect score of 1.00, confirming the adherence of the dataset to ethical standards, while Conversation Naturalness improved from 0.67 to 0.95, suggesting more natural, human-like dialogues. Additionally, our Evaluation Scripts Quality and Evaluation Scripts Scalability scoring 0.98 and 0.94, respectively, prove efficient solution to be against with data leakage problems. Overall, these results highlight the effectiveness of human calibration in enhancing both the conversation quality and the robustness of DESCISION COMPANY and high quality of COTA.
Human evaluation is also conducted with a focus on the actions. Figure 4 illustrates the consensus that all actions in CoTA are both necessary and commonly observed in real-world data analysis scenarios. Furthermore, our simulated scenarios conversational data analysis conversations.
successfully capture and reflect key characteristics of these
28

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Table Package Diversity of Our Dataset
Category
pandas
matplot
Machine Learning (sklearn, scipy, seaborn)
numpy
Percentage
%
56.47%
7.90%
16.78%
12.64%
O.2. Dataset Diversity
We acknowledge the importance data diversity and believe our benchmark, CoTA, effectively demonstrates across multiple dimensions. We measure data diversity through several aspects:
Domain Topic Diversity, shown in Figure 8(a).
Result Type Diversity, covering wide range of query types requiring different code techniques, as displayed in Figure 12(b).
Action Diversity, presented in Figure 3, where we demonstrate the comprehensive coverage various action types in conversational data analysis. To our knowledge, we are the first offer such extensive action type diversity this domain.
Table 8. Average Unique 3-grams
Metric
CoTA (1013)
DS-1000
Average unique 3-grams in eeference_answer
46.64
11.63
(including codes & multi-choice answers)
Average unique 3-grams surrent_query
84.54
115.31
Average unique 3-grams in prompt_ with context (history)
1033.40
115.31
Table 9, Total Unique 3-grams Counts
Metric
CoTA (1013)
DS-1000
Total unique 3-grams in reference_ answer
47,245
11,633
Total unique 3-grams in current_quent
85,643
115,305
Total unique 3-grams in prompt_ prompt_with_coee wwth_wcotee.._eeee (history)
1,046,832
115,305
Additionally, we evaluate data diversity through two more specific metrics:
Package Diversity: Table shows the distribution of query topics that cover various Python packages commonly used in data analysis, such as pandas, matplotlib, and machine learning libraries (sklearn, scipy, seaborn), with pandas dominating at 56.47%, followed by machine learning packages (16.78%) and numpy (12.64%).
Query Diversity: Following the methodology in (Li et al., 2023a), we compute n-grams (n = 3) to reflect the diversity of each query. We compare CoTA against DS-1000 (Lai et al., 2023), a popular data analysis benchmark, to show the diversity of queries in our dataset.
Average Unique 3-grams: Table 8 illustrates that CoTA provides much higher diversity in reference answers, averaging 46.64 unique 3-grams compared to 11.63 of DS-1000. For current queries, CoTA maintains comparable level diversity, but when context is considered (prompt DS-1000, demonstrating its ability to handle complex, context-driven conversations with an average of 1033.40 unique 3-grams.
ciampttwithh__t with context), COTA vastly outperforms
29

---

Are Large Language Models Ready for Multi- Turn Tabular Data Analysis?
Total Unique 3-grams Count: It show the diversity of the whole dataset. Table 9 further supports the diversity of CoTA, with the total number of unique 3-grams in reference answers reaching 47,245, significantly surpassing 11,633 of DS-1000. The total unique 3-grams in current queries stand at 85,643, and in prompt_with with_connext, COTA achieves an outstanding 1,046,832 unique 3-grams, compared to 1 115,305 of DS-1000. This further highlights the rich contextual conversations captured by COTA.
Overall, these metrics, along with the diversity in domains, result types, and actions, underscore extensive coverage of real-world data analysis tasks in COTA. It surpasses existing benchmarks in capturing the full range of complexity and diversity needed for robust evaluation of conversational data analysis.
P. Ethical Statement
The application of LLMs for automatic data generation requires a rigorous examination of ethical implications. The primary concern is the potential for LLMs to generate contents that could be considered harmful biased. To mitigate these risks, human annotators already filter and fix all problematic cases in Section 3.2. Also, LLMs may disseminate private or sensitive information. Therefore, we employ anonymization techniques wherein personal identifiers are systematically altered. For example, the name strings are replaced randomly, and any information of personas are switched as well. And the geographical locations of John Smit th will be replaced with locations of Carlos Carcia to prevent any linkage to real-world individuals or entities. These procedures are conducted in Section Moreover, we are committed to ensuring that the outputs generated by our LLM, referred to COTA, are free from political sexual biases. To this end, each output, including conclusions and generated responses, is rigorously reviewed by the authors. In nutshell, framework is built on a foundation of transparency, accountability, and proactive stance towards mitigating any ethical concerns associated with the use of L.Ms.. The measures we have implemented reflect our commitment to upholding the highest standards of ethical research practice with LLMs
ethical
Q. DECISION COMPANY Prompt
The process begins with the generation of client personas, shown in Figure 13, where the Administrator agent prompted to create meaningful personas. Following this, simulate diverse analysis scenarios using In- In-Context Learning (ICL), which is depicted in Figure 14, allowing us explore wide range of potential outcomes. A critical aspect of the system is the discussion of analysis plans, where the conversation between the Data Scientist agent and the Client agent, illustrated in Figure 15, results in the generation of a series of analysis plans. To further support the process, conversation logs are annotated to capture the essence of conversations, with Figures 16 and 17 showing the perspectives of the Data Scientist Agent and the AI Chatbot Agent, respectively. Lastly, the evolution of our private library is detailed in Figure 18, which demonstrates the framework for prompting GPT-4 to generate code automatically, while human intervention plays a key role in minimizing bias and correcting errors.
R. Implementation Prompt
R.1. CODE GENERATION
The Figure 19 describes how we prompt LLM model to generate code to answer user queries. And Figure 20 describes how we prompt LLM in Agent to generate code to answer user queries following with chain-of-thought (Wei et al., 2022). Finally, Figure 21 describes how we prompt LLM in Inter-Agent generate code to answer user queries with our proposed ACR. And Figure 25 describes how we prompt LLM in Model-Base version to ask for clarification.
R.2. MULTI-CHOICE
The Figure 22 describes how we prompt LLM to answer user queries. And Figure 23 describes how we prompt LLM in Agent to answer user queries following with CodeAct (Wang et al., 2024a). Finally, Figure 24 describes how prompt LLM w/ Inter-Ace -Agent answer user queries with our proposed ACR.
30

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Find Target Group:
In Decision Dynamics, company specializing in data-driven decision-making support, CEO Patrick Reynold leads team af experts dedicated to assisting clients and organizations in making informed choices for any domains based on comprehensive data analysis.
You an administrator at Decision Dynamics, Sarah Johnson. responsible for receiving client inquiries and managing business budge make sure profits. Your role involves collaborating with data scientists calculate the costs of each case and determining whether to accept or decline orders. Additionally, you oversee the cost control of dialogue turns generated while resolving cases. Today, you need to find out target clients for tabular data about fastfood with the following columns: ['restaurant, 'item', 'calories', 'cal_fat', 'total fat', fat_fat', 'trans trats_fat', "cholesterol, 'sodium', 'total, carb', 'fiber', 'sugar', 'protein', 'vit a', 'vit c,, 'calcium', 'salad").
Please search out 3 groups of clients whose busniess needs this data. Fill them in the JSON file: { ('Goal': [Relation to this tabular datal, 'Group:: [Group namelB. Each JSON should be split by || and no numbers are needed. Make sure 3 Group have 3 different business needs:
Agent Output:
('Goal': 'Recommend restaurants based users' nutritional needs', 'Group': 'Health and Wellness Mobile App Developers') I| ('Goal': 'Evaluate and improve restaurant menu offerings', 'Group': "Restaurant Owners and Managers') |I ('Goal': "Compare fast food options for healthier choices', "Group': 'Health and Fitness Bloggers') || ('Goal': 'Create healthier fast food alternatives', 'Group':
"Food Product Development Companies'
Find Client Personas:
In Decision Dynamics, a company specializing in data driven decision maaiiig support, CEO Patrick Reynold leads of experts dedicated assisting clients organizations making informed choices for any domains based on comprehensive data analysis.
You are an administrator Decision Dynamics, Sarah Johnson, responsible for receiving client inquiries and managing business budgets to make sure profits. Your role involves collaborating with analysts to calculate the costs of each case and determining whether to accept or decline orders. Additionally, you oversee the cost control of dialogue turns generated while resolving cases. Today, you receive tabular data about Fastfood. Today, she purchased an tabular data called fastfood containing the following columns: I'restaurant', 'item', 'calories', 'cal_fat', "total_fat" fat', 'sat_fat', "trans_fat' 'cholesterol", 'sodium', 'total_ carb_,, 'fiber', 'sugar', 'protein', 'vit_a', a", 'vit_ct 'calcium', 'salad").
There groups clients who maybe interested this data:
Group Health and Wellness Mobile App Developers.
Group Restaurant Owners Managers.
Group Health Fitness Bloggers. Group 4: Food Product Development Companies.
-- Generate client personas for each group by filling the JSON: ('Name': [name], 'Location': [location), "Job': .Job title), 'Background': [background of persona], Each JSON should be split by II and no numbers are needed. Note job title should not be related to analyst, researchers, etc:
Agent Output:
('Name': 'Emma Thompson', 'Location': 'San Francisco, CA', 'Job': 'Mobile App Developer', 'Background': 'Emma has degree computer science and specializes developing health and wellness apps. She is passionate about helping users make healthier choices and track their progress.") ') ('Name': Martinez', 'Location': York, NY', 'Job': 'Restaurant Owner', 'Background'; 'David owns successful chain fast-food restaurants and always looking for ways to improve the nutritional value of his menu items while maintaining customer satisfaction.') II ('Name': 'Sophia Lee', 'Location': 'Austin, TX,, 'Job': 'Health and Fitness Blogger', 'Background': 'Sophia is a certified nutritionist and fitness
enthusiast who shares her knowledge and experiences through her popular blog. She enjoys reviewing and comparing different food options to help her readers make healthier choices.") II ('Name':
'Michael Johnson', 'Location': 'Chicago, IL', 'Job': 'Food Product Developer', 'Background': 'Michael has a background in food science and works for leading food product development company. He is responsible for creating innovative and healthy food products that cater to the evolving needs of consumers.']
Figure 13. The prompt of Client Persona Generation
31

---

Are Large Language Models Ready for Multi-Tun Tabular Data Analysis?
Analysis Scenario:
You are client named Emma Thompson, living in San Francisco, CA. You work as a/an Mobile App Developer. You have degree computer science and specializes developing health and wellness apps. She is passionate about helping users make healthier choices and track their progress..
You have the data fastfood contains the following columns: ['restaurant', 'item', 'calories', lllfat', 'total fat', $sa__fat,,, 'trans fat', 'cholesterol', 'sodium', 'total carb', 'fiber', 'sugar', 'protein', 'vit a', 'vit_ 'calcium', 'salad"). Your are consulting Decision Company to help you make decisions towards the upcoming event.
Introduce your events by filling Json file: ('Scenario': [the scenarios where you experiencing by starting 'There ...'l, "Scenario Name': [summarize the name of Scenario), 'Decision Making': [what kind goals you want to make]). scenario one-step related to your work background. Each Json should be split
and numbers are needed:
Agent Output:
"Scenario": "There is an upcoming health and wellness conference in San Francisco, and you have been invited to showcase your latest mobile app. You want to make sure the app is well-received and demonstrates effectiveness in helping users make healthier choices.", "Scenario Name": "Health and Wellness Conference", "Decision Making": "Determine which features to highlight during the presentation and how to best demonstrate the app's effectiveness promoting healthier choices."
Ill
"Scenario": "You have been approached by a local gym to create custom mobile for their members. The gym wants to encourage members to make healthier food choices and track their progress the app. You need to decide which features include the app and how tailor it the gym's specific needs.",
"Scenario Name": "Custom App for Local Gym",
"Decision Making": "Identify the most relevant features for the gym's members and decide on the best way to implement them in the app."
}ll{
"Scenario": "You have noticed trend in user feedback requesting more personalized recommendations for healthier fast food options. You want to update your app include feature that provides personalized suggestions based on users' dietary preferences and restrictions.", "Scenario Name": "Personalized Recommendations Update",
"Decision Making": "Determine how incorporate personalized recommendations into the app and decide on the best to utilize the fast food data to provide
accurate and helpful suggestions."
Figure 14. The prompt of analysis scenario generation
32

---

Are Large Language Models Ready for Multi-Tun Tabular Data Analysis?
Data Scientist Ales Moraan
Thank you providing outline of your requirements, most accurate analysts possible.
questions fully understand your provide
1.ln the ATP tennis datases, do have any specific players would like me focus should analype all the players in the dataset? analysing
there any particular perlods specific tournaments you would 3.When analyzing performance on different surfaces, do you want me focus specific surfaces, or should consider all available surfaces the dataset? you
A.Far the visualizatioos
for the types graphs you like see?
Clintt Emna Brown
Thank you questions, Alex, Here ansovers:
1. would like you to anelyze all the players in the detaset, but possible, please provide a sepurate analysis for the players from our academy.
2. Please focus on the last three years data and prioritize the analysis major touramennts such Grand Slams, Masters 1000, and ATP Finals.
surfaces, please tournoment in London will be played on grovs. the visualizations summaries, would your expertise to chaose the most appropriate visualization for each type of analysis. (pertain to 1, 2,
3. analyzing performance
dataset, but pay special attention to grass since the upcoming
bar
well
player matchups. Please
hope information helps you provide the most accurate analysis possible. you have any further questions need clarification, please don't hesitate to ask.
Analysis Plan
from last five years? Result What the win rate each of top players on grass courts the last five years? Result Type: Data table with player namee,, number of matches played, number of matches won, and win rate percentage.
Who the ranked players
player along with their rankings.
How the players perform apainst head to head win-loss records between each pair of the top 10 players.
head-to-head
on grass
Typer displaying
Figure 15. The example of plan discussion. The final output should be a plan of analysis involving questions and their or result types.
33

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Data Scientist View:
You are a male data scientist with an impressive array of skills including data analysis, statistics, machine learning, and proficiency in Pandas.
have the data credit_customeee customers containing the following columns: ['checking_staee status', 'duration', 'credit cendit_isttory,, pprrpose', _onditt_ 'savings_stateee st__ttttus', 'employment', 'installment cttpreett_ccmmmmmettttt "personal_statiee stttuss',, 'age', scther_ayyment_llane 'housing', coisting_crediits/ 'job', "num_ dependents', 'own telephone', 'foreign_wrrker" 'class'). "num_deppeed"
Trsidence_sincee
observed the first lines of the by running:
import pandas pd
Load the dataset
credit_ cotititrsstttssssee pd.read_csy "rreditccstommrrscsxy"" credit_ besstomers.headd33
chacking status
duration
purpose
credi anount
<0
Critical/other existing credit existing paid
rad2o/tv
1169
0<cx-208
43
radio/tv
5951
checking
12 critical/othe existing credit
aducal
2096
42
existing paid
furnsture/oqujoeee
7882
There questions that you to solve:
W.ich clients in credit_ customers dataset have high credit amounts longer loan durations?
Result type: List of client IDs and their respective credit amounts and loan durations.
2.Among these clients, who have a history of late payments defaults in their credit history?
Result type: List of client
with history of late payments defaults.
W.hhic these clients have multiple existing credits and high installment commitments?
Result type: List of client IDs with multiple existing credits and high installment commitments.
4. many clients in the filtered dataset are aged between 25 and 55?
Result type: Count of clients aged between 25 and 55.
5.Among these clients,
employed and preferably have stable employment? Result type: List client IDs stable employment.
6.wow many clients in the final filtered dataset reside rented or owned housing, excluding those living rent-free?
Result type: Count of clients residing in rented or owned housing.
7.hhat the common characteristics clients who may benefit from debt consolidation in the filtered dataset?
Result type: Summary common characteristics, such as average credit amount, average duration, and most common employment status.
8.Are there any patterns or trends in the data, such relationships between credit history, loan duration, and employment status?
Result type: Insights patterns trends observed in data, including any correlations relationships between variables.
Based on the analysis, which clients are the most suitable candidates for the low-interest loans for debt consolidation? Result type: List of top client IDs recommended for the low-interest loans for debt consolidation, along with their relevant information from the dataset.
Begin your interaction with the Al Assistant Tapilot to help you finish these questions. Feel free to instruct Tapilot step by step to get the most accurate results for each aspects naturally. Don't worry about generating code, as Tapilot can do that for based your instructions. You have to tell Tapilot with result types for each question.
In order to prevent Tapilot from collecting your private data, responses from Tapilot should be codes and you are required to execute them by your own and generate code to answer questions from Tapilot if has questions about data content. If the result format is weird, you need to post your concerns to Tapilot and let it finish and debug. [You (data scientist]]:
Figure 16. The prompt of Data Science Agent in conversation log generation.
34

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Chatbot View:
You are an Al assistant that aids users in performing data analysis using Python and Pandas to find information.
There the
credit_customers customers containing the following columns: ['checking_ stattuu 'duration', 'credit "'redit_historr', 'purpose', "credit_amoune"" amount', "savings_sste ststtutt,,, 'employment', 'nntalineet_commimmentt commitment', 'personal_stalee ctter_parties/ 'residence since', property_magnitude" 'age', 'num_ dependents', You observed the lines of data by running:
paaneettt_pnn,,,, 'housing', 'existing_credits" 'job', telephone', "orrign_worker," worker', 'class'].
import pandas as pd
Load dataset
credit_customers customers pddread_cs"""rredit csv("crediit "ecereddi__csttmmrrsssss"" credit_ cetiiccsstomersheaadi..
checking
duration
history
purpose
crstical/othe existing
Sc=X<200
redio/tv
checking
bothher existing
education
2096
existing poid
furnsture/equ.ppent
7882
To perform a more reliable analysis for users, are required to ask necessary questions about data when want to assume conditions.
Considering contents from the dataset result types from user, you only need to generate codes and notations.
Conversation begins:
[USER (data scientist)):Let's by answering the first question. will find clients with high credit amounts and longer loan durations. We can consider high credit amounts as above 75th percentile and longer loan durations as those above the 75th percentile as well. Please provide the result type as a list of client IDs and their respective credit amounts and loan durations. [YOU assistant)]:
Figure 17, The prompt the Chatbot Agent conversation log generation.
Three-Step Prompting
1. my prototype code snippet, find all functions, such including pandas, seaborn numpy: (Prototype Code Snippet)
Output:
(A List Functions)
2. Then Convert these functions into customized functions via new names. Each function should contain string, please:
(A List of Functions)
Output:
(New Customized Private Functions w/ Human Ca Calibration)
3. Finally, rewrite my prototype code, via the customized functions: (Prototype Code Snippet)
Output:
(Code Snippet Via Private Libraries and Human Calibration)
Figure 18. The prompt of conversion from prototype code towards the code with private libraries.
35

---

Are Large Language Models Ready for Multi-Tun Turn Tabular Data Analysis?
Model Base prompt for code generation
There data: credit_ccstomeee costomerssssssseeee containing following columns: l'checking status', 'duration', 'credit 'credit_history, ppurpose', 'credit_ann amount', saviingssstatus', 'empplyment', 'installment, commitment', 'personal_ status', 'other_parties 'residence since", property_magnituue 'age', bllpppaaa___plnnsss__ 'housing', cistting_rredit/' 'job', 'num "ddeendeeetts'' jwwn_telephone) 'foreign worker' 'class"].
The description for each column data is:
Considering contents from dataset and requirements from THAT HAVE SET!
CHANGE
Interactions begin:
Interaction History:
[Interaction _history
Query: (New_query)
[YOU assistantj]::
Figure 19. The prompt of LLM in Moodel-Base version in CODE GENERATION mode.
Agent prompt for code generation
There credit_custoneeee customers containing following columns: ['checking status', 'duration', 'credit history', 'purpose', 'credit_ amount', 'savings_ 'savings_status' sstatus' 'emppoyment', 'istallment_coom _ommmttment',, 'personal status', jatter_patties, 'residence, since', property_magniinn
"istallment_commmmmenn",, lllpppaaan__ppange 'housing', cristing_credits, 'job', 'nummnduuaaa dimn_ddenndent', www_ttelephone'
'age',
'foreign, worker', 'class"].
The description for each column this data desurndddscrptiion
Considering contents from dataset and requirements from THAT HAVE
CHANGE FILE VARIABLE NAMES
Interactions begin:
History:
(Interaction_ E
Query: (New_query)
[YOU assistant)]: need first write step-by-step outline then write code:
Figure The prompt LLM with data analysis agent in CODE GENERATION mode. The COT prompt text red color.
36

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Inter-Agent prompt for code generation
There is the date: credit_ custto_csss containing the following columns: l'checking, status', 'duration', 'credit_history/ 'purpose', ammout", "savings_ status,, 'employment' 'installment_come "atalmment_cmmmmmmnnnt 'personal_ stalus', 'other_parties rresidence, property_ maanituduee 'age', foreign_worker" 'dass").
htt_paymenn_pplnns "housing, "existing_ credits', joob, 'num dependenns,, 'own_teke telephone',
The description for each tolumn this data Is:
[Column, description)
Considering THAT HAVE SETI Interactions
requirements
NOT CHANGE VARIABLE NAMES
Interaction History:
[Interaction history)
As a data science professor, your mission is to guide your students in creating Python code tackle data science probloms "New Query", analyzing the dialague History and pseudorode and Pythan code generated the successful example below. Your task to firstly develop psoudocode that will guide students to write carrect python code to answer the "New Query". While creating pseudocode, consider these aspects:
stratagy for reusing variables from the prior interaction. Identify the trigger words or situations that mandate the reuse these variables. (2) Ascertain the level detail in necessary pseudocode.
create
artattnnteetnt Example Starts: tennttennettnnn JUSER (data scientist]]: Hey Tapilot, I've got this dataset called "creditt_ustommeeeeee customers.cov csv" and need to preprocess for machine learning madel. you help load pandas?
|YOU [Al assistantt]::
Step 1: pseudocode:
<pseudocode>>
!/ import the pandas package IMPORT pandas as
// Define a function load CSV file FUNCTION load_cooffff ccufflll_pathh pathh: // Use pandas read the CSV file DATAFRAME pd.read csv(tlle_aath)
// Return the loaded data
// Define function check missing values dataset FUNCTION check, missing, values]dataffamee:
// isnall() and suml) mothods count missing vallues per column MISSING, VALUES e dataframe. aataframe.snullj.. isnull)).smm(
// Return series with missing value counts
RETURN MISSING VALUES
// Main program exccution starts here // path
FILE PATH <. "credit pustamers.ca/
// Call the function load the dataset customers
PATH)
// missing_valie values e- check, checkkmiisinng misssnng cesssosscreelllccsssooeeess
missing
// Display the missing values to the user DISPLAY missing </pseudocode>
# Step 2: Python code:
import pandes as pd import pickle
UMAA AUAAAAAAAA Example Ends поАяИИлАНИия
New Query
(New_query)
assistant]]:
Step 1: pscudocode:
<pseudocode>
Figure 21. The prompt of L.L with conversational data analysis agent in CODE GENERATION mode. The ACR prompt text are in green color, which are generated by LLM itself by learning from successful history.
37

---

Are Large Language Models Ready for Multi-Tun Tabular Data Analysis?
Model Base prompt for multi-choices
There the data: ATP_ _rP_nennis containing the following columns: ('Tournament', 'Date', 'Series', 'Court', 'Surface', 'Round'; 'Best of, 'Player_11 'Player_2 'Winner', 'Rank_1 'Rank_2 2,, 1Pss_1,, 'Pts_ 'Odd 'Odd 'score'). The description for each column this data is:
(Column_desse d
Considering contents from the dataset and requirements from user. Please note DO NOT CHANGE FILE AND VARIABLE NAMES THAT HAVE SET!
Interactions begin:
Unteraction_histone
We interested exploring the existence of notable trends shifts in how court surfaces Influence player performance within the ATP tennis dataset across different years. To accomplish this, we plan to conduct Time Series Analysis, which will include the creation of line charts for visual representation, trend analysis to identify any patterns, and the application of statistical tests to confirm our findings. Following the analysis, could identify if there type of court surface for which no significant trend player performance observed?
[USER (data scientist)]::
A, Hard
B. Grass
C. Clay
D. Carpet
E. None of above
Please generate the python code (with pandas version 2.0.3 and matplotlib version 3.7.4 between <code>... s/cooee to answer the first question and based the answer choose the most appropriate option and directly provide the choice between <choice>.. cchoice>..</choice>.
[YOU (Al assistant)]:
<choice>
Figure 22. The prompt of LLM in Model -Basc version in MULTI-CHOICE mode.
38

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Agent prompt for multi-choice
Solve question answering task with interleaving Thought, Code, Action, Results steps. Thought can reason about the current situation, and Action
three types:
(1) [eccee which execute the provided code with python and returns the code output if exists.
(2) Terminate| (ansswes)), which returns the answer and finishes the task. examples.
Example Start:
(Example)
Example End
database atp_tenteeeeeeeeeeee tennis shown follows:
Tournament Date Series Court Surface Round
Best
Player_1
1 Player_2 2 Winner Rank_1 RRnk__ Pts Pts
Odd 1 Odd 2 score
Brisbane International 2012-12-31 ATP250
Outdoor
Hard
1st Round
Mayer F. Giraldo Mayer 28 1215
[778|1.36|3.066 46-4
Brisbane International 2012-12-31
ATP250
Outdoor Hard
1st Round
3 Benneteau J.
Nieminen J.
Nieminen J. 35
Brisbane international 2012-12-31
ATP250 Outdoor
Hard
1st Round
Nishikori K. Matosevic M. Nishikori
49] (45/).2553.33555555 7--662
History:
Considering contents
provided requirements
CHANGE
VARIABLE NAMES THAT HAVE SET!
Interactions begin:
(Interaction, history)
[USER (data scientist)]: We are interested in exploring the existence of any notable trends or shifts in how court surfaces tennis
influence player performance
across different years. accomplish Series Analysis, which will include the creation of line charts for visual representation, trend analysis to identify patterns, and the application statistical tests to confirm our findings. Following the analysis, could you identify there any type of court significant player performance observed?
plan conduct
surface for
A. Hard
Grass
Clay
Carpet
None of above
NOTE: Please generate ONLY ane turn this time and wait for User give Result based on your generated code segment, Do NOT generate the whole code single turn! And you can give the final enswer after "Answer:" at any turn when you are confident.
[YOU (Al assistant]]: Let's break down the code generation into several turns and solve the multi cholce question turn by turn! БиАНИииАыUллИААИИ The Answer Starts Here: #НАUИАИИИАИИИИИИЕ Turn 1:
i 5 turns left to provide final answer. Please only generate code segment in 'Code' (with proper print) and 'Act' in this turn, no need generate 'Result', NOT generate whole code single turn!
Thought First import all packages needed and load the dataset.
Code 1:
import pandas
atp_tennis pd_read_esv/ pead__sv(''ee gand__cv''tt__eeneeee htnn'atttenniisooo print/atp tenniis
Act 3: Exec[Code 1]
Result
Tournament Date Series Court Surface
Round
Best of Player 1 Player 2 Winner Rank 1 Rank. Pts Pts
Odd
score
Brisbane International 2012-12-31 ATP250 Outdoor
Hard
1st Round
3 Mayer F. Giraldo Mayer 28 1215
(778|1..6/3.006--55 international
Outdoor
Round
Benneteau
J.
41] 2 63-6666-1666622166
Brisbane International 2012-12-31
ATP250
Outdoor
Hard
1st Round
3 Nishikori K.
Matosevic M. Nishikori 19
49| 1452:3337/.
Turn 2:
answer. Pleese generate step TTouugh', segment 'Code' proper print) and 'cct' in this turn, no need generate 'Result'. Thought
Figure 23. The prompt of L.L with data analysis agent in MULTI-CHOICE, mode.
39

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Inter-Agent prompt for multi-choices
Solve question answering task with interleaving Thought, Code, Action, Results steps. Thought can reason about the current situation, and Action be three types:
[1] Execfcode), which execute the provided code with python and returns the code output it exists.
lerninatfansweer,, which
finishes the task.
Here an examples.
Example Start:
(Example)
Example End
The database table atp_ so_tennis is shown as follows:
Tournament Date Series Court Surface
Round Best
Player 1 Player Winner |Rnkk 1| Rank, 2 Pts Pts.
_dd_1 Odd_2 score
Brishane International 2012-12-31
ATP250 Outdoor Hard
1st
3 Mayer F. Giraldo Mayer 28 57
[778|1.96]3./|6-644
Brisbane International| 2012-12-31
ATP250 Outdoor Hard
3 Benneteau Nieminen Nieminen J. 1st
(1:107//27/2.221.6611
Brisbane International 2012-12-31
ATP250 Outdoor
Hard
1st Round
3 Nichikori K.
Matosevic Nishikori
19
49|1833| (11580//5)).22553777555555
History:
Considering contents from the table provided above and requirements from user. Please note DO NOT CHANGE FILE AND VARIABLE NAMES THAT
Interactions begin:
[Interaction jereacttio_histtry
[USER (data scientist]]:
or influence player performance within the ATP tennis dataset across different years. To accomplish this, plan to conduct a Time representation, analysis identify any patterns,
interested exploring
surfaces
Analysis, which the application of statistical tests confirm our findings. Follawing the analysis, could you identify there is any type of court significant in player performance was observed?
creation
surface for A. Hard
C. Clay
Carpet
E. None of above
Please firstly analysis the given pseudocade and follow my example between "Example Start" and "Example End" above to answer question interleaving Thought, Code, --Pseudocodde Starts---
(Pseudocode)
--PPsudocodd Ends--
Please generate ONLY one turn this time and wait for User to give Result based on your generated code sepment, DON'T generate
NOTE:
single you give final
"Answer:" any you confident,
assistanti]: break
generation
multi-choice
adaastennteennn The Answer Starts Here: tnasunssuneee
Turn # turns left to provide final answer: Please only generate a code segment in 'Cade' (with proper print) and 'Act' in this turn, no generate 'Result", NOT generate whole code single
Thought 1: First import all packages needed and load the dataset.
Code 1:
import pandas
rdar covl tonnis.csv
vrin(ettpttenns]]
Act Exec(Code 1]
Result 1:
Tournament Date Series Court | Surface Round Best Player: 1 Player Winner |Rank, 1| Rank, 2 Pts Pts.2
Oddd 1 [ddd_ score
Brisbane International 2012-12-31 ATP250 Outdoor Hard
1st Round
3 Mayer F. Giraldo S. Mayer 28 1 57 1215
(778|1.36|3.0|6-4
Brisbane International 2012-12-31 ATP250 Outdoor Hard
Benneteau J. Nieminen J. Nieminen J. 35 1st Round
1st Round
[1/105//22/22221..111 21
Brisbane International 2012-12-31
ATP250 Outdoor Hard
3 Nishikori K. Matosevic M. Nishikori 19
49| 1830 (118330//5/1.2//333 845/1.25|33.99/7-522
Turn
4 turns left provide final answer. Please only generate a small step in 'Thought', a code segment 'Code' (with proper print) and 'ec' in this turn, need generate 'Result'. Thought
Figure 24, The prompt of I.I.M with conversational data analysis agent in MULTI-CHOICE mode, The ACR prompt text are in green color. And the pseudocode is generated by L.M itself by learning from successful history
40

---

Are Large Language Models Ready for Multi-Turn Tabular Data Analysis?
Clarification Mode in code generation
There is the data: credit_ customers containing the following columns: ['checking staatus' 'duration', 'credit, "'redit_history!. 'purpose', 'credit amount', 'savings_ status', 'emplymmnnt' 'insta mnstallment_com conemttemmmmmentt 'personal status', 'other_ 'other_parties', parties', 'residence _inceene_ssnce',, since', "property_magee magnitude', 'age', satrr_pamme___aaggg 'housing', "existing_creddee crediits, 'job', 'nuummmmeee muum_dppendents 'own__te @wn_telephone 'foreign worker', "class"..
The description each column this data
[Column_ description)
Considering contents from the dataset and requirements from user. Please note DO NOT CHANGE FILE AND VARIABLE NAMES THAT HAVE SET! Interactions begin:
Interaction History: [Interaction hiisorr)
New Query:
[USER (data scientist]]: Please filter dataset to include only main course items such as sandwiches, wraps, and salads, exclude side dishes and desserts. Then, provide the filtered dataset containing only main DataFrame, filter include only rows where the 'item' column Please the keywards related to food (making search case insensitive), finally, save filtered snippet is: -BEGINN CODE TEMPLATE-
"astfood.csw dataset
of several pickle file.
My template
import pandas import numpy as np import pickle
atp_tennis pd.reedd_cs('ttppp csv('atp_teentii tenniisenns.)
YOUR SOLUTION BEGIN:
<codel>
[COMPLETE YOUR CODE]
</code1>
YOUR SOLUTION END
print(federer match_iateee
fickle.dump(fedeee pratch_ids,openff aaapeen""/dddedeeeeeeeeee result/federer__aachh... "wob"" --END CODE TEMPLATE--
"wb"))
Please that you have generate the WHOLE python code instead of code segments based the code snippet using Pandas library 2.0.3 version and Matplotlib library 3.7.4 version. You must keep comments code snippet unchanged.
talking your and your goal address the user's questions, It's very task that you have make all requirements from the user be fulfilled without any uncertainty. Any missing details wrong assumptions may lead to failing and you will be fired! Now, you have chance ask User most ONE question between <question>YOUR, QUESTION</questin you uncertain about the latest query. Otherwise, user query.
very certain, you directly
Ask clarification:
Could you please specify the keywords related to fast food items that you want to filter by?
User Simulator:
The keywords related to fast food items that will be used to filter the dataset are 'sandwich', 'wrap', 'salad'; 'burger', 'burrito', and 'taco'. Now have to generate Python code based the code snippet to answer the latest User query.
Figure 25. The prompt of LLM in Model-Banr -Bas version in CLARIFICATION mode.


Reviews:
[{"review_id": "MQQP7RWTDV", "reviewer": "Reviewer_ez8j", "summary": "This paper proposed a benchmark, namely COTA, and a multi-agent environment named Decision Company to evaluate the performance of  LLMs in the task of Tabular generation. The paper used the proposed benchmark and multi-agent environment to evaluate the performance of  8 LLMs in the task of tabular generation.\n\nI have concerns in Client Persona Generation. In this step, the 'Persona' are generated by actually prompt engineering using a LLM. Although expert supervision is involoved, the output generated from LLMs may still differ from the real human. Moreover, the agents are given certain information of a faked person, which may not be representative to all the professionals in that task. The authors may consider use RLHF to enable the agents to mimic the professionals in a better way.\n\nAlso, the proposed benchmark does not include other fields which maybe more challenging, such as medical, marting, and supply chain.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Concern 1:Client Persona Generation**\n\n**Ans:** Thank you for this valuable feedback. While LLMs were utilized in persona generation, we implemented a rigorous multi-stage validation process specifically to address potential discrepancies between LLM-generated content and real human behavior. Our approach includes:\n\nExpert supervision at every stage of persona development, where data analysis professionals with 10+ years of experience reviewed and modified the personas based on their real-world client interactions.\nHigh inter-annotator agreement (92.78%), suggesting strong consistency in the evaluation of these personas by multiple domain experts.\nComprehensive human evaluation on multiple metrics, including Scenario Diversity and Reasonableness (0.96), Conversation Topic Coherence (0.93), and Conversation Naturalness (0.95), demonstrating the real-world applicability of these personas.\n\nFurthermore, our human-in-the-loop approach was intentionally designed as a practical middle ground between purely synthetic data generation and expensive expert crowdsourcing.\n\n**Concern 2: Persona Representativeness**\n\n**Ans:** We appreciate the suggestion regarding RLHF for better mimicking professionals. Our current approach prioritizes diversity and domain expertise. The personas were intentionally designed to cover a wide range of domain-relevant scenarios (18 analysis topics across 5 common domains) with varied backgrounds and needs.\nActually, the proposed DECISION COMPANY framework already incorporates expert feedback loops that serve a similar purpose in guiding agent behavior toward realistic professional practices. Our evaluation with 10 data analysis experts outside the author team validated that the scenarios and interactions were highly representative of real-world data analysis tasks.\nFor future work, we are exploring how RLHF techniques could further enhance the quality of agent-based simulations within our framework, especially for extending COTA to additional domains.\n\n**Concern 3: Domain Coverage**\n\n**Ans 3:** Thanks for suggestion. Our current focus on financial, sports, food, consumer electronics, and housing was intentional for three key reasons:\n\n- These domains feature widely available open-source tabular datasets with minimal privacy and ethical concerns, enabling broad accessibility of the benchmark.\n- These domains represent common data analysis scenarios that don't require highly specialized domain expertise to validate, ensuring reliable quality assessment.\n- Additionally, our current data already covers advanced topics such as Healthcare (Food),  Financial (Credit Card, Bank) shown in Fig. 9.\n\nWe view COTA as a foundation that can be expanded to more challenging domains. The evaluation framework, conversation action types, and metrics we've developed will transfer well to specialized domains in future extensions of this work.\n\n**Question:**\n\n**Ans 4:** Thank for asking this. Due to limited resource at this time, we test the performance of Claude-3.5-Sonnect against GPT-4 and CodeLlama on CoSQL a conversational text-to-SQL benchmark:\n\nPerformance (EX) comparison on other datasets. \n| Model Name | CoSQL |\n|----------|----------|\n| CodeLlama | 35.7 |\n| GPT-4 | 68.2 |\n| Claude-3.5-Sonnet | 58.6 |\n\nFrom this table, we can observe that Claude-3.5-Sonnet didn't outperform GPT-4, which means different advanced models maybe strong in different programming languages due to different training corpus. In our Python-based data analysis code generation, claude-3.5-sonnet is the strongest model.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Do not apply.", "experimental_designs_or_analyses": "Yes. I checked the validity of the experimental design.", "relation_to_broader_scientific_literature": "This paper contributes to the task of Tabular data generation. It provides a dataset and methodology to evaluate the Tabular generation.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Consider to enhance the Persona Generation step. Also consider to include more domains.", "other_comments_or_suggestions": "No.", "questions_for_authors": "The Claude-3.5-Sonnet outperforms other models according to Figure 6. Is this the same case when using other dataset as shown in Table 1?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3Xo6RCc2Q", "reviewer": "Reviewer_xy4y", "summary": "# Summary\n\nThis paper constructs a novel benchmark for the task of \"conversational tabular data analysis.\" The benchmark is constructed using a complex process of interacting agents, aided with human annotation, starting from a set of 5 \"data sources\" and 18 \"topics\". This ultimately yields a benchmark of approximately 1k examples, over which the authors conduct a benchmarking study.\n\nThis is a very important area of research and it requires both rigorous empirical studies, and reliable high-quality benchmarks; the current study is an attempt at both. However, I have some concerns about the current study. The dataset creation process is extremely complex. This has two consequences: (1) it is difficult for the authors to describe it with sufficient detail and clarity; (2) it makes it difficult to assess whether their design choices (of which there are many) affect the results of the benchmark. I also have some concerns about the value of the empirical insights gleaned from this study. Overall, I think that this is an important direction, but that the paper is not ready for acceptance in its current form.\n\n# Major comments\n\n* The dataset design process is almost incomprehensibly complex, and it is made even more difficult to understand by the writing in the paper. The are so many points that aren't clearly described; here are just a sample:\n- The authors state (L78) that \"multiple choice questions\" are a response type. How can this be evaluated, and how can all responses be distilled into either code generation or multiple choice types? What is an example of this kind of response? This is not clear.\n- The authors introduce 6 action types, saying simply that \"we identify 6 common actions during conversations\". How are these identified? How do we know that these represent a complete action space for conversational tabular data analysis tasks? These actions are also not clearly described; for example they define the Plot_QA action by simply saying \"The Plot_QA action helps users understand plot-derived insights\" -- but this is not a definition.\n\n* The data used as the basis for the benchmark are not clearly described. Relegated to the Appendix, in B.3 and Figure 9(a), we see that there are \"18 topics and 5 sources of COTA\". This still does not answer basic questions, such as: how many tables comprise the benchmark, and exactly which ones? What is the difference between a table and a \"source\"? This is critical, fundamental information about the benchmark that is not clearly provided in the paper, and makes the results nearly impossible to reliably assess.\n\n* Simple procedures, such as evaluation, are also not clearly described. For example, the paper says that \"Each question is provided by an expected result type, such as dataframes, lists, or various plot types\" but does not enumerate the expected result types. Similarly, the evaluation metrics are not clearly defined. For example, the definition of AccR, one of the core evaluation metrics, is simply limited to \"we extend Acc to include a recall-based adjustment for instances involving private libraries\" which does not give any details about the metric.\n\n* Collectively, the above issues make the empirical results nearly impossible to assess. Furthermore, the authors' evaluation of the existing empirical results is quite limited and does not seem to lead to new insights about how to improve models.\n\n# Minor comments\n\n* The paper relies a lot on annotations by the authors. For example, the authors do the human-sandbox annotation in Section 2, and the error analysis in 7. It would be more reliable (less prone to bias) to have external annotators conduct these annotation steps.\n\n* Why is only GPT-4-32k shown in Figure 7? It is not the best-performing model, and there isn't any other clear differentiator that makes this model particularly interesting to plot versus the others.\n\n# Typos etc.\n\n* \"Conversational Tabular Data Analsis\" in abstract does not need to be capitalized.\n* Section 1: \"Among the vast types of data available, tabular data stands out as one of the most prevalent and interpretable formats organized by rows and columns\" -- isn't tabular data the only format organized by rows and columns, by definition?\n* In several places the paper uses the word \"codes\" where the authors seem to mean \"code\" (as in, Python code).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Additional Evidence & Paper Reference in: https://anonymous.4open.science/r/additional_materials-E646**. We will use `B.X` to index evidence in the following rebuttal:\n\n**C1: Why the annotation is complex and annotators are also authors?**\n\n**A1:**\nBecause the task that we are researching is complex. Unlike traditional NLP or mathematical tasks that rely on common knowledge or basic skills, our benchmark demands specialized expertise in tabular data analysis, statistical reasoning, coding proficiency, and domain experience. These requirements make conventional crowdsourcing approaches impractical for several reasons:\n\n- The technical depth required exceeds what typical annotation platforms can support.\n- The extended timeline is necessary for comprehensive task development and refinement.\n- The iterative workflow requires continuous expert feedback and adaptation.\n\nThis is also reflected in very recent related work: BIGCODEBench and SPIDER 2.0 (ICLR 2025) as Reviewer K6vh mentioned. These projects similarly implemented sophisticated workflows for their tasks and recognized substantial annotator contributions through authorship. \n\nOur benchmark presents greater complexity than previous efforts. Therefore, our benchmarks deserve more sophisticated method to construct and authorship for annotators for their intellectual contributions, which is also trending of complex task benchmarking.\n\n**C2: Code Generation & Multi-Choice Annotation:**\n\n**A2:** In COTA, we deliberately designed two distinct answer types to enable objective and consistent evaluation:\n\n- Code Generation: This involves LLMs generating Python code to analyze tabular data. This code is evaluated through execution-based test case scripts.\n- Multiple-Choice Questions: After code execution, users often need to interpret results and make decisions. Rather than accepting free-form text analysis, which is difficult to evaluate objectively, we convert analytical questions into multiple-choice format.\n\nFor example, after analyzing ATP tennis data, instead of asking \"What trends do you see in player performance?\" (subjective, and require LLM-as-Judge to evaluate), we frame it as:\n```\nWhich surface shows no significant performance trend over time?\nA. Hard\nB. Grass\nC. Clay\nD. Carpet\nE. None of above\n```\nThe evaluation process of multi-choice is straightforward - we compare the model's selected option against the ground truth answer. Each multiple-choice question has a single correct answer determined during dataset construction and verified by our expert annotators. This can eliminate ambiguity in subjective assessment, which may introduce evaluation bias.\n\n**C3: 6 action annotation and plot_QA**\n\n**A3:** Our identification of these 6 common actions stems from multiple rigorous sources of evidence rather than mere assertion. As we discussed in L187-188, annotators summarized actions and inject into conversations by two resources:\n- The annotators have> 10-year experience of data analysis (L135), ensuring strong expertise. This expertise is further validated by our high initial inter-annotator agreement rate (detailed in line 217). Also human expert (**outside annotator**) evaluation in Section 5 (**Action-wise Metrics**) shows its high Action Commonness. Detailed evaluation script in Appendix P. \n- Also, the action types were derived through systematic analysis of prior literature and empirical observation. Each action is grounded in but not limited to established research:\n\n**Update Code (for debugging)**: given code generation is crictical in data analysis [1, 2, 3]. \n**Fast Fail**: See Appendix J, [4, 8]\n**Clarification:** [6-9]\n**Best Guess:** [4, 5]\n**PlotQA:** Chart/Plot is one of most common data format during data analysis [1, 9].\n\n**C4: Data Source Description**\n\n**A4:** Please see Tab. 1-5 of `B.1` in our anonymous link, it contains 45 large professional tables (> 5k rows, > 40 cols on avg), the \"source\" means domains.\n\n**C5: Details about metrics and Result Type:**\n\n**A5:**\n- Evaluation Metrics: We will feel grateful if you cloud give a look at Appendix K, M\n- Result Type Enumeration: We have catalogued all result types and their distributional characteristics in Figure 13(b) of Appendix K. To improve navigability, we will add direct cross-references in the revised manuscript. Thank you.\n\n**C6: Why is only GPT-4-32k shown in Figure 7?**\n\n**A6:** \nThe visualization demonstrates how the Inter-Agent approach with ACR (light blue) generally outperforms both the base model and standard Agent configurations across most categories. GPT-4-32k likely serves as this case study because it represents a middle-ground performer (and one of most available resources). We have results on Claude-3.5-Sonnet which contains similar findings in Tab. 6 in `B.2`.\n\n**C7: More Reference**\n\n**A7**. Due to page limit, we have included a more comprehensive literature review in Appendix O. We will expand Section 8 in the camera-ready version if accepted.", "claims_and_evidence": "See above.", "methods_and_evaluation_criteria": "See above.", "theoretical_claims": "See above.", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "See above.", "essential_references_not_discussed": "See above. I would suggest additional references to existing works that apply language models to tabular data tasks (of which there are several).", "other_strengths_and_weaknesses": "See above.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zPpwNaTaVd", "reviewer": "Reviewer_K6vh", "summary": "This paper introduces CoTA, a benchmark to evaluate the effectiveness of LLMs in multi-turn conversational tabular data analysis scenarios. The authors' motivation is to address the lack of realistic, quantitative evaluation datasets by creating conversational data through an innovative multi-agent sandbox environment. CoTA includes diverse conversations scenarios to rigorously assess conversational abilities of LLMs. Additionally, the authors propose ACR, a self-generated reflection strategy to improve LLM performance, achieving notable enhancements over baseline approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Paper Reference in: https://anonymous.4open.science/r/additional_materials-E646**.\n\n**C1: Comparison with Spider 2.0**\n\n**A1:** In summary, our benchmark differs from Spider 2.0 in several important aspects: \n- We focus on **conversational multi-turn** interactions for Python code, while Spider 2.0 primarily evaluates **single-turn** text-to-SQL capability; \n- Our benchmark includes more data analysis and scientific questions, including statistical and machine learning tasks such as clustering and linear regression; \n- Spider 2.0 emphasizes schema understanding and retrieval since their input data is larger (> 1000 columns), whereas our work prioritizes conversational compatibility and realistic data science problems. We will add these discussion in the Related Work section and Tab. 1 if fortunately given 1 additional page in camera-ready version. Thanks!\n\n**C2: Annotation Cost**\n\nThanks for asking. Similar to Spider 2.0 and BigCodeBench for complex task annotation, we recognize annotators through authorship acknowledgment rather than direct financial compensation. Fortunately, to maintain data privacy and prevent GPT API abuse, annotators accessed our controlled annotation system, which documented a total working time of 3,677 minutes and accumulated GPT usage costs of  71.39 USD. While we did not provide monetary compensation to annotators, we can estimate the equivalent human expert cost by referencing established rates from research involving real data analysis experts [13-14], which indicates an average rate of 0.37875 USD per minute. Using this conversion, total human annotation would amount to 1,392.66 USD.\n\nTherefore, the comprehensive benchmark development cost totals 1,464.03 USD. This represents a cost-effective and scalable approach, with each data point in COTA costing approximately **1.45 USD**, significantly more economical than comparable benchmarks such as BIRD-SQL (**6.13 USD**) and TableBench (**6 USD**), despite COTA's increased complexity and deeper expertise.\n\n**C3: Is it possible to generate conversations using open-source models like Llama?**\n\n**A3:** We appreciate this question. Currently, we found that weaker LLMs struggle with this task. When we initially experimented with GPT-3.5-Turbo (the most popular LLM at that time), it exhibited significant hallucination issues after 2.5 conversation turns on average and failed to follow the Analysis Plan generated in Section 3.1. This required more expert workflow even than was generated totally by themselves.\n\nTherefore, we utilized GPT-4, the most capable model available when starting this project, as base model for sandbox construction. As shown in Figure 3, our sandbox based on this can support very long-turn conversations (14.15), similar to realistic complex task-solving scenarios. External human evaluations in Section 5 confirm the quality of these conversations.\n\nRecently, we observed that newer models like Llama 3.3 70B demonstrate improved capabilities for high-quality data annotation and data analysis questions, as shown in Table 3. This model could potentially serve as a replacement for GPT-4 in future studies. Thanks for your suggestion.\n\n**C4: Potential Limited Diversity**\n\n**A4:** The main reason why we use GPT-4 is its broad and diverse knowledge due to intensive pre-training. In the paper, we show scenario diversity evaluation in Tab.2 of **Section 5** and detailed fine-grained diversity analysis in **Appendix P.2** in terms of domain, result type, action, query, package, which actually shows our work already covers most comprehensive aspects of data analysis compared to related works as far as we know.\n\n**C5: What are the practical applications of conversational tabular analysis?**\n\n**A5:** Tabular data analysis is ubiquitous in daily operations.  Automatic tabular data analysis can help users make informed decisions through natural language interactions, without requiring specialized skills in complex tabular data understanding, coding or even domain knowledge, which also can improve efficiency for data scientists or relevant users. \n\nAs we stated in L 23-32, users rarely express their intentions completely in a single turn and often have follow-up questions based on previous responses or other actions (as we summarized and listed in the paper). Therefore, conversational capability is necessary for complex tasks. All Leading AI assistants like ChatGPT, Claude, and DeepSeek support multi-turn interaction because conversation is the most natural form of human communication. Therefore, conversational tabular data analysis can accelerate and improve decision making for finance, health care, policy making, which usually store their records and data in more structrued format. \n\nIn this case, a comprehensive, scalable (to prevent data leakage) benchmark should be proposed to make users understand the capabilities of LLMs.", "claims_and_evidence": "1. The authors argue that the ACR method significantly improves conversational agent performance, and empirically evidenced.\n\n2. The authors claim that CoTA is a scalable and realistic benchmark for conversational data analysis. However, the authors may show its scalability in the aspect of the cost (since they use LLMs to generate them). In addition, while their purpose is really similar to text-to-SQL tasks, CoTA lacks comparison with them.", "methods_and_evaluation_criteria": "1. They use Accuracy as metric, to measure correctness of code generation and answers.", "theoretical_claims": "As best as I know, there are no theoretical claims.", "experimental_designs_or_analyses": "The authors tested several advanced LLMs, including Mistral, Llama, Claude, and GPT-families, across four conversation scenarios, i.e., Normal, Action, Private, and Private Action. They also performed extensive error analysis.", "relation_to_broader_scientific_literature": "The literature is closely related on LLM-based code generation such as text-to-SQL.", "essential_references_not_discussed": "I think Spider 2.0 [1] should be discussed in the paper, since it is also a benchmark that tries to make realistic text-to-SQL scenarios where tabular data is really messy and the generated code (i.e., SQL) is really long.\n\n[1] Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows, ICLR 2025.", "other_strengths_and_weaknesses": "**Strengths**\n1. Comprehensive benchmark addressing realistic multi-turn conversational scenarios.\n\n2. Effective use of a novel multi-agent sandbox for dataset creation.\n\n3. Extensive experimental results, including multiple model comparisons and error analysis.\n\n**Weaknesses**\n1. Limited explicit discussion on scalability constraints or the cost implications of human-in-the-loop annotations for widespread practical use.\n\n2. Relies heavily on GPT-4 based agents for conversation generation, possibly limiting diversity in generated data.", "other_comments_or_suggestions": "See above weaknessess.", "questions_for_authors": "1. What is the main differences of benchmark characteristics, compared to Spider 2.0.\n\n2. Can you provide a used cost for generating conversations, and also used of human annotators?\n\n3. Is it possible to generate conversations using open-sourced models like Llama?\n\n4. What can be the practical applications of such conversational tabular analysis?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wb0xpP97je", "reviewer": "Reviewer_Wqoz", "summary": "The authors create a synthetic conversational dataset with dialogues about the data tables. The authors scrape the tables from Kaggle. To generate the conversation, the authors organize a multi-agent multi-turn conversation where each agent is an LLM instance prompted to play a specific role.The authors then perform extensive human expert validation of the generated conversions and ground truth formulation as MCF or code. A thorough comparison to 8 competing datasets/benchmarks is performed. Having the dataset in place, the authors propose Adaptive Conversation Reflection (ACR) - an agentic setup that learns from the conversational dataset and improves the scores over the naked LLM and CoT-LLM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Additional Evidence in Anonymous Link: https://anonymous.4open.science/r/additional_materials-E646**. We will use `B.X` to index evidence in the following rebuttal:\n\n**C1: The name Decision Company is not clear, specifically, why “decision”?\"**\n\n**A1:** The name \"DECISION COMPANY\" refers to the multi-agent sandbox environment created by the authors to simulate realistic data analysis scenarios. Based on the paper, this environment is designed to support data-driven decision-making processes. The name likely reflects its purpose: facilitating decisions through data analysis in a company-like setting where different agents (Administrator, Client, Data Scientist, AI Chatbot) interact to answer analysis questions for decision making. \n\n**C2: Suggested Formula of Action Types**\n\n**A2:** Thank you for the suggestion. Given that our benchmark evaluates LLMs on separate actions, we initially didn't formulate each action individually. For more mathematical rigor, we can indeed extend our notation as follows:\n\nLet $s_i \\in S$, where $s_i$ represents the selected action mode from an enumerated set of available actions listed in Section 2. \n\nThe task formulation could then be improved as generating answers:\n$a_{it} = f_\\theta(u_t, H, T, s_i)$\n\nThis represents that given conversation history $H$, tabular data $T$, and the current query $u_t$, the agent's response $a_{it}$ is conditioned on the specific action mode $s_i$. \n\n**C3: What is the definition of a “logic”? What are curly brackets in “Re-Org One-Shot Reasoning” section? (This is the most IMPORTANT question).**\n\n**A3:** Thanks for this question.\n\nIn Figure 11, \"logic\" represents the intermediate reasoning process that connects natural language queries to executable code or analytical answers. More formally, $m_{t-1}$ denotes the **inferred** reasoning pathway between a user query $u_{t-1}$ and the corresponding answer $a_{t-1}$ from the previous turn. This intermediate representation functions as pseudocode, which is a structured thought process that bridges natural language intent and formal execution.\n\nThe curly brackets in the \"Re-Org One-Shot Reasoning\" section serve as an organizational construct. The notation $p_{t-1} = (u_{t-1}; \\lbrace m_{t-1}; a_{t-1} \\rbrace)$ mathematically represents our one-shot example structure, where:\n- $u_{t-1}$ is the previous user query\n- $\\lbrace m_{t-1}; a_{t-1} \\rbrace$ represents the pairing of inferred logic and answer. **In the prompt, this part is specifically highlighted with special symbols** to lead LLMs to follow this reasoning procedure.\n\nBy structuring examples to show the reasoning process $m_{t-1}$ followed by its corresponding answer $a_{t-1}$, we enable the model to learn the pattern of first generating logical reasoning steps before producing final answers. This significantly improve performace of LLMs in conversational data analysis tasks in a simple manner.\n\n**C4: Data Resources & Dataset Liscense**\n\n**A4:** Thank you for this important question regarding our dataset resources and licensing. \n\nWe have indeed provided comprehensive license information in our paper's Appendix B. Specifically, in Appendix B.1, we clearly state that the COTA dataset is available under the CC BY-SA 4.0 license (Creative Commons Attribution-ShareAlike 4.0 International).\n\nRegarding the source data, Appendix B.2 documents that all tabular data utilized in constructing COTA were obtained from Kaggle under either:\n1) Public Domain Mark designation, or\n2) CC BY (Creative Commons Attribution 4.0 International) licensing\n\nAlso, we provide all detailed liscenses in `B.1` in the Anonymous Link. We would appreciate if you could go through it. Thanks.", "claims_and_evidence": "All the claims of the paper are well supported.\n\n1. The reasoning over the table data is an important field of study\n2. The dataset creation procedure is rigorous enough. Even though the original dialogues are synthetic, they are well validated and curated by human experts.\n3. Formal (objective) automated evaluation ground truth and metrics are worked out.\n4. The feature coverage compared to the baseline benchmarks is rich.\n5. ACR seems to improve the scores over a fairly strong baseline of LLM-CoT.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "I am not aware of any essential literature that was missed.", "other_strengths_and_weaknesses": "The idea of having a “private” version of pandas API is interesting and helps to assess how well LLMs close the Open/Closed source domain gap.", "other_comments_or_suggestions": "The citation of “Li et al., 2024a” is incorrect, should be “Li et al, 2023”.\n\nThe term “Action” is not very clear. The mentioned term “Clarification” scenario would be more suitable as the main name.", "questions_for_authors": "Writing could be slightly improved.\n\n1. The name Decision Company is not clear, specifically, why “decision”? \n\n2. The consistency of the terminology is not ideal. The authors write:\n\n094  efficient creation of COTA\n\nWhereas:\n\n034 In this paper, we introduce COnversational Tabular data Analysis (COTA).\n\nThe derived phrase “creation of analysis” is not correct.\n\n3. “Action mode” and “Action types” lack mathematical notation for them. How are they related to the notation of the Task Formulation section?\n\n4. What is the definition of a “logic”? What are curly brackets in “Re-Org One-Shot Reasoning” section? (This is the most IMPORTANT question).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Legal Compliance (e.g., GDPR, copyright, terms of use)"], "ethical_review_concerns": "What are the licenses of all the tables that you had harvested from Kaggle?"}}]

Author rebuttal:

Meta-review: This paper proposes a synthetic conversational tabular data benchmark. While I understand the strengths of the paper, the reviewers mention few weaknesses, such as complex setup, unnatural data creation, and presentation issues. As is, this makes a weak accept.
Decision: Accept (poster)