InsightBench: Evaluating Business Analytics Agents Through Multi-Step Insight Generation
Authors: Gaurav Sahu
Abhay Puri
Juan A. Rodriguez
Amirhossein Abaskohi
Mohammad Chegini
Alexandre Drouin
Perouz Taslakian
Valentina Zantedeschi
Alexandre Lacoste
David Vazquez
Nicolas Chapados
Christopher Pal
Sai Rajeswar
Issam H. Laradji
Venue: ICLR 2025 Poster (2025)
Keywords: Automated Data Analysis
Data Analytics Benchmark
LLM agents
Code Generation
LLM Evaluation
Primary area: datasets and benchmarks
TL;DR: We propose a comprehensive benchmark to evaluate LLM-based agents on their ability to perform multi-step data analysis and discover interesting insights in data.

Abstract: Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We introduce InsightBench, a benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business use cases such as finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement a two-way evaluation mechanism using LLaMA-3 as an effective, open-source evaluator to assess agents’ ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance of open- and closed-source LLMs and various evaluation strategies. Overall, this benchmark serves as a testbed to motivate further development in comprehensive automated data analytics and can be accessed here: https://github.com/ServiceNow/insight-bench.

Published as a conference paper at ICLR 2025
INSIGHTBENCH: EVALUATING BUSINESS ANALYTICS AGENTS THROUGH MULTI-STEP INSIGHT GENERATION
Gaurav Sahu' 16 Abhay Puri' Juan Rodriguez!:2 12,4 Amirhossein Abaskohi
Mohammad Chegini" Alexandre Drouin' Perouz Taslakian'
Valentina Zantedeschi' Alexandre Lacoste' David Vazquez' Nicolas Chapados
Christopher 1.33 Sai Rajeswar Mudumbal Issam Hadj Laradji
ServiceNow Research
Mila Quebee AI Institute
$Canada CIFAR AI Chair
'École de Technologie Supérieure
University British Columbia University Waterloo
7 University of Victoria
ABSTRACT
Data analytics is essential for extracting valuable insights from data that can assist organizations in making effective decisions. We introduce InsightBench, benchmark dataset with three key features. First, it consists of 100 datasets representing diverse business cases such finance and incident management, each accompanied by a carefully curated set of insights planted in the datasets. Second, unlike existing benchmarks focusing on answering single queries, InsightBench evaluates agents based on their ability to perform end-to-end data analytics, including formulating questions, interpreting answers, and generating a summary of insights and actionable steps. Third, we conducted comprehensive quality assurance to ensure that each dataset in the benchmark had clear goals and included relevant and meaningful questions and analysis. Furthermore, we implement two-way evaluation mechanism using LLaMA-3 as effective, open-source evaluator to assess agents' ability to extract insights. We also propose AgentPoirot, our baseline data analysis agent capable of performing end-to-end data analytics. Our evaluation on InsightBench shows that AgentPoirot outperforms existing approaches (such as Pandas Agent) that focus on resolving single queries. We also compare the performance evaluation strategies. Overall, this benchmark serves testbed to motivate further development in comprehensive automated data analytics and can be accessed here: attps://github. com/ www.o//eeiieeooi.e..eeooo insight -bech
open- and closed-source L.MMs and various
INTRODUCTION
Businesses frequently leverage vast datasets to perform data analytics to uncover insights, discover pattems.. and analyze trends in order to support effective decision-making (MeAfee Brynjolfsson, 2012 Colson 2019 Bean 2022). The task of end-to-end data analysis starts with stating high-level goal; the analyst then alternates between identifying key questions to explore, and extracting valuable insights from their answers, working towards the goal. This iterative process continues until they have comprehensive summary of insights and recommended actions (shown in Figure D.
Analysts often carry out such tasks by leveraging tools like Jupyter notebooks and dashboards (Bean 2022 Yin et al. (223). which, unfortunately, require both significant human effort as well as data science and domain expertise. Fortunately, agents based on large language models (LIMMs) have emerged as promising

---

Published as a conference paper at ICLR 2025
Existing Benchmarks
Expert
Araver wihoo whhoo Acdoviblo inigtt
Exper Quention Speciili Tank:
Thy saueen waluts anund
reaectee oollit caca
lrsn and tacc sesccilluuiii dufuudyy the slts son dl cauft usee of direrninadi
wifh ve stily
comtiia autuncatie dilis ribicc toap p012 rol right sugur
It.
InsightBench (Ours)
General User
AgentPoiret
Surmay
take
spplone the aire.
Specitc Tack
of
Inigh
Seak
& Predectee Speetlo Tesk
ty Premripht Speake Tork
Figure 1: Existing benchmarks (top) assess the agents' ability to solve highly specific data analytics query with pre defined output templates. They often require expert user ask the questions. InsightBench (bottom), the other hand, evaluates the LLM- based agents on the complete comprehensive data analytics processes. This includes evaluating the agents' ability to answer high-level questions by general user, recommend the best specific tasks to address a specific goal, extracting insights across descriptive, diagnostic, predictive, and prescriptive categories, and summarizing both findings and recommending next steps (as demonstrated by AgentPoirot).
assistants users in performing data science tasks (OpenAl 2022 2023). But these agents (such as
Pandas Agent (LangChain) (2024) and Code Interpreter (OpenAl (2044) only focus on solving narrow, single-step code-completion (or data analyties) tasks (E.g.. "What the R² value for linear regression model on this dataser?").
One of the reasons most existing agents have limited data analysis capabilities is because current benchmarks focus on evaluating agents on single-step code-compp code-completion tasks (Lai et al. Chen et al.) 2021a| Hu et al. 2024). Another reason is that it is challenging to evaluate LLMM-based agents accurately without including substantial information about the problem details, expected code structure, and expected outputs (such as identifying variable correlation) in the prompt. Recent works such as InsightPilot (Ma) (et al. 2023) and InsightLens (Weng et al. 2024) propose agents that can perform end-to-end data analytics, but they were evaluated using human assessments, which require significant effort and often lack consistency due variability in individual biases and interpretations of evaluation criteria.
[he2n
Constructing a standardized, automated benchmark to address the consistency and reliability concerns involves tackling two main challenges. The first concern is to ensure that the agent explores the most informative and interesting questions, given the data and goals. Second, we must ensure that the insights derived from these questions are many ways to ask interesting questions about the data and different ways to interpret their answers. To address these issues, we need datasets with clearly defined data and goals, equivalent analysis performed by an expert data scientist.
accurate and relevant. These challenges stem from the fact that there
This inspired us to propose InsightBench, which consists of 100 datasets whose structures have been acquired from the ServiceNow (ServiceNow (2044) platform, which focuses on business operations and workflows. This platform allows us to generate synthetic tabular data that mimics real-life enterprise data. We used it to create datasets for Incident, User, Finance, Inventory, and Enterprise Goal Management. We chose this data because an agent capable of effectively extracting insights from it can be deployed to help organizations optimize their business operations.
To ensure the quality of goals and questions in our datasets, we had expert annotators set clear goals and manipulate the data to plant interesting insights. Each insight includes a question, generated code, and plot values from which the insight is extracted. The annotators then aggregate these insights, summarize them, and provide recommended actions. We ensure that our insights are discoverable, good LLM-basee agent performing analytics on the data should recommend similar questions and extract similar insights the ones planted by the annotators.
2

---

Published as a conference paper at ICLR 2025
To effectively score the agent's ability to predict the most relevant insights, need scoring method that can compute the semantic and factual similarities between two free-form texts. In our case, we need to score how closely the predicted insights match the ground-truth insights.
Therefore, we draw inspiration from G-Eval which a state-of-the-ar technique that uses GPT-4 to evaluate the quality of generated texts in manner that aligns closely with human judgment (Liu et al. 2023). Since GPT-4 is costly and closed-source, we replace with Llama-3 (Touvron et al. 2023) and call the technique LLaMA-3-Eval. Our experiments show that LLaMA-3-Evall and G-Evalproduce consistent outcomes when ranking the performance of agents in extracting insights.
We benchmark LLL--baee agents on InsightBench, where we variant of the Pandas Agent that perform end-to-end data analysis by recommending questions and extracting insights from their answers. We also propose baseline agent, AgentPoirot, which can extract deeper insights and questions by carefully designed prompts covering Descriptive (what happened), Diagnostic (why it happened), Predictive (what will likely happen), and Prescriptive (what actions to take) analytics. Our evaluation shows that AgentPoirot obtains higher LLaMA-3 --vaall score compared to Pandas Agent.
To summarize, our paper makes the following contributions:
We propose Insightßench, the first benchmark specifically designed to evaluate M.-M-based agents for performing end-to-end, multi-step data analytics.
We present comprehensive set of experiments contrasting open-source and closed-source models and different prompting strategies for evaluation on our benchmark. Our results show that AgentPoirot outperforms existing analytics methods such as Pandas Agent and LLaMA-3-Eval is feasible alternative to using the dlssed-oource G-Eval.
2 InsightBench AN ENTERPRISE DATA ANALYSIS BENCHMARK
InsighuBench's s data were derived from the ServiceNow (ServiceNow) 2024) platform, which is used to manage business workflows crucial for enterprise operations. It consists of tables that store and manage records and data entities relevant to various organizational functions. Common tables in ServiceNow include incident tables (for incident records), user tables (for user information). and asset management tables (to oversee infrastructure). InsightBench consists of 100 tabular datasets acquired from the ServiceNow platform covering five distinct themes (see Figure 3}.
Overview of Benchmark Creation. InsightBenchh is an automated data analysis benehmark that can rigorously assess the performance of L.L.-based data analysis agents in real-world scenarios. Building InsightBench consisted of four stages: 1) selecting relevant data tables from the ServiceNow data tables and extracting a list of relevant columns to define the schema (Section 1.I.I) 2) formulating trends to inject in the data (Section 2.1.2. 3) creating synthetic data entries to populate the tables that follow the trend formulated the previous step (Section 2.1.3). and 4) creating a ground-truth analysis notebook for the generated data (Section 2.1.4.. See Figure for an overview of the multi-stage process. A detailed list of the datasets sorted by themes is presented in Appendix [B.]] We propose a multi-step evaluation mechanism measure the performance of an agents on InsightBench (See Section 2.2)
2.1 CREATING InsightBench DATASETS
Each dataset consists of 500 synthetically generated entries, stored as CSV file. We choose size of 500 entries keep the volume of data manageable for analysis and substantial enough to simulate typical enterprise data loads. To emulate a realistic enterprise environment, InsightBench uses a hybrid approach combining actual data structures with synthetic insights.
2.1.1 DATA TABLES
To create the datasets, we first select relevant data tables from ServiceNow system tables. As guiding example, consider an incident table, which contains records of any disruption to normal service operations. The incidents can range from server outages to hardware malfunctions, such as non-operational printer. This table is structured with fields (aka columns) relevant to managing the lifecycle an incident, such as the time it opened, the description of the incident, and assigned agents (See Figure a)). We
3

---

Published as a conference paper at ICLR 2025
servicenow
Inddent
Iraight
Insight 2
Insight (Fleg)
Data Agent
tnoight type
description
Figure 2: Stages of Benchmark Creation: (a) Shows demo incidents table from ServiceNow for the Incident Management theme, detailing schema and data fields. (b) Demonstrates process of creating one of datasets the benchmark that embeds linear increasing trend incident resolution times, also highlighting the role of the slope parameter in dictating the trend's 's strength. (c) Displays the multi-step annotation analysis, with cach step involving a question, corresponding plot, insightful description, and classification of the insight type.
choose appropriate columns that allow us to plant realistic trends (as explained in the next subsection) while excluding irrelevant columns.
2.1.2 PLANTING INSIGHTS
We embed systematic anomalies and trends into the datasets to certain types of columns the datasets, referred to as controllable columns. These trends were implemented using standard mathematical models or algorithms, such as regression, to ensure consistency and detectability of the planted insights. For instance, manipulated the tome-ooresolutiinn (TTR) trend for incidents in service management dataset to reflect an increasing trend over time. This was achieved by using a linear model to generate the TTR based the creation date of each incident: TTR 1 slope-(incident (incident open date data start date), where (slope) a parameter that controls the rate of increase in resolution time (See Figure
Types of Insights. Every insight in InsightBench
be categorized into four types, where each type serves specific function in data analysis, contributing uniquely to the utility of the insights derived. Descriptive: These insights describe what has happened and are vital in summarizing large datasets into understandable plots. F.g., plot of the distribution of incident categories Diagnostic: These insights explain the why or the cause behind observed trends using tools such as segmentation and correlation. E.g., wordeloud of the most common words in incident description.
the past year.
3. Predictive: These insights use statistical methods to forecast future outcomes based on past data. E.g.,
a plot forecasting the future volume of incidents based on current trends in resolution times.
4. Prescriptive: These types of insights suggest actions to tackle the current issues. E.g., an insight recommending strategies to mitigate this projected increase in incident volume.
2.1.3 SYNTHETIC DATA GENERATION
We employed two primary methods to populate the non-controllable columms- coluumss that are not directly manipulated to embed synthetic insights, where each method is chosen to maintain authenticity and alignment with real-world data characteristics. The entire data generation pipeline implemented in Python, allowing for reproducibility and scalability. We now describe the two methods below:
Random Sampling: For fields like IDs, categories of incidents or assets, transaction dates, and status codes, used random sampling from carefully curated lists of plausible values.
Context Generation Using Large Language Models (LLMs): To introduce complexity and realism, certain text-based fields like incident or goal descriptions and user feedback were generated using LLMS such as GPT-4 OOenAA 2023). These models were tasked with creating coherent and contextually relevant entries that align with the data schema, enhancing the datasets with natural- and appropriate text data.
We include more details in Appendix B.I and the prompt structure used can be found in Appendix D
4

---

Published as a conference paper at ICLR 2025
2.1.4 GROUND-TRUTH ANALYSIS NOTEBOOKS
A key component of constructing InsightBench was developing 100 expert-annotated Jupyter notebooks, each tailored for specific context, such as incident management or financial operations. The notebook structure and contents are as follows (see Figure
Dataset overview and SMART Goal: Each notebook begins with a comprehensive dataset overview, outlining its relevance and structure. It is accompanied by SMART (Specific, Measurable, Attainable, Relevant, and Timely) goal (E.g.. "Analyze the discrepancy and imbalance in the distribution of incidents assigned across categories," aimed at identifying the primary causes of hardware failures in an organization.)
Sequential Analysis: The notebook contains series of questions designed to uncover layers of insights gradually. Each question has a Python code block that generates a plot to answer the question, and each plot's data is summarized as JSON metadata outlining the insights. We ensure that each question builds sequentially onto the previous one, leading up to the final insight.
Insight Summary: The final section of each notebook summarizes the findings and proposes actionable steps. This might include recommendations like "Open a Ticket" for further investigation.
2.2 EVALUATING AGENTS ON InsightBench
To evaluate agents on InsightBench, compare the ground-truth annotations (GT) in Jupyter notebooks against the list of insights (A) provided by an agent. Formally speaking, we compare two natural language texts. While multiple metrics have been proposed in the past, like ROUGE (Lin) 2004). METEOR (Banerjee & Lavie 2005), and BERTSCore (Zhang et al. 2019). they do not align well with human preferences. Recently, G-Eval (Liu et al., 2023) was shown to be highly aligned with human preferences for variety of tasks, including text summarization. Since our task involves comparing two free-form sentences, we adopt an L.-M-based evaluator as well. Specifically, we use -LMM--3- variant of G-Eval that uses ATLaM-33-700 instead of GPT model. We use I.LAMA-- as the primary metric to measure the correctness of agent- srent--rovided insights. We chose LLaMA-3-Evar over G-Eval because LLaMB- 1LLaM--3-777 3--0bb is open-sourced, allowing to avoid API costs for using GPT models from OpenAI and b) fix the model weights to obtain a stable and reliable evaluation metric, unlike G-Eval, whose output scores can change due to periodic updates to GPT endpoints.
InsightBench performs a two-way evaluation:
Sunmary-level. To obtain the summary-level score, we simply compute the LLaMA-3-Eval score between the agent-provided summary of insights, and the ground-truth summary.
Insight-level. First, we select the most appropriate insight a A to evaluate against given ground-truth insight (gt GT),, and then average the scores for all the ground-truth insights. The insight-level score can be expressed as in Equation where |GT| is the number of ground -ruth insights in a dataset, and M represents LLaMA-3-Eoe evaluator:
score= Egterr ar
GGT
We average the summary-level and insight-level scores for all 100 datasets in InsightBench obtain measure of the agent's performance. Additionally, we compute scores using ROUGE-1 for comparison.
2.3 DATASET STATISTICS
The reason is that it seems like InsighiBench covers distinct range of business analytics themes and varying difficulty levels, each chosen to reflect typical scenarios encountered in enterprise settings The benchmark includes 100 datasets with total of 475 insights spread across five key topics. The distribution across the topic is shown in Figure 3 and the data tables used are outlined in Appendix [.2]
Distribution by Dataset Category, The benchmark has 20 datasets on Incident Management, 20 datasets on Asset Management, 15 datasets on User Management, 15 datasets on Finance Management, 10 datasets on Goal Management, 10 datasets on Asset & User Management combined, and 10 datasets on Finance & User Management.
Distribution by Difficulty. The datasets are assigned a difficulty level on a scale from (easy) to (hard).
5

---

Published as a conference paper at ICLR 2025
Easy (Level 1-2) comprises 30 datasets that primarily involve direct data retrieval and basic analysis. Medium (Level 3) includes 36 datasets that may require the integration of mul- tiple data sources or applying moderate data transformations. Hard (Level 4-5) consists of 34 datasets that require the calculation of multiple intermediate quantities or significant transformation of data variables. detailed list of datasets with their respective topics and difficulty range is outlined Appendix [B.3
Gral Mgnt.
Mignt.
12.5%
25.0%
10.8%
Finance
Auast Mgmmt..
18.0%
Uner Ngntt
2.4 QUALITY ASSURANCE
To ensure our datasets are of high quality, we designed a ques- tiomnaire and interface that displays dataset to users. We had 20 volunteers with basic data science skills evaluate whether datasets relevant and interesting questions, and provided substantial insights. The exact questions used, the interface design, and the results are shown in B.4 The average rating was 4 out of 5 across these questions, and we received helpful comments that improved parts of the dataset.
Figure 3: InsightBench breakdown across five key thematic areas.
accurately described, contained
EXPERIMENTAL SETUP
The main experimental setup in our benchmarking process involves inputting the dataset schema and an overarching goal for each dataset in InsightBench and letting the agent perform exploratory data analysis. As discussed in Section 2 the goals are carefully designed to provide a meaningful signal to the agent without revealing the ground-truth answer.
In addition to the main experiments, we conduct the following ablation studies to understand the importance of different parameters of InsightBench and their effect on agent performance.
Effect of Using Generic Goals. Instead of using the carefully designed goal from the ground truth annotations, we use the generic goal "I want to find interesting trends in this dataset." We aim to study the effectiveness of our goals steering the agents toward performing meaningful data analysis.
Effect of Trend Intensity. We generate variations of Dataset 2 (see T the complete list), which has the planted insight, "Time to resolution increases linearly over time." " We vary the slope of the linear curve for that insight from -0.1 to 0.9. Through this experiment, we study whether agent needs trend to be blatantly obvious (high intensity) to retrieve or the agent can retrieve nuanced trends (low intensity) well. Notably, also consider negative slopes in this experiment to test if the agents output a false positive (discovering a trend even when it is not present).
Effect of Question Diversity. We restrict the agent generate only "descriptive" questions follow-ups instead of letting generate different types of questions.
Effect of LLM Sampling Temperature. We vary the sampling temperature of the LLM from 0 to 1.0 to study its effect on the agent's 's performance.
3.1 BASELINES
We run the following data analytics agents on InsightBench:
a) Pandas Agent (PA) (hangChain 2024). ).
data science agent by LangChain optimized for question- answering Given a data frame and a question related to it, Pandas Agent first generates Python code and then executes it to produce the answer. In our experiments, we input the goal the agent and let iteratively generate question and their answers. We then pass the list of answers to the agent again and ask to generate a summary of those answers.
b) AgentPoirot (Ours). We propose AgentPoirot that, given dataset (D) in InsightBench and goal (G), performs systematic data exploration aimed achieving G. Figure shows the working of AgentPoirot:
I Chainn MIT Licensed Python library for huilding context-aware reasoning applications

---

Published as a conference paper at ICLR 2025
1. Extract Dataset Schema (S): The dataset schema contains information about every column in the dataset. AgentPoirot extracts the name, type, total unique values, and total NA values for every column. Additionally, if a column contains numerical data, it extracts the minimum, maximum, mean, and standard deviation; if a column contains dates, it extracts the minimum and maximum dates; otherwise, it extracts the top 5 unique values from the column.
2. Generate Insights: Given the triplet (D,G,)), AgentPoirot uses the Question Generation Prompt to generate k-=3) high-level questions. answers each high-level question by first generating Python code and then interpreting the output of the code. Then, it generates ni( (4=) follow-up questions for each high-level question and then answers them as well to obtain a total of generates (n+ 1) x insights(see Figure 7 to see a complete overview of AgentPoirnt).
3. Generate Summary: AgentPoirol summarizes the insights from the previous step (using Prompt
c) AgentPoirot (Ours) w/ generic goal. We use the generic goal, "I want to find interesting insights in this dataset," instead of the carefully designed goal and run AgentPoirot on InsighBBench as in b).
d) AgentPoirot (Ours) non-diverse follow-ups. We use PromptXFinstedd instead of Prompt 3 to generate only one type of follow-up questions and run AgentPoirot on InsighBBench as in b). Refer to Figure 7 in the Appendix for a visualization of our proposed method.
In addition to PandasAgent, we also considered the following baselines: Insightpilot (Ma et al. 2023 but neither the code nor the prompts are available; Data-Copilot (Zhang et al. 2023b), but it is limited to Chinese financial data, which isn't suitable for task; OpenAgents (Xie al.) 2023). but the public version only allows 10 requests, which isn't It enough for meaningful benchmarking: MuiiAgent-aabennn (Hu al. 2024). but we excluded it due to its extremely poor performance in this benchmark. It handles only simple tasks like "Generate Python code to compute the mean of a list," but struggles with more complex queries like "What is the distribution of incidents by category in this dataset?" Lastly, we considered using PowerBI but it is a closed-source tool that is not suitable for large-scale benchmarking. We conducted a small-scale evaluation of PowerBI and found that obtained low-quality insights and generated inconsistent outputs for the same prompt. Our benchmark emphasizes transparency and reproducibility, therefore we focussed on methods that the scientific community can easily inspect and scrutinize, which is not possible with the aforementioned baselines.
3.2 IMPLEMENTATION DETAILS
We use Python's opena package to access the family of GPT models for our experiments and v1lm to host 1LLMM----700 g model on 4 A100 GPUs We use different LLMS as backbones in our experiments to benchmark, including gpt-44 4o, apt-4-turbo, 3pt-3.5 5..5trb,, and 1 1lama 3-70b All results are reported for the sampling temperature of 0.0, unless otherwise stated. We use the evaluate Python package to compute ROUGE-1 scores and Prompt in Appendix Appendix[]forr [Door computing LLaMA-3-Eval scores. We also fix the temperature of LLaMA-3- 3-70b to 0 during evaluation. We repeat all our experiments for 5 seeds and report the mean and standard deviation in our results.
4 BENCHMARK RESULTS
Main Results. Table !| shows the performance of different data analytics agents on InsightBench. First, we note that AgentPoirot outperforms Pandas Agent in terms of ROUGE-1 and I.LMAA3-Fval scores. We note that using gpt-4o consistently achieves the best overall performance. We further see that AgentPoirot with LLaMA- 3-70b outperforms AgentPoirot with gpt-3. 3.t-3.5-mm0 5-turbo and is close to gpt turboo in terms of LLaMA-3 scores.
Our performance analysis by dataset categories (Figure 4ab shows agents performing best in "" "Asset Man- agement," " "Goal Management." and "Finance Management." While AgentPoiror with LLaMA-3-77 Ob underperforms on combined datasets due context limitations, PandasAgent excels in these categories. Performance varies by difficulty (Figure 4b). with strong results on "Easy" and "Medium" tasks but declining on "Hard" ones. Across insight types (Figure 4c). agents show decreasing performance from
http::///apppp.wweze/ wwwpwwwwwww///aeeeeoooo/o/ttopee nome
https: htwp:ichhbooooooaeeeooommmoo htt.mmmtttalll 11amm 11amm3
We use gpt-4-turbo-202-2
and gpt- 3.t-3.5-turre 1..5-turbb-O00 10trro specifically.
7

---

Published as a conference paper at ICLR 2025
Table 1: Performance of difterent agents InsightBench. All results are for 5 different seeds.
Insight-level Scores
Summary-leve Scores
Agent
ROUGE-1
LLaMA-3-Evar
ROUGE-1
LLaMA-3-Evvl
PA (gpt-4o)
0.35-ome
0.54 =0.1
0.35 =0:1
0,40 =:004
Ours (gpt-4o)
0.32 =000
0.60 =o.r
0.37 =000
0,44 #:00
w/ generic goal
0,30 000
0.40 20:0
0,30 10.05
0,33 1:012
Ours (gpt-4-turbo)
0,30 =002
0,56 =001
0.35 =0005
0,35 1:00
w/ generic goal
0.28 00.0
0,38 10:0
0.29 :0002
0.27 10011
Ours (gpt-3.5.igg --buub)
0.34 Loot
0,50 10.0
0.27 10.14
0,31 10006
w/ generic goal
0,30 =0/2
0,36 =0:03
0.24 =003
0.25 #0006
Ours (llama-3-70b)
0.33 =002
0,52 20044
0.36 =001
0,33 20001
non-diverse follow-ups
0.29 =000
0,51 =000
0.32 =00
0.28 +:00
w/ generic goal
0.27 =000
0.35 =000
0.23 =003
0.23 +0002
1.0
Ouri (gpt-3 3.5 (u/b))) Ours (opt curbo)
Ous (gp044)
AA (g)44.)
1.0
Ours (lmma 3-700)
(gpt: turbb)
Ours (llama 3-70))
Ours (gpt turbo)
PA (gpt-4o) 4o)
Qurs (gpt-44o
0.6
LLMA---EVal (Soft
0.6
0.4
0.2
0.2
0.0
Hard
Medium Dataset Difficulty
Easy
Dotssee Carogory
(a) Performance by dataset category.
(b) Performance by clataset difficulty.
10
Durs (ppe-1 1ape-155 --tubbe)) Duns hmpe----urbee)
Durs (gpe-te)
PA cgp--tr)
Dure 3-70b-
ULaMA-3 Eval (Soft Scores)
0.9
8.4
0.2
0.0
Insight Types
(c) Performance by insight type.
Figure 4: Performance of different agents on InsightBench grouped by difficulty and dataset category.
Descriptive (0.52-0.62) through Diagnostic, Prescriptive, and Predictive insights. Table in Appendix C provides qualitative comparisons between AgentPorrot and PandasAgent.
also shows the performance of AgentPoirot that uses a generic goal. We notice drastic overall decrease in the ROUGE-1 and LLaMA-3-Eval scores for all the backbones. For instance, the insight-level LLaMA-3-Ews score for AgentPoirot (cpt-4o) drops from 0.60 (# 0.03) to 0.40 (#: 0.03) and the summary-leve level LLaMA-3-Ewe score drops from 0.44 (+t0.03 to 0.33 (+0.12) (see "Ours (gpt-4o)" v/s "Ours (gpt-4o) w/ generic goal" rows Table TT. This highlights the importance of including a SMART goal in the agent's performance.
Effect of Using Generic Goals. Table
Effect of Exploring Diverse Questions. Table I Iincludes the performance AgentPoirot when gener- ating only single type of follow-up questions (see "Ours (1lama-3-706 non-diverse follow-ups"), We observe a notable decrease in ROUGE-1 scores and a slight decrease in LLaMA-3-Ewa scores, confirming that discovering diverse set of questions during data analysis leads to better performance.
8

---

Published as a conference paper at ICLR 2025
Effect of Trend Intensity Figure 8b shows the performance of AgentPoirot with different backbones on variations of dataset 2 in InsightBench. First, we note that all the models fail to discover insight when the slope is less than 0.1 (including negative slopes). This suggests that our model did not generate false positives (we assume the flag is discovered if LLLMAA3-EEvl is greater than 0.5. Overall, for slope discovered the insight every time, I.L.MA-3-000
greater than 0.1, gpt-4o and gpt-4- discovered it nearly half the times, and gpt appt-3.. 3.5. 5 5-turbo had a particularly hard time discovering the insight. While both sometimes (as showcased by the fluctuations in LLaMA-3- overall discovery rate of the insight compared to gpt-3. 3. 5-turbo.
and gpt - 3 turbo have difficulty discovering the trend
scores), LLaMA-3-700 shows a better
Effect of Sampling Temperature. Figure Bajin Appendix C shows the performance of AgentPoirot with 1.-MM--377b for different sampling temperatures. The performance of the agent peaks at temperature 0.2 before it starts to decrease. Higher temperatures also lead to unstable performance, as indicated by bigger standard deviations the plot. Overall, we note that the temperature should be low but not completely 0 for optimal agent performance InsightBench.
G-Eval v/s -LaMA-3-Eval Table 4 in Appendix C shows results for small-scale study where we compare G-Eval and LLaMA-3-Eval. We note that both scores are similar in range and follow the same trend for all agents. This suggests that LLaMA-3-700 is a good alternative to using gpt-4o for evaluation.
One-to-Man w/s Many-to-Many Evaluation. We conduct another study to compare approach with an alternative many-to-man version, where, instead of computing a score for each ground truth and prediction pair, first prompt the LLM to match every ground truth response with an appropriate prediction. We perform this experiment because the one-to one-to-many evaluation approach can be time- conseminn as it needs compute a LLaMA-3-Eval score for every ground nuth--peddicton pair determine the most appropriate prediction to evaluate against a given ground truth and using many-to-many matching may result in a faster evaluation process. We also show justifications generated by LLaMA-3 for its scores for high and low scoring examples in Table 7 of Appendix C
Table 6 shows the quantitative results of our experiments. We note that using many-to- many evaluation generally leads to lower score; however, upon inspection, we find that many-to-man prompt often mismatches a ground truth insight with wrong prediction (even though more appropriate prediction was present in the pool of predicted insights). shhows some mismatches generated by the many- maany-tommnn many evaluation approach compared to the proposed one-to-many evaluation approach that intuitively does not miss good prediction if it is present in the list of generated insights.
5 RELATED WORK
Our work intersects with the literature on Data Science Benchmarks, LLM Evaluation Frameworks, and Texl-to-nnalyiccs Agents.
Data Science Benchmarks With enhanced abilities code generation, utilization of LLM- M-based data analytics assistants is becoming increasingly prevalent. This has led to the development of numerous data science benchmarks (Lai et al. [2022] Chandel et al.) [022] Zan et al. 2022 Hu et al. 2024: Zhang et al.) 2024b). D000 (Lai al.] 2022) focuses on code generation for diverse data science questions sourced from StackOverflows Similarly,
(Hu et al. (024) assesses the end-to-end problem- solving ability of LLMs by asking questions based on provided CSV files. DSP (Chandel et al. 2022). evaluates a model's 's proficiency through a code-infilling task within Jupyter Notebooks. DSEval Zhang al. 2024b) focuses on the overall behavior of data science agents without getting into the nuances of code generation techniques. On the other hand, Text2SQL and tabular data reasoning benchmarks assess the ability of models to parse queries, extract information, and synthesize the retrieved data formulate responses Beaiooiianis-Meimmraaa existing benchmarks primarily evaluate agents on single-step code generation, InsightBench shifts the focus to multi-step analysis (Delen & Ram 2018).
& Koutrika 2021 ZZong et al. 2018 Chen et al.] 2021b). While
LLM Evaluation Frameworks Existing LLM evaluations frameworks focus on handling structured outputs and predominantly rely on pre- formmtttee prompts to assess code completion (Wu et al.) 2023
Zhang et al. 2023a] 2024a Yao et al. 2023). ). While recent advancements have seen autonomous agents
specializing intricate data science tasks, including analysis, visualization, and modeling (Qian et al.
2023
2024 Zhang et al.) 2023a), evaluations for these methods often depend on extensive human effort
heng
al. 2023) use more powerful LLMS assess the output (Dubois et al. 2023 Belyi et al.
2024

---

Published as a conference paper at ICLR 2025
Another work uses the "Capture the Flag" principle, where insights are planted into a dataset as flags to evaluate whether models can uncover them [Laradji et al.) 2023). G-Eval (Liu et al. 2023) a recent technique used evaluate the quality of free-form texts in terms of factuality and coherence. In this work, we use a variation of G-Eval to score how well the predicted insights are aligned with the ground-truth insight.
Texl-to-Analytics Agents Chen et al. (2023) explore the application of GPT variants within a data
visualization context, highlighting the strengths and limitations of these models. More recent LLM-based data amalysis agents include Code Interpreter (OpenAl 2024] and Pandas Agent (Lang 2024) that are capable of processing multiple data formats and answering questions about them. IMa et al. (2023) propose InsightPilot, an advanced automated tool that leverages LLMs to enhance data exploration by automatically identifying goals and generating targeted intentional queries. Vacareanu al. (2024) showed that L.Mss can also perform regression tasks, enhancing their predictive abilities. Additional studies assess the data analysis capabilities of GPT-4 and propose an end-to-end framework for automating data processes (Cheng al. 2023; Wang et al. 2023 2024 Hong al. 2024). Inspired by InsightPilot, we propose AgentPoirot that can perform end-to-end data analysis that includes extracting descriptive, diagnostic, predictive, and prescriptive insights.
CONCLUSION
We have introduced InsightBench, benchmark with 100 diverse datasets that evaluates agents on their ability to perform end-to-end data analytics, including suggesting questions, interpreting answers, and summarizing insights. We ensured each dataset has clear goals and meaningful analysis as ground-truth to evaluate agents reliably. Using LLaMA-3-Eval. an open-source evaluator, we assessed how well the agents extracted insights compare to the ground-truth truth. We showed how our proposed agent, AgentPoirot, can perform data analytics from high-level goals using both open-source and flosed-source models and showed that outperforms existing approaches like Pandas Agent. We believe Insightßench will significantly drive advancements data analytics. For future work, we look into expanding this benchmark to include more categories of data such as healthcare data, social media trends, environmental data, e-commerce analytics, and educational statistics.
7 LIMITATIONS
Benchmarks have the risk of reinforcing existing biases or oversimplifying complex decision processes. The design of InsightBench necessitates continuous reviews ensure that the insights generated by agents do not perpetuate biases or lead to misinformed decisions. Additionally, agents' performance on this benchmark should be continually assessed against evolving business practices and technological advancements to maintain relevance and effectiveness. More discussions on limitations are presented in Appendix A
REPRODUCIBILITY STATEMENT
To facilitate the reproducibility of our results, we have taken the following measures:
1. Code Availability: All our code, including data used for model implementation, training, and evaluation, available in the supplementary material.
Computing Infrastructure: All LLaMA-3 experiments were conducted on 2 x 80G A100 GPUs.
4. Ablations: A complete list of hyperparameters used in our experiments is provided in Appendix C of the paper. Additionally, all prompts used this work are included in Appendix D.
5. Random Seeds: To ensure reproducibility, have
experiments for 5 random seeds.
Evaluation Metrics: We provide a detailed description of all evaluation metrics used Section 2.2 of the paper. Implementation of these metrics is included in our codebase.
Experimental Procedures: Section of the paper includes a comprehensive description of our experi- mental procedures.
Dependencies: A complete list of software dependencies, is provided in the 'requirements.bx file of the supplementary material.
10

---

Published as a conference paper at ICLR 2025
By specifying these details, we aim to ensure that our work can be readily reproduced and built upon by the research community.
REFERENCES
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Randy Bean. Why becoming data-driven organization is so hard. Harvard Business Review, feb 2022.
Masha Belyi, Robert Friel, Shuai Shao, and Atindriyo Sanyal. Luna: An evaluation foundation model to catch language model hallucinations with high accuracy and low cost. arXiv preprint arXiv:2406.00975. 2024.
Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. Training and evaluating Jupyter notebook data science assistant. ArXiv, abs/2201.12901 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, David W. Cummings, Matthias Plappert, Fotios Chantzis, Blizabeth Barnes, Ariel Herbert-Voss Voss, William H. Guss, Alex Nichol, Igor Babuschkin, S. Arun Balaji, Shantanu Jain, Andrew Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford. Matthew M. Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code. ArXiv, abs/2107.03377 2021a.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguistics.
Zhutian Chen, Chenyang Zhang, Qianwen Wang. Jakob Troidl, Simon Warchol, Johanna Beyer, Nils Gehlenborg, and Hanspeter Plister. Beyond generating code: Evaluating GPT on data visualization course, 2023.
Liying Cheng, Xingxuan Li, and Lidong Bing. Is GPT-4 good data analyst? In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
Beerum?d=PxEhhrr iBB0
Eric Colson. What di-driven decision making looks like. Harvard Business Review, jul 2019.
Dursun Delen and Sudha Ram. Research challenges and opportunities in business analytics. Journal of Business Analytics, 1(1):2-12, 2018.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Danyang Li, Jiaqi Chen, Jiayi Zhang Jinlin Wang. Li Zhang. Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Wenyi Wang, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zongze Xu, and Chenglin Wu. Data interpreter: An 1lm agent for data science, 2024.
Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang. Jing Su, Jingjing Xu, Ming Zhu, Yao Cheng, Jianbo Yuan, Jiwei Li. Kun Kuang, Yang Yang. Hongxia Yang, and Fei Wu. infiagent-dabenee Evaluating agents on data analysis tasks. In The Twelfth International Conference on Machine Learning (ICMI.,, 2024.
11

---

Published as a conference paper at ICLR 2025
George Saatoggianni-Meimaadde and Georgia Koutrika. A deep dive into deep leaming approaches for text- to-sql systems. In Proceedings of the 2021 International Conference on Management of Data, SIGMOD '21, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 1784433333331
Yuhang Lai, Chengxi Li, Yiming Wang. Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. ArXiv, (1t/2211.11100). 2022.
LangChain.
Pandas dataframe.
Attps:/ langchain. com/vo. 2/ //coss/ 2024. Accessed: 2024-06-03.
Issam Laradji, Perouz Taslakian, Sai Rajeswar, Valentina Zantedeschi, Alexandre Lacoste, Nicolas Chapa- dos, David Vazquez, Christopher Pal, and Alexandre Drouin. Capture the flag: Uncovering data insights with large language models. arXiv preprint arXiv:2312.13876, 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004, Association for Computational Linguistics. URL
https: http//aaaannt httpss///ccaatthoooog... o wwtp:////////00000000000
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), EMNLP, 2023.
Pingchuan Ma. Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. Demonstration of InsightPilot: An LLM-empowered automated data exploration system. CoRR, abs/2304.00477, 2023.
Andrew McAfee and Erik Brynjolfsson. Big data: The management revolution. Harvard Business Review, oct 2012. From the Magazine (October 2012).
OpenAL. Introducing ChatGPT. hutps://openni.ccmmmnnexxxaaggg 2022.
OpenAL GPT-4 technical report. wwXii,abb/23033,8877422000
OpenAL. Code interpreter.
httpps://platformm.oog openat
code interpreeerrrr
2024. Accessed: 2024-06-03.
Chen Qian, Xin Cong. Wei Liu, Cheng Yang, Weize Chen, Yusheng Su, Yufan Dang, Jiahao Li, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Communicative agents for software development, 2023.
Chen Qian, Jiahao Li, Yufan Dang, Wei Liu, YiFei Wang, Zihao Xie, Weize Chen, Cheng Yang, Yingli Zhang, Zhiyuan Liu, and Maosong Sun. Iterative experience refinement of software-developing agents, 2024.
ServiceNow. Vancouver release notes. Online, 2024. Available at: httpps:///////. httpss///ddccsssevvcerrro. c mesereeeeeeeeeeeeeeeeeeeeese
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bushlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 207.v:230..09288, 2023.
Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, and Mihai Surdeanu. From words to numbers: Your large language model secretly a capable regressor when given in-context examples, 2024.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan- and-solve prompting: Improving zero-shot thain-of-thous reasoning by large language models. arXiv. abs/2305.044011, 2023.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents, 2024.
Luoxuan Weng, Xingbo Wang, Junyu Lu, Yingchaojie Feng, Yihan Liu, and Wei Chen. Insightlens: Discovering and exploring insights from conversational contexts in large- moder-p data analysis, 2024.
12

---

Published as a conference paper at ICLR 2025
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen IIm applications via multi-agent conversation framework, 2023.
Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the wild. arXiv
preprint rrXiv:2310106644 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Confer- ence on Learning Representations, 2023. URL httph:ttt///// httppp://oeenrevvie. net/ forum? id=
VluYUL-X
Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation interactive data science notebooks. In Association for Computational Linguistics (ACL), 2023.
Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou. CERT: Continual pre-training on sketches for library-oriented code generation. The 2022 International Joint Conference on Artificial Intelligence, 2022.
Lei Zhang, Yuge Zhang, Kan Ren, Dongsheng Li, and Yuqing Yang. Micopilot: Unleashing the power of large language models in solving machine learning tasks. arXiv preprint arXiv:2304.449 2023a.
Shaokun Zhang, Jieyu Zhang, Jiale Liu, Linxin Song, Chi Wang, Ranjay Krishna, and Qingyun Wu. Training language model agents without modifying language models. ICML'24, 2024a.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv: 1904.09675, 2019.
Wenqi Zhang, Yongliang Shen, Weiming Lu, and Yueting Zhuang. Data-copilot: Bridging billions of data and humans with autonomous workflow. arXiv preprint arXiv:2306.072000 2023b.
Yuge Zhang, Qiyang Jiang, Xingyu Han, Nan Chen, Yuqing Yang, and Kan Ren. Benchmarking data science agents. arXiv preprini 1711 2024b.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2SQL: Generating structured queries from natural language using reinforcement learning, 2018. URL http:///////eeeeeee.eeeee. Byx6bz Ab
13

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND POTENTIAL SOCIETAL IMPACT
InsightBench is designed to advance the field of Exploratory Data Analysis (EDA) by providing chal- lenging framework for evaluating open-ended multi-step data analytics tools and models. The dataset evaluates and helps build conversational data agents with significant potential enhance decision- and operational efficiencies in organizations, particularly for end-users who may not have deep technical expertise data science. Yet, there are important considerations and limitations concerning the current data construction and the benchmark's application in the real-world setting.
Diversity and Scope of Data Simulation. While InsightBench covers spectrum of business areas and themes, the datasets are synthesized based on patterns observed in widely-used ServiceNow. Consequently, the synthetic "flags" are designed to mimic real-world anomalies and trends but may not capture genuine enterprise data's 's full complexity or unexpected behavior.
Expanding the Benchmark. The current structure of InsightBench allows for scalable expansions to include more nuanced scenarios, themes, and more datasets that could involve unstructured data like reports, logs, or emails. Crucial information and actionable insights can also be derived from visual representations such as charts and plots. Further development could also integrate diverse business norms, the ability to analyze the plots and graphs from a visual context alongside language by the agents. Moreover, incorporating feedback from real-world deployments could help refine the datasets simulate underlying business data better. Lastly, InsightBench allows interpreting data types and evaluating outputs for practical utility. Its adaptive framework, which evolves through multi-step interactive feedhack, may align with agile development practices. This positions our InsightBench future setup for integrating intelligence into the end-to-end software development process.
B APPENDIX: OVERVIEW
Our supplementary material includes the following sections:
Section A: Contains details of data accessibility, generation process, and breakdown of domains
covered by InsightBench.
Section B: Contains results of some ablation experiments and qualitative examples.
Section C: Discusses relevant literature for our work.
Section D: Contains all the prompts used in our experiments.
Dataset Maintenance Authors committed to ensuring the dataset's regular upkeep and relevance, will place system for users to report issues or suggest updates. A feedback form will be available for users contribute their input. We commit actively reviewing these suggestions and making the necessary adjustments to the dataset. We also invite contributions from the community through pull requests on InsightBench's 's GitHubb repository.
B.1 EXAMPLE PROTOCOL FOR DATA GENERATION AND INSIGHT PLANTING
Here, we outline a practical case of how we created flag dataset for an incidents table with a trend in which the duration of incident resolution increases over time:
1. Schema Exporting and Standardization: Initially, the schema for the incidents table is exported and standardized by inspecting demo instance. Fields such as "number", "opened att", and "closed at", among others, are defined.
2. Data Generation Parameters: Parameters for generating the data include factors that influence and define the trend. For this example, the parameters include slope to model the trend in resolution time, Dates ranging from start date to end date are used to set the temporal context of the data. In the experiments section, we ablate and study the effect of the key parameters.
3. Synthetic Data Creation: The entire synthetic data generation pipeline is implemented in Python, leveraging libraries such as Pandas for data manipulation and custom scripts for quality control. For example, for fields like IDs, categories incidents or assets, transaction dates, and status codes, used random sampling from carefully curated lists of plausible values. The lists were created by analyzing common patterns in actual operational data to capture typical distributions and variances. For example:
14

---

Published as a conference paper at ICLR 2025
ID fields: We generated unique identifiers based specific formats to simulate real data structures, such as GUIDs or composite keys for hierarchical datasets.
Date fields: Transaction dates and timestamps were randomly sampled within specific ranges, often constrained by meaningful boundaries (e.g.. fiscal quarters or business hours) to reflect real-life constraints.
Categorical fields: Categories like incident types and asset classifications were randomly sampled to create balanced distribution that mirrors real datasets' diversity. Sampling strategies included uniform and weighted distributions to reflect frequency differences in actual usage patterns.
Additionally, we applied statistical validation ensure that the generated datasets reflected the anticipated distributions and complexities.
4. Trend Implementation:
The time to resolution is calculated using a linear model where the resolution time increases over the duration of the dataset. The parameter chosen in step is used to model the linear function. The closing time of an incident is determined based on the resolution time and its opening time.
5. LLM Utilization:
For fields that require diverse and realistic inputs like short descriptions of the incidents, an LLM generates descriptions based on the incident category, ensuring that the data retains an authentic and varied narrative quality.
This approach provides flexible and scalable framework to produce synthetic datasets that closely resemble real-world business data, allowing us introduce realistic patterss, variations, and inconsistencies essential for robust model training and evaluation. Further details on our data generation approach, including specific parameters and code snippets, provided in Appendix B.
B.2
DATA TABLES ACROSS THEMES
Here is an exhaustive and descriptive outline of the topics covered in the benchmark:
Incidents Management: Focuses on tracking, analyzing. and resolving workplace incidents.
extensive description of this data table is discussed afore- Sec
Datasets in this theme are derived from sys sys_user system table of ServiceNow. This table tracks all user profiles within the platform. It contains information about each user, such as their roles, department affiliations, contact details, and activity status. Key fields include the user's ID, name, email, department, and last login time. This table is used in InsightBench for insights related roles and permissions of employee agents, focusing on schedules and activity patterns.
User Management:
Finance Expenses: Datasets in this topic examine detailed records of expenses uncover patterns and optimize budget allocations and prioritization. Datasets are built from fm im_expense_ _ system table that logs detailed entries of financial transactions and expenses as part of the financial management module. It includes data points like the expense amount, date, associated user, department, category, and processing status.
Inventory Management: Manages data regarding hardware assets and procurement, aiming to un- derstand patterns inventory systems. Datasets are derived from alm hardware, a system table that manages records of all physical assets, particularly IT hardware. Fields in this table detail each asset's s tag number assigned to the user, status, location, purchase date, and warranty expiration. It supports asset tracking, lifecycle management, and maintenance activities within organization.
Enterprise Goal Management: Datasets in this theme evaluate the alignment of departmental per- formance with overarching goals, focusing the effectiveness and achievement rates. Datasets derived from sn_gf goal system table of ServiceNow, which is essential for evaluating goal management efficiency and aligning departmental outputs with organizational objectives. Fields in this table consist of goal description, start and end dates, current status, owner, priority, and percentage completion.
B.3 LIST OF DATASETS AND PROBLEMS
Table 2 shows the names of all the datasets in InsightBench along with their difficulty and category. Table presents example of question-insight pair along with corresponding plots for each category.
15

---

Published as a conference paper at ICLR 2025
CH
(a)
Figure 5: Screenshots of the Interface used for Dataset Quality-check:
B.4 QUALITY-CHECK INTERFACE
To ensure the high standards of data quality and relevance, InsightBenchhas implemented a comprehensive quality-check process involving volunteer contributors. This process is facilitated through a specifically developed gradio interface, which guides the expert reviewers through a structured review of the ground- truth analysis notebooks included the benchmark (Figure 5). The evaluation is divided into three main sections:
Section 1: Dataset Overview Evaluation. Volunteers assess how clearly the dataset and its objectives are described. This includes reviewing the clarity of the dataset description and the specificity and measurability of the stated goals (Figure 5ap.
Section 2: Question and Insights Evaluation. This section focuses on the engagement and relevance of the questions posed in the notebooks. Volunteers use slider interface to review each question and corresponding insight sub-sections, evaluating how effectively the plotting code answers the questions and how well the insights align with the analysis goals (Figure 5b).
Section 3: Summary of Insights Evaluation. The final section requires volunteers to determine whether the summary accurately encapsulates the key insights and conclusions drawn from the data analysis. This involves checking if the final flag provides a comprehensive and concise summary of the notebook's 's findings (Figure (5c).
The average rating scores for questions across the three sections is depicted in Figure 6
B.5 DETAILS THE PROPOSED AgentPoirot
Here we describe the proposed baseline AgentPoirot in further detail. Figure depicts our pipeline. Initially, AgentPoivot uses the dataset and its schema obtain set root questions about the data, aligning with the user's Goal or Role. Each of the root questions can be injected into a Code Generation prompt to obtain and execute code, as well as obtain textual descriptions the insight found. From these outputs, we generate follow-up questions and start the process again, obtaining model that navigates the data in depth and in breadth to find interesting insights.
16

---

Published as a conference paper at ICLR 2025
(b)
Figure 6: Distribution of Dataset Quality-check Scores across the three sections:
Database / Schema
CSV
CG Prompt
Code </>
Piots/Figures
IE Prompt
Follow-Ui Questions
Answer/insight Jusiificaton
Question
Questio
Question
Top Question
Question
FQ Prompt
Prompt
Question
Question
Conversition
Figure 7: AgentPoirot the "Question Generation (or QG) Prompt" (Prompt I in Appendix D) that takes input the dataset schema and goal to generate & high-level questions. Then, uses the "Code Generation (or CG) Prompt" (Prompt 2 in Appendix(D [D) to generate plots answering the high-level questions. Using the "Insight Extraction (or IE) Prompt" (Promp:[5]nn Appendix D). and the outputs from the previous step, generates an insight and the justification of that insight. Finally, AgentPoirot uses the "Follow-Up Up Question (FQ) Prompt" (Prompt Appendix(D) generate diverse follow-up questions, and selects the top one using the "Question Selection QS) Prompt" (Prompt 6 in Appendix D). This question injected back the CG prompt to start cycle again.
17

---

Published as a conference paper at ICLR 2025
Table 2: Overview of Insigh(Bench Datasets and Problems. This table enumerates subset of the first 30 datasets included in the benchmark, detailing their respective topics, difficulty levels, and titles. This enlists diverse scenarios that InsightBench covers evaluate the abilities of data analytics agents.
Dataset
Dataset ID
Difficulty
Category
Hardware Incident Dataset
4
Incident Management
Incident Resolution Time Dataset
3
Incident Management
Incident Assignment Distribution Dataset
2
Incident Management
Incident Category Trends Over Time
4
Incident Management
Time to Resolution Trends Across Incident Cate-
2
Incident Management
gories
Agent Performance Analysis Over Time
4
Incident Management
Incident Assignment and Resolution Efficiency Analysis
3
Incident Management
Caller Incident Impact Analysis
2
Incident Management
Hardware Incident Analysis During Specific Time Windows
4
Incident Management
Incident Resolution Time Trends Analysis
10
3
Incident Management
Category based Incident Trends Analysis
4
Incident Management
Hardware Incident Easy Dataset
12
1
Incident Management
User Agent Wellbeing and Incident Volume Analy-
13
2
Incident Management
sis
Performance Trends in Employee Agents Manage-
14
4
User Management
ment
Workload Distribation and Efficiency Analysis
15
4
User Management
Asset Warranty Analysis
16
2
Asset Management
Asset Cost Analysis by Department
17
3
Asset Management
Asset Warranty and Purchase Date Analysis
3
Asset Management
User Management
Expense Management Discrepancies
19
3
Finance Management
Travel Expense Rejection Analysis
20
2
Finance Management
Expense Rejection Trends for New Employees
21
2
Finance Management
& User Management
Expense Processing Efficiency Analysis
22
3
Finance Management
Expense Claim Pattems and Fraud Analysis
23
3
Finance Management
Expense Processing Time Analysis
3
Finance Management
Expense Processing Dynamics Analysis
25
2
Finance Management
Asset Warranty Analysis
26
2
Asset Management
Management Staffing Analysis in IT Department
27
3
User Management
Goal Achievement Rate Analysis in IT Department
28
2
Goal Management
Goal Management Analysis Category Focus
29
2
Goal Management
Goal Management Analysis Cost Reduction
3
Goal Management
18

---

Published as a conference paper at ICLR 2025
Table 3: Instances of insights from InsightBench for each category.
Dataset Category
Insights
Plot
Incident Management
Question:
What is the trend in time to resolution (TTR) of incidents across categories?
Insight: TTR starts increase linearly for hardware
incidents abruptly during particular time period.
User Management
What is the distribution of reporters per Man- ager by Department?
Question:
Insight: The average number of reportees per manager in the IT department significantly higher than other departments. It is 50.5 compared to 8.8 in Customer Support, 11.6 Finance, 12.8 HR. and 13.0 in Sales.
Assets Management
Question:
What is the relationship between the purchase date of assets and their warranty periods?
Insight: There a significant correlation between pur- chase dates and warranty periods in that the later purchases an asset the higher the warranty which a surprising insight.
Expense Management
Question: What is the distribution processing times of expense reports across different cost brackets? Insight: counterinttuttvelly expenses within lower cost brackets experience significantly longer process- ing times.
Goal Management
Question: How long do goals under 'Cost Reduction" take achieve?
Insight: As time passes, there is an increase in the time takes to achieve 'Cost Reduction' goals.
19

---

Published as a conference paper at ICLR 2025
LLaWA-3-Eval (Saft Scare)
1.0
0.0
0.8
0.6
0.6
LaMM-33
0.4
0.4
(gpt
0.2
Temperature
Slope
Effect of varying sampling temperature (per-
(b) Effect of varying trend intensity (for spe- cific insight dataset 2).
formance reported across 100 datasets in
Insigh(Bench)
Figure 8: Sensitivity Analysis.
C ABLATION RESULTS
Table shows the difference between various evaluation strategies, and Tahlo qualitative results for many-to-many and one-to-many evaluation.
Table 4: Comparison of different evaluation methods on the first 10 datasets of Insigh/Bench.
Soft Scores
Summary Scores
Agent
G-Eval
LLaMA-3-Eval
LLMaA-3-Eval (many-to-many)
G-Eval
I.IMMA3-Evaal
Ours (gpt-4o)
0.53 :0.01
0.57 :0.02
0.43 :003
0.55 +0001
0.58 ::003
Ours (gpt-4-turbo)
0.52 :0.02
0.55 #0.02
0.40 -000
0.54 10008
0.58 +0002
Ours (gpt-3.5-turbo)
0.46 +0.01
0.50 :0.01
0.35 =0.02
0.48 +0.01
0.50 10001
Ours (llama-3-70b)
0,45 =0.01
0.50 #0.02
0.34 =0003
0.47 +0.01
0.49 #0.01
20

---

Published as a conference paper at ICLR 2025
Table 5: Comparsion of insights generated by PandasAgent and AgentPoirot. Parts in bold show why insight a close approximation the ground truth insight. We note that AgentPoirot consistently generates to-the-point insights compared to PandasAgent, which often generates rolated but slightly off-topic insights.
Grvund Truth Insight
Insight Prodicted by Pandaskgent (gpt -lo
Insight Prodicted by AgestPhirot (gpt-40)
Datreet Category: Incident Management
The time resolution of incidents is in- creasing over time
Network assignment group consisiently has longer resolution times compared other group, with an average resclution time of upprosimaely 1074 hours
Yes, there is clear trend of increasing inckdent resolution timues over the months with the ovenge resolation time tasing from 113.00 houns Junurry 2023 3150 S6 bours June
The incresso volume of incilents only for one particulir category ia. Hand- Ware
The ansigedd saries significaenly facst assigament groups with "Hadwwse" grou hoving the highesi number of mideess (314), followad by "Sochware" (56), "Network" (48), "Darahase" (43), and "Service Dest" (39).
dif-
The colegory 'Hardwwrre lus shown highest increase the number lncidents the given period, total incrane of 32.0 incidents.
Dataset Cotegory: Managessent
Frod Ludkyy has a tignificantly higher sw- esage TTR compore ofher agents
There spacific types of incidents where the specifial hu- agest Fred Luidy, concistearly performs thas others.
The absence Howsed Johnson signfficnntl inguets the worklaadd performance
agents, evideneed the high mean resuuution time,, purticularty for Fred Fuilyy at 410,65 hours, compoeed to ofher agais like Reth Angin at 102.210 hour..
There as no signdficntt change in the dis- uibution incident categonics Beth Auglin Lake Wilson during the other agenns" PIO
Durag pesk business hours, Lake Wilson and Beth Anglan rocerve the highest sumber of incident asoguments, while off- peak hours show more balanced distrbbtiion among ageats.
No specific periods ideniified where the number of incidenss assigned agents significanly corretation with agents" leave perfods has been estihlished.
or
Datastt Category: Asect Managrment
Servers Web Servers peusive asset cutegories on average, fol- lowed by computers.
Yos, are cost outliens within specific model categories as the mastimum cost an asset is 4786, which is significntly higher than the 75th pencentile cost 2834.
Yes, certain casegories such as and "Com- puter' have higher varlance asset costs compared to others, with vurianzes of 5,441 227/02 and 1J063, 157.4177.44 re- spectively, these categories mean warmaty periods 73028 1,228.88 days.
relatively longer
There excessive rumber compui- ers per user in IJR Department
Yes, in the HR deportment "Compute" mode cotegory has significantly higher costs compared to other modeb, sotal- ing 61215.
Yos, HR degartment has diprospotionntlly high number of Computers (19) compared to other model cate- pories such as Wab Serves (5) and Serven (4).
Dataset Category: Finance Managemment
Muse the declined exponses ure belung to a few spacific usess in IT Degortment
There is curelation between the user submittngg the expense and the likelinsoal of rejection, some espersas are declined, indicating rejorion, while whers procosed pending.
Yes. specifiic users such as Veroon Engelman and Haleme Theng from the IT department have higher rukes of experss rejections, porticnlarly the 'Assets' cotegory.
There is linear positive correlation he- employee's dates and their experse rejection rates
There are differences in the approval rates of expense sub- and experienced cuppkoees.
The rejection rate of expense suhmissions tends to be higher for emplovees with shorter eupdoyment durations, indcated higher rejection observed inrial days of employment.
misdons between
with newer empleyees having higher rates of suhmision or sass and rejections compared to mare exper leneed empioyees.
Dutaset Category: Goad Monugement
unifoem distribution gool pri oriics in the Faranc departinent
There signfficmt comelation between owner a goal in the Finasce deparamett and the time takes for completion.
The priocity level goals in Finance depurtment adively halanced. with counts 58 for Low, 52 for Medium, 52 for High, and 51 fo Cittccll pioities.
The IT' department exhibiss a higher rum- ber of Criical and ligh phority gouls companed other departments
The Marketing depamment has the highest mimber of gouls. followed by IRR, IT. and Finance.
The distrihution of goal priorities within the IT depart- ment is 56 Hlghh SI Critcal, 10 Law, and 9 Medium, which shows higher concettationn High Crilical priori- liks congared to other departments.
21

---

Published as a conference paper at ICLR 2025
Table Mismalched examples by many-to-many compared to one-to-many evaluation scheme. Parts bold indicate why the prediction is appropriate to compare against the ground truth insight.
Ground Truth Insight
Prediction Matched by Many-to-Many Evaluation Scheme
Prediction Matched by One-to-Many Evaluation Scheme
The time resolution of incidents is increasing over time
The average resolution time for incidents (67..2 hours *1 Critical' priority,
Yes, there is clear trend of increas- ing incident resolution times over the months, with the average resolution time rising from I 113.03 hours in January 2023 3150 310..86 hours in June 2024.
1554.63 hours for '2 High' priority, and 1584.00 hours for '3 Moderate' priority.
The time to resolution of in- cidents uniform over time
Beth Anglin has shown the highest effi- ciency in resolving incidents with resolution time 153.90 minutes, and her workload has included handling 105 incidents over the given period.
There are no noticeable patterns in the types of incidents (categories) that are being resolved more quickly or slowly over time.
The volume of Hardware in- cidents is elevated during specific time windows
Yes, certain 'caller id' individuals such as 'Don Goodliffe' report more incidents the 'Hardware' category, and there is a trend showing that 'Bud Richman' and 'David Loo' also frequently report inci- dents in 'Hardware' and 'Software' cate- gories.
Yes, there are patterns in the types of incidents reported over time, with 'Hardware' incidents peaking signifi- cantly July August.
There is specific agent, Fred Luddy, who is signed significantly more cidents than others
The assignment group with the highest number of incidents assigned to them is 'Network' with 310 incidents.
Fred Luddy consistently handles a higher number of incidents compared others, with 385 incidents assigned him.
There are higher expense jection rates for employees with shorter tenure
Yes, 'Assets' and 'Travel' categories have a higher rate of submission errors or rejections, with counts of 91 and 72 respectively, compared to 'Services' and 'Miscellancous" which have 16 and
The rejection rate of employees tends to be higher for employees with shorter employment durations, with many instances of 1.0 rejection rates ob- served the initial days of employment, it gradually decreases as employment duration increases.
There are higher success rates in Critical and High priority goals within the IT department
Yes, there are correlations between the metric types and the success rates projects, as indicated by the plot statis- ties showing range of data points for percent complete and target percentage of projects.
The datasct reveals trend where projects with 'Highh and 'Critical' pri- ority levels generally have higher com- pletion rates, often exceeding 80%, pared those with 'Low' 'Medium' priorities, which show more variability and lower completion rates overall.
22

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of Generated Insights with Ground Truth
Example
Ground Truth
Generated Insight
LLaMA-3-Eval Score and Justification
(GT)
(Good)
There significantly higher costs in the HR Department.
The average cost of assets in the HR de- partment is $4874.25, which is significantly higher compared to the average cost of $1967 26 in other de- partments.
Score: 0.81
The generated text accurately conveys that the HR department has higher asset costs compared to other departments, with spe- cific numerical values supporting the claim. The score is slightly less than because "significant" used in general rather than statistical sense. Additional context could further improve this.
(Bad)
There is a weak comrllttio be- tween the num- ber of users and the high cost of computer assets in the HR de- partment.
The distribution as- set types within the HR department in- cludes 19 computers, web servers, and 4 servers, with com- puters being the most prevalent asset type.
Score: 0.18
The generated text does not address the correlation between the number users and the cost of computer assets. Instead, it discusses asset distribution, which is irrel- evant the GT claim, resulting in a low score.
(Bad)
There is a strong positive conrelation between the increase training hours and improved employee performance.
The training program was held over a span of 6 months for six agents.
Score: 0.11
The generated text fails address the cor- relation between training hours and em- ployee performance. Instead, it provides irrelevant details about the training pro- gram's 's duration and the number of partici- pants, resulting in a low score.
23

---

Published as a conference paper at ICLR 2025
D PROMPT DESIGN
In this section, we illustrate the use of tailored prompis to enhance the analytical capabilities of our models for extracting insights and conducting advanced analytics.
Prompt 1: Prompt used for Extracting Questions from Data
SYSTEM MESSAGE:
You the manager of data science team whose goal is to help stakeholders within your company extract actionable insights from their data. to
of highly skilled data scientists that answer complex questions about the data. You call the shots and they do the work. Your ultimate deliverable is repert that summarizes the findings and makes hypothesis for any trend anomaly that was found.
You have
PROMPT:
### Instruction:
Given the following context: (contett>(cottett)
Given the following goal: <goal>(goal></goal>>
Given the following schema:
<schema> (schema) </ schema>>
Instructions:
Write a list of questions to be selved by the data scientists your tean to explore my data and reach my goal.
Explore diverse aspects of the data, and ask questions that are relevant to my goal.
must ask the right questions to surface anything interesting (
trends, anomalies, etc.)
Make sure these can realistically be answered based en the data schema.
The insights that your team will extract will be used to generate report.
Each question should only have one part, that a single '?' the end which only require a single answer.
Do not number the questions.
You produce at most (max_questions) questions. Stop generation after that.
Most importantly, each question must be enclosed within <question<</ question> tags. Refer to the example response below:
Example response:
<questicn>What the average age customers?< customers3<<quesiion>>> <questicn>What is the distribution of the customers based on their age?</ question>
Response:
24

---

Published as a conference paper at ICLR 2025
Prompt 2: Prompt used for Generating and Executing code for answering Question
PROMPT:
Given the geal: : An {goal) Given the schemarkn (schema)
Given data path:\n (database_path)
Given the list of predefined functions in Zba.toola module and their example usage:\n\n
ffunctio_ddccs]
Give me the python code required to answer this question "[question)" and put comment top of each variable. AnAn
Make a single code black for starting with 'python Do not produce code blocks for languages other than Python.
Import cba.tools at the beginning. You must only plot.
the predefined functions mentioned above to make the
You must generate one single simple plot and save it as ips file. For the plct, save stats json file that stores the data of the plot. For the plct, save x_axis.json and y_axis. json file that stores a maximum of 50 of the most important and y axis data points of the plot, respectively.
Save each json file using the cba tba.save_json function For the json file must have a "name", "description", and "value" field that describes the data.
The content of the json file should less than 4500 characters
Call the fix_fnames function in cba. coolt at the end of your code. End your code with 10,
Output code: An
25

---

Published as a conference paper at ICLR 2025
Prompt 3: Prompt used for Generating Diverse Follow-up Questions based on the previous answer
SYSTEM:
You the manager of data science team whose goal is to help stakeholders within your company extract actionable insights from their data. You have access to team highly skilled data scientists that can answer complex questions about the data.
You call the shots and they do the work.
Your ultimate del iverable is report that summari zes the findings and makes hypothesis for any trend anomaly that was found.
PROMPT:
Hi, I require the services of your team to help reach goal.
(Cotteett((otteeex
<goal>
<schema> (Schema>(schema)) s/chema>>>
<questicn> (questiion </ </quettion>
<answer> (answea
Instructions.
Produce list of follow up questions explore data and reach my goal
Note that we have already answered <question> and have the
answer?, do not include question similar to the one above.
Explore diverse aspects of the data, and ask questions that are relevant to goal.
You must ask the right questiens to surface anything interesting
trends, anomalies, etc.)
Make sure these can realistically be answered based on the data schema. The insights that your team will extract will be used to generate report.
Each question that you produce must enclosed in <question>content</ question> tags.
Each question should only have one part, that is a single '?' at the end which only require a single answer.
not number the questions.
You can produce most (max_questions) questions.
26

---

Published as a conference paper at ICLR 2025
Prompt 4: Prompt used for Generating only type of uollow-up Questions based the previous answer
SYSTEM:
You the manager of data science team whose goal is to help stakeholdera within your company extract actionable insights from their data. You have access to team highly skilled data scientists that can answer complex questions about the data.
You call the shots and they do the work.
Your ultimate del iverable is report that summari zes the findings and makes hypothesis for any trend anomaly that was found.
PROMPT:
Hi, I require the services of your team to help reach my goal.
<goal>(goal)
<schema><schemaa<< s/chhma>
Sanswer>(answar </ answer>
Instructions:
Produce list of follow-up questions explore my data and reach my goal.
Note that we have already answered <question> and have the answer at <
answer>, do not include question similar to the one above.
Explore diverse aspects of the data, and ask questions that relevant to my goal.
You must ask the right questions to surface anything interesting (
trenda, anomalies, etc.)
Make sure these
realistically be answered based on the data schema.
The insights that your team will extract will used generate report.
The question has to adhere to the type of question that is provided in the <question_types tag
The type of question is either descriptive, diagnostic, prescriptive, or predictive.
Each question that you produce must be enclosed in <question/contentc/ question> tags.
Each questien should only have one part, that is single '?' at the end which only require single answer.
not number the questions.
You can produce at most (max_ mixx_question] questions.
27

---

Published as a conference paper at ICLR 2025
Prompt 5: Prompt used for Interpreting the Solution
PROMPT:
### Instruction:
You are trying answer question based on information provided by data scientist.
Given the context: <context> You need to answer questicn based information provided by data scientist. </context>
Given the goal: (goa)>ggal)<<gezt)
Given the question: (
Given the analysis: <analysis>
<message>
(message/
</message>
(insights)
</ s/analysis>
Instructions:
Based on the analysis and other information provided above, write an answer to the question enclosed with <question<<qqeettion>>>>>>> tags.
The answer should be a single sentence, but it should not be too high level and should include the key details from justification. Write your ansver in HTML-like tags, enclosing the answer between answer></answer> tags, followed by justification between Dustification><//ue fication> tags.
Refer to the following example response for the format of the answer and justification.
Example response:
<answer>This is a sample answer</answer
<justification-Thie is sample jusiffcct.oo///jutfflaattoo
### Response:
28

---

Published as a conference paper at ICLR 2025
Prompt 6: Prompt used for Selecting the best Question based previously asked questions and the goal
SYSTEM MESSAGE:
You the manager of data science team whose goal is to help stakeholders within your company extract actionable insights from their data.
You have access to team highly skilled data scientists that can answer complex questions about the data.
You call the shots and they do the work.
Your ultimate del iverable is report that summari zes the findings and makes hypothesis for any trend anomaly that was found.
PROMPT:
Hi, I require the services of your team to help reach goal.
<goal>(goal)</gaa)
Instructions:
Given a context and goal, select one follow up question from the above list to explore after prev_question that will help me reach my goal.
not select question similar to the prev_questions above. Output only the index of the question in your response inside question_idx<<quetttie
tag.
The output questions id must be 0-indexed.
29

---

Published as a conference paper at ICLR 2025
Prompt 7: Prompt used for Summarizing the Insights
SYSTEM_ MESSAGE:
You the manager of data science team whose goal is to help stakeholders within your company extract actionable insights from their data. You have access to team highly skilled data scientists that can answer complex questions about the data.
You call the shots and they do the work.
Your ultimate del iverable is report that summari zes the findings and makes hypothesis for any trend anomaly that was found.
PROMPT:
Hi, I require the services of your team to help reach goal.
<goal><goall</goal>>
(hist
Instructions:
Given context and goal, and all the histery of <question i> pairs from the above list generate the 3 top insights that will help me reach my goal. Output each insight within this tag <insight>></neight>>>
30

---

Published as a conference paper at ICLR 2025
Prompt 8: Prompt used for generating the synthetic data for non-controllable elements of an incidents dataset.
fields
("short_deccrpptionn,
"assignmett_group"
field info join.
<field>"
[rantttttff/ammmmpppooooooooddd. </typp>>"
{
#"<chiice>/coL_in(([111 choices' ] |/</choloos>" if "choices" in col_info[f] else
"</Sield>"
for 1 in gpt fields
prompt
Create fake ServiceNox incident by generating realistic values for the following fields:
<fields>
(gpt_ficlds)
</ficlds>
Here the types of each of these fields and the permitted values: <field_info>
(field_info)
</field
Produce jsen dictionnary with single value for each these fields. Make sure respect allowed values and their full diversity.
Example:
seshortdesccipptoon "My laptep is broken"))
Respond with the json only.
valid_data
while len (val id_data) num num_samples:
try:
completion client chat wwwpletios..co.ooooco.... creatt model==ggt-4-turbb", messages=|
"role": "system",
"content"; "You' re going help generate
simulated data that looks like really came from ServiceÑow Glide tables.".
b
("role": "user", "content"; prompt),
incident = json. loads (completion choices [0] coopiestion.coooess
.esssage........................ content)
Validate values assert set fiel_fe
set (incident. )))
for gpt_ fields:
if "choices" in col infolf]]
assert incident [f] in col info f] ["choices"]
31

---

Published as a conference paper at ICLR 2025
Accept
append valid_data, append (incident)
print (E" {len (valid_data)) / (nun_samples)
except: print ."... invalid value, trying again")
Prompt 9: Prompt used for LLaMA-3-Eval Scores
SYSTEM:
You are high school teacher evaluating student respenses to question. are tasked with grading the resporse based on how well it answers the question. You are to provide a numerical rating for how well the response answers the question based on the ground truth answer.
PROMPT:
Below is an instruction that describes a task. Write response that appropriately completes the request.
### Instruction:
Provided Answer:
(answer)
Ground Truth Answer: (gt_answer)
Follow these instructions when writing your response:
On scale of 1-10, provide numerical rating for how close the
provided answer is the ground truth answer, with 10 denoting that the provided answer the same as ground truth answer.
Your response should contain only the numerical rating. DONOT include
anything clse like the provided answer, the ground truth answer, or explanation of your rating scale in your response. tags.
Wrap your numerical rating inside <rating><</atinng
Check very carefully before answering.
Follow the output format as shown in the example below:
Rxample response:
<rating>7</zating>
### Response:
32

---

Published as a conference paper at ICLR 2025
Prompt 10: Prompt used for many-to-many LLaMA-3-Eva Eval Scores
SYSTEM:
You are high school teacher evaluating student responses to some questions. Refore scoring their answers, you need first match each ground truth answer with the most appropriate answer providea by the student.
PROMPT:
Below an instruction that describes a task. Write response that appropriately completes the request.
### Instruction:
Predicted Answers:
(pred_list]
Grouth Truth Answers: (gt_list)
For each ground truth answer above, provide the index of the most appropriate predicted answer (1-indexed). Each line must contain single integer value denoting the id of the matcned prediction.
If there is no appropriate prediction for a ground truth answer, write -1.
Check very carefully before answering.
### Response:
33


Reviews:
[{"review_id": "BNLZUP9nG6", "reviewer": "Reviewer_AkMH", "summary": "This paper introduces InsightBench, a benchmark designed to evaluate the effectiveness of large language model-based (LLM-based) agents in performing comprehensive data analytics tasks. Unlike previous benchmarks, InsightBench focuses on end-to-end data analytics, assessing capabilities in question formulation, insight extraction, and summary generation across diverse business scenarios such as finance and incident management. InsightBench includes 100 datasets derived from the ServiceNow platform, enriched with specific insights and questions to evaluate an agent’s ability to recommend tasks and extract actionable insights. A baseline agent, AgentPoirot, is introduced and evaluated against other existing tools, with results showing it outperforms simpler query-based agents. The evaluation also incorporates LLaMA-3-Eval, an open-source alternative to GPT-4, offering a cost-effective and stable approach to assessing agent performance in text-based insight matching.", "questions": "- It would be valuable to test InsightBench on actual enterprise data and synthetic data to better understand its applicability to real-world tasks.\n- Given the evolving nature of LLMs, a dynamic benchmark that updates or adjusts insight difficulty could be beneficial for testing agent adaptability over time.\n- Further breakdown of evaluation scores (e.g., insight precision vs. general text similarity) could improve the interpretability and reliability of agent assessments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- InsightBench represents a significant step forward from single-query benchmarks, testing agents' holistic capabilities in an end-to-end analytics process more reflective of real-world data analysis.\n- The paper showcases careful design in creating datasets with embedded trends, anomalies, and comprehensive goals that simulate actual enterprise challenges, making the benchmark realistic and versatile.\n- By introducing LLaMA-3-Eval, the authors provide a stable and open-source alternative to GPT-4-based evaluations, reducing costs and ensuring reproducibility, a valuable contribution to the LLM community.\n- AgentPoirot effectively demonstrates the viability of performing multi-step data analytics, including descriptive, diagnostic, predictive, and prescriptive tasks, positioning it as a useful baseline for future improvements in analytics agents.", "weaknesses": "- LLaMA-3-Eval offers advantages, but its consistency with GPT-4 evaluations could benefit from more rigorous comparison across various agent tasks beyond simple insight extraction.\n- Despite the variety of themes, the benchmark may not fully capture the complexity of certain domains, potentially limiting generalizability to more complex or highly regulated fields like healthcare or law.\n- Synthetic data generated from ServiceNow templates may lack the nuances of actual enterprise data, which could affect the evaluation's applicability to real-world use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rTeCKZmDx", "reviewer": "Reviewer_DA8a", "summary": "This paper introduces InsightBench, a benchmark dataset designed to evaluate the performance of data analytics agents in generating multi-step insights from business data. The authors propose AgentPoirot, a new agent model that outperforms existing tools such as the Pandas Agent. The benchmark dataset includes 100 datasets across various business themes, each with planted insights for the agents to identify. A two-way evaluation mechanism using LLaMA-3 is implemented to assess agent performance.", "questions": "1. How does InsightBench handle noisy or incomplete data? While the synthetic datasets are well-curated, it would be valuable to know how robust InsightBench and AgentPoirot are to missing data or noise, as these are common in real-world applications.\n\n2. What is the scalability of InsightBench for larger datasets? Since InsightBench datasets are capped at 500 entries, how would the benchmark scale to larger, more complex datasets? Would AgentPoirot's performance degrade with larger data volumes?\n\n3. Are there plans to incorporate real-world data into InsightBench? A dataset that mixes synthetic and real-world data could potentially improve the benchmark’s applicability to practical scenarios.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "· Originality: The introduction of InsightBench is a significant innovation in the evaluation of data analytics agents. Unlike previous benchmarks, InsightBench focuses on multi-step analytics, providing a comprehensive framework for assessing agents' abilities to perform holistic analysis rather than isolated tasks.\n\n· Clarity: The paper is clearly written, with a logical progression from motivation to methodology and results. Figures and tables effectively support the text, particularly in explaining how InsightBench differs from existing single-step benchmarks.\n\n· Significance: InsightBench is a meaningful contribution that could substantially impact the development of data analytics agents. By offering a platform for evaluating agents' end-to-end performance, InsightBench encourages the creation of more versatile and context-aware analytics models, filling a notable gap in the data analytics research community.", "weaknesses": "· Limitations in Real-World Validation: While InsightBench provides synthetic data and insights, the effectiveness of AgentPoirot and other evaluated agents in handling real-world, noisy datasets remains untested. To demonstrate its practical value, further evaluation on real-world data (beyond synthetic datasets) would enhance the relevance of InsightBench.\n\n· Interpretability of LLaMA-3-Eval Scores: The LLaMA-3-Eval scoring method could benefit from additional explanation regarding its interpretability and stability, particularly in cases where scores may fluctuate with subtle prompt changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cJG5vD0Xxy", "reviewer": "Reviewer_nFSq", "summary": "InsightBench is a new way to test how well AI language models can handle complex business analysis tasks. Unlike older tests that only look at simple questions, this one checks if AI can do a whole analysis from start to finish - asking questions, finding important information, and suggesting what to do next.\n\nThe creators made 100 different datasets about things like finance and managing problems. Some of the data is made up, and some is real, to make it feel like actual business situations. They also came up with their own AI agent called AgentPoirot, which did better than other AI tools at figuring out different kinds of insights from the data.\n\nTo make sure anyone can use and improve on their work, the team used an open-source AI called LLaMA-3 to judge how well the AI agents do. They hope this will help push forward the development of AI that can do complete data analysis for businesses without needing humans to guide it every step of the way.", "questions": "1. Synthetic data\n   * How do you ensure that InsightBench datasets capture the complexity of real-world enterprise data?\n   * Could you elaborate on the methods used to simulate unpredictable or \"noisy\" data patterns?\n   * Have you incorporated real-world anomalies or edge cases into your data generation process?\n   * Have you validated the synthetic data against any real-world datasets?\n1. Dependency on structure\n   * Do you have plans to incorporate unstructured data (like text logs or emails) that is common in businesses?\n1. Scale\n   * Do you have a process in place for regularly updating the dataset, or have you considered partnering with industry experts to ensure the dataset remains relevant to current business practices?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Diverse coverage: The dataset covers diverse business domains like finance and incident management, so can be widely useful, although one can argue that the dataset is not focused on a particular domain.\n1. Quality: The dataset insights are evaluated by experts, hence there is an assurance of quality. The two way evaluation (summary + insights) will also help users of the platform be more confident in the results.", "weaknesses": "1. Synthetic data: This is a long standing tussle for evaluation -- how realistic the data can get. InsightBench datasets depend on synthetic data generation which can make the data quality lower as the data regresses from real world scenarios. Particularly, it is unclear how well the data represents the unhappy paths / scenarios often found in real world. One suggestion might be to further split up the datasets into Happy path and Challenge datasets, with challenge datasets using less synthetic data, and being more realistic of real world scenarios.\n1. Dependency on structure: InsightBench focuses on structured, tabular data, but increasingly other data formats like unstructured text, images, or complex graphs are becoming relevant for business analytics.\n1. Scale: It is unclear how the dataset can be kept up to date with evolving business tasks, how quality will be maintained, how new domains will be added etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lVrpGKYyoB", "reviewer": "Reviewer_FGNL", "summary": "This paper presents a new benchmark dataset called InsightBench, which evaluates end-to-end analytics tasks' performance in real-world enterprise troubleshooting. The benchmark includes 100 datasets extracted from the ServiceNow platform. The current benchmarks mainly focus on single round-trip query-answer evaluations. This proposed benchmark aims to evaluate an agent's holistic problem-solving skills.\n\nIn addition to proposing a new benchmark dataset, the authors also developed a new baseline agent called AgentPoirot, which demonstrated superior performance compared to existing popular agents such as Pandas Agent.", "questions": "1) Do you have any evidence that the distribution of your data (for example, insight types--incident, user, asset, finance, goal management) resembles real-world business scenarios?\n\n2) Agent Poirot works best with a temperature of 0.2. This value looks interesting to me. Do you know why the agent works best at such a low temperature?\n\n3) It was encouraging to see that LLaMa-3-Eval produced consistent outcomes compared to G-Eval, which is known to be well aligned with human judgments. Was there any way to evaluate whether LLaMa-3-Eval output is also directly aligned with human judgments?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1) This paper is a leap forward in laying a foundation for building more robust enterprise-grade business analytics agents. I agree that the current benchmarks primarily just focus on simple code completion or single-query tasks. Real-world problems almost always involve multi-step reasoning and insight development.\n\n2) The benchmark building process is very well-documented and detailed, which was a huge plus from me. It was also clever to use the real-world data schema from a popular troubleshooting ticketing platform, ServiceNow, which ensures the applicability of this approach.\n\n3) Using LLaMA-3-Eval as an alternative to GPT-4 which is a closed API. For the sake of science, using LLaMa is a huge boost in transparency.\n\n4) Plus, AgentPoirot showed impressive performance against Pandas Agent in end-to-end analysis. This result shows that properly designed benchmarks can drive the development of superior agents.\n\n5) The quality assurance process using both expert reviewers and extensive validation was impressive.", "weaknesses": "1) Whether synthetic data represents the real-world context is questionable. The synthetic data generation part is less well-documented than other parts of the paper. The authors should explain better how their synthetic data generation process works and how they ensured the representativeness of the synthetic data.\n\n2) The authors mentioned the four types of analytics up front (descriptive, diagnostic, predictive, and prescriptive). However, this notion of four types of analytics was not as prevalent in the later part, where the results were presented. Consider looping back to these four types of analytics when presenting the results and performance.\n\n3) I understand Pandas Agent is probably the most popular agent aiming to achieve similar goals. However, I would like to see the comparison against some other agents, if there are any.\n\n4) This paper focuses too much on showing the strengths of the benchmark and the specialized agent, Agent Poirot. Providing the boundary conditions and limits where the agent fails would be helpful for future studies in this line.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)