Is GPT-4 a Good Data Analyst?
Authors: Liying Cheng
Xingxuan Li
Lidong Bing
Venue: EMNLP 2023 Findings (2023)
Keywords: data analyst
GPT4
TL;DR: We quantitively evaluate whether GPT-4 is a good data analyst.

Abstract: As large language models (LLMs) have demonstrated their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by artificial intelligence (AI). This controversial topic has drawn great attention in public. However, we are still at a stage of divergent opinions without any definitive conclusion. Motivated by this, we raise the research question of "is GPT-4 a good data analyst?" in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data analysts.

Is GPT-4 a Good Data Analyst?
Liying Cheng 1.2
Xingxuan Li+ Lidong Bing
'DAMO Academy, Alibaba Group, Singapore HHupan Lab, 310023, Hangzhou, China *Nanyang Technological University, Singapore
fliying.cheng xingxuan. 1i, 1.bing) @ @alibaba- inc. com
Abstract
Oytiznal Ioput
Compuliocy lepua
Dutabase
External
As large language models (LLMS)) have demon- strated their powerful capabilities in plenty of domains and tasks, including context under- standing, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by artificial intelligence (AI). This controver- sial topic has drawn great attention lic. However, we are still at a stage of diver- gent opinions without any definitive conclusion. Motivated by this, we raise the research ques- tion of "is GPT-4 good data analyst?" in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose frame- work to tackle the problems by carefully de- signing the prompts for GPT-4 to conduct ex- periments. We also design several task-specific evaluation metrics to systematically compare the performances between several professional human data analysts and GPT-44 Experimental results show that GPT-4 can achieve compa- rable performance humans. We also pro- vide in-depth discussions about our results to shed light on further studies before reaching the conclusion that GPT-4 can replace data an- alysts. Our code, data and demo are available at: https:/ / http//gttttttbtt wtwttbub..oommmm/ cot....oocc..oo... Dwwp//DMMOO-----SSEE GPT4 -at-aa--aaaaaaaaaaaeaaaaaa
Knowledge
Question
Schemn
Raw Data
Retrieval
Code Execution
Online Information
Code
datu
pub-
Outpst
Analysis
figuro.pdf
Figure I: A figure showing the flow of our proposed framework regarding GPT-4 as a data analyst. The com- pulsory imput information containing both the business question and the database is illustrated in the blue box on the upper right. The optional input referring to the external knowledge source circled in the red dotted box on the upper left. The outputs including the ex- tracted data (i.e., "data, txt "), the data visualization (i.e., ffgurr.pdd"") and the analysis are circled in the green box at the bottom.
finance (Wu et al., 2023c), healthcare (Han et al., 2023; Li et al., 2023b), biology (Zheng et al., 2023), law (Sun. 2023), psychology (Li et al., 2023a), etc. Most of these researches demonstrate the effec- tiveness of LLMs when applying them to different tasks. However, the strong ability in understand- ing, reasoning, and creativity causes some potential anxiety among certain groups of people.
As LLMs are introduced and becoming popular not only in the NLP community but also in many other areas, those people in and outside of the NLP community are considering or worrying whether artificial intelligence (Al) can replace certain jobs (Noever and Ciolino, 2023; Wu et al., 2023a). One such job role that could be naturally and controver- sially "replaced" by Al is data analyst (Tsai et al., 2015; Ribeiro al., 2015). The main and typical job scopes for a data analyst include extracting rele- vant data from several databases based on business partners' requirements, presenting data visualiza- tion in an easily understandable way, and also pro-
1 Introduction
Large language models (LLMs)) such as OpenAl's GPT series have shown their strong abilities on var- ious tasks in the natural language processing (NLP) community, including data annotator (Ding et al., 2023), data evaluator (Chiang and Lee, 2023; Luo et al., 2023; Wang et al., 2023; Wu et al., 2023b), etc. Beyond NLP tasks, researchers also evaluate the LLM abilities in multiple domains, such as
Xingxuan Li is under the Joint Ph.D. Program hetween Alibaba and Nanyang Technological University.

---

viding data analysis and insights for the audience. This job involves relatively routine scope, which may become repetitive at times. It also requires several technical skills, including but not limited to SQL, Python, data visualization, and data analysis, making it relatively expensive. As this job scope may adhere to relatively fixed pipeline, there is heated public debate about the possibility of an Al tool replacing data analyst, which attracts signifi- cant attention.
gest further studies are needed before concluding whether GPT-4 is good data analyst. To summa- rize, our contributions include:
We for the first time raise the research question of whether GPT-4 is a good data analyst, and quantitatively evaluate the pros and cons. How- ever, further research is still required reach definitive conclusion.
For such a typical data analyst job scope, we propose an end-to-end automatic framework to conduct data collection, visualization, and analysis.
In this paper, we aim to answer the following re- search question: Is GPT-4 good data analyst? To answer this question, conduct preliminary stud- ies on GPT-4 to demonstrate its potential capabili- ties data analyst. We quantitatively evaluate the pros and cons of LLM as a data analyst mainly from the following metries: performance, time, and cost. Specifically, we treat GPT-4 (gpt-4-0314 4)' as a data analyst to conduct several end-to-end data analysis problems. The flow of our proposed frame- work is shown in Figure 1. According to the given question, the model has to identify the relevant tables and schemes in the databases that contain the necessary data, and then extract the data from the databases and organize it in a way that is suit- able for figure generation. Then, is required to analyze the data to identify trends, patterns, and insights that can help answer the initial question. Since there is no existing dataset for such data analysis problems, we choose one of the most re- lated datasets NyBench (Luo al., 2021), and add the data analysis part on top. We design several automatic and human evaluation metrics to compre- hensively evaluate the quality of the data extracted, charts plotted and data analysis generated.
We conduct a systematic and professional hu- man evaluation of GPT-4's outputs. The data analysis and insights with good quality can be considered as the first benchmark for data anal- ysis in the NLP community.
2 Related Work
2.1 Related Tasks and Datasets
Since our task setting is new in the NLP community, there is existing dataset that is entirely suitable for our task. We explore the most relevant tasks and datasets. First, the NyBench dataset (Luo al., 2021) translates natural language (NL) queries to corresponding visualizations (VIS), which covers the first half of the main job scope of a data ana- lyst. This dataset has 153 databases along with 780 tables in total and covers 105 domains, and this task (NL2VIS) has attracted significant attention from both commercial visualization vendors and academic researchers. Another popular subtask of the NL2VVS task is called text-to-SQI,, which con- verts natural language questions into SQL queries (Zhong et al., 2017; Guo et al., 2019; Qi et al., 2022; Gao et al., 2022). Spider (Yu et al., 2018), SParC (Yu et al., 2019b) and CoSQL. (Yu et al., 2019a) are three main benchmark datasets for text- to-SQL tasks. Since this work is more focused imitating the overall process of the job scope of a data analyst, we adopt the NL2VIS task which has one more step forward than the text-to-SQL task.
Experimental results show that GPT-4 can beat an entry-level data analyst and an intern data ana- lyst in terms of performance and have comparable performance to a senior-level data analyst. In terms of the cost and time of our experiments, GPT-4 is much cheaper and faster than hiring data an- alyst. However, since it is preliminary study on whether GPT-4 is good data amalyst, we conduct some additional experiments and provide fruitful discussions on whether the conclusions from our experiments are reliable in real-life business from several perspectives, such as whether the questions are practical, whether the human data analysts we choose are representative, etc. These results sug-
For the second part of data analysis, we also ex- plore relevant tasks and datasets. Automatic chart summarization (Mittal et al., 1998; Ferres et al., 2013) is a task that aims to explain a chart and summarize the key takeaways in the form of natu- ral language. Indeed, generating natural language summarics from charts can be very helpful to in- fer key insights that would otherwise require a lot
'The most advanced model in the GPT series at the time of this paper was written.

---

of cognitive and perceptual effort. In terms of the dataset, the chart-to-text dataset (Kantharaj al., 2022) aims to generate a short description of the given chart. This dataset also covers a wide range of topics and chart types. Another relevant NLP task is called data-to-text generation (Gardent et al., 2017; Dušek et al., 2020; Koncel-Kedziorss et al.. 2019; Cheng et al., 2020). However, the output of all these existing works is descriptions or sum- maries the form of one or few sentences or a short paragraph. In contrast, data analysts are required to provide more insightful comments in- stead of intuitive summaries. Furthermore, in the practical setting of data analytics work, one should highlight the analysis and insights in bullet points to make them clearer to the audience. Therefore, in this work, we aim to generate the data analy- sis in the form of bullet points instead of short paragraph.
munication and problem-solving In this work, we regard GPT-4 as data analyst to conduct our ex- periments.
3 GPTT-4 as a Data Analyst
3.1 Background: Data Analyst Job Scope
The main job scope of data analyst involves uti- lizing business data to identify meaningful patterns and trends from the data and provide stakeholders with valuable insights for making strategic deci- sions. To achieve their goal, they must possess a variety of skills, including SQL query writing, data cleaning and transformation, visualization genera- tion, and data analysis.
To this end, the major job scope of data ana- lyst can be split into three steps based on the three main skills mentioned above: data collection, data visualization and data analysis. The initial step in- volves comprehending business requirements and deciding which data sources are pertinent to an- swering them. Once the relevant data tables have been identified, the analyst can extract the required data via SQL queries or other extraction tools. The second step is to create visual aids, such as graphs and charts, that effectively convey insights. Finally, in the data analysis stage, the analyst may need ascertain correlations between different data points, identify anomalies and outliers, and track trends over time. The insights derived from this process can then be communicated to stakeholders through written reports or presentations.
2.2 Abilities of GPT-3, ChatGPT and GPT-4
Researchers have demonstrated the effectiveness of GPT-3 and ChatGPT on various tasks (Ding et al.. 2023; Chiang and Lee, 2023; Shen et al., 2023; Luo et al., 2023; Wang et al., 2023; Wu et al., 20236; Li et al., 2023a; Han et al., 2023; Li et al., 2023b). For example, Ding et al. (2023) evaluated the perfor- mance of GPT-3 show that GPT-3 performs better on simpler tasks such as text classification than more complex tasks such as named entity recognition (NER). Wang et al. (2023) treated ChatGPT as an evaluator. They used ChatGPT to evaluate the performance of nat- ural language generation (NLG) and to study its correlations with human evaluation. They found that the ChatGPT evaluator has high correlation with humans in most cases, especially for creative NLG tasks.
data annotator. Their findings
3.2 Our Framework
Following the main job scope data analyst, we describe our task setting below. As illustrated Figure 1, given a business-related question and one or more relevant database tables with their schema, aim to extract the required data, generate fig- for visualization and provide some analysis and insights.
GPT-4 is proven to be significant upgrade over the existing models, as it is able to achieve more advanced natural language processing capabilities (OpenAI, 2023). For instance, GPT-4 is capable of generating more diverse, coherent, and natural language outputs. It is also speculated that GPT-4 may be more capable of providing answers to com- plex and detailed questions and performing tasks requiring deeper reasoning and inference (Bubeck al., 2023). These advantages will have practical implications in various industries, such customer service, finance, healthcare, and education, where AI AT-powered language processing can enhance com-
To tackle the above task setting, we design an end-to-end framework. With GPT-4's abilities in context understanding, code generation, and data storytelling being demonstrated, we aim to use GPT-4 to automate the whole data analytics pro- cess, following the steps shown in Figure 1. Ba- sically, there are three steps involved: (1) code generation (shown in blue arrows), (2) code exe- cution (shown in orange arrows), and (3) analysis generation (shown in green arrows). The algorithm of our framework is shown in Algorithm

---

Algorithm GPT-4 as a data analyst
Require: Question q; Database schema s; Database table t; Online
Require: Instruction prompts for code generation Prodes analysis generation Panalysis
Require: LLM f()); LM decoding temperature
Require: An external knowledge retrieval model g(.)
Require: Python compiler i.))
Q f(g.& 8. Peodes T)
D Generate SQL query (Q) and Python code (C).
D,G <- h(Q,C,8,b)
D Execute code to get data (D) and graph (G).
if o is true then
Only use online information when instructed. Query information from external knowledge source. Generate analysis (A) from data (D) and online information (I).
I 4 ((g, D)
A + f(g. Panalysis: T)
return D, A
else o false then <- {g: Panalysis: T)
D Generate analysis (A) from data (D).
return D, A
end
Question: [question]
Question: [question]
conn = sgliite,ccmmect((datbbbae file name))
[extracted data]
[database schema]
Generate analysis and insights about the data in bullet points.
Write Python code to select relevant data and draw the chart. Please save the plot to "figure.pdf and save the label and value shown in the graph "data.txi".
Table Prompt for the third step our framework: analysis generation. Text in blue: the specific question and the extracted data as shown in "data.txt".
Table I: Prompt for the first step in our framework: code generation. Text blue: the specific question, database file name and database schema.
conn
file name))" shown in Table 1 in the code, the massive raw data involved this step. By executing the Python code, we are able to get the chart in "figure.pdff and the extracted data saved in "data.txt".
Step 1: Code Generation. The input of the first step contains question and database schema. The goal here is to generate the code for extracting data and drawing the figure in later steps. We utilize GPT-4 to understand the questions and the relations among multiple database tables from the schema. Note that only the schema of the database tables is provided here due data security reasons. The massive data is still kept safe offline, which will be used in the later step. The designed prompt for this step is shown Table 1. By following the instructions, we can get piece of Python code containing SQL queries. An example code snippet generated by GPT-4 shown in Appendix A.
Step 3: Analysis Generation. After we obtain the extracted data, we aim to generate data analysis and insights. To make sure the data analysis is aligned with the original query, question and the extracted data as the input. Our designed prompt for GPT-4 of this step is shown in Table 2. Instead of generating a paragraph description about the extracted data, we instruct GPT-4 to generate the analysis and insights in 5 bullet points to emphasize the key takeaways. Note that we have considered the alternative of using the generated figure as input well, as the GPT-4 technical report (OpenAI, 2023) mentioned it could take images as input. However, this feature was not open to the public at the time of this paper was written. Since the extracted data essentially contains at least the same amount of information
both the
Step 2: Code Execution. As mentioned earlier in the previous step, maintain data safety, execute the code generated by GPT-4 offline. The input in this step the code generated from Step and the raw data from the database, as shown in Figure 1. By locating the data directory using

---

the generated figure, we only use the extracted data here as input for now. From our preliminary experiments, GPT-4 is able to understand the trend and the correlation from the data itself without seeing the figures.
Metrics
Group
Group 2
Average
Correctness
0.77
0.78
0.78
Figure
Chart Type
0.99
1.00
0.99
Aesthetics
2.48
2.51
2.50
Correctness
0.94
0.94
0.94
Data
Complexity
2.30
2.28
2.29
In order to make our framework more practical such that it can potentially help human data an- alysts boost their daily performance, we add an option of utilizing external knowledge sources, as shown Algorithm 1. Since the actual data analyst role usually requires relevant business background knowledge, we design an external knowledge re- trieval model g(.) to query real-time online infor- mation (I) from an external knowledge source (e.g. Google). In such option, GPT-4 takes both the data (D) and online information (I) as input to gen- erate the analysis (A).
Analysis
Alignment
1.00
1.00
1.00
Fluency
3.00
3.00
3.00
Table 3: Performance of GPT-4 as a data analyst.
The information correctness and chart type correct- ness are calculated from 0 to 1, while the aesthetics score is on scale of O to 3.
A Analysis Evaluation For each bullet point gener- ated in the analysis and insight, we define evalua- tion metrics below:
correctness: does the analysis contain wrong
4 Experiments
data or information?
4.1 Dataset
alignment: does the analysis align with the question?
Since there no exact matching dataset avail- able, we select the most relevant one, known as the NyBench dataset. We randomly choose 1000 questions from various domains, featuring differ- ent chart types and difficulty levels, to conduct our main experiments. The chart types cover bar, stacked bar, line, grouping line, scatter, grouping scatter and pie. The difficulty levels include: easy, medium, hard and extra hard. The domains include sports, artists, transportation, apartment rentals, col- leges, etc. On top of the existing NyBench dataset, we additionally use our framework to write insights drawn from data in 5 bullet points for each instance and evaluate the quality using our self-designed evaluation metrics.
complexity: how complex and in-depth is the analysis?
fluency: is the generated analysis fluent, gram- matically sound and without unnecessary repe- titions?
We grade the correctness and alignment on a scale of O to 1, and grade complexity and fluency in a range between 0 to 3.
To conduct human evaluation, 6 professional data annotators are hired from two data annotation companies to annotate each figure and analysis bullet points on the evaluation metrics described above following the detailed annotation guidelines shown in Appendix B. The annotators are fully compensated for their work. Each data point is independently labeled by two different annotators.
4.2 Evaluation
To comprehensively investigate the performance, we carefully design several human evaluation met- rics to evaluate the generated figures and analysis separately for each test instance.
4.3 Main Results
GPT-4 performance. Table shows the perfor- mance of GPT-4 (gpt-4-0314) as a data analyst on 1000 samples. We show the results of each individual evaluator group and the average scores between these two groups. For chart-type correct- evaluation, both evaluator groups give almost full scores. This indicates that for such simple and clear instruction such "draw a bar chart", "show pie chart", etc., GPT-4 can easily under- stand its meaning and has background knowledge about what the chart type means, so that it can plot
Figure Evaluation We define 3 evaluation met- rics for figures:
correctness: is the data and information shown in the figure correct?
chart type: does the chart type match the re- quirement the question?
aesthetics: the figure aesthetic and clear with- out any format errors?

---

Figure
Data Analysis
Annotator
Sumples
Correctness
Chart Type
Aesthetics
Time (s)
Correctness
Complexity
Alignment
Fluency
Time (s)
Senior
0.79
0.96
2.96
472
0.98
2.01
0.98
2.98
324
30
GPT44
0.73
0.96
2,41
59
0.82
2.18
1.00
3.00
40
Junior
0.96
2.66
645
0.95
1.98
0.86
3.00
388
30
GPT-4
0.71
0.98
2.75
50
0.94
2.32
1.00
3.00
34
Intern
0.74
0.91
2.40
648
0.86
1.59
1.00
3.00
173
40
GPT44
0.73
0.97
2.45
55
0.91
2.28
1.00
3.00
33
Table 4: Overall comparison between several seniorrjunior/intem data analysts and GPT-4 on 100 random examples in total. Time spent shown in seconds (s).
the figure in the correct type accordingly. In terms of aesthetics score, it get 2.5 out of on av- erage, which shows most of the figures generated are clear the audience without any format errors. However, for the information correctness of the plotted figures, the scores are not so satisfactory. We manually check those figures and find most of them can roughly get the correct figures despite some small errors. As shown in Appendix B, our evaluation criteria very strict, such that long any data any label the x-axis y-axis is wrong, the score has be deducted. Nevertheless, it has room for further improvement.
Median/Average Annual Salary
Cost per instance
Source
Level
(USD)
(USD)
Senior DA
90,421
9.92
levels.fyi
Entry Level
37,661
5.36
Senior DA
86,300
9.47
Glassdoor
Junior DA
50,000
7.12
Intern DA
14,400
1.63
Senior DA
11.00
Our Annotation
Junior
7.00
Intern DA
2.00
GPT-4
0.05
Table 5: Cost comparison from different sources.
For analysis evaluation, both alignment and flu- ency get full marks on average. It verifies that generating fluent and grammatically correct sen- tences is definitely not problem for GPT-4. We notice the average correctness score for analysis is much higher than the correctness score for figures. This interesting that despite the wrong figure generated, the analysis could be correct. This is because, as mentioned, most of the "wrong" fig- ures only contain some small errors. Thus, only I or 2 out of the 5 bullet points related the error parts from the figures may be generated incorrectly, while most of the bullet points be generated correctly. In terms of the complexity scores, 2.29 out of on average is reasonable and satisfying. We will show few cases and discuss more on the complexity scores in Section 4.4.
ing, GPT-4's performance is comparable to human data analysts, while the superiority varies among different metries and human data analysts.
Among different levels of human data analysts, overall speaking, the senior group performs the best, followed by the junior group, and finally the interns, especially on the analysis correctness and complexity. Comparing human data analysts with GPT-4,
notice that GPT-4 outperforms both junior and intern data analysts on most of the met- rics, while still having some gap with senior data analysts on three metrics: figure correctness, figure aesthetics and analysis correctness.
Apart from the comparable performance be- tween all data analysts and GPT-4, we can notice the time spent by GPT-4 much shorter than hu- man data analysts. Table shows the cost compari- son from different sources. We obtain the median annual salary of data analysts in Singapore from level.fyi² and the average annual salary of data an- alysts in Singapore from Glassdoor We assume there around 21 working days per month and
Comparison between human data analysts and GPT-4. To further answer our research question, we hire professional data analysts to do these tasks and compare them with GPT-4 comprehensively, The profiles of the data analysts are described in Appendix C. We fully compensate them for their annotation. Table shows the performance of data analysts of different expert levels from different backgrounds compared to GPT-4. Overall speak-
hhttps://ww/ 1ewels. hettps://www.glaseee

---

Question
Please list the proportion number cach winning aircraft.
Question
List position of players and the averuge num-- ber of points of players each position. Visualize by har chart, and could you sort by the total num- ber in ascending?
SQL Query SELECT
a.Aircrat, COUNT(m Winning_Aicrrut) as wins
FROM airematt a JOIN match
Figure
ON L.Arrrrftt = m. Winning_Ainen Aircraft
GROUP BY a.Aircraft
ORDER BY wins DESC
Figure
Proport
Analysis
1. There 10 positions. Based the names, this dataset is about Rugby. Rugby group sport that is commonly seen the US, Canada and Commonwealth countries.
2. Stand Off and Scrum Half' are having more than 100 average points while the rest are below 40. This is a bit surprising as usually the Right Wing and Left Wing are the star scorers.
Analysis
1. The Robinson R-22 and Mil Mi-26 are the most successful aircraft in the dataset, with euch having 2 wins.
3. Prop has lowest average points only 9.33, as Prop's main role is to be in the first row and push against the opposition' 's Prop.
2. The CH-53E Super Stallion, CH-47D Chinook, and Bell 206B3 JetRanger each have win, mak- ing them less successful compared to the Robin- son R-22 and Mil Mi-26.
4. Full Bck, Loose Forward and Second Row hus the same points (20), while Hooker, Right Centre and Right Wing has the same points (36).
5. Overall dataset is slightly different from the general understanding. If there's no data issue, it might worth to reevaluate the players and think about a better position assignment.
3. The Robinson R-22 and Mil Mi-26 together account for 50% of the total wins, indicating their dominance the competition.
The dataset includes diverse range of aireraft, from helicopters like the CH-53B Super Stallion to smaller aircraft like the Robinson R-22, show- casing the variety of winning uircruft. 5. Despite the variety of uireraft, the data suggests that certain models, such as the Robinson R-22 and Mil Mi-26, have higher likelihood of win- ning. potentially due their design, performance, or other factors.
Table 7: Case Study 3 by Senior Data Analyst 2.
and draw a proper and correct pie chart accord- ing to the given question. In terms of the analy- sis, GPT-4 is capable of understanding the data by conducting proper comparisons (e.g., "most suc- cessful", "less successful", "diverse range"). In addition, GPT-4 can provide some insights from the data, such as: "indicating their dominance in the competition". These aforementioned abilities of GPT-4 including context understanding, code generation and data storytelling are also demon- strated in many other cases. Furthermore, in this case, GPT-4 can also make some reasonable guess from the data and its background knowledge, such as: "potentially due to their design, performance, or other factors". However, in another case shown in Appendix D, we notice some numerical errors done by GPT-4 which is very likely due to its issue of hallucination.
Table 6: Case study by GPT-4.
the daily working hour is around hours, and cal- culate the cost per instance in USD based on the average time spent by data analysts from each level. We pay the data analysts based on the market rate accordingly, to roughly match the median or av- erage salaries from two sources. Specifically, discuss the pay with each data analyst case by case. For our annotation, the cost of GPT-4 is approxi- mately 2.5% of the cost of an intern data analyst, 0.71% % of the cost a junior data analyst and 0.45% of the cost of a senior data analyst.
4.4 Case Study
Case by the senior data analyst. As shown in Table 7, we can notice that this expert human data analyst can understand the requirement, write the code to draw the correct bar chart, and analyze the
Case by GPT-4. In the case shown in Table 6, GPT-4 is able to generate a Python code containing the correct SQL query to extract the required data,

---

extracted data in bullet points, Apart from this, we can summarize three main differences with GPT-4, First, different from GPT-4, the human data ana- lyst can express the analysis with some personal thoughts and emotions. For example, the data an- alyst mentions "This is a bit surprising ...". In real-life business, personal emotions are important sometimes. With the emotional phrases, the au- dience can easily understand whether the data is as expected or abnormal. Second, the human data analyst tends apply some background knowl- edge. For example, as shown in Table 7, the data analyst mentions "... is commonly seen ...", which is more natural during a data analyst's actual job. While GPT-4 usually only focuses on the extracted data itself, an experienced data analyst is easily linked with one's background knowledge. How- ever, this might be the reason causing the slightly lower alignment scores in Table To mimic a hu- man data analyst better, our framework, we add an option of using Google search API to extract real-time online information when generating data analysis. We explain our additional experiment in- tegrating the optional online information in Section 4.5. Third, when providing insights suggestions, human data analyst tends to be conservative. For instance, in the 5th bullet point, the human data analyst mentions "If there's no data issue" before giving a suggestion. Unlike humans, GPT-4 usually directly provides the suggestion in a confident tone without mentioning its assumptions.
4, senior data analyst and junior data analyst. The detailed results are shown in Appendix E. For such practical and general questions, the senior data analyst and GPT-4 perform much better than the junior data analyst. The performances of the senior data analyst and GPT-4 are basically on par with each other.
Online Information Integration. In Figure 1, we show the optional input of external knowledge in our proposed framework. In some cases, data analysts are not only required to interpret the data in the databases, but also to understand and inte- grate some industry background knowledge. For such questions, we design an optional module that queries online information from Google and gen- erates the data analysis with the incorporation of the online information. Through our preliminary experiments, this module helps GPT-4 to combine additional knowledge. We show one case in Ap- pendix F.
4.6 Findings and Discussions
Generally speaking, GPT-4 can perform compara- ble to a data analyst from our preliminary exper- iments, while there are still several issues to be addressed before we can reach a conclusion that GPT-4 is good data analyst. First, as illustrated in Section 4.4 and Appendix D, GPT-4 still has hallucination problems, which is also mentioned in GPT-4 technical report (OpenAI, 2023). Data analysis jobs not only require those technical skills and analytics skills, but also requires high accuracy to be guaranteed. Therefore, a professional data analyst always tries to avoid those mistakes includ- ing calculation mistakes and any type of hallucina- tion problems. Second, before providing insightful suggestions, professional data analyst is usually confident about all the assumptions. Instead of di- rectly giving any suggestion or making any guess from the data, GPT-4 should be careful about all the assumptions and make the claims rigorous.
4.5 Additional Experiments
More Practical Questions. The questions in the experiments above randomly selected from the NyBench dataset. Although the questions indeed cover a lot of domains, databases, difficulty levels and chart types, they are still somewhat too specific according to human data analysts' feedback. The existing questions usually contain information such as specific correlation between two variables, and a specific chart type. In more practical setting, the requirements are more general, which requires data analyst to formulate a specific question from the general business requirement, and determine what kind of chart would present the data better. Therefore, we carefully design five practical and general questions that are acknowledged by a few senior data analysts. To evaluate the comprehensive abilitics such as the problem formulation ability of GPT-4, we compare the results among GPT-
5 Conclusions
The potential for large language models (LLMs) like GPT-4 to replace human data analysts has sparked controversial discussion. However, there is no definitive conclusion on this topic yet. This study aims to answer the research question of whether GPT-4 can perform as a good data ana- lyst by conducting several preliminary experiments.

---

We design a framework to prompt GPT-4 to per- form end-to-end data analysis with databases from various domains and compared its performance with several professional human data analysts us- ing carefully-designed task-specific evaluation met- rics. Our results and analysis show that GPT-4 can outperform intern data analyst junior data analyst, and can achieve comparable perfor- mance to senior data analyst, but further studies are needed before concluding that GPT-4 can re- place data analysts.
Acknowledgements
We would like to thank our data annotators and data evaluators for their hard work. Especially, we would like to thank Mingjie Lyu for the fruitful discussion and feedback.
References
Sébastien Bubeck, Varun Chandrasekaran, Ronen EI- dan, John Gehrke, Eric Horvitz, Fice Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv, abs/2303.122 12712.
6 Limitations
More Experiments. As mentioned in Section 4.5, the questions from the NvBench dataset con- tain very specific information, which is somewhat disconnected from real work scenarios. In terms of the broader questions that are more closely re- lated to real work scenarios, only 5 questions designed and evaluated in this work. Our next step is collect more practical and general questions to further test the problem formulation ability of GPT-4.
Liying Cheng, Dekun Wu, Lidong Bing. Yan Zhang, Zhamming Jie, Wei Lu, and Luo Si. 2020. Ent-desc: Entity description generation by exploring knowl- edge graph. In Proceedings of EMNLP.
Cheng Han Chiang and Hung-yt yi Lee. 2023, Can large language models be alternative human evalua- tions? Proceedings of ACL.
Bosheng Ding, Chengwei Qin, Linlin Liu, Lidong Bing. Shafiq Joty, and Boyang Li. 2023. Is gpt-3 good data annotator? In Proceedings of ACL.
We did not systematically conduct a large num- ber of experiments using online information as well. The reason is similar to the above. The original questions from the NvBench dataset largely de- pend on the data stored in the database and rarely require additional knowledge. Therefore, we leave the design of such open questions to future work.
Ondfej Dušek, Jekaterina Novikova, and Verena Rieser. 2020. Evaluating the late--of-the-ar of end-to-end natural language generation: The e2e nlg challenge. Computer Speech & Language.
Leo Ferres, Gitte Lindgaard, Livia Sumegi, and Bruce Tsuji. 2013. Evaluating a tool for improving acces- sibility to charts and graphs ACM Trans. Comput. Hum. Interact., 20:28: 1-28:32.
Choice of Annotators. The quantity of human evaluation and data analyst annotation is relatively small due to budget limitations. For human evalua- tion, we strictly select those professional evaluators in order to give better ratings. They have to pass our test annotations for several rounds before start- ing the human evaluation. For the selection of data analysts, we are even more striet. We verify if they really had data analysis working experience, and make sure they master those technical skills before starting the data annotation. However, since hiring a human data analyst (especially for those senior and expert human data analysts) is very expensive, we can only find few data analysts and ask them to do few samples.
Chang Gao. Bowen Li, Wenxuan Zhang, Wai Lam, Bin- hua Li, Fei Huang. Luo Si, and Yongbin Li. 2022. Towards generalizable and robust text-to-sql parsing. In Findings of EMNLP.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Bertz-Beltr.chii.. 2017. The webnig chal- lenge: Generating text from rdf data. In Proceedings of INLG.
Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian- Guang Lou, Ting Liu, and D. Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of ACL.
Tianyu Han, Lisa C. Adams, Jens-Michalis Papaioan- nou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K. Bressem. 2023, Medalpaca an open-source collection of medical conversational ai models and training data.
Ethics Statement
The purpose of this work is not replace the data analyst role or to create anxiety. Instead, we would like to explore the potential of GPT-4 to aid human data analysts in more efficient working.
Shankar Kantharaj. Rixie Tiffany Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and

---

Shafiq Joty. 2022. Chart-to-text: large-scale bench- mark for chart summarization. In Proceedings of ACT, pages 4005-4023.
Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin, 2023a. Ai-generated content (aige): A survey.
Rik Koncel-Kedziorsk, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph trans- formers. In Proceedings of ACL.
Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. 2023b. Large language models are diverse role-players for summarization evaluation. arXiv preprint arXiv:2303. 2303.
Xingxuan Li, Yutong Li, Shafiq Joty. Linlin Liu, Fei Huang. Lin Qiu, and Lidong Bing. 2023a. Does gpt-3 demonstrate psychopathy? evaluating large language models from a psychological perspective.
Shijie Wu, Ozan Irsoy. Steven Lu, Vadim Dabravolski. Mark Dredze, Sebastian Gehrmann, Prabhanjan Kam- badur, David Rosenberg, and Gideon Mann. 2023c. Bloomberggpt: A large language model for finance.
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023b. Chatdoctor: A medical chat model fine-tuned on llama model using medical do- main knowledge.
Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Victoria Lin, Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasumaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vin- cent Zhang. Caiming Xiong, Richard Socher, Walter Lasecki, and Dragomir Radev. 2019a. CoSQL: A conversational eext-to-SQL challenge towards cross- domain natural language interfaces to databases. In Proceedings of EMNLP-IJCNLP, pages 1962-1979.
Yuyu Luo, Nan Tang, Guoliang Li, Chengliang Chai, Wenbo Li, and Xuedi Qin. 2021. Synthesizing nat- ural language to visualization (n12vis) benchmarks from nl2sql benchmarks. In Proceedings of SIG- MOD.
Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2023. Chatgpt as factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621.
Tao Yu, Rui Zhang. Kai Yang, Michihiro Yasunaga, Dongxu Wang. Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. In Proceedings of EMNLP, pages 3911-3921.
Vibhu O. Mittal, Johanna D. Moore, Giuseppe Carenini, and Steven Roth. 1998. Describing complex charts in natural language: A caption generation system. Computational Linguisties. 24(3):431445
Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang. Tao Chen, Emily Ji. Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vin- cent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. SParC: Cross-domain se- mantic parsing in context. In Proceedings of ACL. pages 1511-4523.
David Noever and Matt Ciolino. 2023. Professional certification benehmark dataset: The first 500 jobs for large language models.
OpenAI. 2023. Gpt-4 technical report.
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang. Quanshi Zhang, and Zhouhan Lin. 2022. RASAT: Integrating relational structures into pretrained Seq2Seq model for text-to-SQL. In Proceedings of EMNLP, pages 3215-3229.
Zaixiang Zheng, Yifan Deng, Dongyu Xue, Yi Zhou, Fei YE, and Quanquan Gu. 2023. Structure-ifformed language models are protein designers.
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv: 17X0:07000000033
André Ribeiro, Afonso Silva, and Alberto Rodrigues da Silva. 2015. Data modeling and data analytics: A survey from big data perspective. Journal of Software Engineering and Applications, 08:617-634)
Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing, 2023. Are large language models good evalua- tors for abstractive summarivation?
Zhongxiang Sun. 2023. A short survey viewing large language models in legal aspect.
Chun- Chun-Wei Tsai, Chin-Feng Lai, H. C. Chao, and Athana- sios V. Vasilakos. 2015. Big data analytics: a survey. Journal of Big Data, 2:1-32.
Jiaan Wang, Yunlong I Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. chatgpt good nig evaluator? a preliminary study. arXiv preprint N3V:22303.04044

---

A Example Code
B Detailed Annotation Guidelines
In this section, we present our detailed annotation guidelines for human evaluators.
import sqlite3
B.1 Figure Evaluation
2 import natplotlib.pypler as plt
3
For the figures generated by the model, scores will be given based on the following three criteria, using the correct figure (and correct data) as reference:
Connect to the database
sqlite3. connect( commect(sirccraftssqieee cursor()
7
8 Query to get the count of each winning aircraft
Information Correctness. The information cor- rectness can be chosen from O, 0.5 and 1. First, if the information is correct, 1 point is awarded. Second, if there are minor errors, 0.5 points are awarded. The minor errors mean that the data is mostly correct, but missing one or two data points or showing indexes instead of x-axis/y-axis names, and it does not affect the overall data conclusion. Third, if any important data is incorrect, no points are awarded. The errors include data errors, extra data, missing data, incorrect x-axis/y-ai names, etc. Errors do not include inconsistent color, inac- curate data, inconsistent order, etc.
9 query
.iinring.Airrraftt a.Aircraft, COUNT (uininn.iinnin Aircraft) as wins FROM aircraft
JOIN natch a.Aircraft_ID n. Airerat
13 GROUP BY a.Aircraft
ORDER BY wins DESC
Execute the query and fetch the results execute(query) 19 results cursor. retchall()
Extract the labels values the chart labels [row[0]
results] values [row[1] [1) for in results]
24
Save the labels and values to data. txt with open("data.txt" "w")
for label, (Laabte(f"(laeel)) (value)\n")
zip(labels, values):
Chart Type Correctness. Since chart type is pretty straightforward, the scores will be binary. it matches the chart type required in the question, 1 point is awarded; otherwise, O point is awarded. For example, if pie chart is required in the ques- tion, but line chart is drawn, 0 point is awarded.
29
30 Create the pie chart plt.pie(values, labels=labels, autopet="<1. (1.
("Propoooooooottoonn Wins by Aircraft")
Save the chart to tigure.pdf
35
connection
conn.close()
The score of this metric is on scale of O to points. all information clear, it will receive full marks (3 points). there are minor format issues, points are awarded. If it affects the reader's understanding to some extent, point is awarded. If it seriously affects reading, O points awarded. Subjectivity is relatively high for this metric, and thus we show the annotators few examples.
Aesthetics.
Figure 2: An example of complete code generated by GPT-4. This code is to answer the question shown in Table
Figure 2 shows an example code generated by GPT-4. First, we can notice that GPT-4 is capable of writing SQL queries with several commands, such JOIN, GROUP BY, ORDER BY extract the required data. Sccond, GPT-4 knows how to use multiple Python packages including sqlite and matplotlib, which help to connect the databases and draw charts respectively. Third, GPT-4 can under- stand the requirement in the question to save the data and figure it into the correct files accordingly. Last but not least, it can also generate comments understandable by readers, which is aligned with the goal of helping human data analysts boost their daily performance. In the case when the wrong code generated, a human analyst can easily un- derstand which part goes wrong with the aid of the comments.
B.2 Data Analysis Evaluation
For each data analysis bullet point generated by the model, we evaluate from the following four aspects based on the correct data and the correct figure.
The scores of this metric are binary. The sentence gets 1 point if the information in the sentence is correct. If the sentence contains any false information, it will get score.
Correctness.
A Alignment. The scores of this metric are binary as well. The bullet point gets point it is relevant to the question, and 0 points if irrelevant.

---

Complexity. The score of complexity is o scale of to 3 points. The bullet point gets 0 points if is a general description. For example: "This is bar chart, the x-axis represents.. the y-axis represents ...". This information is considered very general, which can be obtained without seeing the actual data. The bullet point gets point for directly visi- ble data points or information. For example, "the quantity wednesday reached 50.", "There are total of data points.", etc. The bullet point gets 2 points for analysis obtained by comparison or calculation. For example, "the range of certain data is from 23 to 699". This actually includes which one has the highest and lowest scores, and is ob- tained by comparison, so it is labeled as 2 points. Similarly, another example is, "Wednesday has the highest score, reaching 50". Other types of exam- ples are: "The data increased from 5 to 8", "the sum of and on Wednesday is 67", "there is a positive correlation trend".. Lastly, The bullet point gets 3 points the sentence has some insights. For example, "the number on Wednesday reached 50, indicating/suggesting ...'. Most of the keywords used are "indicates//uugestt//he sically be worth 3 points. The premise is that the insight should be about the content of the figure. It can be an insight obtained from specific numbers or the positive/negative correlation/trend of the axis/y-axis variables. The insights generally can obtain some useful conclusions. ii.
years. The intern junior data analyst is doing a data analysis internship at consulting firm.
D More Case Studies
Question
Show about the correlation between Height and Weight scatter chart.
Figure
Conceastion
24
34
355
2.0
1:54
100
a82
Analysis
1. The data provided consists of 9 data points, each representing the height and weight of an in- dividual.
2. The height range the dataset from 188 cm to 202 em, while the weight range is from 82 kg
to 94 kg.
3. The average height of the individuals in the dataset is approximately 195.3 em, and the aver-
which can ba-
age weight is approximately 88.1 kg.
4. There seems be positive correlation be- tween height and weight, taller individuals gen- erally have higher weights.
5. A seatter chart would help visualize this cor- relation, with height on the x-axis and weight on the y-axis. The data points would likely form an upward trend, indicating that as height increases, weight also tends increuse.
Fluency. Fluency and Grammar soundness is on scale of 0 to 3 points. (a) Very smooth, gram- mar errors: 3 points.
Table 8: Another Case Study by GPT-4.
(b) Some minor grammar or spelling errors that do not affect understanding: 2 points. (c) Some grammar errors that affect un- derstanding to some extent: 1 point. (d) Serious grammar errors that significantly affect reading and understanding: 0 points.
Table shows another question addressed by GPT-4. Again, GPT-4 able to extract the correct data, draw the correct scatter plot and generate reasonable analysis. Although most of the bullet points are generated faithfully, if we read and check carefully, we can notice the numbers of the average height and weight are wrong. Apart from the well- known hallucination issue, we suspect that GPT-4's calculation ability is not strong, especially for those complex caleulations. We also notice this issue in several other cases. Although GPT-4 generates the analysis bullet points in a very confident tone, but the calculation is sometimes inaccurate.
C Profiles of Our Data Analysts
We hire 5 data analysts of different levels in total, 2 of them are senior data analysts, 2 of them are junior data analysts and I of them is an intern data analyst. One senior data analyst has more than 6 years of data analysis working experience in the fi- nance industry. The other senior data analyst works in the internet industry 5 years. One junior data analyst has data analysis working experience in a consulting firm within years. The other junior data analyst works in the supply chain industry as a data analyst for around
data analyst for over
E More Practical Questions
In this section, we present 5 more practical ques- tions that do not have clear or specific requirements. The questions designed are more likely to be open

---

Question Junior DA Senior GPT-4
Table 13, All three annotators provided the same answer. GPT-4 ranked first because it provided more insights in most of the bullet points. The senior data analyst is ranked second because he/she suggests another metric to measure the popularity of the rooms. Although the jumior analyst gave the same answer, 3 out of 5 bullets are irrelevant to the question.
2
1
2
1
3
1
2
Table 9: Scores 5 more practical questions.
Table 14 shows the results of the last practical question. The senior data analyst's answer is no doubt to be the best, as the figure is clearer and the bullet points are more reasonable than the other two. For the junior data analyst and GPT-4, the answers mentioned in the analysis cannot be seen clearly from the figures. The junior data analyst is ranked last as he/she wrote few irrelevant analyses again.
questions, which do have one fixed answer. This requires the data analysts and GPT-4 to have good problem formulation ability. We score them based on their ranking of cach question. The one who is ranked the first gets a score 3, the second one gets a score of 2 and the last one gets score of 1. The results are shown in Table 9. We discuss the results of these 5 questions one by one evaluate how human data analysts and GPT-4 perform.
F Online Information Integration
Table 10 shows the first more practical ques- tion. This question simply asks "Which candidate should accept?" without specifying the exact re- quirements for candidate acceptance. We rank the senior data analyst's answer the first among these 3. Instead of only considering the support rate, the senior data analyst also considered the oppose rate provided in the database, and proposed a new met- ric named net polarity rate. GPT-4 gave the answer by sorting on the support rate. However, GPT-4 mentioned other candidates could be considered if additional factors are taken into account. This indicates the potential of GPT-4 to be trained be a professional data analyst who has comprehensive thinking.
Table shows one case done by GPT-4 with and without integrating online information. When gen- erating analysis without querying online informa- tion, the analysis bullet points are purely from the given database. When incorporating external knowledge, GPT-4 is able to generate relevant addi- tional information from online to make the analysis more powerful.
The results of the second question are shown in Table 11. All annotators gave the same answer by ranking the students based on their average grades. However, the junior data analysts did not specify the names of the students and wrote few irrelevant analysis bullet points, thus ranked the last by us. In contrast, the human senior data analyst explained the reason for choosing this metric clearly.
Among the results of the third question shown in Table 12, we rank the GPT-4 the first, followed by the senior data analyst and finally the junior data analyst. Both GPT-4 and the senior data analysts analyzed the data based on different regions. Given the limited employee data, GPT-4 still mentioned the potential possibilities of cost reduction in 3 bul- let points, while the senior data analyst mentioned it in the last bullet point.
The results of the fourth question are shown in

---

Question
Which cunditaate should accept?
Junior DA
3.0
439
0.6
Based dataa candidale has the highest average support
Newsweek Poll has the highest average support score amount three different poll sources.
Candidate 2 has extremely high unsure rate given by WNBC/Marist Poll.
4. Candidate 4 has the highest oppose rate.
5. The variance of consider rate is low among all the different rates.
DA
Candidate wupportiorpsoonnee.ooo
6.-
-6.2
dioe
Co03in side
LAT2 Kergne
Vinern Miglore Arliga
Hemo
Sarnics
Laurent Capst
Combining the Support Rate, Oppose and Net Polarity Rate, should accept Loic De Kergret.
Overall. all candidates have higher oppose rate than the support rate. For the support much higher oppose rate.
Loic ranks second. Although Vincent Montmeant has higher support rate than Loic, Vincent also has
4. We consider the New polarity rate to be the best metric to take. This metric is calculated as Support Rate substrate the Oppose rate.
We think it's important for the candidate to have a high support rate, but also important to have low oppose rate. Hence the Net
Polarity rate the one we take.
GPT-4
Average Support Bais for Each Cancidats
3.00
Vincent Montméat the highest overage support among candidates, score of 0.33, making him a strong contender acceptance.
and 0.18, respectively, indicating that Guillaume Samica and Lote De Kergret have similar average support scores of 0.24, placing them in the middle of the pack and suggesting that they could be posential alternatives Vincent Montméat not selected.
Stéphane Antiga and Laurent Capet the lowest average support. with
they may the most suitable candidates for the position.
4. Hubert Henno has slightly higher average support score than Guillaume Samica and Loic De Kergret, with score of 0.25, making him another potcatial altemnttive candidate.
Overall, the data suggests that Vinceut Montméat the most favored candidate, but Hubert Henno, Guillaume Samica, and Loic De Kergret could also be considered if additional factors need to be taken into account,
Table 10: Comparison GPT-4, senior and a junior data analyst on the first more practical questions.

---

Question
Choose students perform well representatives.
Junior DA
13
20499 39451
X04MM
X0406
XI4X
39424
2000
Based GPA, lop
324257, 324274.
ACCT department the amount of students dataset, which is Students date birth before 1981.
The student hours has high variance. out of students transfer students.
Senior DA
1. We choose to measure the student by their average grade.
The calculation method is to take the GPA for every student and take the average. This is to prevent the case where student takes multiple courses.
The top students ranked by grade are Robertson Gerald, Smithson Anne, Katinga Raphael.
Robertson Gerald's score the highest 3.87, followed by Smithson Anne 3.27, Katinga Raphael at 3.15. While Rohertson significantly than the rest, Smithson Katinga not leading the following much.
GPT-4
Shuderts by
35
38
0
10-
00
The data provided consists three students" their respective performance scores, presumably scale of O
5.
Among the three students, Gerald Robertson has the highest test performance score with a 3.87, indicating that be performs exceptionally well on tests.
3. Annz Smithson has the sccond-hight soore with a 3.27, which suggests that she also performs well on tests, although not as
high as Gerald Robertson.
4. Raphael Katinga has the lowest score among the three students with a 3.15, but this score still indicates relatively good performance on tests.
Based the data, three students can be considered as representatives those who perform well on tests, with Gerald Robertson being the top performer, followed by Anne Smithson and Raphael Katinga.
Table 11: Comparison of GPT-4, senior and junior data analyst on the second more practical questions.

---

Question
How reduce human shifting employees from different departments among these regions?
Junior DA
3199
3339
199-
Based salary cmployees all departments, department highest salary, followed by accounting and
public relations
The lowest paid departments purchasing and administrationund shipping
Amoung the titles, purchusing clerk, slock clerk and shipper clark lowest paid while managers and presidents highest paid
4. Shipping and sales have the most amount of employees while accounting and administration have lowest amount of employees
5. Based un ahove, it is shown that it's not feasible in move people from accouning. admin depurtments in sales, shipping there
are very little people in those department. Instend, finance and IT are higher paid and having more people and could be good target of restructruing
Senior DA
1. We plotted out employee count by the department and colored hy region with blue the American and orange for Fumppe Europe has only employees, with from Human Resources and from Public Relationship think it may not very efficient to set up an EU office with only two employces Usually, Human Resources employees closely
EU, department. Hence we would suggest shilling the keep in EU
resource function
PR, the marketing However, we would advise company consider labour different places well
GPT-4
The data shows a high concentration employees in the Shipping department in the Americas region, which could indicate a potential area for cost reduction by redistributing employees to other regions or departments. The Americas region diverse range departments, Finance and Purchusing having a significant number employeen regions there shortage of staff
This could present opportunity shift employees similar roles.
departments
Europe has very limited number of departments represented, with only Human Resources and Public Relations having employee each. This suggests that there may need for additional staff in other departments the Europe region, which could be addressed transferring employees from the Americas region.
4. The Executive depariment in the Americas region has three employees, which could be an area consider for cost reduction by evaluating the necessity of all three positions and potentially consolidating roles or shifling employees to other departments or regions.
Overall, duta suggests that there may he opportunities for cost reduction and incressed efficiency by evalunting the distribation employees across departments and regions, and considering strategic shifls of employees to address imbalances and reduce redundancy in certain areas.
Table 12: Comparison of GPT-4, a senior and junior data analyst on the third more practical questions.

---

Question
Which is popular?
Junior DA
"Canvoke and sanguine" and Trugal upropos" und 'immulable before decorumm are popular they have the
most reservations,
Frugal apropos has the highest average based reservations.
3. Most of moms either has traditional or modern decoration.
4. Base price is usually associate with the max number of occupancy.
5. Reservation checkin and chockout data are not in standard format, it's unclear which year it's referring to.
Senior DA
CAS is the most popular if considering reservations, with reservations.
It only wins by reservations compared with the room with the second most reservations. FNA.
The minimum reserved room is AOB, with 51 records.
4. Overall, the gup between the rooms is not hugely different.
5. Besides reservations, we could also measure the "popularity" by the number of guests.
GPT-4
Prouuer by Booms
Foom
The most popular room is "Convoke and sanguine" with 69 visitors, indicating that it may have the most appealing features or activities for guests.
The least popular room 'Abscond bolster" with visitors, suggesting may not as attractive or engaging as other
("Conveke and sanguine." "Frugal apropos," "Immutable before decorum' have relatively close number visitors, ranging from 63 indicating that they may share similar characteristics that make them popular among guests.
The top
4. The bottom three rooms ('Mendicant with cryptic." "Riddle to exculpate." and "Abscond or bolster") have more significant difference in visitor numbers, ranging from 51 to 56, suggesting that there may more variation their appeal guests. Overall, there is relatively even distribation of visitors acrows the moms, with only 18-visitor diffeerence between the most and least popular rooms. This could indicate that guests have diverse preferences and interests, or that the rooms offer variety of experiences that cater to different tastes.
Table 13: Comparison of GPT-4, a senior and junior data analyst on the fourth more practical questions.

---

Question client's budget and docsn't like Zinfandel, which wines shall recommend him?
Junior DA
1. We should recommend wine Papas Block him this wine is within his budget and not made of Zindandel and has highest
Score.
2. The mean score are roughly the same across all different type of grapes.
All wines are from California.
Most wines have "now" as value drink hut there are also some other years appear in this column.
Most wines are ussociated with year 2007 which is suspected to be the made year, followed by year 2008 and 2009.
Senior DA
1. The metric we choose to measure ROI. It's calculated as Soore/Price. We interpret it as the score value gained for every dollar. Further, we set the minimum score required to be 90.
Based on this metric, we would recommend Rose Wine Creek Ranch.
Rose Wine Creek Ranch's 's score not high (only 90), however, only dollars. This super compared rest. for dollar.
would
Wine Creek Ranch, provides
GPT-4
(ocluing Zeresaal
The majority of the wines listed within the client's budget providing wide range of options for recommendations. Sauvignon Blanc. Pinot Noir, and Chardonnay the most frequently listed wine varieties. suggesting that these popular choices and could be suitable rcommnnndttins for the clicat.
The price range for the wines listed is quite broad, from as low to high 60, indicating that there are options available for various preferences and budgets.
4. The client specifically meationed not liking Zinfandel, and none of the wines listed are Zinfandel, cnsuring that all options provided are suitable for the client's 's taste preferences.
Some wines have multiple price points, such as Sauvignon Blanc and Pinot Noir. which could indicate different vintages. vineyards, or quality levels, providing further varicty for client to choose from.
Table 14: Comparison of GPT-4, senior and a junior data analyst on the fifth more practical questions.

---

Question
Combining the data of the phone market in yeurs and the database, which phone popular?
Figure
Seearty
Analysis
The iPhone the popular phone among given options, total stock of 4.324 units.
Without
The iPhone 7 has the lowest popularity, with only 874 units stock, indicating lower demand compared other models.
Online
The iPhone and iPhone X have moderate popularity, with 2.914 and 2,540 units in stock, respectively.
Information
4. The data suggests that older iPhone models (Ss and 65) still have significant preseace in the market, possibly due their affordability and continued support from Apple.
The lower stock of iPhone 7 could be attributed to its position between the older. more affordable models and the newer, more advanced iPhone X. making it less appealing to consumers.
Queried Online
1. This graph shows the market share of mobile operating systems worldwide from May 2022 May 2023. Apple has 31,12%, Xiaomi bas 23.42% and Samsung has
Information
Android maintained its position as the leading mobile operating system worldwide in the first quarter of 2023 with market share 77.4 1.
Later. big data specificalll call detail record data collected from millions cell phones was hyped as useful stopping the
disease by tracking
This reseurch was commissioned develop more depth understanding smarphhone user purchasing behaviour UK smartphone market... ... Where does each iPhone rank
peers? Here's reckon they all compare. from Welcome the interactive ENISA Smartphone Guidelines Tool, To information about the domains available click the names below:.
latest model.
Analysis With
The iPhone is the popular among given options, with total stock of 4,324 units.
The iPhome cornes second place with total slock of 2,914 units, indicating that older iPhone models still have significant
Online
presence the market.
Information
The iPhone X. a more recent model, has lower total stock of 2,540 units, suggesting that it not be popular as the older
ihone 6s and 5x models.
4. The iPhone 7 bas the lowest total stock of 874 units, indicating that the least popular among the given options.
Despite the popularity of iPhones in this data, important to note that Android maintains larger market share worldwide
(71,4%), 3, which may impact the overall popularity of iPhones comparison Android devices.
Table Comparison of the GPT-4 analysis results with and without online information.


Reviews:
[{"review_id": "MKPpcWlxjB", "reviewer": "Reviewer_AeQb", "summary": "", "questions": "1. Can you conduct more experiments with more human evaluators.\n\n2. Can you provide some result of a wider range of large language models.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and comments.\n\n**W1 and Q1:** As mentioned in the limitation section, we can only do the evaluation on 1000 samples due to the budget limit. Since the evaluation requires some data analysis skills, the evaluator selection was rigorous where they had to pass several rounds of test annotations. We will try to conduct more experiments with more evaluators in future work. \n\n**W2 and Q2:** As the paper title infers, we aim to investigate whether GPT-4 is a good data analyst. GPT-4, being acknowledged as the most advanced model currently available in the market, prompts us to examine the extent of its proficiency. During our preliminary experiments, we experimented with several other commonly used LLMs, such as Claude, but their performance could have been more satisfactory. We will leave the comparison with other LLMs for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A sound new research direction to show the ability of GPT-4 to conduct data analyze, which may have further impact to reduce the working load for data analytics.\n\n2. Extensive evaluations with 1000 questions and many case studies.\n\n3. Good written and easy to follow.", "reasons_to_reject": "1. Limited evaluation with only 6 human evaluators.\n\n2. Only the result for GPT-4 is available, and there is no result for other language models.", "questions_to_authors": "1. Can you conduct more experiments with more human evaluators.\n\n2. Can you provide some result of a wider range of large language models.", "paper_topic_and_main_contributions": "This paper studies an interesting topic about whether GPT can be a good data analyst.  In detail, the authors regard GPT4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains and propose a framework to do data analyze with GPT. Evaluations by human metrics show the effectiveness of the frametwork and demonstrate GPT-4's ability.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None at the moment", "Typos_Grammar_Style_and_Presentation_Improvements": "None at the moment", "Justification_for_Ethical_Concerns": "None at the moment"}, "extra_text": {}}, {"review_id": "bN3G62S9fk", "reviewer": "Reviewer_pggJ", "summary": "", "questions": "- Subjective feedback in human-subject experiments is typically elicited on a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) of 5.  What is the motivation behind using a 4-level (0-3) response scale? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and thoughtful comments.\n\n**W1: data contamination**. First, according to the GPT-4 Technical Report (Page 10), “GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its pre-training data cuts off in September 2021”; while the arXiv version of the nvBench paper was submitted on 24 Dec 2021. Consequently, it is unlikely that data contamination issues would arise due to the chronological order of information. Additionally, in Section 4.5, we created practical questions that were intentionally excluded from GPT-4's exposure. GPT-4 still demonstrates proficiency in answering those questions. Lastly, it is important to note that data contamination remains an open question and a separate research area. In our work, we did not prioritize investigating this aspect.\n\n**W2: using external knowledge**. GPT-4 is designed to encompass public web knowledge, but it does not have access to real-time information, especially after September 2021. Furthermore, its primary focus is often centered on the given data itself, rather than explicitly integrating online information. In the revised version, we will provide more details on how external knowledge is incorporated to address these concerns.\n\n**W3: integer-valued alignment and fluency scores**. Indeed, GPT-4 performs well on these two metrics, achieving full marks. We will add the std and p-values in the revised version.\n\n**W4: practical cases for data analysts**. As stated in Section 4.5, we worked toward this to evaluate whether GPT-4 has good problem formulation ability. However, it is very challenging to comprehensively conduct such an evaluation as explained in the limitation section. In this paper, our primary focus lies on the initial step, specifically when dealing with specific questions. The exploration of more practical scenarios falls within the scope of our ongoing work, which we consider as future research directions.\n\n**Q1: score scales**. In an effort to mitigate the subjectivity of the evaluation process, we have taken a deliberate approach by meticulously designing specific standards for each score of every metric, as outlined in Appendix B. Instead of adhering to the conventional Likert scale with five levels, our intention is to ensure clarity and precision. Through our careful analysis, we determined that utilizing a 4-level response scale adequately captures the necessary nuances for certain metrics, making it sufficient for our purposes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Addresses an intriguing and high-impact question and offers nuanced discussion. \n\n- Provides substantial initial analysis and obtains results on which future work can be built. \n\n- Acknowledges the importance of open-ended (exploratory) analysis. \n", "reasons_to_reject": "- GPT-4  is trained on public Web data and more. So, it is likely already trained on the NvBench benchmark dataset. It needs to be clarified how the data contamination is handled here. \n\n- An essential suggestion by the work is to use external knowledge to mimic a data analyst’s domain understanding and experience. However, it is unclear how Google search provides that beyond the recency of information. GPT-4 is already supposed to encapsulate public web knowledge. Also, details on how Google search is used and how search results are filtered to feed them back to GPT-4 are missing.  \n\n- Tables 3 and 4 show integer-valued alignment and fluency scores. Given the large number of crowd ratings, they look infeasible. Showing the standard deviation values and analyzing the statistical significance of differences (for Table 4) would be helpful. \n\n- In practice, data analysts spend most of their time getting data ready for analysis (data preparation). Then, they perform exploratory and confirmatory analyses, determining what questions to ask and finding answers, if possible (data rarely tells clear answers). This process is noisy and iterative, more like a multi-turn conversation than answering one independent question at a time. In many cases, data analysts don’t know which questions to ask.  \n", "questions_to_authors": "- Subjective feedback in human-subject experiments is typically elicited on a [Likert scale](https://en.wikipedia.org/wiki/Likert_scale) of 5.  What is the motivation behind using a 4-level (0-3) response scale? ", "paper_topic_and_main_contributions": "The paper evaluates the performance of GPT-4 for data analysis using  questions randomly selected from the NvBench benchmark and compares it with data analysts at different experience levels, all with the help of crowd annotators. Results show GPT-4 can perform better than or on par with data analysts.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VXStikSLpL", "reviewer": "Reviewer_jqtG", "summary": "", "questions": "How was the correctness determined for practical questions?\n\nGPT-4 was also allowed to search google to gain domain specific knowledge. What kind of search queries were actually made? Are they similar to what a human would make?\n\nDid you find the results to be sensitive to prompt?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the positive feedback and insightful questions.\n\nRegarding your suggestion on **the open-ended question**, this is a meaningful direction. As stated in Section 4.5 and Appendix E, we worked toward this to evaluate whether GPT-4 has good problem formulation ability. Tables 10-14 show the performance. The practical questions here are actually open-ended and do not have definite answers. \n\n**Q1:** We do not have correctness scores for practical questions since they do not have definite answers. In Appendix E Lines 911-918, we mentioned we evaluated the performance based on comparisons among a junior DA, a senior DA, and GPT-4.\n\n**Q2:** In order to guarantee that the retrieved information is highly relevant to the questions, we input both the original question and the processed data as search queries. This strategy resembles human behavior, as humans also tend to use data alongside their questions to achieve more accurate results when performing searches.\n\n**Q3:** The results may differ given different prompts, but overall are not that sensitive to prompts. As highlighted in CoT (Wei et al., 2022) Section 3.4, prompt wording (different terminologies) has little impact on results, as long as the structure remains similar. Furthermore, due to cost concerns, we do not conduct quantitative evaluations on different prompting methods.\n\nWe will **make the dataset available**, as stated in Lines 28-29.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Deals with an interesting problem. Nicely written and easy to follow.\n\nExhaustive experiments were performed to justify the findings.\n\n", "reasons_to_reject": "The paper does not report what happens when the question is open-ended. For example, a data analyst is often given a data without a specific question. The task is to identify patterns/correlations in the dataset. Will GPT-4 be able to deal with that? This is not a reason to reject per se.\n\nThe dataset could be made available.", "questions_to_authors": "How was the correctness determined for practical questions?\n\nGPT-4 was also allowed to search google to gain domain specific knowledge. What kind of search queries were actually made? Are they similar to what a human would make?\n\nDid you find the results to be sensitive to prompt?", "paper_topic_and_main_contributions": "The paper investigates the ability of GPT-4 to be a data analyst. The model is provided with a question and a database and is tasked to write a python code to obtain the relevant information, execute the code and finally analyze the obtained results to generate insights along with visualization. The performance of GPT-4 is then compared with data analysts at intern, junior and senior levels with GPT-4 outperforming the intern and junior data analyst across several use cases.  ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dtmdxFSFLO", "reviewer": "Reviewer_QeUo", "summary": "", "questions": "Q1. How do you eliminate the factor of subjectivity in the proposed evaluation metrics?\nQ2. Did you consider different difficulty levels in selecting the queries?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and thoughtful review.\n\n**W1:** We designed an end-to-end new framework for the task. Details can be referred to Section 3.2. Our designed framework can outperform junior data analysts and achieve comparable performance to a senior data analyst. On the other hand, the works (e.g. https://aclanthology.org/2023.acl-long.626.pdf accepted at ACL 2023, https://aclanthology.org/2021.acl-long.172.pdf accepted at ACL 2021) presenting interesting analysis without any new solution should not be regarded as flawed. In contrast, they offer valuable insights and analysis from an interesting perspective. These findings contribute to the readers' understanding of a particular model or domain and can also make meaningful contributions to the NLP community.\n\n**W2 & Q1:** In an effort to mitigate the subjectivity of the evaluation process, we have taken a deliberate approach by meticulously designing specific standards for each score of every metric, as outlined in Appendix B. We acknowledge that defining these metrics can be challenging, which is why we have opted for a more limited scale. Using a common scale of 0-5 could introduce greater subjectivity. Metrics such as fluency are commonly employed in tasks like summarization and generation, and their inclusion is essential for evaluating the quality of generated outputs.\n\n**W3:** Thanks for your suggestion on more complex and surprising patterns/insights. Based on our experiments, we have observed that GPT-4 can sometimes analyze correlations, distributions, and outliers within the given data. We will provide illustrative examples in the final version. However, it is important to note that many of the questions in the nvBench dataset are relatively straightforward, where complex insights may not be necessary. Additionally, nvBench is currently the only available dataset that aligns with the steps involved in the data analysis process. In this work, our primary goal is to validate GPT-4's capability to automate the data analysis process, rather than solely focusing on generating surprising insights.\n\n**Q2:** The nvBench paper classified each question into different difficulty levels. Among our 1000 samples, 199 are easy, 392 are medium, 286 are hard, and 123 are extra hard. The scores are shown below:\n|             |             | easy + medium | hard + extra hard |\n|-------------|-------------|:--------------:|:------------------:|\n| Figure      | Correctness |     0.79      |       0.75       |\n|             | Chart Type  |     0.99      |       1.00       |\n|             | Aesthetics  |     2.52      |       2.46       |\n| Data analysis | Correctness |     0.95      |       0.93       |\n|             | Complexity  |     2.32      |       2.25       |\n|             | Alignment   |       1.00       |       1.00       |\n|             | Fluency     |       3.00       |       3.00       |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "S1. The paper is about a recent topic which has high potential to attrack audience.\nS2. A detailed experiment setup  and evaluation is presented. ", "reasons_to_reject": "W1. While the paper presents an interesting analysis, it does not include a contribution towards a new solution.\nW2. The proposed evalution metrics are prone to subjective assesment (particularly Aesthetics, Complexity and Fluency). It is not very straightforward how correctness is measured either.\nW3. There is no analysis as to extraction of more complex or surprising patterns/insights from the data. ", "questions_to_authors": "Q1. How do you eliminate the factor of subjectivity in the proposed evaluation metrics?\nQ2. Did you consider different difficulty levels in selecting the queries?", "paper_topic_and_main_contributions": "The authors present an analysis as to how well GPT4 can perform data analysis. They break down the data analysis into three sub tasks of data collection through SQL, data vşusalization and providing insights on the collected data. The experiments are conducted on 1000 queries in natural language from NvBench data set and the results are evaluated by and compared against data analysts. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies an interesting topic about whether GPT can be a good data analyst. The paper is well-motivated and easy to follow, the experiments are solid, and it has the potential real impact to significantly reduce the workload for data analytics. However, reviewers also point out the paper could add more in-depth analysis, the data and label generated are subjective, the evaluation results have some issues.
Decision: Accept-Findings