Large Language Models for Predictive Analysis: How Far Are They?
Authors: Qin Chen, Yuanyi Ren, Xiaojun Ma, Yuyang Shi
Accepted at: acl25

Large Language Models for Predictive Analysis: How Far Are They?

Qin Chen* 1, Yuanyi Ren* 1, Xiaojun Ma† 1, Yuyang Shi2,
1Peking University, 2Harvard University,
{chenqink, yyren, mxj}@pku.edu.cn , yuyangshi@fas.harvard.edu

Abstract
Predictive analysis is a cornerstone of modern decision-making, with applications in various domains. Large Language Models (LLMs) have emerged as powerful tools in enabling nuanced, knowledge-intensive conversations, thus aiding in complex decision-making tasks. With the burgeoning expectation to harness LLMs for predictive analysis, there is an urgent need to systematically assess their capability in this domain. However, there is a lack of relevant evaluations in existing studies. To bridge this gap, we introduce the PredictiQ benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. We design an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis. Generally, we believe that existing LLMs still face considerable challenges in conducting predictive analysis. See Github.

1 Introduction
Predictive analysis (Kumar and Garg, 2017) involves making predictions about future outcomes based on past data, using statistical modeling, data mining, and machine learning techniques. It is widely used in decision-making across various fields like business (Che et al., 2024) and healthcare (Dixon et al., 2024).

To facilitate off-the-shelf predictive analysis for users without data analysis expertise, Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Anil et al., 2023; Abdin et al., 2024) serve as powerful tools, supporting users with coherent and contextually relevant responses. Figure 1 shows how users engage with LLMs by submitting predictive queries based on specific datasets.

Given the substantial potential of LLM-based predictive analysis, a comprehensive evaluation of leading LLMs is essential. While no studies specifically assess LLM-based predictive analysis, related studies (Zhao et al., 2023b; Chen, 2022; Saeed et al., 2023; Gao et al., 2023; He et al., 2023; Abolghasemi et al., 2024; Pratt et al., 2024; Hong et al., 2024) in relevant fields primarily focus on evaluating either the model’s outcomes only (e.g., a total sales figure answering a database query) or the generated codes that, when executed, produce these outcomes. However, the former raises scalability issues as LLMs may struggle to process the entire lengthy dataset due to limited context length. The latter lacks textual explanations, such as justifications for algorithm selection, which constrains its applicability and undermines user trust. This makes existing evaluation efforts insufficient when directly adapted to predictive analysis, where tasks are inherently more complex.

LLM-based predictive analysis typically requires handling tasks such as data preprocessing, algorithm selection, result interpretation, and so on. These tasks require both textual explanations and code implementations. In practical applications, textual analysis is crucial for explaining algorithm selection and enhancing reliability. Moreover, effective alignment between text and code improves user comprehension of LLM-generated solutions. Thus, textual analysis, code generation, and text-code alignment are integral to the functionality of AI-driven assistants.

In this paper, we propose the PredictiQ (Q stands for query) benchmark to evaluate the potential of current LLMs in predictive analysis. We first collect datasets covering eight common fields, including economics, traffic, and more. Diverse datasets form the basis for PredictiQ to assess LLMs’ performance in predictive analysis. Based on the collected datasets, we collaborate with data analysis experts to formulate predictive queries tailored to
Figure 1: An example of users conducting predictive analysis via Large Language Models.

the datasets. Each query is limited to a single, self-contained question that relies solely on the dataset, without requiring external knowledge. We then formulate the data-specific queries and corresponding datasets into prompts, which are provided to LLMs to generate responses. We propose an evaluation protocol encompassing three domains: textual analysis, code generation, and text-code alignment, focusing on seven key aspects. The responses from the LLMs are then evaluated by (i) data analysis experts and (ii) LLMs. The LLM whose responses align most closely with those of the human experts is selected as the primary evaluator. In this study, we adopt GPT4Turbo, which achieves an average alignment ratio of 90.5% with human experts. PredictiQ involves 1130 queries from 44 datasets across 8 fields, requiring 300 human hours for query generation and examination. Our evaluation on PredictiQ costs 900 human hours for response evaluation, 72.18 million input tokens, and 20.4 million output tokens.

We conduct extensive experiments on PredictiQ with twelve renowned LLMs. We find that (i) Fine-tuning LLMs on code enhances both text analysis and code generation, occasionally pushing the model beyond its parameter limits. (ii) Code generation and text analysis are interconnected processes that shape LLMs’ overall predictive analysis proficiency. (iii) Several LLMs’ predictive analysis abilities vary significantly across different fields, with wide margins, and exhibit diversified score distributions. We believe existing LLMs are far from mastering predictive analysis, both in terms of performance and efficiency. Our contributions are summarized as follows.

• We propose the PredictiQ benchmark—a comprehensive evaluation framework for LLMs in predictive analysis. It covers the entire analytic process by integrating 44 datasets across 8 real-world fields, 1,130 rigorously curated data-specific queries, and evaluation protocols. This approach goes beyond existing benchmarks that focus only on basic descriptive tasks.

• We systematically evaluate the performance of twelve LLMs on the PredictiQ benchmark, highlighting insights that could improve the efficacy of LLMs in predictive analysis.

2 Preliminary

Predictive Analysis refers to the use of statistical algorithms and machine learning techniques to analyze data patterns, predict future events, trends, or behaviors (Siegel, 2013; Kumar and Garg, 2018). It is widely used in various fields, including finance, marketing, healthcare, and risk management, to anticipate potential risks, identify opportunities, and inform the decision-making process. See Figure 1 for an illustration.

Problem Definition. The predictive analysis process with LLMs is formalized as follows:

\[
(\text{query}, \text{data}) \xrightarrow{\text{LLM}} (\text{text}, \text{code}).
\]

The input consists of a predictive analysis query associated with corresponding data. A query is a data-specific predictive question that requests future predictions, identifies implicit data patterns, and so on. The data consists of multiple columns and rows. The output includes text, which explains how the prediction is made and justifies the chosen method, and code, which implements the analysis. Ideally, the text should provide clear and in-depth analysis based on the query and data. The code should provide related and functionally correct implementation that solves the problem raised by the query.
3 The PredictiQ Benchmark

This section outlines the details of the PredictiQ benchmark, covering (i) data collection, (ii) query formulation, and (iii) response evaluation.

3.1 Data Collection

The PredictiQ benchmark evaluates LLMs’ predictive analysis capabilities across diverse fields. It includes various publicly available datasets from real-world scenarios, ensuring the benchmark’s diversity and validity. The datasets are organized into eight fields that are prevalent in real-world predictive analysis applications, as presented in Table 1. See Appendix A for detailed descriptions.

3.2 Query Collection

<table>
  <tr>
    <th>Field</th>
    <th>#Datasets</th>
    <th>#Queries</th>
  </tr>
  <tr>
    <td>Economics</td>
    <td>12</td>
    <td>270</td>
  </tr>
  <tr>
    <td>Marketing and Sales</td>
    <td>6</td>
    <td>200</td>
  </tr>
  <tr>
    <td>Industry Analysis</td>
    <td>7</td>
    <td>180</td>
  </tr>
  <tr>
    <td>Traffic</td>
    <td>5</td>
    <td>130</td>
  </tr>
  <tr>
    <td>Healthcare</td>
    <td>4</td>
    <td>130</td>
  </tr>
  <tr>
    <td>Social Study</td>
    <td>4</td>
    <td>110</td>
  </tr>
  <tr>
    <td>Human Resource</td>
    <td>3</td>
    <td>80</td>
  </tr>
  <tr>
    <td>Education</td>
    <td>3</td>
    <td>70</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>44</th>
    <th>1130</th>
  </tr>
</table>

In this subsection, our goal is to formulate predictive questions tailored to each dataset. To achieve this, we engaged data science experts to develop well-defined queries following our instructions and examples. The provided instructions clarify the context of predictive analysis and outline the constraints on expected outcomes, as detailed below.

Context of Predictive Analysis. Based on our research into the application of predictive analysis across various fields (Wazurkar et al., 2017; Mishra et al., 2023; Saranya and Asha, 2019; Liu et al., 2023), we categorize common questioning methods into two types: those focused on predicting future trends and those focused on identifying previously unrecognized patterns within the dataset. We also encourage participants to consider queries related to time-series data.

Constraints on Expected Outputs. To ensure the quality of the outputs, we have implemented three specific instructions: (i) Participants are required to articulate a clear predictive target using unambiguous language. (ii) Queries should be based solely on the information within the dataset, excluding any reference to external data that could compromise the evaluation’s integrity. (iii) Each query should be restricted to a single question, ensuring a more balanced distribution of workload across all queries. In addition, to improve real-world relevance and adaptability, participants are provided with background information on the sources and construction methods of the dataset.

To ensure diversity, we engage experts to select unique questions that cover distinct aspects of the dataset. The number of queries for each dataset is then tailored to its informativeness. For further details, see Appendix A. In total, we have generated 1130 queries from 44 datasets, covering common application scenarios in predictive analysis. The process of generating and examining the queries requires roughly 300 human hours.

3.3 Response Generation

With the datasets and collected queries, we organize the data to formulate the input prompts. Each prompt consists of three parts:

• Query and Instruction: We instruct the LLM to perform predictive analysis, assuming the role of a professional data scientist.

• Data Summary: We provide a list of all columns in the dataset with their respective data types (e.g., int). We also include details such as the maximum and minimum values for numerical columns and the total number of categories for categorical columns. These summaries help the LLM better understand the dataset.

• Data Details: The dataset is provided in a CSV format, with columns separated by commas and rows by line breaks. This detailed format gives the LLM the essential information needed for predictive analysis.

The prompts are then submitted to the LLMs to generate responses. An example of a prompt and its corresponding response is provided in Appendix E.

3.4 Evaluation Protocols

As mentioned in section 1, LLM-based predictive analysis involves both code and textual descriptions. It is essential to evaluate all three components: code generation, textual explanations, and
the alignment between the two. Textual analysis offers key insights that enhance reliability, such as justifying algorithm choices, while code implementation operationalizes the analysis. Proper alignment helps users better understand the solutions generated by LLMs. Building on this, we propose an evaluation protocol spanning three domains and seven aspects, each rated on a scale from 0 to 4. A summary is provided below, with full details available in Table 9.

Text Analysis. The text generated by LLMs is evaluated on two aspects: Relevance, which assesses how closely the analysis aligns with the given data and the specific question, and Depth, which evaluates the comprehensiveness of the justification for selecting a particular model or algorithm. We expect the textual analysis to provide a detailed and thorough examination of the analysis process, tailored to the specific data and queries.

Code Generation. This domain focuses on the quality of the code produced by LLMs with two aspects. Firstly, Usefulness evaluates how well the code snippet addresses the given problem. Secondly, Functional Correctness examines the code’s execution correctness. We expect the generated code to accurately and effectively implement the associated predictive analysis.

Text-Code Alignment. We further evaluate the congruence between the generated code and textual analysis across three aspects: (i) Descriptive Accuracy, which assesses how precisely the text reflects the code; (ii) Coverage, evaluating whether the generated text addresses all relevant aspects of the code, including its functions and nuances; and (iii) Clarity, which examines the clarity of the alignment between the code and text. We expect the generated text and code to be well-aligned, and comprehensive, enhancing the user’s understanding of the entire analysis process.

The proposed protocol offers a comprehensive evaluation of the responses. Advanced LLMs enable large-scale, stable, and effective assessment of the responses. To investigate how different LLM evaluators agree with experts’ ratings and choose the appropriate evaluator, we employ (i) human experts and (ii) LLMs to assess the responses according to this protocol. We then compare the score distributions from both groups to determine which LLM aligns most closely with human experts. Among the LLMs tested — GPT4Turbo, GPT4O, and Phi3Medium — GPT4Turbo demonstrated the highest alignment with human evaluations, as shown in our experiment in Figure 4. Consequently, we adopt GPT4Turbo as the primary evaluator. For assessing Functional Correctness, we manually execute the generated code and evaluate the ratio of successful executions without errors.

4 Evaluation Results on PredictiQ

4.1 Involved LLMs
We evaluate eight popular LLMs on PredictiQ. (i) GPT Family. GPT3.5Turbo, GPT4Turbo, GPT4O, GPT4O1, and GPT4O3Mini (Brown et al., 2020; OpenAI, 2023, 2024) are potent large-scale language models fine-tuned for both chat and code generation. (ii) Llama Family. This family includes CodeLlama2-7B, ChatLlama2-7B, ChatLlama2-13B, and ChatLlama2-70B (Touvron et al., 2023). CodeLlama2-7B is fine-tuned for both chat and code generation. Other versions are reported to be fine-tuned for chat. (iii) Other Models. We also incorporate other popular LLMs, including Phi3Medium (Microsoft, 2024a), Phi4 (Microsoft, 2024b) and CohereRPlus1, into our evaluation. For LLM settings, please refer to Appendix C.

4.2 Evaluation on PredictiQ
GPT4O3Mini outperforms all others, achieving the highest total score of 24.11 out of 28. Within the GPT family, GPT4O1 incurs significantly higher token costs, up to twice those of GPT4O. GPT4O3Mini, a refined version, improves both efficiency and performance compared to GPT4O1. Models from the Llama family (from 7B to 70B), whether fine-tuned for code generation or not, exhibit relatively low scores. Generally, larger parameter scale enhances overall performance but also increases token usage.

Finding 1: LLMs often fail to produce complete and executable solutions, frequently overlooking steps like data pre-processing. While larger models handle import errors better, they still struggle with logic errors.

LLMs often overlook essential procedures like data pre-processing (e.g., handling missing values) and filtering in the generated code, as revealed in Table 3. For GPT4O3Mini, only 49% of the generated code includes proper pre-processing and filtering, while the rest operate on unclean data. This

1https://docs.cohere.com/v2/docs/command-r-plus
Table 2: Evaluation results on PredictIQ across seven perspectives (scored 0–4) outlined in Table 9. Functional Correctness represents the ratio of executable code, mapped to a 0-4 scoring scale. We also report average token costs, with additional reasoning costs for GPT4O1 and GPT4O3Mini.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="2">Text</th>
    <th colspan="2">Code</th>
    <th colspan="2">Text-Code Alignment</th>
    <th rowspan="2">Total</th>
    <th rowspan="2">Token Cost</th>
  </tr>
  <tr>
    <th>Relevance</th>
    <th>Depth</th>
    <th>Usefulness</th>
    <th>Functional Correctness</th>
    <th>Descriptive Accuracy</th>
    <th>Coverage</th>
    <th>Clarity</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>3.00<sub>±0.96</sub></td>
    <td>1.76<sub>±0.89</sub></td>
    <td>2.40<sub>±0.86</sub></td>
    <td>2.12 (53%)</td>
    <td>2.66<sub>±1.01</sub></td>
    <td>2.47<sub>±0.94</sub></td>
    <td>2.80<sub>±0.96</sub></td>
    <td>17.21<sub>±5.19</sub></td>
    <td>1934.34</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>3.39<sub>±0.79</sub></td>
    <td>2.18<sub>±0.64</sub></td>
    <td>2.78<sub>±0.72</sub></td>
    <td>3.12 (78%)</td>
    <td>3.09<sub>±0.80</sub></td>
    <td>2.95<sub>±0.77</sub></td>
    <td>3.18<sub>±0.77</sub></td>
    <td>20.68<sub>±4.23</sub></td>
    <td>2072.95</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>3.60<sub>±0.65</sub></td>
    <td>2.39<sub>±0.71</sub></td>
    <td>3.12<sub>±0.67</sub></td>
    <td>3.24 (81%)</td>
    <td>3.36<sub>±0.68</sub></td>
    <td>3.31<sub>±0.67</sub></td>
    <td>3.41<sub>±0.64</sub></td>
    <td>22.43<sub>±3.82</sub></td>
    <td>3390.74</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>3.61<sub>±0.51</sub></td>
    <td>2.80<sub>±0.53</sub></td>
    <td>3.45<sub>±0.60</sub></td>
    <td>3.40 (85%)</td>
    <td>3.47<sub>±0.68</sub></td>
    <td>3.48<sub>±0.67</sub></td>
    <td>3.48<sub>±0.67</sub></td>
    <td>23.70<sub>±3.40</sub></td>
    <td>6534.45</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>3.63<sub>±0.53</sub></td>
    <td>2.91<sub>±0.42</sub></td>
    <td>3.53<sub>±0.55</sub></td>
    <td>3.48 (87%)</td>
    <td>3.52<sub>±0.61</sub></td>
    <td>3.52<sub>±0.61</sub></td>
    <td>3.52<sub>±0.61</sub></td>
    <td>24.11<sub>±3.13</sub></td>
    <td>4402.94</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>2.01<sub>±0.87</sub></td>
    <td>1.31<sub>±0.65</sub></td>
    <td>1.49<sub>±0.71</sub></td>
    <td>0.72 (18%)</td>
    <td>0.83<sub>±0.82</sub></td>
    <td>0.85<sub>±0.79</sub></td>
    <td>1.14<sub>±0.94</sub></td>
    <td>8.34<sub>±4.50</sub></td>
    <td>1763.87</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>2.04<sub>±0.91</sub></td>
    <td>1.34<sub>±0.67</sub></td>
    <td>1.64<sub>±0.71</sub></td>
    <td>0.60 (15%)</td>
    <td>0.99<sub>±0.90</sub></td>
    <td>1.00<sub>±0.89</sub></td>
    <td>1.22<sub>±1.00</sub></td>
    <td>8.83<sub>±4.91</sub></td>
    <td>1786.59</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>1.97<sub>±0.88</sub></td>
    <td>1.24<sub>±0.64</sub></td>
    <td>1.53<sub>±0.69</sub></td>
    <td>0.72 (18%)</td>
    <td>1.02<sub>±0.84</sub></td>
    <td>1.03<sub>±0.79</sub></td>
    <td>1.24<sub>±0.89</sub></td>
    <td>8.75<sub>±4.49</sub></td>
    <td>2032.84</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>2.32<sub>±0.90</sub></td>
    <td>1.51<sub>±0.67</sub></td>
    <td>1.78<sub>±0.73</sub></td>
    <td>0.84 (21%)</td>
    <td>1.25<sub>±0.92</sub></td>
    <td>1.27<sub>±0.90</sub></td>
    <td>1.60<sub>±0.96</sub></td>
    <td>10.57<sub>±4.85</sub></td>
    <td>2487.18</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>2.90<sub>±1.25</sub></td>
    <td>1.74<sub>±1.01</sub></td>
    <td>2.33<sub>±1.04</sub></td>
    <td>1.64 (41%)</td>
    <td>2.45<sub>±1.21</sub></td>
    <td>2.33<sub>±1.14</sub></td>
    <td>2.58<sub>±1.18</sub></td>
    <td>15.97<sub>±5.50</sub></td>
    <td>3314.55</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>2.94<sub>±0.24</sub></td>
    <td>2.55<sub>±0.58</sub></td>
    <td>2.87<sub>±0.55</sub></td>
    <td>2.16 (54%)</td>
    <td>2.84<sub>±0.43</sub></td>
    <td>2.82<sub>±0.45</sub></td>
    <td>2.88<sub>±0.39</sub></td>
    <td>19.06<sub>±2.90</sub></td>
    <td>3815.88</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>2.89<sub>±0.96</sub></td>
    <td>1.70<sub>±0.64</sub></td>
    <td>2.38<sub>±0.79</sub></td>
    <td>1.68 (42%)</td>
    <td>2.50<sub>±0.92</sub></td>
    <td>2.42<sub>±0.86</sub></td>
    <td>2.62<sub>±0.87</sub></td>
    <td>16.20<sub>±4.86</sub></td>
    <td>2851.20</td>
  </tr>
</table>

issue is even more pronounced in smaller models. As shown in Table 3, we observe the following: (i) Smaller models frequently face import errors, such as using non-imported functions, which decrease as model capacity grows. (ii) Logic errors, including syntax issues and function call errors, persist across models, indicating the need for future improvements. (iii) Fine-tuning on code helps CodeLlama2-7B reduce import errors but increases logic errors, resulting in a lower executable code rate than ChatLlama2-7B.

Table 3: Analysis on the portion of the codes without pre-processing, and the frequencies of error types.

<table>
  <tr>
    <th>LLM</th>
    <th>No Pre-processing</th>
    <th>Import Error</th>
    <th>Logic Error</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>71%</td>
    <td>3.8%</td>
    <td>43.2%</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>66%</td>
    <td>1.3%</td>
    <td>20.7%</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>66%</td>
    <td>0.4%</td>
    <td>18.6%</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>50%</td>
    <td>0.3%</td>
    <td>14.7%</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>51%</td>
    <td>0.3%</td>
    <td>12.7%</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>92%</td>
    <td>4.1%</td>
    <td>40.1%</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>89%</td>
    <td>38.2%</td>
    <td>46.8%</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>87%</td>
    <td>36.1%</td>
    <td>45.9%</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>87%</td>
    <td>15.0%</td>
    <td>64.0%</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>72%</td>
    <td>5.9%</td>
    <td>53.1%</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>58%</td>
    <td>3.8%</td>
    <td>42.2%</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>78%</td>
    <td>4.6%</td>
    <td>53.4%</td>
  </tr>
</table>

Finding 2: Fine-tuning on code generation improves the overall performance of models, sometimes allowing them to perform better than their parameter size would suggest.

With 46% fewer parameters, CodeLlama2-7B outperforms ChatLlama2-13B in total scores, aligning with the conclusion of (Zhou et al., 2023). Fine-tuning on code remarkably improves model performance in code relevance, code-text alignment, and the quality of textual description.

Finding 3: Fine-tuning for code generation increases the frequency and length of code produced by CodeLlama2-7B, but it negatively impacts the model’s ability to generate executable data analysis code.

To our surprise, CodeLlama2-7B achieves 15% of the executable code rate, falling behind even ChatLlama2-7B. We find that, at this parameter level, fine-tuning on code generation helps to boost the performance of other domains, including textual analysis and text-code alignment. However, in terms of executable code rate, its performance is downgraded. This may be attributed to the overly specialized fine-tuning. Also, the reason may be that fine-tuning on code does not always focus on the data-analysis-related codes. For example, there may be some C++ kernel code that is not related to data analysis.

We systematically analyze the lengths and lines of the generated codes in Table 4. Despite our prompt instructions to generate codes, we find that models from the Llama family often ignore code generation. It turns out that fine-tuning on code only helps CodeLlama2-7B generate code more frequently, with a tendency to generate longer code. However, for a complex task like data-aware predictive analysis, it won’t improve the quality of the generated code at this scale.
Table 4: Analysis of the average length and number of lines of the generated code, as well as the ratio of cases with no code.

<table>
  <tr>
    <th>LLM</th>
    <th>Length</th>
    <th>#Lines</th>
    <th>Absent</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>967.45</td>
    <td>27.22</td>
    <td>4%</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>1276.49</td>
    <td>32.46</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>2423.53</td>
    <td>65.79</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>4375.41</td>
    <td>127.37</td>
    <td>0</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>3561.61</td>
    <td>90.63</td>
    <td>0</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>660.72</td>
    <td>17.41</td>
    <td>64%</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>863.85</td>
    <td>21.64</td>
    <td>60%</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>627.51</td>
    <td>16.68</td>
    <td>63%</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>611.32</td>
    <td>16.54</td>
    <td>53%</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>1336.94</td>
    <td>30.79</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>1469.85</td>
    <td>33.85</td>
    <td>0</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>1749.12</td>
    <td>52.90</td>
    <td>0</td>
  </tr>
</table>

Finding 4: LLMs vary in score distribution across data source domains. GPT4O3Mini and GPT4O are well-balanced, while ChatLlama2-70B excels in the Education domain.

![Bar charts showing score distributions of LLMs on eight fields.](page_120_670_600_300.png)

Figure 2: Score distributions of LLMs on eight fields. For clarity we present the total scores of text, code, and their alignment.

We present the performance of leading models — GPT4O3Mini, GPT4O, Phi4, and ChatLlama2-70B — within their respective families across eight fields (see Figure 2). The data highlight the varying effectiveness of these LLMs across domains. GPT4O3Mini and GPT4O demonstrate balanced performance, with deviations of 1.67 and 1.82 points, respectively. In contrast, Phi4 and ChatLlama2-70B exhibit greater variability, with score differences of 4.38 and 5.16 points. The score distributions differ across fields. For instance, ChatLlama2-70B excels in *Education*, exceeding its average score by 31.4%, suggesting domain-specific strengths and weaknesses among models.

![Line chart showing analysis on impact of context length limit.](page_800_670_600_300.png)

Figure 3: Analysis on impact of context length limit.

4.3 Impact of Context Length

We evaluate the impact of context length on GPT4O, GPT4O1, and GPT4O3Mini in Figure 3. At 4k tokens, GPT4O performs well, and increasing the token limit does not improve its performance. However, GPT4O1 and GPT4O3Mini, which need extra tokens for reasoning, perform much worse initially\footnote{As stated in Appendix C, we set the context length to 32,768 for GPT4O1 and GPT4O3Mini to unlock their full potential, resulting in the outcomes in Table 2.}. As context length increases, their performance improves rapidly, with GPT4O3Mini achieving similar results to GPT4O1 using fewer tokens.

4.4 LLMs as Evaluators

Table 5: Results on PredictiQ with different evaluators.

<table>
  <tr>
    <th>Evaluator</th>
    <th>GPT4Turbo</th>
    <th>GPT4O</th>
    <th>Phi3Medium</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>17.21±4.82</td>
    <td>21.68±3.69</td>
    <td>25.49±1.29</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>20.34±3.92</td>
    <td>25.34±2.08</td>
    <td>26.45±1.42</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>22.43±3.72</td>
    <td>26.13±1.97</td>
    <td>26.86±1.93</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>23.70±3.40</td>
    <td>26.45±2.33</td>
    <td>26.96±1.22</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>24.15±3.13</td>
    <td>26.64±2.05</td>
    <td>27.04±1.50</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>8.34±4.50</td>
    <td>13.94±5.02</td>
    <td>22.93±4.56</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>8.83±4.91</td>
    <td>14.59±4.89</td>
    <td>22.62±4.51</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>8.75±4.49</td>
    <td>15.75±4.13</td>
    <td>23.67±3.34</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>10.57±4.85</td>
    <td>16.36±4.54</td>
    <td>23.54±4.13</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>15.97±5.49</td>
    <td>21.19±4.67</td>
    <td>25.23±1.43</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>19.06±2.26</td>
    <td>21.19±4.67</td>
    <td>25.23±1.43</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>16.20±4.83</td>
    <td>20.87±4.54</td>
    <td>25.07±1.28</td>
  </tr>
</table>

To investigate how different LLM evaluators align with human preferences and select the proper evaluator, we engage (i) five experts in data analysis and (ii) LLMs including GPT4Turbo, GPT4O, and Phi3Medium to grade the responses according to the evaluation protocol \footnote{We exclude GPT4O1 and GPT4O3Mini as evaluators due to their high computational cost and slow running speed.}. We then calculate
Figure 4: Alignment scores of different evaluators with human experts. See Figure 5 for full results.

the average scores from human experts and compare them with those from LLMs. Score pairs with an absolute difference smaller than 4 are labeled positive (1), while those with a larger difference are labeled negative (0). The alignment between LLM and expert scores is visualized in Figure 4, with higher scores indicating better alignment. Full results are presented in Figure 5.

Finding 5: GPT4Turbo’s evaluations align most closely with those of the experts, while GPT4O tends to be more positively biased. Phi3Medium is not applicable because it lacks differentiation in its evaluations.

As shown in Figure 4, GPT4Turbo aligns most closely with expert ratings, matching 90.5% of instances and maintaining a score distribution consistent with human evaluators across all target models. In contrast, GPT4O shows weaker alignment, particularly with the Llama family, where it frequently assigns disproportionately high scores to poor responses. Phi3Medium demonstrates the poorest alignment, assigning nearly indistinguishable scores across all targets. This may be attributed to the round number bias (Honda et al., 2022; Stureborg et al., 2024), where certain scores are assigned more frequently, irrespective of the context, impeding precise quality assessments. This could result from biased training data, sentiment-supportive fine-tuning, or limitations in reasoning abilities (Murugadoss et al., 2024; Wei et al., 2024; Wang et al., 2024).

Finding 6: The tendency of LLMs to assign themselves the highest scores when serving as evaluators is reduced in the context of predictive analysis.

Studies (Ye et al., 2024; Panickssery et al., 2024; Koo et al., 2024) show that when LLMs are used as evaluators, they often exhibit a bias towards assigning higher scores to their own responses. However, in our experimental setup, this bias appears to be mitigated. As demonstrated in Table 5, the gap between total scores (\( \text{Score}_{\text{GPT4O}} - \text{Score}_{\text{GPT4Turbo}} \)), when evaluated by GPT4Turbo, is 2.09. In contrast, the gap narrows to 0.79 when evaluated by GPT4O. Interestingly, when the evaluator is switched from GPT4Turbo to GPT4O, the performance gap narrows rather than widens, indicating that the "ego-centric" (Koo et al., 2024) or "self-preference" (Panickssery et al., 2024) scoring tendency is reduced in this evaluation.

4.5 Ablation Study

Table 6: Ablation study. For clarity, we present the total scores of three domains.

<table>
  <tr>
    <th></th>
    <th>Text</th>
    <th>Code</th>
    <th>Alignment</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>5.99±1.22</td>
    <td>6.36±1.07</td>
    <td>10.08±1.00</td>
    <td>22.43±3.82</td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.78±1.14</td>
    <td>6.22±0.97</td>
    <td>9.94±1.07</td>
    <td>21.94±3.88</td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>5.81±1.12</td>
    <td>6.34±0.88</td>
    <td>9.76±1.17</td>
    <td>21.91±3.97</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>6.54±1.78</td>
    <td>7.01±1.05</td>
    <td>10.56±1.83</td>
    <td>24.11±3.13</td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>6.56±1.86</td>
    <td>7.31±1.07</td>
    <td>10.35±1.90</td>
    <td>24.22±4.07</td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>6.25±1.56</td>
    <td>7.18±1.07</td>
    <td>10.22±1.76</td>
    <td>23.65±3.13</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>3.98±1.46</td>
    <td>2.66±1.78</td>
    <td>4.39±2.64</td>
    <td>11.03±4.86</td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>3.94±1.64</td>
    <td>2.65±1.82</td>
    <td>4.19±3.10</td>
    <td>10.78±5.56</td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.84±1.65</td>
    <td>2.64±1.77</td>
    <td>4.22±3.05</td>
    <td>10.70±4.46</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>5.49±0.70</td>
    <td>5.03±0.85</td>
    <td>8.54±1.21</td>
    <td>19.06±1.26</td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.15±1.07</td>
    <td>4.45±1.54</td>
    <td>7.83±2.20</td>
    <td>17.61±2.46</td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>4.85±1.38</td>
    <td>4.68±2.47</td>
    <td>7.97±3.85</td>
    <td>17.50±3.15</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>4.59±1.50</td>
    <td>4.06±1.79</td>
    <td>7.54±2.57</td>
    <td>16.20±4.96</td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.01±1.54</td>
    <td>4.21±1.80</td>
    <td>8.20±2.79</td>
    <td>17.42±5.13</td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>5.04±1.39</td>
    <td>4.21±1.72</td>
    <td>8.24±2.38</td>
    <td>17.49±4.50</td>
  </tr>
</table>

Finding 7: Prompt engineering primarily influences textual descriptions and text-code alignment, with relatively little impact on code generation.

We conduct ablation studies using a subset of 160 queries covering all fields from Table 1. We present the results of the top-performing models from each family in Table 6, with the full results provided in Table 16. The results show that prompt engineering primarily impacts textual descriptions and text-code alignment, with minimal effect on code generation. They enhance the performance of most models, except for CohereRPlus. We also find that prompt engineering has little impact on GPT4O3Mini, likely because the reasoning modules have similar functionalities, which is consis-
tent with the findings in its report 4.

4.6 LLMs for Predictive Analysis
Based on the aforementioned evaluations, existing LLMs still have significant room for improvement in predictive analysis, particularly in terms of performance and efficiency. While GPT4O3Mini achieves relatively high scores, it lacks depth in providing justifications for algorithmic choices (Depth score: 2.91 out of 4) and struggles with data pre-processing (51% of codes omit this step). Additionally, its high token consumption (with an average exceeding 4K tokens) and the requirement for a large context window (up to 32K tokens) present challenges for real-world applications, especially in privacy-sensitive scenarios where only local, smaller models are viable.

5 Related Works

5.1 Large Language Models
Recent advancements in natural language processing (NLP) are largely attributed to Large Language Models (LLMs) (Zhao et al., 2023a; Chang et al., 2023) like GPT series (Brown et al., 2020; OpenAI, 2023, 2024), Gemini series (Google, 2024), Llama series (Touvron et al., 2023; Rozière et al., 2023; Meta, 2024), and so on. These models undergo extensive pre-training on vast text datasets and are further refined through techniques like reinforcement learning from human feedback (RLHF) (Kirk et al., 2024) and instruction fine-tuning (IFT) (Brown et al., 2020).

In parallel, models specifically fine-tuned by code-formatted data emerged, considering the wide-ranging demand for code understanding and program generation from natural language prompts. For example, Codex (Chen et al., 2021), CodeT (Chen et al., 2022), Code Llama (Rozière et al., 2023) and HiRoPE (Zhang et al., 2024) have displayed excellent performance in code-related tasks, such as code completion and description-to-code generation. Meanwhile, these fine-tuned LLMs have shown extra progress in solving math problems (Zhou et al., 2023; Xu et al., 2024).

5.2 LLMs for data analysis
LLMs have recently demonstrated significant potential in various data analysis fields. Some studies concentrate on rudimentary data tasks in preparation for further analysis like early-stage data preparation (Zhang et al., 2023) and table-to-text summarization (Zhao et al., 2023b). Some studies (Chen, 2022; Saeed et al., 2023; Gao et al., 2023) focus on LLMs’ performance in transforming texts into SQL queries with their execution on the corresponding database, mainly for elementary descriptive analysis. For more advanced data analysis, most existing research works (Yu et al., 2023; Jin et al., 2023; Schoenegger and Park, 2023; Pratt et al., 2024; Hong et al., 2024) are limited to domain-specific time series forecasting tasks and lack generalizability. Text2Analysis (He et al., 2023) introduces queries for four types of advanced data analysis. However, their evaluation is exclusively centered on code generation, neglecting the essential elements of textual explanation and the alignment between text and code. Generally, there is a noticeable lack of comprehensive evaluations of LLMs in predictive analysis.

6 Conclusion
In this paper, we propose the PredictiQ benchmark, addressing the gap in evaluating LLMs’ capabilities in predictive analysis. The benchmark provides domain-diversified datasets, data-specific queries, and in-depth evaluation protocols. We evaluate twelve popular LLMs on the benchmark, highlighting their capabilities and limitations in predictive analysis. From the evaluation, we find the following key insights: (i) Fine-tuning LLMs on code boosts their performance, sometimes exceeding the limits of model parameters. (ii) Code generation and text analysis abilities are interconnected, jointly influencing LLMs’ predictive analysis skills. (iii) LLMs exhibit varying levels of predictive ability across domains. Overall, existing LLMs still have significant room for improvement in achieving both high performance and efficiency in predictive analysis.

7 Limitations
This work exhibits the following limitations. (i) As discussed in section 2, this work centers on evaluating the performance of LLMs in predictive analysis, and does not extend to other advanced data analysis fields such as prescriptive analysis or diagnostic analysis. Future research is expected to investigate broader analysis dimensions and assess effectiveness of LLMs across diverse analytical domains. (ii) The dataset employed in this study is confined to a limited number of commonly
encountered fields, potentially overlooking requirements and challenges present in less-represented or novel fields. This limitation may impact the generalizability of the findings and suggests that future work should incorporate more diverse datasets to explore how LLMs perform across a broader range of fields.

8 Ethics Statement

This work benchmarks the efficacy of predictive analysis in LLMs. The evaluation involves datasets collected from publicly available platforms, detailed in Appendix A. We ensure that the collected datasets do not contain any personally identifiable information. Additionally, we rigorously verify the data licenses and copyright permissions to confirm that they authorize public use for research purposes. We need to further clarify that this study focuses on the evaluation of language models and does not introduce new model architectures or deployment strategies. While we acknowledge that flawed model predictions, particularly in sensitive or high-stakes domains, can contribute to biased or harmful outcomes, addressing such risks requires domain-specific considerations and interventions that fall beyond the scope of this work. The evaluation intends to inform and support future research aimed at improving model reliability and mitigating downstream risks in applied settings.

In the study, volunteers consisting of master’s students in data science with an Asian background conducted human annotation to generate and evaluate the queries. They also participate in manual code execution and LLM response evaluation. While these annotators possess a solid foundation of predictive analysis, there is a potential risk that individuals from a specific cultural background may exhibit biases in their comprehension of query content and LLM responses.

We have used ChatGPT to assist us in refining the expression of our paper.

References

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripir, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.

Mahdi Abolghasemi, Odkhishig Ganbold, and Kristian Rotaru. 2024. Humans vs. large language models: Judgmental forecasting in an era of advanced ai. International Journal of Forecasting.

Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology.

Chang Che, Zengyi Huang, Chen Li, Haotian Zheng, and Xinyu Tian. 2024. Integrating generative ai into financial market prediction for improved decision making. Preprint, arXiv:2404.03523.

Bei Chen, Fengji Zhang, A. Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. Codet: Code generation with generated tests. ArXiv, abs/2207.10397.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidi Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.

Qin Chen, Liang Wang, Bo Zheng, and Guojie Song. 2025. Dagprompt: Pushing the limits of graph prompting with a distribution-aware graph prompt tuning approach. In Proceedings of the ACM on Web Conference 2025, pages 4346–4358.

Wenhui Chen. 2022. Large language models are few (1)-shot table reasoners. arXiv preprint arXiv:2210.06710.

Dursun Delen and Sudha Ram. 2018. Research challenges and opportunities in business analytics. Journal of Business Analytics, 1(1):2–12.
Diny Dixon, Hina Sattar, Natalia Moros, Srija Reddy Kesireddy, Huma Ahsan, Mohit Lakkimsetti, Madhia Fatima, Dhruvi Doshi, Kanwarpreet Sadhu, and Muhammad Junaid Hassan. 2024. Unveiling the influence of ai predictive analytics on patient outcomes: A comprehensive narrative review. Cureus, 16.

Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-sql empowered by large language models: A benchmark evaluation. Preprint, arXiv:2308.15363.

Google. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. Preprint, arXiv:2403.05530.

W.H. Greene. 2019. Econometric Analysis, Global Edition. Pearson Education.

Xinyi He, Mengyu Zhou, Xinrun Xu, Xiaojun Ma, Rui Ding, Lun Du, Yan Gao, Ran Jia, Xu Chen, Shi Han, et al. 2023. Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries. arXiv preprint arXiv:2312.13671.

Hidehito Honda, Rina Kagawa, and Shirasuna Masaru. 2022. On the round number bias and wisdom of crowds in different response formats for numerical estimation. Scientific Reports, 12.

Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binbao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinxing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, and Chenglin Wu. 2024. Data interpreter: An llm agent for data science. Preprint, arXiv:2402.18679.

Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al. 2023. Time-llm: Time series forecasting by reprogramming large language models. arXiv preprint arXiv:2310.01728.

Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. Understanding the effects of rlhf on llm generalisation and diversity. Preprint, arXiv:2310.06452.

Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Jae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics ACL 2024, pages 517–545.

Vaibhav Kumar and M. L. Garg. 2017. Deep learning in predictive analytics: A survey. In 2017 International Conference on Emerging Trends in Computing and Communication Technologies (ICETCCT), pages 1–6.

Vaibhav Kumar and ML Garg. 2018. Predictive analytics: a review of trends and techniques. International Journal of Computer Applications, 182(1):31–37.

Xuyuan Liu, Yinghao Cai, Qihui Yang, and Yujun Yan. 2024. Exploring consistency in graph representations: from graph kernels to graph neural networks. arXiv preprint arXiv:2410.23748.

Zichen Liu, Xuyuan Liu, Yanlong Wen, Guoqing Zhao, Fen Xia, and Xiaojie Yuan. 2023. Treeman: tree-enhanced multimodal attention network for icd coding. arXiv preprint arXiv:2305.18576.

Xiaojun Ma, Qin Chen, Yi Wu, Guojie Song, Liang Wang, and Bo Zheng. 2023. Rethinking structural encodings: Adaptive graph transformer for node classification task. In Proceedings of the ACM web conference 2023, pages 533–544.

Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Microsoft. 2024a. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.

Microsoft. 2024b. Phi-4 technical report. Preprint, arXiv:2412.08905.

Harsh Mishra, Siddharth Swarup Rautaray, and Manjusha Pandey. 2023. Review on big data analytics and its impact on marketing strategy. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), pages 424–429.

Bhuvanashree Murugadoss, Christian Poelitz, Ian Drosos, VuLe, Nick McKenna, Carina Suzana Negreanu, Chris Parrin, and Advait Sarkar. 2024. Evaluating the evaluator: Measuring llms’ adherence to task evaluation instructions. Preprint, arXiv:2408.08781.

OpenAI. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

OpenAI. 2024. Gpt-4o system card. Preprint, arXiv:2410.21276.

Arjun Panickssery, Samuel R. Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Preprint, arXiv:2404.13076.

Sarah Pratt, Seth Blumberg, Pietro Kreitlon Carolino, and Meredith Ringel Morris. 2024. Can language models use forecasting strategies? Preprint, arXiv:2406.04446.

Yuanyi Ren, Haoran Ye, Hanjun Fang, Xin Zhang, and Guojie Song. 2024. ValueBench: Towards comprehensively evaluating value orientations and understanding of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2015–2040. Association for Computational Linguistics.

Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish
Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosséz, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. 2023. Code llama: Open foundation models for code. Preprint, arXiv:2308.12950.

Mohammed Saeed, Nicola De Cao, and Paolo Papotti. 2023. Querying large language models with sql. arXiv preprint arXiv:2304.00472.

P. Saranya and P. Asha. 2019. Survey on big data analytics in health care. In 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT), pages 46–51.

Philipp Schoenegger and Peter S Park. 2023. Large language model prediction capabilities: Evidence from a real-world forecasting tournament. arXiv preprint arXiv:2310.13014.

Eric Siegel. 2013. Predictive analytics: The power to predict who will click, buy, lie, or die. John Wiley & Sons.

Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. Preprint, arXiv:2405.01724.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

G. J. J. Van den Burg and C. K. I. Williams. 2020. An evaluation of change point detection algorithms. arXiv preprint arXiv:2003.06222.

Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9440–9450, Bangkok, Thailand. Association for Computational Linguistics.

Parth Wazurkar, Robin Singh Bhadoria, and Dhananjai Bajpai. 2017. Predictive analytics in data science for business intelligence solutions. In 2017 7th International Conference on Communication Systems and Network Technologies (CSNT), pages 367–370.

Hui Wei, Shenghua He, Tian Xia, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. Preprint, arXiv:2408.13006.

Yifan Xu, Xiao Liu, Xinghan Liu, Zhenyu Hou, Yueyan Li, Xiaohan Zhang, Zihan Wang, Aohan Zeng, Zhengxiao Du, Zhao Wenyi, Jie Tang, and Yuxiao Dong. 2024. ChatGLM-math: Improving math problem-solving in large language models with a self-critique pipeline. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9733–9760, Miami, Florida, USA. Association for Computational Linguistics.

Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. Preprint, arXiv:2410.02736.

Xinli Yu, Zheng Chen, and Yanbin Lu. 2023. Harnessing llms for temporal data-a study on explainable financial time series forecasting. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 739–753.

Haochen Zhang, Yuyang Dong, Chuan Xiao, and M. Oyamada. 2023. Large language models as data preprocessors. ArXiv, abs/2308.16361.

Kechi Zhang, Ge Li, Huangzhao Zhang, and Zhi Jin. 2024. HiRoPE: Length extrapolation for code models using hierarchical position. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13615–13627, Bangkok, Thailand. Association for Computational Linguistics.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023a. A survey of large language models. arXiv preprint arXiv:2303.18223.

Yilun Zhao, Haowei Zhang, Shengyun Si, Linyong Nan, Xiangru Tang, and Arman Cohan. 2023b. Investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 160–175, Singapore. Association for Computational Linguistics.

Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. 2023. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921.

A Datasets

We have collected tabular datasets from different application scenarios to ensure the benchmark’s diversity and validity. The data sources can be summarized as follows.

• Datasets of the Economics domain cover financial systems at a state or national level. For
Table 7: Detailed statistics of collected datasets and the corresponding queries.

<table>
  <tr>
    <th>Table Name</th>
    <th>Domain</th>
    <th>#Columns</th>
    <th>#Queries</th>
  </tr>
  <tr><td>CarSales</td><td>Marketing and Sales</td><td>7</td><td>20</td></tr>
  <tr><td>MonetPaintingsSales</td><td>Marketing and Sales</td><td>6</td><td>20</td></tr>
  <tr><td>MovieBuzz</td><td>Marketing and Sales</td><td>13</td><td>20</td></tr>
  <tr><td>StoreSales</td><td>Marketing and Sales</td><td>6</td><td>20</td></tr>
  <tr><td>HousePrices</td><td>Marketing and Sales</td><td>81</td><td>100</td></tr>
  <tr><td>GasolineConsumption</td><td>Marketing and Sales</td><td>6</td><td>20</td></tr>
  <tr><td>GPA</td><td>Education</td><td>6</td><td>20</td></tr>
  <tr><td>ProgramEffectiveness</td><td>Education</td><td>5</td><td>20</td></tr>
  <tr><td>GenderEconomicsCourses</td><td>Education</td><td>11</td><td>30</td></tr>
  <tr><td>RunLog</td><td>Healthcare</td><td>5</td><td>20</td></tr>
  <tr><td>Smoker</td><td>Healthcare</td><td>24</td><td>40</td></tr>
  <tr><td>HealthCareOutcomes</td><td>Healthcare</td><td>16</td><td>30</td></tr>
  <tr><td>GermanHealthCare</td><td>Healthcare</td><td>25</td><td>40</td></tr>
  <tr><td>Employee</td><td>Human Resource</td><td>14</td><td>20</td></tr>
  <tr><td>LaborSupply</td><td>Human Resource</td><td>19</td><td>40</td></tr>
  <tr><td>LaborMarket</td><td>Human Resource</td><td>12</td><td>20</td></tr>
  <tr><td>StockPrice</td><td>Economics</td><td>9</td><td>20</td></tr>
  <tr><td>USD_ISK_Exchange_Rate</td><td>Economics</td><td>6</td><td>20</td></tr>
  <tr><td>CostFunction</td><td>Economics</td><td>10</td><td>20</td></tr>
  <tr><td>InvestmentData</td><td>Economics</td><td>8</td><td>20</td></tr>
  <tr><td>Macroeconomics</td><td>Economics</td><td>14</td><td>20</td></tr>
  <tr><td>LongleyData</td><td>Economics</td><td>5</td><td>20</td></tr>
  <tr><td>ExpenditureAndDefault</td><td>Economics</td><td>14</td><td>30</td></tr>
  <tr><td>IncomeAndExpenditure</td><td>Economics</td><td>7</td><td>20</td></tr>
  <tr><td>MunnellProductivity</td><td>Economics</td><td>11</td><td>20</td></tr>
  <tr><td>KleinModel</td><td>Economics</td><td>10</td><td>20</td></tr>
  <tr><td>MunicipalExpenditure</td><td>Economics</td><td>5</td><td>20</td></tr>
  <tr><td>LaLondeEarnings</td><td>Economics</td><td>15</td><td>40</td></tr>
  <tr><td>Titanic</td><td>Social Study</td><td>12</td><td>20</td></tr>
  <tr><td>ExtramaritalAffairs</td><td>Social Study</td><td>11</td><td>30</td></tr>
  <tr><td>CrossExtramaritalAffairs</td><td>Social Study</td><td>10</td><td>30</td></tr>
  <tr><td>ShipAccidents</td><td>Social Study</td><td>14</td><td>30</td></tr>
  <tr><td>HourlyTrafficVolume</td><td>Traffic</td><td>9</td><td>20</td></tr>
  <tr><td>AirPassengerTraffic</td><td>Traffic</td><td>6</td><td>20</td></tr>
  <tr><td>CostData_For_US_Airlines</td><td>Traffic</td><td>6</td><td>20</td></tr>
  <tr><td>TransportationEquipmentManufacturing</td><td>Traffic</td><td>5</td><td>20</td></tr>
  <tr><td>SwissRailroads</td><td>Traffic</td><td>21</td><td>50</td></tr>
  <tr><td>MoleFractionOfCarbonDioxide</td><td>Industry Analysis</td><td>8</td><td>20</td></tr>
  <tr><td>ShanghaiLicensePlatePrice</td><td>Industry Analysis</td><td>5</td><td>20</td></tr>
  <tr><td>GasolineMarket</td><td>Industry Analysis</td><td>11</td><td>20</td></tr>
  <tr><td>BaseballAttendance</td><td>Industry Analysis</td><td>7</td><td>20</td></tr>
  <tr><td>SpanishDairyFarmProduction</td><td>Industry Analysis</td><td>29</td><td>50</td></tr>
  <tr><td>TravelModeChoice</td><td>Industry Analysis</td><td>7</td><td>20</td></tr>
  <tr><td>CaliforniaUtilities</td><td>Industry Analysis</td><td>10</td><td>30</td></tr>
</table>
<table>
  <tr>
    <th>Type</th>
    <th>#Queries</th>
  </tr>
  <tr>
    <td>Classification</td>
    <td>285</td>
  </tr>
  <tr>
    <td>Regression</td>
    <td>324</td>
  </tr>
  <tr>
    <td>Forecasting</td>
    <td>220</td>
  </tr>
  <tr>
    <td>Clustering</td>
    <td>137</td>
  </tr>
  <tr>
    <td>Anomaly Detection</td>
    <td>126</td>
  </tr>
  <tr>
    <td>Other</td>
    <td>38</td>
  </tr>
</table>

Table 8: Detailed statistics of query task categories.

example, the yearly records of Gross Domestic Product (GDP), Gross National Product (GNP), and other economic metrics.

• Datasets of the Marketing and Sales domain cover sales records and marketing schemes for specific brands or products. This includes, for instance, annual data on costs, sales, and profits for various car models across multiple brands.

• Datasets of the Industry Analysis domain cover the development of a certain industry. For example, they might include data on the distribution of residential electricity demands within the utility sector.

• Datasets of the Traffic domain cover different types of traffic records. For example, the monthly records of passenger volumes in several airports.

• Datasets of the Healthcare domain cover healthcare records. For example, the records of physical examination indexes for both smokers and non-smokers.

• Datasets of the Social Study domain cover specific sociological issues. For example, the records of demographic and personal variables as well as the occurrence of extramarital affairs.

• Datasets in the Human Resource domain encompass aspects of human resource management, such as detailed records of employee information, including salary, department, and other related data.

• Datasets in the Education domain include records related to educational activities, such as college students’ grades and attendance data.

Datasets are acquired from various public platforms, including the data science platform Kaggle 5, the TCPD benchmark for time-series analysis (Van den Burg and Williams, 2020) and the eighth edition of Econometric Analysis (Greene, 2019). We ensure that the collected datasets do not contain personally identifiable information. Additionally, we verify the data licenses to ensure they are permitted for public use in research activities. This guarantees that each dataset has undergone a thorough privacy and security review process. Details regarding the collected tabular datasets are available in Table 7. Meanwhile, PredictiQ encompasses a wide range of tasks, including classification, regression, forecasting, clustering, and anomaly detection. We also provide statistics that underscore the diversity of our queries in Table 8.

B Evaluation Protocols

We provide a systematical description of our evaluation protocol in Table 9 regarding text analysis, code generation, and text-code alignment. Functional Correctness is evaluated through the ratio of code execution without errors.

C Settings

For all LLMs, we set the temperature to 0.7 (where applicable), top_p to 0.95 (where applicable), and a maximum token limit of 4096. For GPT4O1 and GPT4O3Mini, which require extra tokens for reasoning, we extend the token limit to 32,768.

D Additional Experiments

D.1 Quality Control During Annotation

To further evaluate the reliability of expert annotations, we report Krippendorff’s Alpha coefficients in Table 10, which measure inter-annotator agreement across evaluation dimensions. The results indicate a generally substantial level of consistency among the annotators, supporting the overall quality and reliability of the annotation process.

D.2 More Detailed Error Analysis

Here We introduce more detailed error categories, including syntax, runtime, library, and data flow errors, to provide clearer insights into the model’s performance and potential failure modes, detailed in Table 11. Our analysis shows that increasing model capacity significantly reduces syntax errors.

5https://www.kaggle.com/
Table 9: The evaluation protocol. Relevance and Depth for text analysis. Usefulness and Functional Correctness correspond to code generation. Descriptive Accuracy, Coverage, and Clarity pertain to text-code alignment.

<table>
  <tr>
    <th>Score</th>
    <th>Description</th>
  </tr>
  <tr>
    <td rowspan="4">Relevance</td>
    <td>0<br>The analysis is not at all relevant to the query. It does not address the core issue or utilize the data appropriately.</td>
  </tr>
  <tr>
    <td>1<br>The analysis is slightly relevant to the query, with superfluous information or focusing on secondary issues.</td>
  </tr>
  <tr>
    <td>2<br>The analysis is relevant, touching upon the central query and making use of the data but not thoroughly enough.</td>
  </tr>
  <tr>
    <td>3<br>The analysis is generally relevant, closely aligning with the given data and query, albeit with minor deviations.</td>
  </tr>
  <tr>
    <td>4<br>The analysis is highly relevant, directly addressing the question with appropriate and substantial use of the data.</td>
  </tr>
  <tr>
    <td rowspan="4">Depth</td>
    <td>0<br>The analysis does not provide any justification for the choice of algorithm used.</td>
  </tr>
  <tr>
    <td>1<br>The analysis provides minimal justification for the choice, with surface-level reasoning without delving into specifics.</td>
  </tr>
  <tr>
    <td>2<br>The analysis moderately justifies the choice, with reasoned arguments but not fully covering all relevant factors.</td>
  </tr>
  <tr>
    <td>3<br>The analysis gives a detailed justification. It is well-reasoned with minor gaps in the argumentation.</td>
  </tr>
  <tr>
    <td>4<br>The analysis thoroughly justifies the choice, showing deep understanding and covering all aspects in detail.</td>
  </tr>
  <tr>
    <td rowspan="4">Usefulness</td>
    <td>0<br>The code is not at all helpful, it is irrelevant to the problem.</td>
  </tr>
  <tr>
    <td>1<br>The code is slightly helpful, it contains information relevant to the problem, but it is easier to rewrite the solution.</td>
  </tr>
  <tr>
    <td>2<br>The code is somewhat helpful, it requires significant changes (compared to the size of the snippet), but is still useful.</td>
  </tr>
  <tr>
    <td>3<br>The code is helpful but needs to be slightly changed to solve the problem.</td>
  </tr>
  <tr>
    <td>4<br>The code is very helpful, it solves the problem.</td>
  </tr>
  <tr>
    <td>Functional Correctness</td>
    <td>0-4<br>Functional Correctness is evaluated through the ratio of code execution without errors, scaled up to 0-4.</td>
  </tr>
  <tr>
    <td rowspan="4">Descriptive Accuracy</td>
    <td>0<br>The text does not describe the code at all; there is a complete mismatch.</td>
  </tr>
  <tr>
    <td>1<br>The text describes less than half of what is represented in the code, missing significant functionalities.</td>
  </tr>
  <tr>
    <td>2<br>The text describes about half of the code, yet there are still substantial gaps in the description.</td>
  </tr>
  <tr>
    <td>3<br>The text describes most of the code adequately, with only minor gaps or discrepancies.</td>
  </tr>
  <tr>
    <td>4<br>The text provides an accurate description of the code, perfectly aligning with all functionalities in the code snippet.</td>
  </tr>
  <tr>
    <td rowspan="4">Coverage</td>
    <td>0<br>The text does not cover any aspects of the code; the explanation is absent or irrelevant.</td>
  </tr>
  <tr>
    <td>1<br>The text covers a few aspects of the code but misses out on explaining several crucial functionalities.</td>
  </tr>
  <tr>
    <td>2<br>The text covers around half of the code’s aspects, providing a partial understanding of the code’s functionalities.</td>
  </tr>
  <tr>
    <td>3<br>The text covers most of the aspects, offering a substantial understanding, with minor aspects left uncovered.</td>
  </tr>
  <tr>
    <td>4<br>The text comprehensively covers all aspects of the code, leaving no functionalities unexplained.</td>
  </tr>
  <tr>
    <td rowspan="4">Clarity</td>
    <td>0<br>The alignment between the text and the code is unclear, making it difficult for users to follow.</td>
  </tr>
  <tr>
    <td>1<br>The alignment is slightly clear; however, users may struggle to correlate the text and code without efforts.</td>
  </tr>
  <tr>
    <td>2<br>The alignment is moderately clear, allowing users to somewhat follow the explanation with a reasonable effort.</td>
  </tr>
  <tr>
    <td>3<br>The alignment is largely clear, aiding users in following the explanation with minimal difficulties.</td>
  </tr>
  <tr>
    <td>4<br>The alignment is crystal clear, offering users an effortless pathway to understand the code through the text.</td>
  </tr>
</table>

Table 10: Krippendorff’s Alpha coefficients showing inter-annotator agreement across evaluation dimensions.

<table>
  <tr>
    <th></th>
    <th>Krippendorff’s Alpha coefficient</th>
  </tr>
  <tr>
    <td>Relevance</td>
    <td>0.89</td>
  </tr>
  <tr>
    <td>Depth</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>Usefulness</td>
    <td>0.84</td>
  </tr>
  <tr>
    <td>Descriptive Accuracy</td>
    <td>0.93</td>
  </tr>
  <tr>
    <td>Coverage</td>
    <td>0.91</td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>0.83</td>
  </tr>
</table>

Moreover, fine-tuning on code further reduces syntax errors, although it tends to increase runtime and data flow errors.

D.3 Evaluation with Other LLMs

We provide the detailed results adopting GPT4O and Phi3Medium as evaluators in Table 12 and Table 13. We find that GPT4O is far more positive when conducting evaluation, compared to GPT4Turbo. Meanwhile, Phi3Medium is non-applicable as a rationale evaluator, as it tends to award high scores to almost all submissions, regardless of evident shortcomings. For example, as we have evidenced in Table 4, many instances from Llama family models have no code at all, while Phi3Medium still scores them with high scores in code and text-code alignment domains. This makes the evaluation by Phi3Medium less meaningful.

We have additionally incorporated Gemini-2.0-Flash 6 as an alternative evaluator in our study. The corresponding results, presented in Table 14, are generally slightly more positive than those from GPT-4 Turbo. We further provide the weighted Cohen’s Kappa scores in Table 15 to assess inter-rater agreement among GPT-4 Turbo, Gemini-2-Flash, and the human experts, with multiple expert evaluations consolidated into unified scores through majority voting. The scores indicate that Gemini-2-Flash generally also exhibits strong agreement with human experts, particularly regarding the Usefulness metric. Based on these findings, it is reasonable to adopt Gemini-2-Flash as the alternative evaluator.

We also provide the full results of expert evaluation against evaluation from LLMs in Figure 5.

6https://deepmind.google/models/gemini/flash/
Table 11: Error analysis across models. The top-level columns include the percentage of code without pre-processing and import errors. The logic error category is further broken down into subtypes of syntax, runtime, library, and data flow errors.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th rowspan="2">No Pre-processing</th>
    <th rowspan="2">Import Error</th>
    <th colspan="4">Logic Error</th>
  </tr>
  <tr>
    <th>Syntax Error</th>
    <th>Runtime Error</th>
    <th>Library Error</th>
    <th>Data Flow Error</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>71%</td>
    <td>3.8%</td>
    <td>15.3%</td>
    <td>7.6%</td>
    <td>7.8%</td>
    <td>12.5%</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>66%</td>
    <td>1.3%</td>
    <td>7.8%</td>
    <td>3.8%</td>
    <td>3.4%</td>
    <td>5.7%</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>66%</td>
    <td>0.4%</td>
    <td>5.1%</td>
    <td>3.9%</td>
    <td>3.8%</td>
    <td>5.8%</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>50%</td>
    <td>0.3%</td>
    <td>3.1%</td>
    <td>3.2%</td>
    <td>3.5%</td>
    <td>4.9%</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>51%</td>
    <td>0.3%</td>
    <td>3.1%</td>
    <td>2.3%</td>
    <td>2.9%</td>
    <td>4.4%</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>92%</td>
    <td>41.8%</td>
    <td>19.3%</td>
    <td>8.3%</td>
    <td>6.2%</td>
    <td>6.3%</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>89%</td>
    <td>38.2%</td>
    <td>17.4%</td>
    <td>11.3%</td>
    <td>8.8%</td>
    <td>9.3%</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>87%</td>
    <td>36.1%</td>
    <td>18.3%</td>
    <td>13.7%</td>
    <td>8.6%</td>
    <td>5.3%</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>87%</td>
    <td>15.0%</td>
    <td>12.5%</td>
    <td>22.7%</td>
    <td>16.3%</td>
    <td>12.5%</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>72%</td>
    <td>5.9%</td>
    <td>15.7%</td>
    <td>12.4%</td>
    <td>13.7%</td>
    <td>11.3%</td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>58%</td>
    <td>3.8%</td>
    <td>12.3%</td>
    <td>10.3%</td>
    <td>9.5%</td>
    <td>10.2%</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>78%</td>
    <td>4.6%</td>
    <td>18.0%</td>
    <td>10.4%</td>
    <td>12.3%</td>
    <td>12.7%</td>
  </tr>
</table>

Table 12: Evaluation results on PredictiQ on seven perspectives, each ranked from 0 to 4 following protocols in Table 9. For Functional Correctness we measure the ratio of executable codes, and map it into scores from 0 to 4 evenly. We adopt the GPT4O as the evaluator.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="2">Text</th>
    <th colspan="2">Code</th>
    <th colspan="2">Text-Code Alignment</th>
    <th rowspan="2">Total</th>
  </tr>
  <tr>
    <th>Relevance</th>
    <th>Depth</th>
    <th>Usefulness</th>
    <th>Functional Correctness</th>
    <th>Descriptive Accuracy</th>
    <th>Coverage</th>
    <th>Clarity</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>3.60±0.57</td>
    <td>2.62±0.72</td>
    <td>3.36±0.77</td>
    <td>2.12 (53%)</td>
    <td>3.36±0.66</td>
    <td>3.13±0.66</td>
    <td>3.49±0.56</td>
    <td>21.68±3.69</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>3.91±0.29</td>
    <td>3.24±0.50</td>
    <td>3.66±0.52</td>
    <td>3.12 (78%)</td>
    <td>3.88±0.33</td>
    <td>3.63±0.49</td>
    <td>3.91±0.29</td>
    <td>25.34±2.08</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>3.95±0.26</td>
    <td>3.48±0.56</td>
    <td>3.83±0.40</td>
    <td>3.24 (81%)</td>
    <td>3.90±0.30</td>
    <td>3.82±0.39</td>
    <td>3.91±0.29</td>
    <td>26.13±1.97</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>3.98±0.09</td>
    <td>3.66±0.27</td>
    <td>3.87±0.69</td>
    <td>3.24 (81%)</td>
    <td>3.94±0.34</td>
    <td>3.85±0.45</td>
    <td>3.91±0.63</td>
    <td>26.45±2.33</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>3.99±0.06</td>
    <td>3.64±0.34</td>
    <td>3.89±0.18</td>
    <td>3.24 (81%)</td>
    <td>3.94±0.07</td>
    <td>3.89±0.14</td>
    <td>3.96±0.09</td>
    <td>26.64±2.05</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>2.74±0.81</td>
    <td>1.93±0.84</td>
    <td>2.27±0.90</td>
    <td>0.72 (18%)</td>
    <td>1.96±0.96</td>
    <td>1.99±0.90</td>
    <td>2.33±0.89</td>
    <td>13.94±5.02</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>2.79±0.81</td>
    <td>1.99±0.76</td>
    <td>2.31±0.92</td>
    <td>0.60 (15%)</td>
    <td>2.19±0.95</td>
    <td>2.18±0.85</td>
    <td>2.53±0.83</td>
    <td>14.59±4.89</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>3.03±0.63</td>
    <td>2.18±0.64</td>
    <td>2.63±0.79</td>
    <td>0.72 (18%)</td>
    <td>2.31±0.84</td>
    <td>2.32±0.71</td>
    <td>2.56±0.76</td>
    <td>15.75±4.13</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>3.09±0.79</td>
    <td>2.24±0.73</td>
    <td>2.66±0.89</td>
    <td>0.84 (21%)</td>
    <td>2.35±0.86</td>
    <td>2.45±0.81</td>
    <td>2.73±0.75</td>
    <td>16.36±4.54</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>3.55±0.63</td>
    <td>2.74±0.71</td>
    <td>3.44±0.73</td>
    <td>1.64 (41%)</td>
    <td>3.32±0.94</td>
    <td>3.14±0.91</td>
    <td>3.36±0.92</td>
    <td>21.19±4.67</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>3.53±0.70</td>
    <td>2.77±0.72</td>
    <td>3.20±0.86</td>
    <td>1.68 (42%)</td>
    <td>3.22±0.87</td>
    <td>3.11±0.83</td>
    <td>3.36±0.73</td>
    <td>20.87±4.54</td>
  </tr>
</table>

Table 13: Evaluation results on PredictiQ on seven perspectives, each ranked from 0 to 4 following protocols in Table 9. For Functional Correctness we measure the ratio of executable codes, and map it into scores from 0 to 4 evenly. We adopt the Phi3Medium as the evaluator.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="2">Text</th>
    <th colspan="2">Code</th>
    <th colspan="2">Text-Code Alignment</th>
    <th rowspan="2">Total</th>
  </tr>
  <tr>
    <th>Relevance</th>
    <th>Depth</th>
    <th>Usefulness</th>
    <th>Functional Correctness</th>
    <th>Descriptive Accuracy</th>
    <th>Coverage</th>
    <th>Clarity</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>3.97±0.18</td>
    <td>3.60±0.54</td>
    <td>3.87±0.33</td>
    <td>2.12 (53%)</td>
    <td>3.98±0.15</td>
    <td>3.95±0.21</td>
    <td>4.00</td>
    <td>25.49±1.29</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>3.96±0.20</td>
    <td>3.62±0.53</td>
    <td>3.84±0.42</td>
    <td>3.12 (78%)</td>
    <td>3.97±0.18</td>
    <td>3.95±0.23</td>
    <td>4.00</td>
    <td>26.45±1.42</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>3.99±0.11</td>
    <td>3.85±0.36</td>
    <td>3.98±0.15</td>
    <td>3.24 (81%)</td>
    <td>3.90±0.61</td>
    <td>3.95±0.44</td>
    <td>3.95±0.44</td>
    <td>26.86±1.93</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>3.99±0.11</td>
    <td>3.95±0.36</td>
    <td>3.98±0.15</td>
    <td>3.24 (81%)</td>
    <td>3.88±0.61</td>
    <td>3.97±0.44</td>
    <td>3.95±0.44</td>
    <td>26.96±1.22</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>3.99±0.08</td>
    <td>3.98±0.19</td>
    <td>3.98±0.10</td>
    <td>3.24 (81%)</td>
    <td>3.95±0.61</td>
    <td>3.95±0.44</td>
    <td>3.95±0.44</td>
    <td>27.04±1.50</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>3.83±0.69</td>
    <td>3.43±0.95</td>
    <td>3.68±0.76</td>
    <td>0.72 (18%)</td>
    <td>3.69±0.96</td>
    <td>3.67±0.97</td>
    <td>3.91±0.48</td>
    <td>22.93±4.56</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>3.84±0.58</td>
    <td>3.41±1.01</td>
    <td>3.54±0.81</td>
    <td>0.60 (15%)</td>
    <td>3.67±0.89</td>
    <td>3.88±0.60</td>
    <td>2.26±2.41</td>
    <td>22.62±4.51</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>3.92±0.47</td>
    <td>3.71±0.64</td>
    <td>3.82±0.55</td>
    <td>0.72 (18%)</td>
    <td>3.80±0.79</td>
    <td>3.77±0.72</td>
    <td>3.92±0.47</td>
    <td>23.67±3.34</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>3.85±0.60</td>
    <td>3.67±0.79</td>
    <td>3.71±0.73</td>
    <td>0.84 (21%)</td>
    <td>3.80±0.80</td>
    <td>3.77±0.79</td>
    <td>3.90±0.61</td>
    <td>23.54±4.13</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>3.99±0.05</td>
    <td>3.81±0.39</td>
    <td>3.94±0.24</td>
    <td>1.64 (41%)</td>
    <td>3.95±0.26</td>
    <td>3.94±0.28</td>
    <td>3.95±0.26</td>
    <td>25.23±1.43</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>3.98±0.15</td>
    <td>3.59±0.59</td>
    <td>3.88±0.33</td>
    <td>1.68 (42%)</td>
    <td>3.98±0.19</td>
    <td>3.96±0.19</td>
    <td>4.00</td>
    <td>25.07±1.28</td>
  </tr>
</table>
Table 14: Evaluation results on PredictiQ on seven perspectives, each ranked from 0 to 4 following protocols in Table 9. For Functional Correctness we measure the ratio of executable codes, and map it into scores from 0 to 4 evenly. We adopt the Gemini-2.0-Flash as the evaluator.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="3">Text</th>
    <th colspan="2">Code</th>
    <th colspan="3">Text-Code Alignment</th>
    <th rowspan="2">Total</th>
  </tr>
  <tr>
    <th>Relevance</th>
    <th>Depth</th>
    <th>Usefulness</th>
    <th>Functional Correctness</th>
    <th>Descriptive Accuracy</th>
    <th>Coverage</th>
    <th>Clarity</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>3.13</td>
    <td>1.83</td>
    <td>2.63</td>
    <td>2.12 (53%)</td>
    <td>2.83</td>
    <td>2.65</td>
    <td>2.9</td>
    <td>18.09</td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>3.55</td>
    <td>2.34</td>
    <td>2.94</td>
    <td>3.12 (78%)</td>
    <td>3.14</td>
    <td>3.12</td>
    <td>3.3</td>
    <td>21.51</td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>3.75</td>
    <td>2.65</td>
    <td>3.45</td>
    <td>3.24 (81%)</td>
    <td>3.56</td>
    <td>3.48</td>
    <td>3.49</td>
    <td>23.62</td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>3.88</td>
    <td>2.9</td>
    <td>3.62</td>
    <td>3.24 (81%)</td>
    <td>3.66</td>
    <td>3.65</td>
    <td>3.55</td>
    <td>24.66</td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>3.93</td>
    <td>3.01</td>
    <td>3.61</td>
    <td>3.24 (81%)</td>
    <td>3.67</td>
    <td>3.67</td>
    <td>3.59</td>
    <td>24.96</td>
  </tr>
  <tr>
    <td>ChatLlama2-7B</td>
    <td>2.31</td>
    <td>1.65</td>
    <td>1.58</td>
    <td>0.72 (18%)</td>
    <td>1.32</td>
    <td>1.25</td>
    <td>1.43</td>
    <td>10.26</td>
  </tr>
  <tr>
    <td>CodeLlama2-7B</td>
    <td>2.33</td>
    <td>1.77</td>
    <td>1.76</td>
    <td>0.60 (15%)</td>
    <td>1.42</td>
    <td>1.42</td>
    <td>1.61</td>
    <td>10.91</td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>2.18</td>
    <td>1.75</td>
    <td>1.66</td>
    <td>0.72 (18%)</td>
    <td>1.53</td>
    <td>1.42</td>
    <td>1.69</td>
    <td>10.95</td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>2.76</td>
    <td>1.83</td>
    <td>1.75</td>
    <td>0.84 (21%)</td>
    <td>1.64</td>
    <td>1.74</td>
    <td>1.87</td>
    <td>12.43</td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>3.05</td>
    <td>1.98</td>
    <td>2.45</td>
    <td>1.64 (41%)</td>
    <td>2.67</td>
    <td>2.7</td>
    <td>2.76</td>
    <td>17.25</td>
  </tr>
  <tr>
    <td>CohereRPlus</td>
    <td>3.06</td>
    <td>1.87</td>
    <td>2.56</td>
    <td>1.68 (42%)</td>
    <td>2.76</td>
    <td>2.84</td>
    <td>2.88</td>
    <td>17.65</td>
  </tr>
</table>

![Bar chart comparing expert evaluation against evaluation from LLMs](page_1012_1342_1012_246.png)

Figure 5: Expert evaluation against evaluation from LLMs.
<table>
  <tr>
    <th></th>
    <th>GPT-4 Turbo vs Human</th>
    <th>Gemini vs Human</th>
  </tr>
  <tr>
    <td>Relevance</td>
    <td>0.85</td>
    <td>0.74</td>
  </tr>
  <tr>
    <td>Depth</td>
    <td>0.74</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Usefulness</td>
    <td>0.78</td>
    <td>0.86</td>
  </tr>
  <tr>
    <td>Descriptive Accuracy</td>
    <td>0.72</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>Coverage</td>
    <td>0.91</td>
    <td>0.88</td>
  </tr>
  <tr>
    <td>Clarity</td>
    <td>0.88</td>
    <td>0.75</td>
  </tr>
</table>

Table 15: Cohen’s Kappa scores assessing inter-rater agreement across GPT-4 Turbo, Gemini-2 Flash, and human experts on six evaluation dimensions.

D.4 Ablation Study

We present the full ablation study of all LLMs in Table 16.

Table 16: Ablation study. For clarity, we present the total scores of three domains.

<table>
  <tr>
    <th></th>
    <th colspan="3">Text</th>
    <th colspan="3">Code</th>
    <th colspan="3">Alignment</th>
    <th colspan="3">Total</th>
  </tr>
  <tr>
    <td>GPT3.5Turbo</td>
    <td>4.75±1.54</td>
    <td>4.52±0.86</td>
    <td>7.93±2.79</td>
    <td>17.21±5.19</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>4.26±1.30</td>
    <td>4.21±0.79</td>
    <td>7.74±2.35</td>
    <td>16.21±4.44</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>4.53±1.22</td>
    <td>4.27±0.71</td>
    <td>7.86±2.39</td>
    <td>16.45±4.31</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT4Turbo</td>
    <td>5.57±1.80</td>
    <td>5.90±0.72</td>
    <td>9.21±2.41</td>
    <td>20.68±4.23</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.27±1.06</td>
    <td>5.50±0.57</td>
    <td>9.18±1.76</td>
    <td>19.94±3.39</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>5.06±1.19</td>
    <td>5.32±0.72</td>
    <td>8.75±2.11</td>
    <td>19.13±4.03</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT4O</td>
    <td>5.99±1.22</td>
    <td>6.36±0.67</td>
    <td>10.08±1.93</td>
    <td>22.43±3.82</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.78±1.14</td>
    <td>6.22±0.67</td>
    <td>9.94±2.07</td>
    <td>21.94±3.88</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>5.81±1.12</td>
    <td>6.34±0.90</td>
    <td>9.76±1.17</td>
    <td>21.91±3.97</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT4O1</td>
    <td>6.42±0.79</td>
    <td>6.85±0.60</td>
    <td>10.44±2.00</td>
    <td>23.70±3.40</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>6.25±0.61</td>
    <td>6.70±0.47</td>
    <td>10.51±2.23</td>
    <td>23.46±3.12</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>6.27±0.57</td>
    <td>6.96±0.14</td>
    <td>10.45±1.86</td>
    <td>23.68±3.40</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT4O3Mini</td>
    <td>6.54±0.78</td>
    <td>7.01±0.55</td>
    <td>10.56±1.88</td>
    <td>24.11±3.13</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>6.56±0.86</td>
    <td>7.31±0.67</td>
    <td>10.51±1.90</td>
    <td>24.22±3.07</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>6.25±0.56</td>
    <td>7.18±0.67</td>
    <td>10.22±1.76</td>
    <td>23.65±3.13</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ChatLlama2-TB</td>
    <td>3.29±1.55</td>
    <td>2.22±0.73</td>
    <td>2.93±1.61</td>
    <td>8.44±4.89</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>2.82±1.64</td>
    <td>2.01±0.81</td>
    <td>1.78±2.53</td>
    <td>6.61±4.96</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.27±1.65</td>
    <td>2.17±0.74</td>
    <td>2.73±2.62</td>
    <td>8.18±5.01</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>CodeLlama2-TB</td>
    <td>3.36±1.46</td>
    <td>2.21±0.72</td>
    <td>3.34±2.60</td>
    <td>8.92±4.97</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>3.27±1.27</td>
    <td>2.12±0.65</td>
    <td>2.72±2.52</td>
    <td>8.10±4.45</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.63±1.37</td>
    <td>2.40±0.71</td>
    <td>3.42±3.15</td>
    <td>9.44±5.29</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ChatLlama2-13B</td>
    <td>3.20±1.44</td>
    <td>2.24±0.72</td>
    <td>3.00±2.59</td>
    <td>8.44±4.74</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>3.16±1.65</td>
    <td>2.24±0.79</td>
    <td>3.15±2.63</td>
    <td>8.55±4.08</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.22±1.48</td>
    <td>2.22±0.81</td>
    <td>3.12±2.61</td>
    <td>8.56±4.90</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ChatLlama2-70B</td>
    <td>3.98±1.46</td>
    <td>2.66±0.76</td>
    <td>3.39±2.64</td>
    <td>11.03±4.66</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>3.94±1.64</td>
    <td>2.65±0.82</td>
    <td>3.19±3.10</td>
    <td>10.78±5.56</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.84±1.65</td>
    <td>2.64±0.77</td>
    <td>4.22±3.05</td>
    <td>10.70±5.46</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Phi3Medium</td>
    <td>4.64±1.99</td>
    <td>3.97±1.04</td>
    <td>7.36±3.47</td>
    <td>15.97±6.50</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>4.23±1.84</td>
    <td>3.55±1.02</td>
    <td>7.13±3.22</td>
    <td>14.92±6.08</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>3.52±2.63</td>
    <td>3.42±1.39</td>
    <td>5.45±4.42</td>
    <td>12.39±8.44</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Phi4</td>
    <td>5.49±0.70</td>
    <td>5.03±0.35</td>
    <td>8.54±1.21</td>
    <td>19.06±2.26</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.15±1.67</td>
    <td>4.45±1.54</td>
    <td>7.83±2.20</td>
    <td>17.61±2.46</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>4.85±3.18</td>
    <td>4.68±2.47</td>
    <td>7.97±3.85</td>
    <td>17.50±3.15</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>CoherentPlus</td>
    <td>4.59±1.50</td>
    <td>4.06±0.79</td>
    <td>7.54±2.57</td>
    <td>16.20±4.86</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Role Play</td>
    <td>5.01±1.54</td>
    <td>4.21±0.80</td>
    <td>8.20±2.79</td>
    <td>17.42±5.13</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>w/o Data Summary</td>
    <td>5.04±1.39</td>
    <td>4.21±0.72</td>
    <td>8.24±2.38</td>
    <td>17.49±4.50</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

prompts related to data. For comprehensive examples, please refer to the supplementary materials.

Prompt
Perform as a professional data scientist. I will provide you with a data analysis question along with a description of the data. Please answer the question and give me all the related analyses and codes. Question: Can we model and predict the effect of increasing prices on customers' selection of electric plans? The columns (and their data types, with additional ranges or num of types) are as PERSON(int64, min: 1, max: 301), NICKNAME(int64, min: 0, max: 100), MREC(int64, min: 32, max: 2048), DATA(int64, min: 0, max: 1), LOCAL(int64, min: 0, max: 1), ONTL(int64, min: 0, max: 51), LOCAL(int64, min: 0, max: 1), KNOW(int64, min: 0, max: 1), TOOL(int64, min: 0, max: 1), SEAS(int64, min: 0, max: 1).

(...data rows...)

Figure 6: Input Prompt

E Examples

This section presents examples of our prompts (in Figure 6) and the corresponding responses generated by GPT4Turbo and ChatLlama2-70B, as shown in Figure 7 and Figure 8. Our objective is to provide clear and representative examples demonstrating how these LLMs handle predictive analysis

F Future Works

This work may be extended in several directions in the future: (i) Broadening the scope of data analysis to include more diverse types, such as prescriptive and diagnostic analyses (Delen and Ram, 2018), as well as exploring related domains like deep learning (Liu et al., 2024; Chen et al., 2025) and AI ethics (Ren et al., 2024). (ii) Diversifying the types of data considered by incorporating a wider range of real-world application areas. This includes expanding the structural forms of data, such as image data or graph-based data (Ma et al., 2023).
GPT4-Turbo Response

Here’s a step-by-step analysis and corresponding Python code using pandas for data manipulation and stats models for logistic regression:

Load the data

Let’s assume the data is in a CSV format and load it into a pandas DataFrame.

import pandas as pd
df = pd.read_csv('electric_plans_data.csv')

Data overview

Take an overview:

print(df.head())
print(df.describe())

Data preparation

Ensure that the data is clean and ready for analysis. For the given data description, it looks like the data is already clean and in numeric format.

Model fitting

Fit a logistic regression model using Price along with other variables that might affect the customer’s choice.

import statsmodels.api as sm

# Logistic regression with all variables included
X = df[["Price", "Ent", "Locat", "Known", "TOD", "SEAS"]]
# Independent variables
X = sm.add_constant(X) # Adding a constant to the model
y = df["Choice"] # Dependent variable

# Fit logistic regression model
logit_model = sm.Logit(y, X)
result = logit_model.fit()

Model Summary

Print a summary of the logistic regression model to interpret the results.

print(result.summary())

Prediction

Make predictions using the model. The logistic regression will provide probabilities of selection. We can choose a threshold (typically 0.5) to classify these probabilities into Choice 0 or 1.

df['Predicted_Choice_Probability'] = result.predict(X)
df['Predicted_Choice'] = (df['Predicted_Choice_Probability'] >= 0.5).astype(int)

Model evaluation

Evaluate the model performance. Common metrics for a classification problem include accuracy, precision, recall, F1 score, and the ROC curve.

from sklearn.metrics import classification_report, roc_auc_score
print(classification_report(df['Predicted_Choice']))
print("ROC AUC score:", roc_auc_score(df['Predicted_Choice_Probability']))

Interpretation

Based on the p-values from the logistic regression, we can determine the significance of each feature, including “Price”, on the “Choice”. A low p-value (typically less than 0.05) suggests that the feature is a significant predictor of the Choice.

Figure 7: Response from GPT4-Turbo

ChatLlama2 Response

Based on the provided data, it appears that the number of accidents has increased over time. To verify this, we can perform a statistical analysis using a regression model.

First, let’s clean and preprocess the data by removing any missing or irrelevant values. We can drop the “Months” column since it only contains unique values and doesn’t provide any meaningful information. We can also drop the “Acc” column since it is not relevant to our analysis.

Next, we can create a new column called “Year” by concatenating the “Y6064”, “Y6569”, “Y7874”, and “Y7579” columns. This will give us a continuous variable that represents the year in which the ship was operated.

Now, let’s create a linear regression model that predicts the number of accidents (Acc) based on the year (Year) and ship type (Type). We’ll use the “TA”, “TB”, “TC”, “TD”, and “TE” columns to create a one-hot encoding of the ship type.

Here’s the R code for the analysis:

# Load the data
data <- read.csv("ship_data.csv")

# Remove irrelevant columns
data <- data[,c("Type", "TA", "TB", "TC", "TD", "TE", "Year")]

# Create a new column for year
data$Year <- paste0(data$Y6064, data$Y6569, data$Y7874, data$Y7579)

# One-hot encode ship type
dataType <- as.factor(dataType)
dataTA_encode <- ifelse(dataType == 1, 1, 0)
dataTB_encode <- ifelse(dataType == 2, 1, 0)
dataTC_encode <- ifelse(dataType == 3, 1, 0)
dataTD_encode <- ifelse(dataType == 4, 1, 0)
dataTE_encode <- ifelse(dataType == 5, 1, 0)

# Create a linear regression model
model <- lm(Acc ~ Year + TA_encode + TB_encode + TC_encode + TD_encode + TE_encode, data = data)

# Summarize the model
summary(model)

The output of the summary function shows the coefficients of the linear regression model. The coefficient for the “Year” variable is positive, indicating that the number of accidents has increased over time. The p-value for the “Year” variable is also less than 0.05, which means that the relationship between year and accidents is statistically significant.

Based on this analysis, we can conclude that the number of accidents has increased over time, and the increase is statistically significant. However, it’s important to note that this analysis only considers the ships that have been in operation from 1960 to 1979, and it doesn’t take into account other factors that may influence the number of accidents, such as changes in ship design, crew training, or safety regulations.

Figure 8: Response from ChatLlama2-70B

Reviewer paper summaries: This paper introduces the PredictiQ benchmark to evaluate the performance of Large Language Models (LLMs) in predictive analysis. The PredictiQ benchmark includes 44 datasets and 1,130 predictive analysis queries across eight diverse fields, covering common scenarios in predictive analysis. The authors designed an evaluation protocol to assess LLMs' responses in terms of text analysis, code generation, and text-code alignment. Through evaluations on 12 well-known LLMs, the paper reveals the strengths and limitations of LLMs in predictive analysis.

The paper introduce the PredictiQ benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. And the paper designed an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis. It concludes that conducting predictive analysis is still challenging for existing LLMs.

Concise Paper Summary
The paper introduces PredictiQ, a large benchmark for evaluating Large Language Models (LLMs), specifically in predictive analysis tasks. PredictiQ comprises 1,130 queries from 44 real-world datasets across eight diverse domains, employing an evaluation protocol that examines text analysis, code generation, and text-code alignment. The study evaluates twelve LLMs, highlighting their strengths and limitations, and indicates significant performance gaps, particularly in executable code quality, explanatory depth, and computational efficiency.
Summary of strengths: 1. Significance of the Research Problem: The paper addresses the potential application of LLMs in predictive analysis and fills a gap in evaluation benchmarks.
2. Reasonable Methodology: The data collection, query formulation, and evaluation protocol are thoughtfully designed to ensure the comprehensiveness and scientific nature of the assessment.
3. Comprehensive Experiments: The paper evaluates 12 LLMs and provides rich experimental data and analyses, including comparisons of model performance across different fields.

1. Data is collected from data-specific datasets, ensuring its high quality.
2. It involves a sufficient number of LLMs, making it a comprehensive assessment.
3. Human evaluation is employed to indicate how LLMs can align with human preferences.

- **Benchmark Contribution**: Addresses a critical gap by introducing a significant benchmark for predictive analysis in LLMs, with standardized evaluation protocols and metrics.

- **Methodological Soundness**: Integrates human expertise and LLM-based evaluation (GPT4Turbo) across three sub-tasks (text, code, alignment) for balanced assessments.

- **Domain Coverage**: Covers 44 datasets from 8 fields (e.g., healthcare, economics) for real-world applicability.

- **Useful and interesting Insights**: Reveals non-trivial findings, such as code fine-tuning degrading executable code quality in smaller models (e.g., CodeLlama2-7B’s 15% executable rate vs. ChatLlama2-7B’s 18%), and context length’s nonlinear impact on performance.

- **Resource Value**: Open-sourcing the benchmark promotes reproducibility and future research in LLM-driven predictive analysis.
Summary of weaknesses: 1. The concept of "predictive analysis" is unclear. Is this term used in any prior work? Is it just a prediction task? or just an analysis task?
2. Why must LLM generate code in its response in your evaluation? what if there is a LLM which cannot generate code, but good at doing prediction based on the previous data?
3. The dataset lacks ground truth, leading to an over-reliance on LLMs as evaluators, which is restrictive. 
4. The paper focuses on query data but fails to carry out statistics to show the diversity of queries.
5. Why does functional correctness vary from 0 to 4? I thought it has just two cases, authors should give the cases of different scores.
6. The paper emphasizes the role of fine-tuning but does not describe the fine-tuning process or include the fine-tuned LLM’s scores in the tables. 
7. While the context window is discussed later, it is not explicitly stated in Table 2 or other tables.
8. Table 5 discusses alignment with human ratings but does not include the exact human scores.

1. The authors claim that there are three contributions, while the first two contributions seem similar: PredictiQ (1st contribution) comprises  1130 data-specific queries (2nd contribution).
2. Despite the comparison between human experts and LLMs, the correlation between is not provided.
3. No quality control mechanism seems to be conducted. For example, how are human experts chosen? What is the inter-rater agreement among human experts?

- **Cultural Bias in Annotation**: Annotators are exclusively Asian master’s students, potentially skewing query formulation and evaluation of domain-specific responses (e.g., social studies).

- **Potential Evaluator Flaws**: Heavy reliance on GPT4Turbo (90.5% alignment with humans) lacks validation against alternative evaluators (e.g., Claude, Gemini) and omits inter-annotator agreement metrics.

- **Incomplete Failure Analysis**: While identifying high rates of missing preprocessing (51–92% across models), the paper does not categorize error types (e.g., syntax vs. logic errors) or quantify their downstream impacts.

- **Shallow Domain-Specific Findings**: Performance variations across domains (e.g., ChatLlama2-70B’s 31.4% higher scores in Education) are noted but not explained, limiting actionable guidance for domain adaptation.
Comments, suggestions and typos: N/A

Na.

- **Mitigate Cultural Bias**: Diversify annotator demographics and validate queries for cross-cultural relevance (e.g., regional economic metrics).

- **Deepen Error Analysis**: Categorize failure modes (e.g., missing preprocessing vs. logic errors) and quantify their impact on predictive accuracy.

- **Enhance Evaluator Validation**: Compare multiple LLM evaluators and report inter-annotator agreement to strengthen reliability claims.

- **Clarify Domain-Specific Mechanisms**: Investigate why models perform well in certain domains (e.g., education)
Overall assessment scores: 2

3.5

3.5