Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data (2024)

Abstract: Quantitative reasoning is a critical skill to analyze data, yet the assessment of such ability remains limited. To address this gap, we introduce the Quantitative Reasoning with Data (QRDATA) benchmark, aiming to evaluate Large Language Models’ capability in statistical and causal reasoning with real-world data. The benchmark comprises a carefully constructed dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models’ quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRTEXT. We evaluate natural language reasoning, program-based reasoning, and agent reasoning methods including Chain-of-Thought, Program-of-Thoughts, ReAct, and code interpreter assistants on diverse models. The strongest model GPT-4 achieves an accuracy of 58%, which has much room for improvement. Among open-source models, Deepseekcoder-instruct, a code LLM pretrained on 2T tokens, gets the highest accuracy of 37%. Analysis reveals that models encounter difficulties in data analysis and causal reasoning, and struggle in using causal knowledge and provided data simultaneously. Code and data are in https://github.com/xxxiaol/QRData.

heading: 1 Introduction
text: Data is everywhere in our daily life and work, from win-loss charts in sports competitions to insurance databases of national health. We rely on our quantitative reasoning skills to process the data, more specifically, to extract information from the data, draw inferences, and make predictions (Bueno de Mesquita and Fowler, 2021).
There is evidence that current Large Language Models (LLMs) can perform fundamental
∗Corresponding author.
data analysis like visualization and summarization (Cheng et al., 2023; Bai et al., 2023). However, little effort has been made to evaluate the more challenging aspects of quantitative reasoning. On the other hand, there are papers evaluating advanced quantitative reasoning abilities of LLMs (Chen et al., 2023a; Jin et al., 2023a), but they have not investigated whether LLMs can apply these skills to real data.
In this paper, we focus on two main areas of advanced quantitative reasoning: statistical reasoning and causal reasoning, with examples shown in Figure 1. Given a data sheet from sample surveys, statistical reasoning aims to infer the underlying probability distribution, addressing questions like what is a 95% confidence interval for the population mean of y; while causal reasoning aims to understand the causal relationship between variables, addressing questions like what is the average treatment effect1 from t to y.
We propose the Quantitative Reasoning with Data (QRDATA) benchmark. It requires models to answer a quantitative reasoning question given data sheets as input. Our benchmark shares a sim-
1Average treatment effect is a measure of causal effect, quantifying the difference in average outcomes between units assigned to the treatment and units assigned to the control.
9215
ilar form with table question answering (Pasupat and Liang, 2015) and data analysis code generation (Lai et al., 2023) tasks, but has its unique challenges: 1) Our benchmark requires advanced quantitative knowledge and skills. Consider the example in Figure 2. To answer the question, models need to know what are common methods of estimating the average treatment effect and how to use them. To the best of our knowledge, few existing benchmarks touch upon concepts in statistical reasoning with data, and none of them involve causal reasoning with data. 2) Our benchmark requires multi-turn reasoning. After execution of model generated code, models need to read the execution results, and write subsequent code or interpret the results to produce the final answer. In the example, models should first check the data structure before safely indexing the variables in step 2. And after the refutation test in step 4, models need to compare the estimation before and after the refutation, to decide if the estimated effect is reliable, or if another estimation method should be used.
To obtain high-quality quantitative reasoning questions and corresponding data, we construct QRDATA from open-source textbooks, online learning resources, and academic papers that are accompanied by data. This ensures that all the questions are reasonable and match well with the data. Through careful manual construction, our dataset consists of 411 questions along with 195 data sheets. To analyze the difficulty of data analysis, we also collect an auxiliary dataset of Quantitative Reasoning with Text (QRTEXT). QRTEXT
contains 290 questions from textbooks and academic papers, and the questions can be solved based on text descriptions only.
We evaluate a wide range of models and reasoning approaches on QRDATA. To find what is a good solution to the task, we implement natural language reasoning with Chain-of-Thought (Wei et al., 2022), program-based reasoning with Programof-Thoughts (Chen et al., 2022), agent reasoning with ReAct (Yao et al., 2022) and the code interpreter assistants of GPT models. We also compare general-purpose LLMs with LLMs finetuned for specific abilities like math, code, and agent reasoning, to find out whether these abilities help in tackling our benchmark. Experimental results show that the best closed-source LLM GPT-4 (OpenAI, 2023) achieves an accuracy of 58% with the code interpreter assistant, and the best open-source LLM Deepseek-coder-instruct (Guo et al., 2024) achieves 37% with Program-of-Thoughts.
As there is much room for improvement in model performance, we further dive into what are the primary difficulties. We find that data analysis is challenging for LLMs, especially for smaller open-source LLMs. And all models including GPT4 and Gemini (Team et al., 2023) struggle with causal reasoning. They acquire causal knowledge from the pretraining corpus, but can hardly integrate the knowledge with the provided data in reasoning.
Our main contributions are as follows: 1) We propose QRDATA, the first benchmark for advanced quantitative reasoning with data, to assess
models’ abilities of data-based statistical and causal reasoning. 2) We evaluate various models and reasoning approaches on QRDATA, and provide insights into effective approaches to tackle the benchmark. 3) We analyze the difficulties of current models in data analysis and causal reasoning.

heading: 2 Benchmark Construction
text: In this section, we introduce how the QRDATA benchmark is constructed. To ensure the quality of our benchmark, we first gather teaching and research resources for quantitative reasoning, and then annotate questions based on the materials. To compare the performance of quantitative reasoning on data and text, we also introduce an auxiliary dataset QRTEXT for analysis.
Resource Selection. We look for quantitative reasoning resources with the following criteria: 1) The resource is related to quantitative reasoning, especially statistical reasoning and causal reasoning. 2) It contains multiple quantitative reasoning questions with corresponding data and answers. 3) It is open-source and allows adaptation.
Eight resources are selected based on these criteria, including two books: OpenIntro Statistics (Diez et al., 2012) and Quantitative Social Science (Imai, 2018); one online learning resource: Causal Inference for the Brave and True2; and five datasets from academic papers: Flow cytometry (Sachs et al., 2005), IHDP (Hill, 2011), Jobs (Shalit et al., 2017), Neuropathic pain diagnosis (Tu et al., 2019), and Arctic sea ice (Huang et al., 2021).
Question Collection. We collect multiple-choice questions and numerical questions from these resources, and ensure that the gold answer is unique. For textbooks and online learning resources, we extract the list of data sheets from their websites. If a data sheet is used in exercises, we use the exercise question and its solution as the question and answer; if it is used in the main content, we annotate questions and answers based on the content.
For existing causal discovery datasets (Flow cytometry, Neuropathic pain diagnosis, and Arctic sea ice), we collect multiple-choice questions like Which cause-and-effect relationship is more likely? A. headache causes groin pain B. groin pain causes headache C. No causal relationship exists follow-
2https://matheusfacure.github.io/python-causalityhandbook/landing-page.html
ing the style of Kıcıman et al. (2023). For causal estimation datasets (IHDP and Jobs), we collect numerical questions of estimating average treatment effect (ATE), average treatment effect on the treated (ATT), and average treatment effect on the control (ATC) like the example in Figure 2. More details are in Appendix A.1.
We also collect data descriptions like the purposes and formats of the data sheets from the resources, and provide them along with questions to models.
Statistics. We provide the dataset statistics in Table 1, and the number of questions collected from each source in Appendix Table 6. The questions we collect cover a wide range of concepts, and we illustrate some of them in Figure 3. We also summarize the topics and question examples in Appendix Table 8. Most questions in QRDATA are based on a single data sheet, while 4% of questions are based on multiple data sheets.
To check whether our dataset has already been memorized by models in training, we apply the pretraining data detection method Min-K% Prob (Shi et al., 2023) on GPT models and Llama-2, and confirm that our dataset is unlikely to appear in their pretraining corpora.
QRTEXT. To separate the challenge of quantitative reasoning from data analysis and analyze whether models master the quantitative reasoning skills, we create an auxiliary benchmark for com-
parison called Quantitative Reasoning with Text. Questions in QRTEXT can be answered without data, with an example shown in Appendix Figure 4. We collect questions from OpenIntro Statistics and two text-based causal reasoning datasets CLADDER (Jin et al., 2023a) and CORR2CAUSE (Jin et al., 2023b). QRTEXT contains 290 questions, 100 for statistical reasoning and 190 for causal reasoning. For ease of model comparison, the ratio of statistical/causal questions of QRTEXT is similar to QRDATA. Details of QRTEXT question collection are in Appendix A.2.
Evaluation Metric. We report the accuracy by comparing the final answer generated by models with the gold answer. As models may encounter measurement errors in numerical questions (e.g., different causal effect estimation methods will yield minor differences in results), we consider any result within ±3% of the gold answer to be correct.

heading: 3 Methods
text: We develop several zero-shot reasoning methods as our baselines. By comparing these methods, we aim to find a good solution to the task of quantitative reasoning with data.
Table Question Answering (Table QA). The task of table QA asks models to answer a question based on tables. To evaluate if our benchmark can be solved by existing table QA methods, we experiment with a competitive Table QA method (Zhang et al., 2023). It constructs a table instruction tuning dataset TableInstruct, and finetunes models on this dataset.
Chain-of-Thought (CoT). CoT prompting enables models to conduct complex natural language reasoning through intermediate reasoning steps. We feed the content of data sheets to models and prompt models to respond to a quantitative question using the provided data ... Let’s think step by step. If the data sheet is too large, we truncate it to fit the length limit. To obtain the final answer from the model output, we employ ChatGPT for answer extraction following Chen et al. (2023a).3
3To assess the reliability of answer extraction with ChatGPT, we randomly sample 50 CoT outputs from all models, and manually verify the extracted answers. ChatGPT achieves an accuracy of 98%, demonstrating its proficiency in answer extraction.
Program-of-Thoughts (PoT). PoT prompting asks models to generate a Python code program, and uses an external computer to execute the code. The output of the code is regarded as the answer. In contrast to TableLlama and CoT prompting, PoT processes the data sheet with code and gets rid of the length limit of data. It also enables models to call data analysis packages for computation.
ReAct-style Prompting. ReAct is a prevalent prompting paradigm for agent reasoning. It combines reasoning and acting in language models for task solving. To address our task, we restrict the action space to Python code execution, and ask models to generate thoughts and actions. Different from aforementioned methods that only allow one-turn generation, ReAct supports multi-turn reasoning. When a piece of code is generated, we execute it and feed the output back to models as an observation. Based on the observation, models will either generate new thoughts and actions, or decide to predict the final answer.
Code Interpreter Assistants. GPT models can be used as agents (called assistants), and a code interpreter tool is built-in, which could execute Python code in a sandbox to interpret the data. Similar to ReAct-style prompting, code interpreter assistants allow multi-turn code generation and execution, and end with a text-formed answer.
Details of prompt design and prompt examples are in Appendix A.3.

heading: 4 Experiments

heading: 4.1 Models
text: We experiment with the CoT, PoT, and ReAct-style prompting methods on the following models:
General-Purpose LLMs. We select Llama-2chat, Gemini-Pro, and GPT-4 as representatives of general-purpose LLMs. They exhibit strong abilities to follow instructions (Touvron et al., 2023; Team et al., 2023; OpenAI, 2023).
Code LLMs. CodeLlama-instruct (Roziere et al., 2023) is initialized on Llama-2, finetuned on a code-heavy dataset, and instruction tuned on a code-related instruction dataset. Deepseek-coderinstruct (Guo et al., 2024) is pretrained on a projectlevel code corpus from scratch and instruction tuned to follow instructions. Both of them perform well among open-source code models on several code benchmarks.
LLM for Mathematical Reasoning. WizardMath (Luo et al., 2023) is finetuned on Llama-2, enhancing the mathematical reasoning abilities with reinforcement learning. It outperforms many larger models on mathematical reasoning benchmarks.
LLM for Agent Reasoning. AgentLM (Zeng et al., 2023) is instruction-tuned based on Llama-2 to enhance the agent capabilities. It uses a combination of instructions from general domains and instructions of interaction trajectories.
For a fair comparison, we use the size of 7B for all the open-source models.4 We use the two latest GPT models GPT-3.5 Turbo and GPT-45 for code interpreter assistants. More implementation details are in Appendix A.4.
We select TableLlama (Zhang et al., 2023) as a representative table QA model. It is finetuned on Llama-2-7B (Touvron et al., 2023) with TableInstruct, using LongLoRA (Chen et al., 2023b) to extend the context length. It exhibits strong generalization ability across table QA
4The size of Deepseek-coder-instruct is 6.7B. 5The specific versions are gpt-3.5-turbo-1106 and
gpt-4-1106-preview.
datasets. As a reference for assessment, we also report the random baseline and the human performance. The random baseline is calculated by randomly choosing from the multiple choices, and all the numerical questions are counted as incorrect.
The human performance is evaluated by asking three graduate students who have taken quantitative reasoning courses to answer a random sample of 50 questions from QRDATA. Each question is answered by at least two students, and the average accuracy is reported. The students are allowed to use a Python executor, and search for concepts rather than specific questions on the Internet. 6

heading: 4.2 Main Results
text: To investigate which reasoning methods and models better solve our task, we summarize the performance of different models paired with different reasoning methods in Table 2. GPT-4 with the code interpreter assistant achieves the best performance,
6The majority of human errors are in causal discovery questions (71%), because state-of-the-art causal discovery algorithms cannot fully resolve this task (Vowels et al., 2022).
and Deepseek-coder-instruct with PoT prompting is the best among open-source models. The best model is 18% worse than human, showing that QRDATA is challenging for LLMs.
Natural Language Reasoning. CoT works best for several open-source models like Llama-2-chat and WizardMath, but the results are close to or even worse than random, as data-based quantitative reasoning requires heavy calculations that are beyond natural language reasoning. The table QA model TableLlama also underperforms across both statistical and causal tasks.
Models can hardly answer numerical questions with CoT, with a maximum accuracy of less than 10%. A limitation of CoT and other natural language reasoning methods is that in many cases, we cannot feed the whole data sheets into the input, hindering models from performing accurate calculations. We further compare CoT and PoT prompting on a subset of QRDATA where the full data sheets can be fed into the base model, and find that CoT does not exceed PoT on this subset. The accuracy of CoT prompting on numerical questions is below 20% across all models, suggesting these models struggle with performing precise calculations on data sheets using CoT.
Program-based Reasoning. PoT best suits Deepseek-coder-instruct, but to our surprise, another code LLM CodeLlama-instruct does not perform well with PoT. The gap in coding abilities can also be seen from the execution success rate, which is 89% for Deepseek-coder-instruct and 73% for CodeLlama-instruct.
Although the overall performance of several noncode LLMs drops with PoT compared to CoT, the accuracy of numerical questions improves with the help of code execution. This exhibits the potential for performance improvement through the hybrid of CoT and PoT rationales (Yue et al., 2023).
Agent Reasoning. Closed-source LLMs like GPT4 and Gemini-Pro perform best with agent reasoning. GPT-4 performance with ReAct-style prompting is close to with the code interpreter assistant, exhibiting that our ReAct-style prompts successfully activate the model abilities.
Smaller open-source models can hardly follow the complex ReAct-style instructions, which limits their performance. Although AgentLM is finetuned for agent reasoning, it cannot generalize well to the quantitative reasoning task. On the
other hand, open-source models except code LLMs perform better with ReAct-style prompting than PoT, because multi-turn code execution gives them chances to fix the generated code, raising the average execution success rate from 25.5% to 35.6%. We also manually review 50 randomly sampled ReAct outputs of these models, and find that in 10% of the cases, the initially generated code is non-executable, and is fixed in the following steps. This suggests that the iterative, multi-turn nature of agent reasoning can contribute to fixing the generated code.
LLMs with Enhanced Abilities. We compare three models finetuned on Llama-2 for enhanced abilities of code, math, and agent reasoning with the general-purpose LLM Llama-2-chat. AgentLM and WizardMath only gain marginal improvement, and CodeLlama-instruct even underperforms Llama-2-chat. This may be due to the limited capability of the base model.
On the other hand, Deepseek-coder-instruct, which masters in code generation, outperforms all other models with similar sizes by more than 10%. We further evaluate Deepseek-coder-instruct33B to see if it could approach the performance of closed-source LLMs, but it gets an overall accuracy of 31.9% with PoT and 27.0% with ReActstyle prompting, even lower than the 6.7B model. Although coding ability is important in solving quantitative questions with data, the performance of code LLMs is still limited compared to larger general-purpose LLMs like GPT-4.
Statistical Reasoning vs. Causal Reasoning. Statistical reasoning focuses on identifying patterns in data, whereas causal reasoning goes a step further to find causal relationships beyond correlation. For all models evaluated, the gap between model performance and the random baseline is larger for statistical questions than causal questions, showing that models obtain better statistical reasoning abilities than causal reasoning abilities. Even the best-performed model GPT-4 only answers 51% of causal questions correctly, demonstrating the large room for improvement in causal reasoning.

heading: 5 Analysis
text: We go deeper into the primary difficulties models face in addressing the task of quantitative reasoning with data. This may provide insights into how to design methods to better solve our task. We also provide an error analysis of GPT-4 in Appendix A.6.

heading: 5.1 Difficulty of Data Analysis
text: We evaluate models7 on our auxiliary benchmark QRTEXT, and compare the performance on QRTEXT and QRDATA to quantify the difficulty of data analysis. As shown in Table 3, GPT-4 achieves the best accuracy of 59.7% on QRTEXT, and Deepseek-coder-instruct gets the best accuracy of 39.0% among open-source LLMs.
The single-turn reasoning methods, CoT and PoT prompting, work best for all models except GPT-4 on QRTEXT. For GPT-4, the performance gap between the code interpreter assistant and PoT
7We do not experiment with TableLlama as the questions do not involve tables.
is merely 0.7%, compared to 8.8% on QRDATA. These demonstrate that single-turn reasoning is almost adequate for quantitative reasoning with text, while multi-turn reasoning is more necessary for quantitative reasoning with data.
All models perform better on QRTEXT than QRDATA from 1.8% to 11.5%, and the gap is larger for smaller models. If we control the knowledge and skills required by restricting questions to the same source (OpenIntro Statistics), models perform 6% better on average. These provide evidence that most models have difficulty in data analysis.

heading: 5.2 Difficulty of Causal Reasoning
text: We observe the performance gap between statistical reasoning and causal reasoning on QRDATA, and the gap remains on QRTEXT. GPT-4 achieves 89% accuracy in statistical questions on QRTEXT, but only about half the accuracy in causal questions. This exhibits the unique difficulty of causal reasoning regardless of data analysis.
Table 5 provides a case where GPT-4 with the code interpreter assistant mis-predicts a causal question from QRDATA. When asked to predict the causal relation between two variables L L1 radiculopathy and R L5 radiculopathy, GPT-4 proposes a wrong plan for calculating the conditional probabilities in Step 2. As correlation does not imply causation, although GPT-4 successfully executes the plan, it makes a wrong prediction.
Easier Said than Done. The unsatisfactory performance on causal questions appears to differ from the findings of some previous papers (Willig et al., 2022; Kıcıman et al., 2023), which show that LLMs are capable of causal reasoning in tasks like causal discovery without data. To figure out the reason for the inconsistency, we conduct an experiment to evaluate if the provided data interfered with the models. Specifically, we evaluate the best three models on multiple-choice causal questions of QRDATA without giving them the data8, and compare their performance with the original setting. We use the best reasoning method for each model: ReActstyle prompting for Gemini-Pro, and code interpreter assistants for GPT-3.5 Turbo and GPT-4.
As shown in Table 4, two out of three models perform better without data access, revealing that it is easier for these LLMs to speculate causal re-
8Note that this setting is different with QRTEXT. Without data access, questions in QRDATA are not self-contained. Models need to predict the answers with their commonsense.
Question Which cause-and-effect relationship is more likely? Please answer with A, B, or C. A. R L5 radiculopathy causes L L1 radiculopathy B. L L1 radiculopathy causes R L5 radiculopathy C. No causal relationship exists Gold Answer C
lations than to conduct causal reasoning with data. The strong causal reasoning abilities exhibited by models in previous works come from their internal knowledge, but models can hardly use their internal knowledge to analyze the data provided.
In the case of Table 5, in the absence of data access, GPT-4 correctly predicts that there is no causal relationship between the two variables. However, when the data is provided, it is misled to calculate the conditional probabilities.

heading: 6 Related Work

heading: 6.1 Data-related Benchmarks
text: Table Question Answering. Table QA (Pasupat and Liang, 2015; Zhong et al., 2017) asks models to answer questions about information from given tables. It evaluates models’ abilities to parse queries, look up information from tables, and integrate the retrieved information to reach an answer. Datasets like Spider (Yu et al., 2018) and SEDE (Hazoom et al., 2021) focus more on the semantic parsing part and design complex questions that need compositional generalization over SQL queries. Some datasets present the challenges of retrieving information from hierarchical tables (Cheng et al., 2022;
Zhao et al., 2022) or a fusion of textual and tabular data (Chen et al., 2020; Zhu et al., 2021). Datasets like FinQA (Chen et al., 2021) and TABMWP (Lu et al., 2022b) require numerical reasoning to figure out the answer, but they mainly focus on basic arithmetic, while our benchmark requires more advanced quantitative reasoning skills.
Furthermore, although some benchmarks require multi-hop reasoning, they can be solved in a single turn, whereas our benchmark needs multi-turn reasoning between the model and data.
Code Generation. Code generation tasks require models to produce code to meet certain intents. Recent benchmarks like DS-1000 (Lai et al., 2023) and ExeDS (Huang et al., 2022) explore LLMs’ abilities to generate data science code, like training models using machine learning packages and making visualizations. KnowledgeMATH (Zhao et al., 2023) and BizBench (Koncel-Kedziorski et al., 2023) evaluate if models could generate code for financial reasoning with tables. To our knowledge, none of them focus on advanced quantitative reasoning. Only a concurrent dataset DAEval (Hu et al., 2024) involves correlation analysis as a component of question concepts, and none of them
include causal reasoning questions.

heading: 6.2 Quantitative Reasoning Abilities
text: There is a growing interest in exploring language models’ abilities of quantitative reasoning. Various benchmarks focus on mathematical reasoning (Hendrycks et al., 2021; Cobbe et al., 2021), and recent works broaden the analysis to the field of science (Lu et al., 2022a; Chen et al., 2023a).
Another line of work focuses on the specific aspect of causal reasoning of LLMs. Bhagavatula et al. (2019); Qin et al. (2019); Liu et al. (2023) explore the ability of commonsense causal reasoning like abductive reasoning and counterfactual reasoning. Willig et al. (2022); Gao et al. (2023); Kıcıman et al. (2023) analyze the mastery of causal knowledge like cause-effect pairs. Jin et al. (2023a,b) evaluate the utilization of causal inference skills.
In this work, we extend the research to quantitative reasoning with data, which poses new challenges for LLMs.

heading: 7 Conclusion
text: We propose QRDATA, a benchmark of advanced quantitative reasoning with data, to explore LLMs’ statistical and causal reasoning abilities with realworld data. We carefully construct the benchmark with questions from teaching and research materials, and compare natural language reasoning, program-based reasoning, and agent reasoning among a wide range of models. Experimental results reveal that models have difficulties in data analysis and causal reasoning. Even though powerful LLMs like GPT-4 have acquired causal knowledge, they can hardly integrate them with the provided data. By proposing this benchmark, we hope to draw attention to the data-based quantitative reasoning ability. We call for more specialized methods to enhance the ability, like training models in the agent reasoning style with data as input, and equipping models with more quantitative reasoning knowledge.

heading: Limitations
text: Scope of the Benchmark. We collect questions related to two primary aspects of advanced quantitative reasoning: statistical reasoning and causal reasoning in this benchmark. We focus on the explicit utilization of these reasoning abilities, more specifically, statistical/causal reasoning with data. And our benchmark is limited to English questions
and data. We look forward to future benchmarks on a broader domain and other languages.
Models and Reasoning Methods. The field of LLM research is developing quickly. We experiment with representative models and reasoning methods in this paper, but we acknowledge that this does not cover all models and frameworks. Methods of constructing training data and finetuning open-source models like Lumos (Yin et al., 2023) can also be explored to enhance the effectiveness of models.

heading: Ethics Statement
text: Intellectual Property. We follow the permission of licenses during the data collection process. And the collected data should not be used commercially.
Potential Risks. Our evaluation reveals that current LLMs are imperfect in conducting quantitative reasoning with data. We suggest readers not trust solutions provided by LLMs.

heading: Acknowledgments
text: This work is supported in part by NSFC (62161160339) and Beijing Science and Technology Program (Z231100007423011). We thank Da Yin and Chen Zhang for their constructive comments. We thank the anonymous reviewers for their helpful discussions and suggestions. For any correspondence, please contact Yansong Feng.

heading: A Appendix

heading: A.1 Question Collection of QRDATA
text: We provide the number of questions collected from different sources in Table 6. Table 8 summarizes the topics covered in QRDATA, and examples of questions related to these topics.
Causal Discovery. We collect both pairwise causal discovery and full graph causal discovery questions from the Neuropathic pain diagnosis dataset following Kıcıman et al. (2023).
We sample 50 causal relations like L1 radiculopathy → IBS from the gold causal graph, and generate pairwise causal discovery questions asking models to predict the direction of the causal relation. The questions are in the form of Which cause-and-effect relationship is more likely? A. R L1 radiculopathy causes IBS B. IBS causes R L1 radiculopathy.
In full graph causal discovery, there are three possible relations between two variables A and B: A → B, B → A, and no causal relation exists. We sample 25 variable pairs that have causal relations in the causal graph and 25 variable pairs that do not have causal relations, and form 50 questions in the form of Which cause-and-effect relationship is more likely? A. headache causes groin pain B. groin pain causes headache C. No causal relationship exists.
For the Flow cytometry and Arctic sea ice datasets, we only collect the more challenging full graph causal discovery questions. As the causal graph of Arctic sea ice contains double-sided edges, we ask models to predict from four possible choices like Which cause-and-effect relationship is more likely? A. Residual_cloud_cover causes Residual_longwave B. Residual_longwave causes Residual_cloud_cover C. The causal relation is doublesided between Residual_cloud_cover and Residual_longwave D. No causal relationship exists.
Causal Estimation. We collect numerical questions of estimating average treatment effect (ATE), average treatment effect on the treated (ATT), and average treatment effect on the control (ATC) from the IHDP dataset. As all the outcomes and counterfactual outcomes are simulated using real covariates, we calculate the values of these estimates as the gold answers. In the Jobs dataset, all the treatments were from randomized experiments, so we can calculate the true ATT values. We collect questions of estimating ATT from the dataset.

heading: Source Question

heading: Failure Reason Percentage (%)

heading: A.2 Question Collection of QRTEXT
text: QRTEXT contains 290 questions in total, 140 multiple choice questions and 150 numerical questions. We collect 100 statistical reasoning questions from OpenIntro Statistics, 94 causal reasoning questions from CLADDER, and 96 causal reasoning questions from CORR2CAUSE. Figure 4 exhibits an example question of QRTEXT.
The OpenIntro Statistics questions are collected from exercises with solutions from the textbook. CORR2CAUSE contains causal discovery questions of determining the causal relationship between the variables given correlational statements. We randomly sample CORR2CAUSE questions from the original test set.
CLADDER contains causal questions where the conditions are described in natural language. It classifies questions into three rungs: associational (rung 1), interventional (rung 2), and counterfactual (rung 3). We only consider questions of rungs 2 and 3 because these are related to causal reasoning. We randomly sample causal estimation questions querying Average Treatment Effect (ATE), Effect of the Treatment on the Treated (ETT), Natural Direct Effect (NDE), and Natural Indirect Effect (NIE), and transform the original yes-or-no questions into numerical questions using the provided metadata. This is to balance the ratio of multiple choice questions and numerical questions, controlling the random performance to be close to QRDATA.

heading: A.3 Prompt Design
text: Table 9 shows an example prompt for Chain-ofThought reasoning. In real prompts, we feed all the content of the data sheets until it meets the length limit of 3500 tokens. We shuffle the data to avoid the bias towards values in the front. After Chain-ofThought reasoning, we use GPT-3.5 Turbo for answer extraction, and the prompt template is shown in Table 10.
Table 11 shows an example prompt for Programof-Thoughts reasoning. We shuffle the data and provide the first ten lines in the prompt. An example prompt for ReAct-style reasoning is in Table 12, and an example prompt for code interpreter assistants is in Table 13. When the question is based on multiple data sheets, we demonstrate the rows of these data sheets respectively in the prompts.
A.4 Implementation Details
We conduct model inference with temperature 0 and maximum new tokens 512. We limit the maximum reasoning steps to 10 in ReAct-style prompting. Experiments are conducted on NVIDIA A800 GPUs. It takes five minutes to evaluate a 7B model on QRDATA with CoT/PoT prompting on a single GPU, and one hour to evaluate a 7B model with ReAct-style prompting.

heading: A.5 Case Study
text: Table 14 demonstrates the full output of GPT-4 on the example case from QRDATA, and Table 15 demonstrates the full output of GPT-4 without access to the data.

heading: A.6 Error Analysis
text: To have a quantitative view of model failures, we conduct an error analysis for the best-performed
model GPT-4 with the code interpreter assistant on QRDATA. We randomly sample 50 incorrect predicted instances, and annotate their main reason for mis-prediction. As shown in Table 7, 34% of the mis-predictions originate from data analysis. The model may fail to retrieve the correct columns according to the instruction of questions, or misinterpret the data values. 44% of the errors occur in knowledge recall, like choosing a wrong hypothesis test, or using statistical methods to address a causal question. Even if the model recalls the correct knowledge to solve the question, it formulates wrong equations in 16% of the cases. And 6% of errors occur in the calculation. This is also the main reason for errors when humans conduct the task, because state-of-the-art causal methods cannot perfectly reach the correct answer.
Data Description The neuropathic pain diagnosis mainly consists of symptom diagnosis, pattern diagnosis, and pathophysiological diagnosis. The csv file Neuropathic_11.csv contains neuropathic pain diagnosis records in the form of tables of which the row represents different patients and the column represents different diagnostic labels. Question Which cause-and-effect relationship is more likely? A. R L5 radiculopathy causes L L1 radiculopathy B. L L1 radiculopathy causes R L5 radiculopathy C. No causal relationship exists Please answer with A, B, or C. Gold Answer C

heading: Future Work
text: We look forward to future benchmarks on a broader domain and other languages. Models and Reasoning Methods. The field of LLM research is developing quickly. We experiment with representative models and reasoning methods in this paper, but we acknowledge that this does not cover all models and frameworks. Methods of constructing training data and finetuning open-source models like Lumos (Yin et al. , 2023) can also be explored to enhance the effectiveness of models