Mathematical Capabilities of ChatGPT
Authors: Simon Frieder
Luca Pinchetti
Alexis Chevalier
Ryan-Rhys Griffiths
Tommaso Salvatori
Thomas Lukasiewicz
Philipp Christian Petersen
Julius Berner
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: datasets
LLMs
ChatGPT
mathematical capabilities
evaluation
benchmarking
TL;DR: We investigate the mathematical capabilities of ChatGPT using an advanced evaluation scheme, covering graduate-level mathematics.

Abstract: We investigate the mathematical capabilities of two versions of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel evaluation scheme. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., mathlib, the Lean Mathematical Library), current datasets of natural-language mathematics used to benchmark language models either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets test, by using 1636 human expert evaluations, whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT and GPT-4 can be used most successfully as mathematical assistants for querying facts, acting as mathematical search engines and knowledge base interfaces. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if you aim to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!

Mathematical Capabilities of Chat GPT
Simon Frieder*: Luca Pinchetti', Alexis Chevalier", Ryan-Rhys Griffiths',
Tommaso Salvatori Thomas Lukasicwiez* Philipp Petersen Julius Berner
"Department of Computer Science, University of Oxford, Oxford, UK VERSES Al Research Lab, Los Angeles, US
School of Mathematics, Institute for Advanced Study, Princeton, US
*Department of Physics, University of Cambridge, Cambridge, UK Departmentt of Computing and Mathematical Sciences, Caltech, Pasadena, US Faculty of Mathematics, University of Vienna, Vienna, Austria
Research Network Data Science, University of Vienna, Vienna, Austria Insstitute of Logic and Computation, Vienna University of Technology, Vienna, Austria https://ghosts: http://osss..iieerrrrr...o friederrr.or.
Abstract
We investigale the mathematical capabilities of two versions of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by lesting them on publicly available datasets, as well as hand-crafted ones, using a novel evaluation scheme. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., mathlib, the Lean Mathe- matical Library), current datasets of natural-language mathematics used to benchmark language models cither cover only clementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first nal ural-language datasets curated by working researchers in mathematics that (1) aim cover graduale-level mat hematiisss,, (2) provide a holistic overview the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets test, by using 1636 human expert evaluations, whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activitics of mathematicians. We benchmark the models on range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT and GPT-4 can be used most successfully mathematiccs assistants for querying facts, acting as hematiicaall search engines and knowledge base interfaces. GPT- 4 can additionally be used for undergraduate-levv graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of graduate student. Hence, if you aim to use ChatGPT to pass a graduato-level math cxam, you would be better off copying from your average peer!
mathematics but fails on
*Corresponding author: simon. stmon.friederrcs.co ac.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

1
Introduction
Since its release in November 2022, the language model Chat Generative Pre-trained Trans- former (ChatGPPT) has rapidly become a widely known guestion-and-anssen ChatGPT has been referenced in mainstrean media across the globe [1-4] and across all major internet platforms [5, 6]. With similar reactions, the relcase of ChatGPT's successor, GPT-4, followed in March 2023 [7].
dialogue system.
The performance of ChatGPT has been analyzed in a large number of exam-related use cases, with varying degrees of scientific rigor, ranging from detailed studics to anccdotal evidence. Use cases include passing the United States Medical Licensing Eramination (USMLE) [8], scoring highly on the Psychology Today Verbal- Verbal-Linguistic Intelligence IQ Test [9], and answering (and genera bine) Operations Management exam questions that were deemed to be within the scope of a typical MBA curriculum [10], all with a performance that elicited a positive sense of surprise from the authors. In turn, the performance of GPT-4 even surpasses that of ChatGPT on large batch of academic and professional exams [7, Table 1]. Such strong task-related performance indicates that large language models (LLMs) could be frequently used as assistants in many domains.
In this paper, we introduce new dataset called GHOSTS, which measures the advanced mathematical abilitics of LLMs. Using this dataset, we perform a detailed analysis of the mathematical capabilities of ChalGPT two of its versions, the 9-January20023 and the 30-Jamuary22023 version. Note that, according to the release notes, the 30-Jamuary-2023 version should possess "improved factuality and mathematical capabilities" [11]. We further examine the performance of GPT-4 on a smaller dataset called miniGHOSTS, which exhibits statistics similar to the larger GHOSTS dataset. We further make available mieroGHOSTS, which itself is a subset of miniGHOSTS, and is designed to facilitate pre-screen of language models, by incurring minimal human evaluation costs. Our analysis includes but is not limited to testing how many of the skills necessary to do professional mathematics can be emulated by these models. Examples of such skills are the ability to answer computational questions, the ability to complete mathematical proofs that have gaps or missing steps, the ability to solve quest ions that are more focused on deep insights and original solutions, such as those of mathematicn olympiads, and the ability to survey the literature and think across domains. None of the previous benchmarks (see Section 2) cover such a broad range of mathematical abilities.
To achieve the goals outlined above, GHOSTS consists of carefully composed prompts aimed at lesting different aspects of LLMs related to mathematical comprehension; see Section 3. This includes both hand-crafted prompts as well as samples from existing datasets that were devised to test models specifically trained for mathematical comprehension [12, 13).
For brevity, use the expression " "(Chat) GPT to refer collectively to both the ChatGPT and GPT-4 language models. We refer to Appendix C for further details regarding different (Chat)GPT versions.
To evaluate the output of (Chat)GPT, we designed thorough testing methodology, including warning and error codes that represent various possible failure modes of (Chat)GPT. We score (Chat)GPTT responsos, report on the results using this methodology, and compare (Chat)GPT to a selection of state-of-thh-art models trained for mathematical comprehension. In summary, the contributions of this article are threefold:
We introduce a new natural-language mathematics dataset, called GHOSTS2, to test the capabilitics of LLMs across range of aspects regarding advanced mathematical comprehension: see Section 3. It consists of two subdatasets derived from state-of-the-art datasets of mathematical queries for language models. Additionally, we devise four hand-crafted subdatasets covering further mathematical tasks. Parts of our dataset consist of problems that were selected have a high probability of not being in the data on which (Chat)GPT was trained; see tags D1-D3 from Table 1.
Benchmark for testing the mathematical capabilities of LLMs:
'The GHOSIS dataset is available at github. wwwp///ieeerr////OOOOOOO and the project is hosted at ghosts. friederrr org.
2

---

Insight for mathematical use of (Chat)GPT: Based on our benchmark, we show for which types of questions and which domains of mathematics, (Chat)GPT may be useful and how it could be integrated into the workflow of a mathematician. On the other hand, we identify the failure modes, as well as the limits of its capabilities. This can aid future efforts to develop LLMs that perform better in mathematics. Our analysis is akin to a mathematical model card [14] in terms of intended use, metrics, evaluation data, and qualitative analyses, mathematical strengths and weaknesses of (Chat) are summarized; see Section 4.
Evaluation of improvements of (Chat)GPT: We can further use our benchmark to track the mathematical capabilities of (Chat)GPPT) variants over time. As first step, we analyze the impact of the upgrade from the 9-January- 2023 to the 30-January-2095 version of ChatGPT, which promiscs better mathematical performance according to the release notes. Then, we procoed to investigate what performance increases the successor GPT-4 brings; see Section 4.1.
2 Related Work
As a language model, (Chat)GPT can be universally employed to perform hamaniiic reasoning and, therefore, has to compete with technologies in this space that are sometimes decades old. Performing mathematical reasoning in an automated way has long history and can be traced back to 1959 [15], the most focus being devoted to proving theorems [16]. According to Harrison [17], there rcalization that classical approaches, using symbolic encoding of mathematics, have reached "plateau".
On the other hand, there is now a growing body of literature on learning mathematical relationships directly in a supervised-learning manner [18-20] or by using LLMs to perform mathematical reasoning directly on mathematics encoded in natural language [21]. Sometimes, the distinction is blurred, because architectures of LLMs can also be used in a supervisod- learning setting and have been employed successfully in learning mathematical relationships, such as bet ween the syntactical form of a function and its integral [13, 22).
Among the supervised approaches, we mention [13], where a Transformer architecture [23] was used to generate symbolic, closed-form solutions to integrals and first and second-order differential equations, which outperformed classical solvers", such as Mathemattca,, MATLAB, and Maple by at least 14% a test set of integration problems. On the task of solving differential equations, the Transformer-based approach still exceeds the classical approach, but by smaller margin (at least 4% in the case of first-order differential equations and with more varied results for second-order equations).
Regarding LLMs, recent ones, for instance, PaLM [24] (released in 2022), are tested only elementary-level mathematical reasoning datasets, such the MathQA or GSMSK datasets [25, 26]. We hypothesize that this is due to a lack of advanced-level natural language mathematics datasets. Moreover, the results obtained indicate that the models at that time had difficulty with much simpler datasets than ours. For example, the version of PaLM with 540 billion parameters only correctly solves 58% of the problems of the GSMSK dataset, even with chain-of-thought prompting and access to an external calculator [24, Table 10]. This model nonetheless outperforms GPT-3 [27], which only achieves 54% on the same dataset. Variations of BERT [28] have been shown to only solve between 28% and 37% of the problems when fine-tuned and tested on the Algebra Question Answering with Rationales (AQuA-RAT)) dataset [29], which is the direct predecessor of MathQA. For some models, such as BLOOM [30] or the LaMDA model [31] (both released in 2022), an evaluation of the mathematical reasoning capability is entirely missing. An up-to-date survey on mathematical datasets and the performance of various LLMs can be found in [32].
Most similar to our datasct is the NATURALPROOFS datasct [33] and the NATURALPRROPS:
GEN dataset [34]. In this paragraph, we illustrate the similarities and differences between these datasets and ours. NATURALPROOFS and NATURALPROOFS-GGE are similar among
'For a given prompt, the computer algebra system is considered to have failed if it does not provide closed-form solution or times out after 30 seconds (in case of Mathematica).
3

---

themselves and cover graduate-level mathematics by focusing on data from ProofWiki (the latter dataset), as well as on the Stacks Project" and two open-source textbooks (the former dataset). Using the TATEX source code, which is available for all these resources, annotated theorems and their proof graphs are extracted. The annotations consist of reference graphs highlighting references to other theorems or definitions, the idea being that these references capture the "skeleton" of a proof. This task resembles the mathematical abilities that the Named Theorem Proof Completion subdataset from the GHOSTS dataset evaluates (see Table 1), although (1) we only retrieve a single reference, and (2) (Chat)GPT,, as far as known, does not use training objectives that make use of information from data annotation, in contrast to models evaluated in [33, 34].
Our framework pertains to general language model evaluation, which may be presented in black-box manner (as is the case for (Chat)GPT), and therefore does not allow to leverage any additional information, such as reference graphs. This is also reflected in the human evaluation schemn introduced in [34, Table 24], which classifies common model mistakes. As reference graphs form the foundation of how the mathematical proofs engineered, many elements of the evaluation schema are strongly tailored toward this representation of mathematical data. Our benchmark is not reference-centrrcc and therefore allows evaluations of any type of proof (inchuding computations, as featured in the -nnteritinnneetttiinn subdataset, which consider to be particular kind of proof). Therefore, our methodology includes further and more general failure modes to make for a more fine-grained evaluation that explains the nature of the errors. We refer to Appendix A for further related works.
The GHOSTS, miniGHOSTS, and microGHOSTS Dataset
We assess the mathematical reasoning capabilities of two ChatGPT versions, 9-Jamuary-2023 and 30-Jamuary20023 and of GPT-4 by first creating a collection of 709 prompts from varions sources, and subsequently evaluating the models on (subsets of) these data points. We rate the corresponding outputs provided by the models and collect statistics, such as error types, output lengths, or the stability of the answer under prompt engineering, see Sections 3.2 and and Appendices B and D. This yiclds a total of 1636 ratings by human experts.
We divide our dataset, the entire collection of prompts, into six subdatasets, called
Grad Teat
Symbolic Integratinn
H ooles-in-Proofs,
MATH,
Soompiad-Problem-SS5
Bearch-Engine-AAe
which, in turn, consists of multiple files, see Table 1. The boldface letters make up the GHOSTS acronym. Details on motivation, composition, collection process, and in- tended uses of the GHOSTS dataset are summarized in our datashect in Appendix H, Sections H.1, H.2, H.3, and H.5, respectively.
GPT-4 was evaluated on a subset of 170 prompls, which we call the miniGHOSTS dataset. Specifically, after having created the GHOSTS dataset, we heuristically selected subset of 10 prompts from each file of the subdatasets included in GHOSTS, having the same mean rating and the same standard deviation (of ChatGPT's output) as the original file; see also our datasheet in Appendix H for more information. In this sense, these subsets can be considered have the most relevance by capturing the "essence" of the model performance in the respective file. The role of miniGHOSTS is thus to reduce the costly evaluation procedure of the full GHOSTS dataset on a new language model. The microGHOSTS dataset represents further reduction of the miniGHOSTS dataset, where one question was extracted from each file making up the miniGHOSTS dataset, for a total of 14 questions. The role of microGHOSTS is to allow rapid pre-screening of a. language model, where each of the 14 questions was chosen to be question representative the mathematical problems in that file, as well as being a problem that (Chat)GPT typically struggled with. Reference solutions, explanations, and known LLM failure modes are provided and discussed for the microGHOSTS dataset in order to aid raters who are not mathematically trained. For more information, see Appendix E.
"nttps://proofwikii org/
http://wwww.eeeeetctct/ttttaaaaa projje
4

---

Table 1: A summary of all the files from all the subdatasets comprising our GHOSTS dataset, together with their size, i.c., the number prompts and their associated attribute tags. The tags Mř, Qi, and Di relate to the level of Mathematical difficulty, the Question type, and the Out-of-Distrbuution type from Section 3.1, respoctivcly. For the Olympiad-Probenn Problem-Solvingg subdataset, 24 further prompts were created where prompt engineering was applied, see Appendix 4.2. These 24 prompts do not count towards the 709 total prompts, only towards the 1636 evaluations.
Subdataset Name
Size
Comprised of the joon file(s)
Tags
Grad- Text
W. Rudin, Functional Analysis (ch. 1)
M3 Q4
15
W. Rudin, Functional Analysis (ch. 2)
M3 Q1
37
J. Munkres, Topology (ch. 1)
M3 Q4
29
J. Munkres, Topology (ch. 2)
M3 Q4
21
R. Durrett, Probability Theory
M3 Q4
oles-in- oles-in-Proofs
Proofs Collection
M3 Q1 Q2 Q5
52
Proofs Collection B Prealgebra
M1 Q5
50
Proofs Collection B Precalculus
M1 Q5
Obimpiad- Praiidd-rooolee---ee Solving
101-244
Olympiad Problem Solving
Q4 D2
S yymbolic-nntegraaten
100
Symbolic Integration
M2 Q3 DI
MATH
50
MATH Algebra
MI M2 M3 Q3 Q4
MATH Counting and Probability
M2 M3 Q3 Q4
18
MATH Prealgebra
M1 Q3 Q4
20
MATH Precalculus
M1 Q3 Q4
Search- Seeel--nnnnn-snsee
Definition Retrieval
M3 Q2 D3
30
Reverse Definition Retrieval
M3 Ql Q2 D3
Named Theorem Proof Completion
M3 Q2 Q5 D8
3.1 Subdatasets
The subdatasets that make up our GHOSTS dataset summarized in Table 1. In the following, we describe cach subdatasct in more detail.
Grad-Text. This subdataset consists of a collection of books (R. Durrett's Probabilily Theory [35]. J. R. Munkres Topology [36] and W. Rudin's Functional Analysis [37] that are widely used in universities to teach upper undergraduate or first-year graduate courses in a degree in mathematics. We have used most of the exercises from these books' first and second chapters, except for [35], where we only used exercises from the first chapter, which was longer than the other books' chapters.
Doles-in-Prroff. This subdataset consists of a number of proofs sourced from math. stackexchange. com, well as some proofs sourced from books (S. Axler's Lônear Algebra Done Right (38) and W. Rudin's Principles of Mathernatical Analysis [39]), and from the MATH dataset [12], where parts of the proofs were intentionally deleted and the LLM was prompted to fill in the gaps: This was done either by (1) using a MISSING token, (2) finishing the proof early and prompting the LLM to complete it, or (3) explicitly asking for certain conditions or results.
This subdataset consists of a selection of exercises from A. Engel's Problem-Soloing Stralegies [40] book, which is often used to prepare for mathematical competitions. We selected and graded the LLM outputs on one hundred exercises drawn from all chapters.
Seempiad-Problem-----
Symmolle-Inteeration. This subdataset consists of random samples of integrals from the test set of [13]. There are three ways in which integrals are generated in [13]: Forward generation (FWD), Backward generation (BWD), and Backward generation with integration by parts (IBP). We sample 21 integrals from FWD test set, 20 integrals from the BWD test set, and 59 integrals from the IBP test set. As these integrals are given in Polish/ prefix notation, natural-language prompt conversion of them is unlikely to be witnessed in the training dataset of (Chat)GPT, The assessment was done by verifying the correct of the output both by using Mathematica, as well as making use of the provided solutions
5

---

(in Polish notation), which [13] generated using SymPy. In particular, we notice that all integrals in this dataset have solutions that can be expressed using elementary functions.
MATH.
This subdataset consists of random sample of problems from the MATH datasct [12]. The latter dataset attaches a level difficulty to cach problom. We focused on two domains, Algebra and Probability Theory, and sampled an equal number of problems at each level of difficully.
Bearch-Engine-Asppe Aspects. This subdataset consists of problems that were not sampled from a particular source but generated by human expert in the field. In the file Named Theorem Proof Completion, we focused on prompting the LLM to provide proof outlines of various theorems that are sufficiently well-known within Functional Analysis to have names. In the Definition Retrieval file, we askod the LLM to correctly state various definitions centered around Functional Analysis and Topology. In contrast, in the Reverse Definition Retrieval file, we verified whether the LLM was able to deduce the name of a mathematical object by describing its properties.
purely textual (at the time of writing), certain types of questions that might be stated and solved in non-text-based fashion (e.g., questions involving graphical diagrams, without text explaining the diagram", 2, as occasionally occur in [40]), have been excluded. Our subdatasets can be categorized along the following dimensions (see Appendix B.1 for more details):
Because input to (Chat)
Mathematical difficulty (ascending): (M1) Elementary arithmetic problems, (M2) Symbolic problems, (M3) problems.
exercises, (M4) Mathematical olympiad
Question type: (Q1) Stating mathematical facts, (Q2) Overview-type review questions, (Q3) Computational questions, (Q4) Theorem proofs or puzzle solutions, (Q5) Proof- completion questions.
Types of high out-of-distrrbuutio likelihood: (D1) Nontrivial problem encoding, (D2) Succinct solution, (D3) Spoken dialogue.
The existing datasets of natural-language mathematics are far from covering all possible combinations across these dimensions. In our well-crafted GHOSTS datasets, we have striven to cover each of these aspects individually, as can be seen in Table 1. The next section specifies the format of our dataset and the methodology for analyzing (Chat)GPPT's output.
3.2 Format
The format of each of the subdatasets that make up our GHOSTS dataset follows the same convention. Each subdataset consists of JSON-formatted files, and our format is similar to, e.g,, the AQuA-RAT dataset [29]. single data point? 7 in a file has the following form:
"prompt": "Let $X$ be topological vector space. All sets mentioned below understood to be the subsets of $x8. Prove the following statement: If $A$ and $B$ are compact, so is $A + B$", "output": "The statement is wrong in general. Consider the example = [-1,1] \times {0}$ and $B = {(} Atimes [-1,1]$. Then $A$ and SBS are compact but $A + = [-1,1] \times [-1,1]$ is not compact."
"rating": "2",
"errorcodes": ["e3", "e5_2", "e5_4"]
"warningcodes": [],
"comment": "The given $A + B$ actually *is* compact.", "mac": ["46A03"],
"ref":
Analysis-Secone Part1-ex3/ /d-ppeee
See, e.g., Exercise 15 in [40, Chapter 2], which asked the reader to inspect a figure which the problem is based.
version of ChatGPT, as identifiable by "The JSON object of an output of the 30-January
the timestamp at which the output was generated, is shown. The prompt comes from the "W. Rudin, Functional Analysis (ch. 1)" file from the Grad- Teat subdataset.

---

"confidence": " "high". , "timestamp": -2023-01-31"
We require each data point to have the same JSON keys as in this example, some of which may be empty depending on the prompt. Among the listed keys, the rating key stands out as the most fundamental one. Its value serves as a condensed representation of the mathematical capability of the tested language model, compressed into a one-dimensional measure ranging from 1 (lowest) to 5 (highest), A more nuanced and fine perspective on the mathernatical capabilities is provided by the errorcodes and warningcodes keys. The msc key denotes the mathemalics subject classification. We explain each JSON key in Appendix B.2. For end-users of (Chat)GPT, it is desirable to avoid having a dialogue to arrive at solution. Therefore, require that (Chat GPP provides us with the correct solution given only the imput prompt without any subsequent interaction.
3.3 Human Input in Dataset Creation and Mathematical Evaluation
For all data points, the values of the keys rating, errorcodes, warningcodes, comment, and confidence were manually labeled, without any automation. The msc, ref, and t timestamp keys were populated in semi-automate way, since their values change only slightly within the same subdataset.
Two of the subdatasets, the MATH subdataset and the Symbolic-Integratte prompts taken from existing datasets, the MATH dataset by [12] and the dataset comprising integrals from [13), respectively. This was done to compare how (Chat) performs against existing statc-of-the-art models that use these datasets, see Section 4. Nonetheless, significant additional annotation effort was involved, since, in both cases, the authors rated the output. Furthermore, in the sccond case, the data are publicly presented in a Polish notation format, and conversion was necessary. The prompts of the other subdatasets were hand-crafted by the authors. We refer to Appendix B.6 for more information on different aspects of human elfort in datasel creation.
subdataset, use
4 Results
Will ChatGPT get you through university math class? No, you would be better off copying from your average peer wurl-sesss it is undergraduate mathematics, for which GPT-4 can offer sufficient (but not perfect) performance.
If we take a rating of 5, the average between the lowest and highest. rating, to be the threshold between success and failure, then Figure 1 shows that for a majority of subdatasets, both versions of ChatGPT will not pass. However, for GPT-4, the situation is different, and, on miniGHOSTS, it passes (sometimes barely) on all subdatasets files, except W. Rudin, Functional Analysis (ch. 2), which tests graduato-leve mathematical knowledge and the Olympiad Problem Solving file, which tests mathematical problem-solving skills. We note that, unless otherwise stated, do not use prompt-cngineere questions in the results presented here (see Appendix 4.2).
We first focus the results of the Jlamuary-2023 version of ChatGPT and note that the results for the 30-January- average, the 1-Jamaary-2222 version achieves rating of 3.20 with standard deviation of 1.23. It performs particularly poorly on proof-based questions in the style of graduate-lovel excrcises mathematical olympiads, well more complicated symbolic calculations. We note that prompt engineering only slightly improved the results for such complex questions; see Appendix 4.2. However, in tasks that only required filling in gaps or stating mathematical facts, ChatGPT was mostly able to achieve a score above 3.5. In particular, ChatGPT was strong at recognizing the context of the question, and the notation of the output almost always matched the one given in the prompt, see Figure 4 in the appendix. Generally, Figure indicates that the ratings closely correspond how mathematicians would rank the difficulty of the exercises. In this context, we note that the length of the prompt does not have a clear effect on the rating; see Figure 9 in the appendix. We present results for different mathematical fields in Figure 5 in the appendix. For a detailed qualitative analysis
20222 2023 are very similar, as can be inferred from the figures. On
We use Bessel's correction term to obtain unbiased estimate of the variance.

---

Grad-Text
Version
W. Rudin, Functional Analysis (ch, 1)
9-Jan.
30-Jan.
W. Rudin, Functional Analysis (ch, 2)
GPT-4
J. Munkres, Topology (ch, 1
J. Munkres, Topology (ch. 2)
R. Durret, Probability Theory
Holes-in-Proofs
Proofs Collection
Proofs Collection Prealgebra
Proofs Collection B Precalculus
Subdataset
Olympiaa-PPobbem--Sollee Problem-Solving
Symbolic-Integration
MATH
MATH Algebra
MATH Counting Probability
MATH Prealgebra
MATH Precalculus
Search-Engine-Aspects
Definition Retrieval
Reverse Definition Retrieval
Named Theorem Proof Completion
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Rating
Figure Average rating for each file in each subdataset (bold) GHOSTS on the 9-January- 2023 and the 30-January-2023 versions of ChatGPT and for miniGHOSTS on GPT-4. Note that the maximal ranking is 5 and the minimal ranking, where the question was at least understood, is 2, Appendix B.4; the lower rating of 1 indicates that the answer completely misses the question. Thus, a reasonable passing grade, i.c., 50% of points, corresponds to score of 3.5, indicated by the vertical dotted line. The error bars represent 95% confidence intervals.
of the results on the different subdatasets, we refer Appendix D.1. Finally, we note that (Chat)GPT almost never expressed any form of uncertainty, even if its output has been completely wrong; see Appendix D.2.
Comparing ChatGPT to the performance obtained by [13), who correctly solved nearly 100% of the integrals in collection of 500 test equations [13, Table 3), the 9-Jamuary20023 version of ChatGPT achieves an average rating of 2.51 (standard deviation: 0.87) on our random sample of their dataset (after conversion from Polish notation to INTEX). Specifically, rating of 2 is dominating 70% of the time, followed by a rating of 3 and 4 for 13% of the prompts each; see also Figure 7 in the appendix. GPT-4 achieves an average of 3.50 (standard deviation: 1 1.43), barely passing grade, on the corresponding subset from miniGHOSTS. These scores trail lar behind the performance achieved by the model [13). The situation is similar when comparing GhatGPT to Minerva [21, Table 3). Their best model achieved
an accuracy of 50% on the MATH dataset. [12]. However, the 9-Jamuary-2023 version of
ChatGPT achieves perfect score only on 29% of our random samples from the MATH

---

9-jan.
30-jan.
GPT-4
Figure 2: A Sankey diagram of how the ratings evolve from 9-January- 2023 ChatGPT to 30-Jamuary- 2022 ChatGPT to GPTT4 (from top to bottom), with all models evaluated on miniGHOST. While grades on the 9-Jamuary and 30-Jamuary models are shuffled wetween the ChatGPT versions, the overall performance remains approximately the same. However, we observe a significant increase in perfect ratings, i.c., a score of 5, for GPT-4.
dataset (which is above the total average of 25% of data points across all subdatasets in which this version achieves a perfect score), see Figures and 7 in the appendix. In contrast, GPT-4 performs substantially better and obtains score of 5 70% of the corresponding questions within the miniGHOSTS dataset, see Figure 7 in the appendix.
4.1 Quantitative Comparison of (Chat)GPT Versions
The ensuing model version, 30-Jamuary-2023 overall performed similarly with an average rating of 3.29 (standard deviation: 1. 1.2 although performance was inconsistent across subdatasets and on some subdatasets marginally better, see Figure 1. A significant jump in performance could only be observed for GPT-4, which achieved a substantially higher average rating of 4. (standard deviation: 1. 12) We note that the evaluation of GPT-4 is only the miniGHOSTS dataset, i.e,, subset of GHOSTS. Nonetheless, these findings send a clear message that the performance of GPT-4 dominates the performance of ChatGPT (both versions), see Figure 1.
Figure 2 shows how the ratings change between the different versions of (Chat) Surpris- ingly, one can see a shuffling of the grades for the two ChatGPT versions, even though the counts in each grade bracket stay approximately the same. For instance, there are roughly the same amount of outputs that received grade 4, yet less than half of the prompts were the same between model changes, Appendix D.4 provides different perspectives on this and reinforces the mixed performance increase that the 30-January2202) model brings. For GPT-4, we see that the percentage of perfect ratings almost doubles, while the percentage of prompts that are not understood or completely wrong (i.e., ratings of 1 or 2) approximately halves as compared to the ChatGPT versions.
Analysis of (Chat)GPT's output and our warning codes reveal that GPT-4 provides even longer ("rambling") answers, whereas ChatGPT usually answered the question without giving any additional context about the lopic, see Figures 6 and 8 in the appendix. The answer style of GPT-4 was often beneficial (resulting in better overall scores) but sometimes reduced the readability of the output. Furthermore, we found the behavior of GPT-4, compared to ChatGPT, to be more opinionated, Finally, despite its better overall performance, GPT-4 still seems to be vulnerable to mistakes in seemingly simple calculations. We refer the reader to Appendix D for further results the models' performance.
4.2 Prompt Engineering
One interesting finding of our study is related to performing prompt. engineering mathe- matical questions. Prompt engineering was solely carried out on questions from the Olympiad- Problem-Soleing subdataset, and prompt-engineered questions consist of lists consisting of

---

two JSON objects. These lists contain the original question that was not prompt-engineered as well as the prompt-engineered question. The latter question is identified, as contains the string <prompt engineered> as the value in the comment key. These lists containing prompt-engineered questions are in the same hierarchy in the JSON file as the other questions from the subdataset.
About 20% of the questions were prompt-engineered: In these cases, ChatGPT was instructed to proceed either step-by-step. by prefixing the sentence "Let's answer this question step by step." or by adding words that formulate the mathematical task in a more explicit way, i.e., by adding "Prove that..." or "Show that..." to the prompt'. Instructing ChatGPT to proceed step-by-step in this way was shown to increase the performance of GPT-3 on datasets that test mathematical reasoning (e.g., GSM8K); furthermore, this is a type of prompt engineering that recommended by OpenAT in their cookbook to improve reliability
As a result of prompt engineering, for the 9-Jamuary-2023 version of ChatGPT, the number of wrong statements and computations (i.c., error codes e2, e3, and e4) decreased, while the number of errors rooted in faulty logic (i.e., error code e5) actually increased. Overall, prompt engineering improves the average rating only slightly, see Figure from the Appendix.
For the questions from Olymppia-Proolemm-- Problem-Solvingg that were selected for the miniGHOSTS dataset, allow to sample from the entire Olympiad- Probiem-Solving subdataset, since the goal of miniGHOSTS is not to measure prompt-engineering effects. Therefore, some of the questions in the miniGHOSTSS version of the Olympiad-Proolem- Troblen-Solving subdataset contain prompt-engineerdd questions. The <prompt engineered> string was therefore removed from the comments in the miniGHOSTS dataset.
Conclusion
We have examined the behavior of (Chat)GPT across various tasks that test different aspects of mathematical skill. Contrary to the media sensation that (Chat)GPT has caused, (Ghat)GPT is not yet ready to deliver high-quality proofs or calculations consistently. At the same time, the quality of the answers can be positively surprising. Morcover, our evaluation of GPT-4 on the miniGHOSTS datasct reveals promising improvements over GhatGPT's performance. In Appendix G, we collect the best and worst results for a number of selected subdatasets. The best responses can be seen to justify the media sensation. It thus seems fair to say that (Chat GPT is inconsistently bad at advanced matherattccs: While its capabilities generally drop with the mathematical difficulty of prompt, it does give insightful proofs in a few cases.
However, (Chat) falls short of achieving the same performance models specifically trained for single tasks. These models, in contrast, lack the flexibility of (Chat) Chat))GPT, which is universal tool suitable for any area of mathematics. In fact, (Chat)GPTTs ability to scarch for mathematical objects, given information about them, is where it shincs. For a user that is already sufficiently mathematically proficient to discern the correctness of (Chat)GPT's output, (Chat) GPT can be intograted as an assistant in the user's workflow. It can function as search engine or knowledge base to speed up various lookup tasks, as they often occur at certain stages of mathematical research.
Due to the prohibitive annotation effort, the GHOSTS dataset is not yet. large enough to significantly improve the mathematical capabilities of LLMs by fine-tuning them on GHOSTS; though we believe it is sufficiently comprehensive to allow evaluation and comparison of LLMs (and more rapid evaluation using miniGHOSTS and microGHOSTS, respectively). We encourage other rescarchers to mine our dataset beyond the descriptive statistics that we computed to gain a deoper understanding of how LLMs behave on mathematical tasks. Finally, hope that our work motivates other mathematicians to contribute to this growing Geld, by evaluating their LLMs on micro/minii/ /GHOSTS in order to establish thorough benchmark for assessing the mathernatica abilities of LLMs.
Some prompts (e.g., the ones taken from the book by Engel [10]) only contain a mathematical statement, without clear instruction; for example, " "An b rectangle can be covered by 1 rectangles iff n|a or nlb. From the context, one must conclude that this statement is correct and should be proven." "eggthuu.coot httpo///wa..o/ooooooooooo//////. ttwpptttott________eee cewproveeeliabbittyy.. .d
10

---

Acknowledgments
This work was partially supported by the AXA Research Fund. The authors would like to thank Karan Desai for helpful discussions and advise on the datasheet.
References
[1] Sascha Lobo. Das Ende von Google, wie wir es kaunten. Der Spiegel, Retrieved 2023- 01-10. https://wwrr wpw.gellww....lllleeeeee...eelle heeeiddd///ee
[ John Naughton. The ChatGPT bot is causing panic now - but it'll soon be mundane a tool as Excel. The Guardian, Retrieved 2023-01-14. settp://wwwe
http://wttwgaa.iaa... com/ httmattt...tteeee/0/////////aaaa hacpapp--bbb---ee---------------
3] Kevon Roose. The Brilliance and Weirdness of ChatGPT. The New York Times, Retrieved 2023-01-24. https://wwwwnttimes.co cctp://////11///////c/o....//// http://///ecccoooooo//ohtttgt chatgpt- ai-twitter. html.
[4] Joe Rogan and Bret Weinstein. What ChatGPT Could Mean for the Future of Artificial Intelligence [Podcast episode). In The Joe Rogan Experience. Episode 1919, Retrieved 2023-01-05. https://www. youtube...bbb........ com//ratcad/vaahaa772777
5) teddy [Steddynpe]] 1 made ChatGPT take a full SAT test. Here's how it did: [Image attached] [Tweet]. Twitter. Retrieved 2023-01-13. https://txitter wwt.cooo////o//e tooo//tddypp///////// (taatss1159866738888888
66 Timothy Gowers [@wtgowers]. It's amusing when ChatGPT makes ridiculous math- emotical mistakes. But of course, it's more interesting to find out what it do well Here's 's one eromple that wasn't bad: gave it a very rough outline of a proof and asked it to fill in the details [Tweet]. Twitter. Retrieved 2023-01-13. https: //twitter. com/ http//ttwwwss//eeeeeeeeeeseeeese http//satusst/t1/ /
[ OpenAT (2023). GPT-4 technical report. arXiv preprint or xri:222003.00 08774, 2023.
[8] Tillany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, and Lorie De Leon et al. Performance of ChatGPT on USMLE: Potential for Al-Assisted Medical Education Using Large Language Models. medRaiv, 2022.
[9] David Rozado. What is the IQ of ChatGPT? Retrieved 2023-01-09. https://
davidrozado. substtck..kukk..k.ookkk www.ssacc.ccmm///aeeeee...e..eee
[10] Christian Terwiesch. Would Chat GPT3 Get a Wharton MBA? Prediction Based Its Performance in the Operations Management Course. Retrieved 2023-01- 04. https://mackinstiiuuee wuteettttttttt..e..eeeeeeeeee wtwpsettttttttttttttttotmo upenn. httpeencce/ppppcccttee////paaaaa Prristtann-erriiess---h--------
[11] Natalie. ChatGPT Release Notes. Retrieved 2023-04-03. tttps://help.openal com/ an/articless68824555 attpaaiclee////--e--aaaeereeeeee
[12] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021.
[13] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXio preprint Xix::0112.01411 2019.
[14] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit. Gebru. Model cards for model reporting. Tn Proceedings of the conference on fairness, accountabilily, and transparency, pages 220-229 2019.
11

---

[15] A. L. Samuel. Some studies in machine learning using the game of checkers. IBM Journal of Reseorch and Development, (03):210-229, 1959.
[16] Jörg Denzinger, Matthias Fuchs, Christoph Goller, and Stephan Schulz. Learning from previous proof experience: survey. Technical report, TU München, 1999.
[17] John Harrison, Josef Urban, and Freek Wiedijk. History of interactive theorem proving. In Computational Logic, volume 9, pages 135 2014.
[18] Malik Amir, Yang-Hut Hui He, Kyu-Hwan Lee, Thomas Oliver, and Eldar Sultanow. Machine Learning Class Numbers of Real Quadratic Fields. arXiv preprint XXXx:22290009550 2022.
[19] Alex Davies, Petar Veličković, Lars Buesing, Sam Blackwell, and Daniel Zheng et al. Advancing mathematics by guiding human intuition with AI. Nature, 600(7887):70-74, 2021.
[20] Yang- Hang-Hui u He. Machine-learning the string landscape. Physics Letters B, 774:564-568, 2017.
[21] Aitor Lewkowyez, Anders Johan Andreassen, David Dohan, Ethan Dyer, and Hen- ryk Michalewski al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, 2022.
[22] Francois Charton, Amaury Hayat, and Guillaume Lample. Learning advanced mathe- matical computations from examples. In International Conference on Learning Repre- sentations, 2021.
[23] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[24] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, and Gau rav Mishra et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
[25] Aida Amini, Saadia Gabriel, Shanchuan Lin, and Rik foonel-Kedzione et al. MathQA: Towards interpretable math word problem solving with operation- based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, pages 2357-2367. Association for Computational Linguistics, 2019.
[26] Karl Cobbe, Vinect Kosaraju, Mohammad Bavarian, Mark Chen, and Hcewoo Jun al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110 2.1.11410.144.. 168, 2021.
[27] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, and Jared Kaplan et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1901, 2020.
[28] Piotr Pickos, Mateusz Malinowski, and Henryk Michalewski. Mcasuring and improving BERT's mathematical abilitics by predicting the order of reasoning. In Proceedings of the 59th Annual Meeting of the Assuciation for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 383 394. Association for Computational Linguistics, 2021.
[29] Wang Ling, Dani Yogalama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 168-167.60 Association for Computational Linguisties, 2017.
[30] Teven Le Scao and Angela Fan et al. BLOOM: A parameeree open-access multilin- gual language model. arXiv preprint ar Xff:22211.0000000 2022.
12

---

[31] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, and Apoorv Kul- shreshtha al. Lamda: Language models for dialog applications. arXiv preprint arXio:2201.08230 2022.
[32] Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Chang. A survey of deep learning for mathematical reasoning. arXiv preprint orXiv:2212. 10535, 2022.
[33] Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. NaturalProofs: Mathematical theorem proving in natural language. arXiv preprint arXix:2104. 01112, 2021.
[34] Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Natural- Prover: Grounded mathematical proof generation with language models. arXiv prepriat arXiv:2205.129 2022.
[35] Rick Durrett. Probability: Theory and Enamples. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
[36] James R. Munkres. Topology- Prentice-Hall, 2000.
[37] Walter Rudin. Functional analysis. McgGraw-Hii Hill, 1991.
[38] Sheldon Axler. Linear algebra done right. Springer, 2015.
[39] W. Rudin. Principles of Mathematical Analysis. International series in pure and applied mathematics. McGraw-Hill, 1976.
[40] Arthur Engel. Problem-Solving Strategies. Springer, 1998.
[41] Tranquil Sea Of Math. Does ChatGPT code LaTeX and write proofs? Youtube. Retrieved 2023-01-12. https://www httt::://ww..yuuubbeee warpr//s.....oeettttt/e/gee/ee 8PO.
[42] Richard Van Noorden Grichhn@@msstoonn.coo social [@Richvn]. Huh. ChatGPT confidently gives the right kind of reasoning to solve this math problem, but whiffss on the algebra in the middle and gels the answer wrong [Tweet]. Twitter. Retrieved 2023-01-09.
[43] Amos Azaria. ChatGPT Usage and Limitations. Retricved 2023-01-15. https://hal.
science/hal-03913383
[44] Ernest Davis. Mathematics, word problems, common sense, and artificial intelligence. arXiv preprint (tt::2301.0972., 2023.
[45] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4, arXiv preprint 1901.12303. 2023.
[46] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint Xie::20009 033000, 2020.
[47] Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint 0r.io:2210017511, 2022.
[48] The mathlib Community. The Lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. ACM, 2020.
[49] Markus N. Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. arXio preprint ar Xiv: 04 75763, 2020.
13

---

[50] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. LLM is like a box of chocolates: the non-determinism of ChatGPT in code generation. arXiv preprint arXiv:2308.0288 2023.
[5]] Sherman Chann. Non-determinism in GPT-4 is caused by sparse MoE, 2023. Accessed on August 5, 2023.
[52] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almcida, and Carroll L. Wainwright et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 2022.
[53] Carroll Wainwright and Ryan Lowe. InstructGPT: Training Language Models to Follow Instructions with Human Feedback. Gitlfub repository, Retrieved 2023-01-09. petps://githun /ot.oaoooooooooloooooooooooooooo hawnss--huaa--eeebeeeeee
[54] Sarah Wiegreffe (sigmoid,social] ) (@sarahwiegreffe) If text-davinci-001 is rough approzimate the model reported in the NeurIPS 2020 paper, and text-davinci-O02
@sarah)
InstructGPT in the 2022 preprint, then what is just 'davinci"? Trying to reproduce results from a time before this naming eristed [Tweet]. Twitter. Retrieved 2023-01-15. https://twittee
[55] OpenAL. GPT-4 API waitlist. Retrieved 2023-06-06. atttp:://oeena..commaaatt apt-4-api.
[56] OpenAT. Documentation Models. Retrieved 2023-06-06. https://plattfrrm openaa. com//ooss/models/gttt
[57] OpenAL. OpenAI API Reference- Chat Completion Endpoint. Retrieved 2023-06-06.
https://platformt openai. wwwm//ocs/apiireeeeeeeeeeeeeeeee
[58] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and Henrique Ponde de Oliveira Pinto et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
[59] Alexander Bogomolny. Pythagorcan theorem, Retrieved 2023-08-10. https://www.cur cut- the- www..not..ooo. ort..rthaoppttrr....
[60] Benj F Yanney and James A Calderhead. New and old proofs of the Pythagorean theorem. The American Mathematical Monthly, 3(4):110 113, 1896.
[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Fd Chi, Quoe V Le, and Denny Zhou. Chain-of-Thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Newral Information Processing Systems, volume 35, pages 24824 24837. Curran Associates, Inc., 2022.
[22] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. xixiv preprint arXiv:2305. 1019905.10000. 10601, 2023.
[63] Xi Ye and Greg Durrett. The unreliability of explanations in few-shot in-context learning. Xio preprinl orxi::2205.032 03401, 2022.
[64] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Commu- nications of the ACM, 64(12):86 92, 2021.
14

---

Appendix
A Further Related Works
B Dataset Creation
B.1 Categorization
16
B.2 Format..
17
B.3 Copyright and Licensing Terms
18
B.4 Data Sources and Labeling Policies
19
B.5 Mitigating Human Errors
21
B.6 Human Effort
22
B.7 Dataset Misuse
23
C Background details on GPh
D Further Results
D.1 Qualitative Analysis of Subdatasets ChatGPT 9-Jamuary-2023
23
D.2 (Chat)GPT's Confidence .eaadennee......................
25
D.3 Figures ChatGPT's Performance (version 9-Jamuary- 2023)
25
D.4 Comparison of (Chat)GPT) Versions
29
The miniGHOSTS and microGHOSTS Dataset
29
Limitations and Reproducibility
G Best-3 and Worst-3 Across Selected Subdatasets
G.1 Grad-Text...
32
G.2 Holos-in-Proofs (Proofs Collection A)
33
G.3 Holes-in-Proofs (Proofs Collection B Prealgebra and Precalculus)
35
G.4 Olympiad-Problen-SSoee
37
G.5 Symbolic-Inlegration
38
H Datasheet for the GHOSTS Dataset
40
H.1 Motivation
40
H.2 Composition
40
H.3 Collection Process
42
H.4 Preprocessing, Cleaning, and/or Labeling
44
H.5 Uses
44
H.6 Distribution
45
H.7 Maintenance
45
15

---

A Further Related Works
In this section, we present further related works. For (Gh most investigations related to mathematical reasoning consist of anecdotal evidence concerning its performance and its failure modes. Notable mentions on social media can, for instance, be found in [5, 6, 41, 42]. Unfortunately, clear methodology is missing, most of the results are scattered on various internet platforms and cannot be casily reproduced.
To the best of our knowledge, the only investigations into the mathematical capabilities prior to the appearance of our first preprint were undertaken by [43, 44]. However, these works only report a small number of qualitative results, often on rather simple mathematica tasks and without specifying the precise versions of (Chat)GPT, The latter reference reports results only on few selected examples, while the former reference investigates ChatGPTTss ability to compute irrational numbers as well as to solve some elementary math word problems.
Recently, the dataset by [45] contains a systematic evaluation of ChatGPT on the GSM8K datasot [26], the MATH datasct [12], and the MMMLU-STEM dataset [46]. These datasets allow for an automatic evaluation using only accuracy as an evaluation metric. The LILA dataset [47] also consists of a set of diverse mathematical problems, spanning different dimensions and problem types. The problems are formulated in such way that solut can be given by Python programs, which precludes problems that involve more advanced forms of mathematical proofs. Further anecdotal examples of mathematical performance are presented in [45].
Among LLMs prior to (Chat)GPT, Minerva [21], based on PaLM (discussed in Section 2), stands out, being trained in equal parts on websites that contain MathJax elements and arXiv preprints (additionally to general natural language data on which PaLM was trained). It achieves a score of roughly 50% on the significantly harder Mathematics Aptitude Test of Heurislics (MATH) dataset [12], which was sourced from various mathematical competitions. One distinguishing feature of the MATH dataset is that its problems admit a unique answer that can be condensed within few characters (a number, for example). This is beneficial for the automatic evaluation of model on such a dataset, since one can simply check the final answer, ignoring the step-by-stey solution.
Finally, we would also like to mention the field of formalized mathematics, where large databases that encode advanced mathematical concepts exist, c.g., the Leon Mathematical Library [48]. Some of the ideas that we have used in this article, such as using prompts that formulate task to fill in gaps in proofs, are echoed in [49] for datasets for formal mathematics consisting of expression trees. Yet, for the purpose of doing mathematics with large language models, these formal datasets cannot be leveraged since no straight forward way exists to convert them to natural language.
B Dataset Creation
B.1 Categorization
Our subdatasets can be categorized along the following dimensions (see Table I):
Mathematical difficulty (ascending):
M1
Elementary arithmetic problems, as found in the MATH dataset [12] at lower levels of difficulty.
M2 Symbolic problems (integration of functions) that can also be solved via a supervised- learning, data-driven approach to mathematics [13].
(Under))raduuate--evll exercises from well-known textbooks [35-39 39] well questions M3
from math. www.ssechacaeeaeeeeeeeee com, spanning diverse domains of mathematics.
M4 Exercises that are in the style of mathematical olympiad problems, such as those taken from Engel's Problem-Solving Strategies book [40].
Question/prompt type:
Using an unknown version of ChatGPT that predates the 9-January 2022 version.
16

---

Q1
Review questions, which ask to state name certain matheratical facts correctly.
Q2
Overview-type review questions, which cut through an entire field of mathematics.
Q3
Computational questions.
Q4
Proof-based questions, which ask for theorem proof or for a puzzle solution.
Q5
Proof-completion questions, where a proof that has gaps or incomplete needs to be completed.
Types of high out-of-distribution likelihood:
D1
Nontrivial problem encoding: The data points from the Symbolic Integration sub- dataset come from [13] and publicly available Since the online training uses Polish notation, it is very unlikely that (Chat)GPT has seen the corresponding prompts in LTTX before.
D2 Succinct solution: The solutions for the Olympiad-Problem--eee Problem-Soloing subdataset are included in the book by Engel [40]. But the solutions are extremely concise, and simply repeating them would not show an immediate understanding of the problem.
D3 Spoken dialogue: The Search- Caarch-Engine-Aae Aspects subdataset is unlikely to be well repre- sented in the data on which (Chat) has been trained since its prompts resemble word fragments that might appear in mathematical dialogue (e.g., oral mathe- matical exam), rather than in a textbook.
B.2 Format
The dataset consists of collection of UTF-8 encoded JSON files. We explain the JSON keys of each data point in our dataset in the following and also indicate whether its value is optional. If the value is optional, the key has to be present, but the value will be an empty array or string.
prompt denotes the input that we provide to (Chat)GPT through its web interface at the URL chat. ceettiopen...... caammai..ommaee /catttt, see also Appendix C. We use a new session for each prompt to avoid (Chat)GPT being biased by previous prompts.
output denotes the raw output that (Chat)GTTP supplies us with. In some cases, mathe- matical formulas were rendered in the web interface such that we copied them in IATEX.
rating is a number from to 5 that shows how many points (Chat)GPT has scored, being a perfect answer and 1 being the lowest rating. A detailed explanation of the rating policy that we followed is contained in Appendix B.4.
errorcodes (optional) highlight list of error types that illustrate the failure modes of (Chat)GPT in more fine-grained way. Not all types of errors apply to all (sub) (sub)datascts: For example, an error code for a missing proof step would not be applicable on dataset that tests whether (Chat)GPT can multiply numbers find prime divisors. The detailed explanation of the error codes (and the warning codes; see below) that was provided to the annotators is contained in Appendix B.4. There, we also include a policy of how ratings and error codes have to be used together.
warningcodes (optional) highlight any problematic aspects of (Chat) for example, GGh might be rambling and providing the user with unrelated information or use a poor (but correct) way of solving problems.
comment (optional) denotes any noteworthy commentary that an assessor of (Chat) (Chat)GPT GPT may make. This can be used to give a more detailed explanation of the output, provide reasoning behind awarding a certain error code or rating, or generally provide context. For some subdatasets, this key was used to indicate the difficulty level of the prompt, as well
"Pgithub.com http:///ccbboreee.r..///oo.baaaa
17

---

as an official solution, if available, see Section 3.1. It was also used to indicate whether we used prompt engineering, see Appendix 4.2.
msc denotes a list of mathematics subject classifications 13 (MSC) that pertain to the output. Note that we do not classify the prompt given to (Chat)GPT with MSC codes, as there may be proper classification; for example, when (Chat) is asked what the most important theorem in all of mathematics is it is meaningless to assign an MSC code to that prompt.
ref (optional) indicates reference to where the prompt was originally taken from (for some subdatasets, such as froles-in-Proofs we have used excerpts from various books or math. stwckexccaaageee.....e . com; the original source was recorded as a value of this key). This key can have an empty value if the question was formulated by the authors and no authoritative source was plansible.
conf idence indicates how confident we have perceived (Chat) to be when presenting us with its output. We allow values of high, medium, and low.
timestamp denotes when the prompt was entered into (Chat)GPT. This can be used to track the version of (Chat)GPT,; see Section 4.1.
The values of these keys within single data point interact in nontrivial ways: If a rating of is given, then it is expected that no error code is present-though there may be warning codes that are used. The error codes and warning codes are loosely in the spirit of a compiler throwing errors and warnings if it is given incorrect or sloppy code-although we have a role reversal, where the human is now the compiler, and the machine produces the code. In this sense, for some prompts, we have used multiple error and/or warning codes, which why the corresponding values are arrays of strings. We use these codes to collect statistics on the behavior of (Chat)GPTT see Section 4.
For most of the subdatasets that make up our GHOSTS dataset, we have used INTEX to encode mathematical formulas in our prompts. Our experiments have shown that (Chat)GPT can process L-Iee-encodded mathematics well.
The usage of MSC codes can be useful for mathematicians who want to integrate GPh in their daily workflow, as it allows them to know in which areas the model performs better and can hence be trusted more. Our dataset is very diverse, having total of 78 MSC codes. The lop short versions of these codes (first two digits) are 26 ("Real functions", 127 occurrences) followed by 00 ("General", 110 occurrences) and 46 ("Functional analysis", 77 occurrences), see also Figure 5.
B.3 Copyright and Licensing Terms
Because our datasel draws on multiple sources, which have different copyright holders, that have issued different licenses (or no licenses at all), we associate licenses on a fine-gr grained level to our data, to differentiate between parts where we are bound by the license of an existing dataset and the parts that we created, that we release under the following Creative Commons license:
4.0 International (CC BY-NC 4.0) 16, By this license, one may not use those parts of the datasct that were created by us for commercial purposes, and one must give appropriate credit when using it non-commercially; if users are building on the GHOSTS dataset, they need to indicate the changes that were made and distribute their contributions under the same license as the original.
Licenses per datapoint: We associate licenses on a fine- ane-ggained level to our dataset in the following way:
For each datapoint, one license applies to the value of the prompt key (which, in some cases, governed by copyright where no license is issued, in which case we do not release it publicly, in that case, the value of the prompt key will be <copyrighted)). Which prompt values are subject to which licenses is fully detailed in Table B.1.
A complete list of MSC codes can be accessed at the URL zbmath www.th.otgaatttteeeeeoeo wwtpssttaai///scc22/0000 par. 14The answer is Pythagoras' theorem, according to (Chat)GPT..
see https://credt wec.rmmovseeooooss...o.oo orgglicceseesbyy-eeee 0/ for the detailed terms of the license.
18

---

Table B.1: A summary of all the files from all the subdatasets comprising our GHOSTS datasct, together with their license. In the case of the prompt values from the "Proofs Collection A file, cither no license was issued from the copyright holder (if the prompt value was taken from the books Linear Algebra Done Right by S. Axler or Principles of Mathematical Analysis by W. Rudin, cf. the ref key associated to that prompt value); or one of three Creative Commons Auribution-ShaareAlikk license, in different versions, depending on the date of the math. stackexchange com post which served the source for the prompt value, see ht https: //apth.//mah..///// /atth stackexchange. com/ hom/help/ie htthhhepppplesssiieeeeeee /1i.esiin..n....
Subdataset Name
Comprised of the json file(s)
License for value of prompt key
Crud-Text
W. Rudin, Functional Analysis (ch. 1)
No license specified""
W. Rudin, Functional Analysis (ch. 2)
No license specified
J. Munkres, Topology (ch. 1)
No license specified
J. Munkres, Topology (ch. 2)
No license specified!?
R. Durrett, Probability Theory
No license specified
H Holes-in-Proos
Proofs Collection A
More than one license
Proofs Collection B Prealgebrs
MIT license
Proofs Collection B Precalculus
MIT license
Olympiad- Prapaaaddd-ooll--SSS--iiii
Olympiad Problem Solving
No license specified
S Integration
Symbolic Integration
CC BY-NC 4.0
MATH
MATH Algebra
MIT license
MATH Counting and Probability
MIT license
MATH Presigebra
MIT license
MATH Precalculus
MIT license
S Seurch- Eeail--ngnnn-nieee
Definition ion Retrieval
CC BY-NÇ 4.0
Reverse Definition Retrieval
CC BY-NC 4.0
Named Theorem Proof Completion
CC BY-NC 4.0
Another license, the CC BY-NC 4.0 license, applies to all other, remaining components of each datapoint; for the datapoints where also created the prompt, these two licenses will coincide, as outlined by able B.1. We note that therefore, in particular, the value of the output, is also released by under the CC BY-NC 4.0 license. This is consistent with OpenAT's terms, which at the time of writing st "You may provide input to the Services (*Irput"), and receive output generated and returned by the Services based on the Irputt ("Output"). Irput and Output are collectively "Content." " As between the parties and to the extent permitted by applicable law, you own all Input. Subject to your compliance with these Terms, OpenAl hereby assigns to you all its right, title and interest in and to Output. This means you con use Content for any purpose, including commercial purposes such as sole or publication if you comply with these Terms."
Some of the subdataset files contain prompts that are protected under copyright, and no specific license has been issued for them by the copyright holder: These are all the prompts from all files from the Grad-Teat and Olympiad-Proleee Problem-Soloinng subdatasots, as well as selected prompts from the "Proofs Collection A" file. In such cases where copyrighted prompt was used for which no license was available, we have opted not to withhold the value of the prompt key in the publicly released dataset. The ref key includes a detailed reference to the page where the original theorem or exercise was presented, so a reader can easily retrieve the original prompt.
Release policy:
B.4 Data Sources and Labeling Policies
Parts of the GHOSTS dataset were collected from books. These prompts were transcribed into TATTX. The output from (Chat) web interface was copied as-is, even if the output was not valid TATEX code. Tf the output contains rendered mathematical expressions, our
httpps: paattps://oppe http://pppaanaccoooaoii///////// offuue
17 Despite our best efforts reach the copyright: holder, we received no response, and, therefore, we do not have a license under which we can (re-)share the prompt. 18 Perrmission was given by the author to reproduce the exercises we used, but unclarities remain who the copyright owner is.
19

---

policy was to transcribe it to ETEX.. Below are the policies that were followed by each assessor of (Chat)GPTTs output regarding the rating, the error codes, and the warning codes:
Rating
+ failure to understand the query (e.g,, the user asks it something about number theory, and it responds with information about differential equations);
> query was understood, but the answer was entirely wrong (e.g., the user asks what the prime divisors of 111 are and responds with and 6);
3 - query was understood, but the answer was only partially correct (e.g., the user asks what the prime divisors of 111 are, and it responds with and 6);
4 + query was understood, and the answer was mostly correct (e.g., the user asks it what the prime divisors of 222 are and it responds with 3 and 37);
- query understood and answer was completely correct.
Error codes
e1 + missing examples or information (e.g., the user asks what the prime divisors of
111 are, and it responds with 3, missing 37); this also applies, if (Chat)GPT ignores part of the prompt (o.g., an equivalence needs to be shown, but (Chat)GPT shows only one direction);
+ few wrong/ /vague statements (e.g., the user asks it what the prime divisors of 30030 are2l and it responds with 2, 3, 5, 7, 13 (wrong); or says that 2, 3, 5, and some other numbers are prime divisors (vague)); it can also denote single statement, that is slightly vague;
e3 + a lot of wrong/too vague statements (e.g,, the user asks it what the prime divisors of 30030 are, and it responds with 2, 5, 8, 12, 13, 15 (wrong); or says that and many other numbers are prime divisors (vague)); it can also denote a single statement, that is highly vague;
e4 - wrong computations (i.e., an additional error flag to disambiguate between statements that are of computational nature not);
e5 + denotes wrong logic or wrong flow of arguments, which we further subdivide into specific flags, we prohibit the use of e5 its own (since it would be uninformative):
e5_ + (Chat) claims that to complete a proof, statements need to shown that are unrelated to the claim;
e5_2 + a proof step is missing;
e5_3 + an edge case has not been considered;
+ an inference step is not supported (e.g., (Chat) claims that from A follows B, but this claim is not true);
e5_5 circular logical argument (i.e., using the hypothesis prove the hypothesis);
e6 > the general set-up is understood, but the legal operations are not respected or misunderstood (c.g., but GG't changes the rules and motivates the solution by allowing the addition of odd integers; or GPh misunderstands an adjective that has multiple mathemation meanings, such as "dual", which can mean either topological dual space or algebraic dual space).
given puzzle where we are only allowed to add even integers,
The following policy applies for error codes: If a rating r with 1 < < 5 has been given, then error code is mandatory to explain the type of error that occurred. For perfect score of 5, no error codes should be assigned (but warning codes can be assigned). If the
Thee 37 and 20Thee are 2, 37, and 3.
Theeey are 3, 5, and 11.
20

---

score is lowest, ie.. rating of 1, error codes can be assigned, but do not have to: In the case where (Chat)GPT has not understood the prompt, there typically no reason to further detail the type of error.
Warning codes
w1 (Chat)GPTT is withholding essential information related to the prompt (e.g,, the user asked it something about the integral J e-* dr, and answers correctly but does not tell the user that the integral was actually famous, named integral, i.e., the Gaussian integral);
#2 + (Chat)GPTT is rambling (i.e., after answering, correctly incorrectly, (Chat)GPTT tells the user much more details than the user wanted to know);
w3 + (Phat)GPT is hallucinating (i.e., after answering, correctly or incorrectly, (Chat)GPT tells the user unrelated information);
w4 + (Chat)GPT behaves weirdly (e.g., by using a weird proof structure (where applicable), using strange mathematical formulations, by adopting strange tone of the conversation or making opinionated statements);
w5 (Chat)GPT changes the notation from the prompt without being instructed to do so (e.g.; the prompt contains a vector space X, but (Chat)GPT calls IF).
B.5 Mitigating Human Errors
Any assessment procedure that has human component is prone to introducing bias in particular, a procodure involving manual work such as rating the model outputs. In the list below describe safeguards that help mitigate bias and human errors (such typos) as well as streamlining procedures that we undertook:
1. Safeguards against ETEX errors and typos:
Various typographical errors may appear due lo incorrect. LaTeX formatting. In this case, we noticed that (Chat)GPT was able to correctly infer what was intended (e.g., $cup$ was correctly interpreted as $ $\cup$). Similarly, we noticed that (Chat)GPT's output was stable under small "perturbations" of the prompt, such as minor typos. Thus, (Chat)GPT therefore provided its own safeguard against these types of errors.
2. Safeguards against encoding issucs:
We presented clear instructions to each author who prompted (Chat)GPT on how to record and save the data in order avoid any file encoding issues. In the end, all JSON files were inspected and streamlined to Unicode, if a different encoding was used.
3. Safeguards against unfair comparisons:
Clear instructions were given to all authors that used (Chat)GPT to ensure that the language model has, to the extent possible, an identical state and starts from a blank chat.
4. Safeguards against missing data and copy-paste errors:
Given a lack of APT access in the early stages of our investigation (see Appendix C), there was fair amount of data being copied from GGhat)GPT. To mitigate any copy-paste errors, several passes over the entire dataset, as well as automatic checks, were made look, e.g., for potential inconsistencies, missing timestamps, and outputs not matching the prompts.
5. Safeguards against datasct misformatting:
Bocause the methodology authors had to adhere is complex (scc Appendix B.2 and B.4, we introduced a number of automatic checks to make sure the general format of each dalapoint is consistent and in line with our rules. In particular, we
have made sure that various entries are in the correct place and take the correct values (e.g., that no warning flag values are in the key for errors, that the ref key value has the correct format, etc.).
21

---

6. Safeguards against other unforeseen errors:
Random somples: Random samples (< 10) were drawn from each dataset, and second assessor reviewed the rating. If deemed problematic, the original assessors were askod to re-evaluate.
Statistical checks: Additional statistical chocks were carried out as plausibility checks make sure no other unforescen errors occurred: If prompts deviated from the average length on that dataset, they were flagged, the output was manually inspected, and, if deemed necessary, a re-evaluation was carried out.
We are aware that these measures are not exhaustive, but given a fixed time budget, we considered them the most feasible. We note that authors evaluated ChatGPT models in multiple stages, as the models became available, and that checks and corrections were made at the end.
Because our checks and corrections pipelines are complex and a number were made after all the evaluations on all models were obtained, we cannot exclude that the strings cont aining the prompis weren't it altered in very minor ways (e.g., converting back to Unicode from a different, original, unknown encoding may introduce slight variations of certain special characters that slipped through inspection). However, because of the sufficiently large number of evaluations that we carried out in each file, which is the smallest unit on which we collect statistics, this will not affect any results. From the viewpoint of reproducing our dataset, this is also not an issue, since even for identical prompts, (Chat)GPP) itself is non-deterministic; this contimues even if its temperature parameter in the API is set to zero [50, 51]. Lastly, as noted above, (Chat)GPPi) answers will almost always still express the same idea, even minor variations of the same prompt used, even individual scores will stay unchanged in such a scenario.
B.6 Human Effort
The evaluation was carried out by a subset of the authors of this paper who have substantial mathematical expertise, ranging from master's degrees in mathematics to postdoc-level and professor- positions at departments of mathematics. Assignment of prompts was done based mathematical difficulty, with more senior mathematicians having received more difficult prompts. No third parties were involved.
Each of the 709 prompts of the GHOSTS dataset was evaluated on both the 9-Jamaary20023 and 30-Jamuary-2023 version of ChatGPT; an additional 24 prompts were used to test the effect of prompt engincering evaluate GPT-4 on the 170 prompts of the miniGHOSTS dataset. This amounts to total of 1636 prompt evaluations of advanced mathematics performed by graduate-level researchers.
a single type of subdataset,
Appendix 4.2. We further
We like to mention that our effort has occasionally unearthed small inconsistencies in existing datasets: For example, the "MATH Counting and Probability" file, which was sourced from the larger MATH dataset [12], contains the prompt " What is the value of 1013- 3- 1012 + 3- 101 18", which is neither about counting, nor about probability, but arithmetic (our MSC codes allow users to find such examples).
We note that is neither possible outsource the creation of these subdatasets to crowdsourcing service, such as Amazon Mechanical Turk, nor is it possible to generate them automatically from code because advanced mathematical insight is required for the creation of each prompt (where applicable) and for providing the fine-grained evaluation of the mathematical capabilities. We note that already in the case of the MATH dataset, which contained less advanced mathematics than some of our subdataset contained, it was noted in [12]: "Because MATH requires a strong mathematical background to perform well on, and a long amount of time to solve problems, we were restricted to assessing six human participants and could not rely on crowdsourcing sites such as Amazon Mochanical Turk". Furthermore, unlike in the case of the MATH dataset, the answer to most of our prompts cannot be condensed into a few tokens (such as a number or a function), e.g., when the answer is a mathematical proof.
This raises the difficulty of the creation of more data since graduate-level (and some cases, PhD-level) mathematics is required. The combined effort of devising mathematically
22

---

insightful prompts and carefully rating the output of (Chat)GPT amounts to 1636 prompt evaluations, totaling several hundreds of person-hours, see Appendix B.6. However, as result of these efforts, our dataset goes beyond all the mentioned mathematical datasets for LLMs in Section 2 in terms of the different aspects of mathematical reasoning that are being tested.
B.7 Dataset Misuse
Our dataset does not contain personally identifiable information. Because the prompts of our dataset, are compiled from a diverse collection, it does not contain systematic collections of mathematical exercises and their answers from a single type of test, which could invalidate certain school-level or unieersity-level tests. We are not aware of any direct way of misusing our dataset, as its primary objective is the detailed evaluation of the mathematical reasoning performance of LLMs.
C Background details on (Chat)GPT
GPT-4 launched on 1st March 2023, is the latest model of the GPT lincage [7], being the successor of various versions of ChatGPT, the first of which was launched on 30 November 2022 [11]. These are all based on InstrucIGPT, which in turn is based on a trained GPT-3 [27], and fine-tuned using reinforcement learning with human feedback [52].
We note that already for models that predate (Chat)GPT, such as InstructGPT, where rescarch articles and model cards [53] have been released, full reproducibility is not possible since the code and exact datasets have not been released. Furthermore, it was confirmed by OpenAl employees that for some of their models, launched prior to 30 November, a slight mismatch exists between the trained model that is accessible via the OpenAl web interface and the model referred in the official paper [54]] This indicates how essential it is to document carefully which model our analysis pertains to and how we have accessed it. In our dataset, we have accordingly included time stamps for each prompt in order to be able to track, based on information provided by OpenAI, any changes in (Chat) version that have occurred.
We have exclusively used the GUI web interface to carry out the evaluation. This was necessary for consistency reasons since, at the beginning of our evaluation, API access was not yet widely available. At the time of writing, API access to GPT-4 is still limited, and a waitlist is employed, which made the use of the GUT web interface a necessity for GPT-4 [55))- We note that there exist no official documents that link the GUT web interface to the different model versions and possible model settings from the API. The 9-Jamaay-2023 ChatGPT versions we evaluated are likely to be earlier instances of the newer model gpt-3. 3..-ttrboo-00000
and 30-Jamuary-208
This model itself is a "snapshot of gpt-3. 5-rbr from March 1st, 2023" that will not receive further updates [56]. For GPT-4, at the time of writing, there exist four models gpt-4, apt-4-0314, gpt-4-32k, gpt-4-324 15-332-0331 that can be used for the chat completion API endpoint [57]. Additionally, for all models, there exist various settings when using the chat completion API endpoint, such as temperature or presence perelcyyeeellyyyeeeeee that influence the models' output, which cannot be controlled via the GUT web interface. It is also not known which values of these settings are used for the GUT web version. The only version identifier in the GUT web version is a generic "model version" " link at the bottom of the page that links to the release notes [11]. The 9-January-202 and 30-January-2083 2023 model versions that we evaluated are the ones presented in the release notes [11] at the respective time.
D Further Results
D.1 Qualitative Analysis of Subdatasets on ChatGPT 9-Jamuary-2023
In this soction, we go through common mistakes performed by ChatGPT, as well as notable observations regarding the output, one subdataset at time. We focus on the 9-Jamary-2023 version, see Section D.4 for more information regarding the other version. We note that the output of (Chat)GPT (and, generally, LLMs) is stochastic and therefore may differ on the same prompt. Nonetheless, clear trends can be observed, which we describe here. Individual outputs can be found in Appendix G.
23

---

5.0
60
Non-Engineered Engineered
4.5
50
4.0
3.5
Sating
Percent
3.0
2.5
2.0
1.5
1.0
Non-Engineered
Engineered
el
e2
e3
e4
e5
Error Codes
Figure 3: Effect of prompt engineering on the rating (left) and the error codes (right) for the 9-January-2023 model.
Grad-Text ChatGPT, version 9-Jamuary20223, performed best on simple set-thoory and logic questions (the first chapter from the book Topology by J. Munkres [36]), which is reflected in its rating, see Figure 1. On the rest of the books, it performed substantially worse Because of the confidence (high) with which it outputs the answer, the of ChatGPT, version 9-Jamuary-0023, is particularly deceiving in this use-case since it may be intensively used by students studying these subjects.
Holes-in-Proofs ChatGPT, version 9-Jamuary-2223, correctly recognized most well-known results concepts (e.g., filling in the mean-value theorem, given a proof that lacked reference to it). However, the ability of ChatGPT to execute algebraic manipulations is surprisingly inconsistent. In some cases, ChatGPT executes complicated symbolic tasks with case; in other cases, it fails on simple arithmetic operations or rearranging terms. The mistakes do seem correlate with the complexity of the algebraic expression. When ChatGPT makes an algebraic mistake, it mostly carries over this mistake reliably to the rest of the computation.
On this subdataset, ChatGPT, version 9-Jamuary20023, performed the poorest. From a mathematical point of view, these questions were also by far the most difficult, they pose difficulties even to professional hatamaiisss. A score of 3 was awarded when the answer started to show promise. However, 75% of the scores are 2 because the answer does not show any promise. No rating of 5 was awarded, and only one rating 4 was achieved. This version of ChatGPT had a tendency to try and solve many questions using induction arguments. While this is not necessarily false, this was very far from the solutions given in the book, and this version's 's inductive proofs were easily seen to contain mistakes. In addition, ChatGPT often had difficulty understanding unconventional puzzles. For example, in the questions involving changing the color squares on a chessboard, the solution offered by ChatGPT did not cover 8 8 chessboard. Sometimes it tried to solve the problem by changing only squares, far from the 32 required. Similarly, the g-Jamuary-2022 version of ChatGPT struggled to respect unusual constraints in the questions, resulting in 8 e6 errors, the highest number of e6 errors out of all subdatasets. In some cases where the problem seemed to require complicated mathematics but was actually solvable by elementary techniques, ChatGPT did not spot this but instead referred to the general theory of, e.g., diophantine equations. Interestingly, ChatGPT would sometimes say, e.g., that the question could be solved with these means but that this was hard, so the confidence score was downgraded in these cases to medium or low.
Cyympiad-Probeem----e
The 4.Jammary-2022 version of ChatGPT was dominated by sys- tems that were trained specifically solve integration problems [13]. In a number of instances, this version got the structure of terms right (for example, the number of summands the output, as well as where factors had to placed before summandds) but failed at concrete computations Even very simple examples were not correct. For example, the antiderivative + x²/2 /2 is evaluated to + x² 22/3+C, C, where is a constant of integration (the correct answer being + x3 /6 C"). For number of prompts, this version claims there
Symbolic-Intogration

---

is no closed-form solution for the integral with complete confidence when, in fact, there is solution; only integrals that have an elementary antiderivative are in this dataset.
MATH On the questions related to Algebra and Probability theory, the 9-Jamuary2202: version of ChatGPT got the reasoning often correctly. However, the most common type of error was e4, occurring 36% of the time (in total 62 times). This version of ChatGPT may struggle when confronted with standard operations, such as inverting fractions, least common multiples, and changing the sign of numbers when moving them from one side of the equal sign to the other. Often, in these questions, a correct solution requires performing multiple operations in sequence. In such cases, most often, at least one operation was wrong. This prevented the model from getting a rating of 5 on the output, which was only achieved for 29% of the questions.
file, the 9-Jamuar-2023 version of ChatGPT knew almost all the theorems that was asked at a basic level but made mistakes when stating them. When came to listing other results required for the proofs, this version typically requested way more than the necessary theory-occasionee occasionally even results that only follow from the theorem which was asked for (error code e5 65_5). On the Definition Retrieval file, this version had quite a good performance: it recited most definitions correctly. It sometimes got confused when being askod about distributions in the sense of clements of the dual space of test functions. ChatGPT, version 9-Jamuary-2023, strongly favors the notion of distributions in the stochastic sense. Similarly, for the adjective "closed", where it chose to pick the context of algebra (instead of topology) and interpreted it to mean "algebraically closed". : On the Reverse Definilion Retrieval file, this version had the strongest performance, being able to recover most definitions from their descriptions, with average raling of 4.30 (standard deviation 14). This indicates the usefulness of ChatGPT as a mathematical search engine. This subdataset is also the simplest from mathematical point of view since no logical thinking is required, but only a name needs to be found.
BearchhEnngne-Aaee
On the Search- Seettt-nnnnn---ppe
D.2 (Chat)GPTT Confidence
(Chat)GPT usually very confident, unlike other LLMs that we have experimented with. As an illustrative example, consider the following prompt lesting the sensitivily to 18T=Xennooeed mathematics vs. Uncoode-encoded mathematics:
Here are two statements: "Let (X, A, 4) be measure space. Show that the signed measures A: A -> form Banach space."; *Let A \mu) $ be a measure space. Show that the signed measures $1lambda A Arightarrow R$ form Banach space." " Are they equivalent?
The response by ChatGPT is not phrased in order to show any maance in terms of confidence (which typical, even ChatGPT is wrong):
Yes, the two statements are equivalent. They both express the same mathematical idea but use different notation and formatting conventions. The first atatement uses parentheses to enclose the elements of the measure space and regular letters represent the functions, while the second statement uses subscripted parentheses and uses the LaTeX formatting to represent the elements of the measure space and the function.
The response by Codex [58], another model that we brielly tested (but whose scope would have exceeded that of single conference article) gives a cautions response and, unlike ChatGPT, capable of voicing doubt:
think they are equivalent but am not sure.
D.3 Figures of ChatGPT's Performance (version 9-January-2023)
In this section, we collect figures that extend the discussion in the main body and provide further views the data and descriptive statistics.
25

---

Error Codes
Warning Codes
1.0
Grad-Text
39
25
50
4
Analysis 1)
W. Rudin, Functional Analysis (ch. 2)
8
3
17
3
3
11
Munkres, Topology (ch. 1)
1
Tapology 2)
R. Durret, Probability Theory
8
3
0.8
Holes-In-Proofs
Proofs Collection
18
7
2
Proofs Collection B Prealgebra
10
8
10
Proofs Collection Precalculus
0.6
5ubdataset
Olympiad-Problem-SSuie
27
51
36
88
16
16
48
16
20
3
Symbolic-integration
24
74
MATH
62
33
MATH Algebra
2
18
14
MATH Counting and Probability
26
19
17
MATH Prealgebra
0.2
Search-Engine-Aspect
19
Retrieval
Reverse Definition Retrieval
Named Theorem Proof Completion
0.0
w1.
W2
W3
w4
65
Figure 4: Counts (annotation) and relative frequencies (color) of error and warning codes by subdatasets (bold) and files for ChatGPT 9-Jamuary-202) on GHOSTS.
(General)
120
26 (Real functions)
33 (Special functions)
28 (Measure and integration)
100
46 (Functional analysis)
54 (General topology)
MSC code
51 (Geometry)
12 (Reld theory and polynomials) (Sequences, series, summability) 60 (Probability theory and stochastic processes) 05 (Combinatorics)
Count
42 (Harmonic analysis on Euclidean spaces)
multilinear algebra; (Mathematical logic foundations)
theory)
theory)
47 (Operator theory)
2
Rating
Figure 5: Average rating over mathematical fields for the 9JJanuary-0023 version of ChatGPT on GHOSTS. The color depicts the occurrence of cach MSC code, and only MSC codes that have at least occurrences are shown. Note that the ranking is not indicative of the complexity of the fields since we do not use equally complicated exercises for all fields. The error bars represent 95% confidence intervals.
26

---

70
Version
60
9-Jan.
30-jan.
50
GPT-4
Percent
40
30
20
0
1
2
4
5
el
C2
c3
c4
e5
w1
W2
W3
w4
W5
Rating
Error Codes
Warning Codes
Figure 6: A comparison of the AJJamaryy20233 model, the 30-Jamuary-2025 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of percentages of ratings (right), error codes (middle), and warning codes (right).
Grad-Text
Holes-in-Proofs
Hyympiad-Problem-Sooe
Version
60
9-Jan.
Percent
30-Jan.
GPT-4
20
0
Symbolic-Inteeration
MATH
Scarch-Engine-Aspects
60
Percent
20
0
1
5
Rating
Rating
Rating
Figure 7: A comparison of the 9-Jamuary-2022 model, the 30-Jamuary-2023 model (both GHOSTS), and GPT-4 (on miniGHOSTS) in terms of percentages of ratings on the different subdatasets.
27

---

19-157
158-320
9-Jan.
321-457
458-751
30-jan.
752-2144
GPT-4
Outpur Length
23-189
190-329
330-470
471-648
649-2076
224-658
659-938
939-1317
1318-1756
1757-2973
1.5
2.0
2.5
3.0 Rating
3.5
4.0
4.5
5.0
Figure 8: A comparison of the 9-Jamary-2023 model, the 30-Jamuary model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of output lengths. Every interval contains 20% of the prompts, and the error bars represent 95% confidence intervals.
21-49
50-65
66-88
89-117
Prompt Length
Version
118-152
9-Jan.
30-jan.
153-189
GPT-4
190-251
252-336
337-461
462-1360
1.0
1.5
2.0
2.5
3.0 Rating
3.5
4.0
4.5
5.0
Figure 9: comparison of the Slauuary-2025 model, the 30-January-2023 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of prompt lengths. Every interval contains 10% of the prompts of GHOSTS, and the error bars represent 95% confidence intervals.
28

---

D.4 Comparison of (Chat)GPT Versions
In this section, we collect figures which illustrate the differences and similarities between versions of (Chat)GPT, We note that even though the 30-Jamuary-2023 version performs very similarly lo the 9-Jaruary-2023 version, there are some differences in the distribution of ratings, error codes, and warning codes, see Figure 6.
On the other hand, GPT-4 strictly dominates the ChatGPT versions in terms of performance. always provides context around the question (whether that was asked for or not) and often gives useful (and correct) pointers that, for example, highlight the importance of particular theorem. Figure 8 depicts the verbosity of different (Chat)GPT versions and the achieved rating. However, we also nole that the optimal level of verbosity can depend on the mathematical background of the user. As a result, there have been significantly more warning codes of type w2 (i.e., rambling) for GPT-4, see Figure 6.
E The miniGHOSTS and microGHOSTS Dataset
The minGHHOSTS and microGHOSTS datasets consist a subset of datapoints from the GHOSTS dataset. Their role is to enable faster evaluation of given language model on advanced mathematics by shrinking the dataset size from 709 prompts (GHOSTS) to 170 prompts (miniGHOSTS), which represent about 24% of GHOSTS and 14 prompts (microGHOSTS), which represent about 2% of GHOSTS.
The miniGHOSTS dataset was created by using a heuristic select a subset of prompts such that the mean and variance of the (0-Jamuary-202 version of ChatGPT on this subset match the mean and variance of the 30-January-201 GHOSTS dataset. The role of the miniGHOSTS dataset is to allow a full assessment of model on a reduced dataset. This dataset is, therefore, not meant to assess the mathematical reasoning performance of other LLMs in absolute terins but rather to provide relative comparison to the 30-January-2025 version Ghat
version of ChatGPT on the entire
The prompts of the mieroGHOSTS were chosen by hand from the worst-performing questions from miniGHOSTS, or those that were deemed to pose mat hematical difficulties to language models. The role of microGHOSTS is to obtain a preliminary assossment of model by reducing the human evaluation cost to close to zero (to have initial information before potentially deciding on embarking on a more comprehensive evaluation).
Our overall evaluation experience leads us to observe that there exist the following broad categories that lead ChaiGPT performing problematically:
instances where complex mathematical definitions involved, which depend in turn on prior definitions (c.g., an absorbing subset of a topological vector space), and where something is asked about how these concepts interact:
instances where the reasoning is predominantly geometric reasoning or where an insight is obtained by thinking geometrically;
long chains of computations;
vague definitions that require ChatGPT to understand that terms in mathematics are overloaded;
problems when constructing examples;
lack of ingenious ideas, such as needed for Olympiad-problem solving.
We have leveraged these insights to include prompts that specifically exploit these weaknesses.
Prompts that belong to the microGHOSTS dataset we tag by using the <microg <microghosts> word as value in the comment key. Further, the values of the comment also contain extensive solutions and explanations of the problem so that good gold standard exists against which given LLM solution can be compared by a human rater. We also include general explanations of the mat concepts that are involved to further aid general machine learning practitioners (who may not have prior mathematical training) if they want to test their language models' reasoning capabilities.
29

---

F Limitations and Reproducibility
In this section, we describe the limitations of what our dataset and our evaluations cover; some of these also are related to issucs regarding reproducibility, which we also discuss below. In terms of:
model selection, we have focused on those models that, at the time of writing, are known to have the best performance among existing LLMs, either based on anecdotal evidence [3, 5, 6] or via previous benchmarks on more elementary mathematical reasoning tasks [7]. Becauss human evaluation has been very time- conneconnuming, we have chosen to invest the available time solely in the evaluation of these models (and not older ones as well) to obtain a picture of their performance that is as accurate as possible.
correctness, we have put a large number of measures in place to guarantee evaluations are as correct as possible (see Appendix B.5). Nonetheless, we cannot guarantee the absolute correctness of each evaluation. There are also some questions where, aside from human errors, fundamental evaluator disagreement occurs regarding the question of what rating should be given; this is well-known problem in psychometrics and, in simple instance, can be resolved by having multiple evaluators evaluate the same question. Given a fixed budget in terms of man-hours, opted to be able to devise a larger dataset and 2) in light of the chocks that we described in Appendix B.5, such an approach would result in diminishing returns on invested time.
have chosen not to follow this approach because 1) we
An issue in this regard is the fact that an absolute ground truth is lacking for a mathematical dataset of this lype that goes beyond simple arithmetical questions. In particular, in the case of proofs, these can be presented in myriad ways: On one hand, for the same mathematical fact, one find conceptually distinct proofs. (See [59] for an extreme example of this, where 122 conceptually distinct proofs of the Pythagorean theorem are given.) On the other hand, one and the same proof can be presented in different ways, since the order in which the arguments are made and connected can be changed
Assuming all the previous problems were, somehow, solvable, further complicating the matters is the fact that not all possible proofs are known, and new ones are being discovered for known statements (staying with the topic of the Pythagorean theorem, even millennia after its publication, new proofs have surfaced [60]). If a model were to output a proof that is not known, it would still require human to judge correctness, as in this case, there may not be any ground truth available to compare against.
All of the above makes it very hard for the ground-truth data to be included in our dataset and if it is included, it may not be meaningful for the outlined reasons. Some clementary datasets, such as the MATH dataset [12], have ground-truth data included in the comment. However, their ground truth is also susceptible to the issues we mentione;; human evaluation is necessary for absolute certainty.
non-determinism, ground-truti -trth absence, and author disagreements notwith- standing, the above shows that an effort to reproduce our results should lead to highly similar scores on each file of each subdataset. Duc to the fact that our analysis is made per subdataset file and not per prompt, we are confident that a double-digit number of evaluations per file makes the conclusions that we draw robust.
prompt engincering type, we have deliberately chosen to avoid more complex types of prompt engineering mechanics, such as using Chain-of-Thoughts [61], or Tree-of- Thoughts [62] because 1) the main focus of the paper is the standard performance of the tested language models, not their the performance augmented by various forms of in-context learning and 2) these types of prompt engineering methods require additional significant human effort to decorate the original prompts with example when carrying out few-shot prompting and to track the output. The human effort for the current dataset was already considerable; adding such prompt engineering methods would have further
E.g. if proving statements C depends on three statements X,Y we could prove first X,Y .Z and then conclude C; but we could also prove Y Z.X and then conclude C' or construct the proof based on any other permutation of X,Y, ,Z. While all of these proof presentations would use the same ideas, syntactically, they would be different proof presentations
30

---

raised the evaluation cost while also increasing the complexity of our non-trivial evaluation methodology and 3) for GPT-3 and certain datasets and prompting engineering techniques, performance was degraded [63].
datapoint-level MSC code coverage, our dataset covers 78 distinct codes spanning most areas of mathematics. Because there are 1636 distinct, MSC-classified outputs, by the pigeonhole principle, some of the MSC codes will necessarily be covered only by small amount of MSC code (Figure indicates those MSC codes that occur at least five times, although some MSC codes appear over one hundred times).
We also note that for particularly easy mathematical questions (e.g., simple arithmetical questions), suitable MSC codes exist to classify the output since MSC codes typically classify more advanced mathematics" Nonetheless, we have attempted to match them as well possible and allow multiple MSC codes in order to classify the output as precisely as possible.
We further note that an exhaustive survey of (Chat)GPTT's performance across every MSC code would necessitate a large, community-driven effort to set up an extensive database. Due to the high cost of rating each output, requiring specialized skills, this is something that no individual research group could reasonably do but we hope that our approach is starting point for such an effort.
subdataset-level attribute tag coverage, we could, in theory, start investigate every combination of these attribute tags for the subdatasets. We recall that the subdatasets have attribute tags assigned to them that indicate difficulty to subdatasets, the type of questions a subdataset contains, and the ways the subdataset is likely to not have been part of training distributions of the considered models (sce the explanation of the tags M1-M4, Q1-Q5 and D1-D3 from Appendix B.1 and Table 1). Thus, for single mathematical domain, represented by one MSC code (such as for elementary arithmetic problems), one could generate subdatasets to cover every possible combination of the 60 possible ways to combine the attribution tag listed above. If one combines this with different MSC codes, this would, above, also lead to a manual curation effort, which is too intensive and consuminn for a single research group.
The MSC codes starting with the numbers "97", which at first glance might be most suitable, are solety reserved to classify content that is related to the educational process of mathematics, rather than the mathematical content itself.
31

---

G Best-3 and Worst-3 Across Selected Subdatasets
We list below the best and worst answers of ChatGPT, version 9-January-2022 over selection of subdatasets. For readability, the prompts and answers are lightly modified so that the bsTeX-basod formulas are correctly displayed, and whitespace removed.
G.1 Grad-Text
Examples from the Grad- Text subdataset, comprising the books Probability Theory by R. Durret [35], Topology by J. R. Munkres [36] and Functional Analysis by W. Rudin [37].
Best-3
"prompt": "Let be a measure, M/ - inf : p(fz 1f(x)) > My) = 0} and ll Igldu. Prove that [iggld S Illll If .".
"output": "Let = Il and <>> 0. Then, by the definition of II have u(fx: [f(x)] M =0. Therefore,
/ [ggdu=
1fgldu+ + S Ifoldy If (M +)) / (sdu / 1fllolda.
iigM++
The second term the right side is zero because ule: [[=)]](>]]>> =0. So have [gidd & (M + Igldp (M elllllh As can be arbitrarily small, we can let it go to zero and we get f M[]g]] - 11911 Hence proved."
"prompt": "If [Ta} is family of topologies on X, show that N. To is a topology on X.", "output": "To show that N. To is topology X, need to show that it satisfies the three axioms a topology: and X are both in Na Ta since 0 and X are in every topology Ta. If (Uiher is a collection of sets in Na then Ulez is also in N. To. Since each is topology. Us is every Tai and so is also 0, Ta If (U,her is a collection of sets in N,Ta, Ta, then Nie Ui is also in N. Ta. Since each Ta is a topology, Nie Ui is in every Ta and so is also N.To. Therefore, N. To is topology on X."
"prompt": "Let X be vector space and and B subsets of X. If A and are balanced, so is A+B.",
"output": "A subset S of a vector space X is called balanced if for every vector in X and every scalar À with W Ae is in $ is in S. To prove that A+B is balanced, we must show that for every vector ==a-b in A+ and every scalar x with (X]
Ar - b) -a -a Xb is in A+B if x is in A+B. Since A and B are balanced, we know that
every a in A and every scalar with N Xa is in A if is in A. Similarly, for every b in B and with [ S1, Xb is in if
every scalar is in A+ B, then is in A and b is in B. Therefore, da is in A and Ab is in B, and thus Ar da Ab in A+ B. Therefore, A+B is balanced."
in So, 2 z
Worst-3
"prompt": "Find function f:R. R that is continuous precisely one point.", "output"; "One example such a function is
f(x) 2 if 40 if x
This function is continuous at x but not any other point."
32

---

"prompt": "Let X be topological vector space. sets mentioned below are understood to be the subsets of X. Prove the following statement: If is compact and B closed, then A+B
closed.", .",
"output" "A topological vector space a vector space with topology such that the vector addition and scalar multiplication are continuous. To prove that A+B is closed, we must show that the complement of A+B is open. Let yEX((A+B). Then there exiats some J1 E and 22 E B such that y= +22 Since A is compact and B is closed, there exist neighborhoods Ui of xı and U2 J2 such that U1 is relatively compact and X\U2 is open. Now consider the set V= + -2 f#1 22::2:1:11 E This set an open neighborhood of and it is contained in (A+ which shows that X (A B) is open
hence 4+ closed."
"prompt": "Let X be topological vector space. All mentioned below are understood to be the subsets of x. Prove the following statement: The sum two closed sets may fail be closed "output": "A counterexample can be used to prove that the sum of two closed sets may fail to closed. For example, let - and let A [0, and B (1,2). Both A and B are closed sets in R, but A- [0.2] is not a closed set in
G.2 Holes-in-Proofs (Proofs Collection A) Examples from the Holes-in-Prooff subdataset, Proofs Collection A file, based the books [38, 39] and questions from math. swwckexchange.ccmmmmmmmm com
Best-3
"prompt": "We want to show that any open subset of is countable union of disjoint open intervals. Let U be a non-empty open subset of R. For Eyy define zny iff [MISSING). It's easily checked that ~ is an equivalence relation on whose equivalence classes
pairwise disjoint open intervals in R. (The Nenquote(interva]] here includes unbounded intervals i.e., rays )) Let I the set c-lasse Clearly U U= I. For each I o choose rational e the map & injective, so I is countable. Can you fill in the missing condition?",
a is
"output": "The missing condition is "there exists an open interval containing and that is contained in U."
"prompt": "The norm on R" is given by | = EE
lax P) We want to We compute
1/p
show show that || I Cl|=ll for arbitrary 1 15p <9
1/P
1/r
|
Σ
Σ
|
no - n 1/1- 1/001/01
Which result has been used in the inequality above?", "output": "The inequality used in the above calculation is Hölder
inequality. It states that for any p.921 such that 1 + 1 =1, and for any vectors zER", yER".
P
1/p
1/9
Σ I
Σ 19: 9
im
i=1
33

---

"prompt": "Let f be real real-valued function continuous on [a, and differentiable on (a.b). Suppose that f exists. Then, want to prove that is differentiable at a and f'(a) = la (x))
Using the definition of the derivative, need to show that
lim fla (a+hh) f(a)
h
exists and is equal to lim (() Note that for h> 0 the Mean Value g-aa
Theorem provides a point ch with a ca <athh h such that [MISSING). Finally. note that on + at h - Can you fill in the missing equation?",
"output": "The missing equation from the Mean Value Theorem
f((c)) fla + h) f(a)
h
Worst-3
"prompt": "We want to compute
2n
3n+1
Let's s first find general formula for the following sum Sm 2 nr". Notice that
m Sm - m+ Σ m1
P- m+1
mr m42
(m 1)r m1
+
1
1
Hence
Sm mr m+2
(m + 1)r
+r
(1-r)²
Can you finish the calculation ?"
"output": "Yes, finish the calculation we can substitute the values of == 2/3 into the formula we found for Sm:
(2) 002
(c0 11+
Sx
(1 2)²
This simplifies to
S00 = 2,
"prompt": "Suppose is an inner product space and T L(V). want to prove that
dett(T) = det VT*T.
By the Polar Decomposition there is an isometry $ L(V) such that
S-STT.
Can you finish the proof?". "output": "Yes, once we have that T SVT+T, we can take the determinant of both sides to get
det det det(T)TTT
Since S is an isometry, it preserves inner products, det(S) +1. Therefore,
= =
Since the determinant of matrix is always non non-negative, we have that = = "
34

---

"prompt": "F a finite field order q. We want to compute the size of (F). In order for an nxn matrix to be invertible, need the rows to be linearly independent. - choices for the first row; now, there span of the first row, so we have d"-q choices for the second rows. Then the set of vectors
you note, we have q vectors in the
row. Now, let v1,22 be the first in the span of 01,02 is of the form (cre) + c2uz/e1, F). This set is of size q²,
have q choices for e and q choices for 62. Thus, we have -9² choices for the third row. Continuing this gives the desired formula. Can you state the formula?",
"output": "The size of (F. is given by the formula -d' g)(a"=+
-g("" - (f- 1)."
G.3 Holes-in-Proofs (Proofs Collection B Prealgebra and Precalculus)
Examples from the Holes-in-Proofs subdataset, Proofs Collection B Prealgebra and Proofs Collection B Precalculus files, based on [12].
Best-3
"prompt": "Find the smallest positive integer that is greater than and relatively prime to the product the first 20 positive integers. Reminder: two numbers are relatively prime if their greatest common divisor 1. Solution: <MISSING> the smallest possible integer "output: "The first 20 positive integers are 1, 2. 3, 4, 6, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, and 20. The product of these numbers is 243302008116640000 To find the smallest positive integer greater than that is relatively prime to this product,
23 Fill in the missing steps.",
start at and incrementally check each number until we find one that has a greatest common divisor (GCD) of 1 with the product. We GCD of each of these numbers with the product is not 1. However, when check 23, find that the GCD of 23 and 2432902008176640000 is 1. Therefore, 23 is the smallest positive integer that greater than and relatively prime the product of the first 20 positive integers.".
check 2, 3, 4, .... 22, and
that the
"prompt": "Find the number of solutions cos cos the interval 0< 24m. Solution: From the equation cos cosæ, cos cos =0, From the sum-to po-pouutt formula, we can write this
5x 3r
-2sin sin =0
<MISSING> Fill in the missing steps.". "output": "Since the product of sines equal zero either sin 5 - fr
B no, where is or sin ir -. Solving for each case: sin g =0 + Sr an integer. This gives the solutions x= w. sin =0 = nm, where is an integer This gives us the solutions == 8n Now we need find the solutions that are 16 24 32 40 48 nT, z 1,,,5 5, a, which are in the given
the interval 0< 24m.
Sn gives solutions interval. I Sa gives solutions 2 x= =- 16 3 which are in the given interval. Therefore, there are 5 solutions for 8 and 3 solutions for == &ng the given interval. The total number of solutions is 5+3-8."
"prompt": "Let a. b, e three vectors such that ||||| |||| J b = and
|b] V3
4
k(b c)
35

---

for some scalar & Find the angle between b and in degrees. Solution: Since bxc is orthogonal to both b and a-b=0 and a.c=0. Since |
(a (a-btc)((abbtt
<MSSINGG> Fill in the missing steps.", "output": "Expanding the dot product on the left side gives: a a b-.baacce b-a.abbbbb----aaeeeeeeeeeeeb----
Since
a a b--a-c-ba--c-a know that =²== - - and subatituting these values in Which means the vectors orthogonal, hence the angle between them 90 degrees."
this simplifies to: +o/ 2b. - 1. We
gives: / c-l Solving for b gives: - - =0
Worst-3
"prompt": "The line described by
2
-1
+
3
is perpendicular to the line described by
?
(1)
and passes through the point (4, Find a Solution: The direction vector the first line
and the direction
vector of the second line is 1
<MISIING> Then t= -1 a= 33
2= so atbtk= -3. Fill in the missing steps
"output": "The direction vector of the first line is and the
To find the point of intersection we can set the two parametric equations equal to each other:
direction vector of the second line
- + 1 2 = -1 + 1
Solving for we get 1=-1,u=-3 3. So the point of intersection is k
-1
= (4, Given that we get = -3 and b 2, then
+ + -3
"prompt": "How many positive integers less than 100 and divisible by are also divisible by 47 Solution: If we start counting from 1, every third consecutive integer is divisible by and every fourth consecutive integer is divisible by 4. <MISSING> 100. Fill in
so there are multiples 12 between the missing steps.",
36

---

"output": "The missing steps are: We need to find the multiples of 12, which are divisible by both 3 and 4. To find the first multiple of 12 less than 100, divide 100 by 12 and round down to the nearest integer, which is 8. To find the last multiple of 12 less than 100, divide 12 by 100 and round up to the nearest integer, which is 8, The number of multiples of 12 between 1 and 100 is the difference between the last and first multiple, which is 8 - 8 = 0. So there are no positive integers less than that are divisible by both 3 and 4.",
"prompt": "The complex numbers 5 10i, 7+2;, and 1l +1+33 three vertices of a rectangle. Find the fourth vertex. Solution: Let a-3+100 b-7+ 24, and e-11+ Let the fourth vertex. Note
that
c-b-4+i
and
a
28(1 +1) (-())==(ee-00
Thus, the angle between the segment joining a and and the segment joining b and is 90" <MISSING> Fill in the missing steps."
"ouputt: "Since the angle between the segment joining and b and the segment joining b and joining and d is perpendicular to the segment joining and b. Since b-b= 00=444 4+1, we have c- d (bb-dd) - -2) Therefore, d=c- (2 8i) 11
90º, it follows that the segment
5r. So the fourth vertex is (9.-5i).
G.4 Olymppat--ProblemSSooie
Examples from the Olympiad- Olympiad-Problem--See Problem- subdataset based on the book [40].
Best-3
"prompt": "n numbers are written on a blackboard, In one step, you may erase any two of the numbers, say and b, and write, instead b)/4. Repeating this step n n-1 times, there is one number left Prove that, initially. if there end, a number which is not less than 1/n will remain.". "output": "This
the board, the
be proven by induction. Base are two ones on the board, after one step, the remaining number (1+1)/41/2, which that the statement is true for n=k. Now consider n n=k+1. If there are k+1 ones on the board, we can erase any two them and write their average, which is not less than 1/k+1 by the induction hypothesis. Therefore, we can conclude that the atatement holds for all n."
n=2. If there
less than 1/2. Inductive case: Assume
"prompt": "The integers I,...,a are arranged in any order. one step you may switch any neighboring integers. Prove that you never reach the initial order after an odd number of steps.", "output"; "This problem can be proven using the concept of parity. Specifically, the parity of a permutation is the number of inversions (i.e. pairs of integers (t.j) such that i<j but the ith integer appears after the jth integer in the permutation) modulo 2 can be shown that the parity of permutation remains unchanged after any single swap of neighboring integers. Since the initial ordering has zero inversions (i.e. has even parity), it follows that an odd number of swaps will always yield a permutation with odd parity, and thus cannot be the same as the initial ordering."
37

---

"prompt": "Let's answer this question step by step. Derive divisibility criteria for 9 and 11.",
"output": "Divisibility Criteria for 9: If the sum of digits of a number is divisible by 9, then the number is also divisible by 9. If the difference between the sum of digits in the odd positions and positions is divisible by then the number is also divisible by 9. Divisibility Criteria for 11: If the difference positions
between the of digita in the odd positions and is divisible by 11, then the number is also divisible by 11. If the alternating minus fourth, etc.) is divisible by 11, then the number is also divisible by 11."
of digits (first minus second plus third
Worst-3
"prompt": "Assume an 8 chessboard with the usual coloring. You may repaint all squares (a) of or column (b) of a 2x2 square. The goal is to attain just one black square. Can you reach the goal?".
"output": "Yes, it is possible to attain just one black square on an 8x8 chessboard. One way do this of the top-left 2x2 square, which will leave one black square in the bottom-right
repaint all the squares
of the chessboard. Another way do this is to repaint all the squares of the first column, which will leave one black square the bottom-right corner of chessboard.
"prompt": "Start with the positive integers 1,..,44-1 In you may replace any two integers by their difference. Prove that even integer will be left after 4n-2 steps.".
"output": "We can prove this by mathematical induction. Base case: When ==1, have the integers 1,2,3,4. We can perform the following moves: (1,2) 1,3,4; (3,4) 1,3; (1,3) 22 an even integer. true for n= We will show that it is also true for n=k+1. We have the integers 1(kk....((k+1]) 1. We can perform the following moves: 1,3, 1
Inductive case: Assume that the statement
1,3,4 4(k (1++)--11 ((k 4) Now we have 4k-2 integers left, 4k-2 consecutive odd integers. By the
+ 1) which induction hypothesis, after 4k - steps, we will be left with an prove
integer. Therefore, by mathematical induction, that an even integer will be left after 4n-2 stepa."
"prompt": "Let's answer this question step by step. Assume an 8x8 chessboard with the usual coloring. You may repaint all squares (a) of row column (b) of 2x2 square. The goal is to attain just one black square. Can you reach the goal?".
"output": "Yes, it is possible to reach the goal of having just black square on an 8x8 cheasboard. One way to do this is to repaint all of the squares in the first black, and then repaint the top-left square of the 2x2 2 square white. This results in one black square and all other squares white."
and first column
G.5 Symbolic-Integration
Examples from our Symbolic-Integration subdataset based on [13].
38

---

Best-3
"prompt": "What is the integral of tr' sin)
"output": "The integral of 4x" (4/8)=* +C, where C is constant of integration. The integral of sim(1 is - -cos(1 +x)+ So the integral of 4x" +s)(( (4/8)=* c0s
prompt": "What the integral of 2048 32. "output": "The antiderivative or indefinite integral of 2088 + 32x with respect
[[2048 32r")dr 2048 (82/4)a² +C where the constant of integration."
"rrmppt: "What is the integral of 12 + coshh(r) "output": "The integral of 12 6cohh(((z is 12r 6sinh(() + C, where C is the constant of integration."
Worst-3
"prompt": "What is the integral of (4+ +))4 3c]le"1(6*)""""
"output": "The integral of (4+ 3allee= not expressible in terms of elementary functions."
"prompt": "What is the integral of 32 coo(c - 3)]"
"output"; "The antiderivative of 3r 4s-iss" (4/3)²* and
antiderivative of [cos(2 + 3)}is sin(z 33. Therefore, the integral of - 4r + 3) x² (1/3)r² sin(a +3) C, where is the constant of integration.""
"prompt": "What is the integral of 1/(2 In(3/2))" "output": "The integral 1/(2+ In(32)) cannot be expressed terms of elementary functions and must be approximated numerically or solved using special functions."
39

---

H Datasheet for the GHOSTS Dataset
This appendix provides a datasheet for the GHOSTS dataset. The format of this datasheet was introduced in [64] and consolidates the motivation, creation process, composition, and intended uses of our dataset as a scries of questions and answers.
H.1 Motivation
Q1. For what purpose was the dataset created? Was there a specific task in mind? Was there specific gap that needed to be filled' Please provide description.
The existing datasets of natural-language mathematics are far from covering all the typical tasks professional mathematicians encounter in daily life, making it unclear whether language models can be of any help in this regard. Existing datasets mostly cover elementary mathematics and 3). Hence, they do not offer any insight into the usage of ChatGPT as tool for mathematicians. In this work, we have made the first attempt towards filling this gap, going beyond math problems that are yes-no rated, and proposed a benchmark made and curated by working rescarchers the field that tests different dimensions of mathematical reasoning.
resemble standard tests like SATs (see Sections
Q2. Who created this dataset (e-g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?
The authors of this work created GHOSTS; see Appendix B.6 for more information.
Q3. Who funded the creation of the dataset? If there is associated grant, please provide the name of the grantor and the grant name and number.
There no associated grant or funding which has been used to create the GHOSTS dataset.
Q4. Any other comments?
No.
H.2 Composition
Q5. What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)! Please provide a description.
GHOSTS consists of textual prompts, in natural language, representing mathematical questions. For each prompt. GHOSTS contains one or more instances of outputs of (Chat)GPT and corresponding fine-grained evaluation by the authors.
Q6. How many instances are there in total (of each type, if appropriate)?
There are 709 prompts in GHOSTS; a selection of 170 these makes up miniGHOSTS and a selection of 14 of those makes up microGHOSTS. For 24 of the questions from the GHOSTS dataset, light prompt engineering variations have been carried out. Each of the 709 + 24 questions from GHOSTS has been evaluated on ChatGPT, version 9-January- 2023 and 30-Jamuary-208 2023, and 170 questions from miniGHOSTS have been evaluated GPT-4 (and thus microGHOSTS too). Thus, in total, (709 24) 2 + 170 1636 outputs and evaluations have been carried out. See also Appendix B.6 for more information.
Q7. Does the dataset contain all possible instances or is it a sample (not neces- sarily random) of instances from larger set? If the dalasel a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic c If so, please describe how this representatioennss was validated/ validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover more diverse range of instances because instances were withheld or unavailable).
GHOSTS tries to cover wide range of mathematical questions from 78 different MSC codes; see Appendix B.1 and B.2. However, due to the prohibitive cost of human evaluation, which cannot be fully automated away (see Section 3.3), it is not feasible to
40

---

represent all mathematical fields across all dimensions of "mathematical behavior" and all types of mathematical questions (overview questions, fact-stating questions, etc.).
Q8. What data does each instance consist of? "Raw" data (e.g.; unprocessed text or images) or features? In either case, please provide descriplion. GHOSTS, miniGHOSTS, and microGHOSTS consist of collection of JSON objects (one for each data point), and each JSON object consists of 10 key-values pairs as detailed in Appendix B.2.
Q9. Is there label or target associated with each instance? so, please provide description.
No, we do not explicitly define label or target for the instances. However, the rating the output can potentially be used to select good and bad mathematical conversations of (Chat)GPT in order to fine-tune models and the errorcodes and warningcodes can be used to make a more fine-ge grained classification possible.
Q10. Is any information missing from individual instances? If so, please provide a description, explaining why this informalion is missing (e.g., because was unavailable). This does not include intentionally removed information but might include, e.9,, redacted texl.
No.
QII. Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If s0, please describe how these relalionships are made explicit.
Relations between instances are explicitly given by the same values on (subsets) of the fields, c.g., the same prompt, the same model version, or the same MSC code. Prompt-cngineered variations of the same question are represented as an array of JSON objects, one object for each variation.
Q12. Are there recommended data splits (e.g., training, development caaaennntttvllaaiion.. testing)? Iʃ so, please provide a descriplion of these splils, expluining the ralionale behind them.
Not applicable.
Q13. Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide description.
The evaluation of the prompts included in GHOSTS underlies human errors. However, we tried to mitigate these errors; Appendix 1.5.
Q14. Is the dataset self-contained, or does it link to otherwise rely external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources,
(a) Are there guarantees that they will ezist, and remain constant, over time?
(b) Are there official archival versions of the complete dataset (i.e., including the external
resources as they erisled al the lime the dalasel was crealed)?
(c) Are there any restrictions (e-g., licenses, fees) associaled with any of the external resources that might apply to a fulure user? Please provide descriptions of all external resources and any restrictions associated with them, well as links or other access points, as appropriate.
The dalaset is self-contained. However, some of the prompls cannot be publicly released since they are taken or adapted from sources that are protected by copyright, for which no license was given, see Appendix B.3; though we do release the output of the models on these prompts, together with our evaluation of the output. Further, we provide a reference to the original, copyrighted materials, that user easily retrieve the original prompts.
Q15. Docs the dataset contain data that might be considered confidential (c.g., data that is protected by legal privilege by doctor-patient confidentiality, data that includes the content of individuals non-public communications)? If so, please provide a description. No.
41

---

Q16. Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
No.
Q17. Does the dataset. relate to people? IJ not, you may skip remaining questions in this section.
No.
Q18. Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations identified and provide description of their respective distributions within the dataset. No.
Q19. Is it possible to identify onc or more natural persons, cither directly or indirectly (i.e., in combination with other data) from the dataset? so, please describe how.
No.
Q20. Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals racial or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide description.
No.
Q21. Any other comments? No.
H.3
Collection Process
Q22.
How was the data associated with each instance acquired? Was the data directly observable (e.g., row text, movie ratings), reported by subjects (e.g., survey responses), or indirectly anferred/derives from other data (e.g., part-of-speech tags, model--asedd guesses for age or language)? If dala was reported by subjects or indireetly inferred/deree derived from other data, was the data ciiuualed/verifeet If so, please describe how.
We collected and constructed prompts from various sources, see Table 1 and Section 3. For the evaluation, we captured the corresponding outputs of (Chat)GPT and rated them according to the instructions in Appendix B.2 and B.4.
Q23. What mechanisms or procedures were used to collect the data (e.g., hard- ware apparatus sensor, manual human curation, software program, soft- ware API)? How were these mechanisms or procedures validated?
To query (Chat) GPT, used the GUI web interface at the URL chat. www.ai.encc.oooo com/ chat; see Appendix B.6 for detailed reasons for using the GUI interface.
Q24. If the dataset is sample from larger set, what was the sampling strategy?
The prompts of the MATH and caabboli-Inteeratton subdatasets have been randomly sampled from [12] and [13], across different files from those datasets.
The prompts taken from all the other sources (books and math. stackexchange.com similarly have been randomly selected.
For our miniGHOSTS datasct, we sampled 10 prompts from each of the 17 files in GHOSTS in the following way: Our results in Section indicate that the 9-January- 2023 and the J0-Jamarry2223 ChatGPT versions have similar overall performance; however, the behavior differs on a more fine-grained level and was marginally better for the 30-January-2023 version. Hence, we assembled miniGHOSTS by computing all subsets of 10 prompts having approximately the same mean rating and standard deviation as the original file from GHOSTS, rated on the 30-January 2023 version
42

---

of ChatGPT. A manual inspection of these subsets, in order to pick subset with appropriate mathematical content (we want to have mathenatically diverse dataset), then led to the final selection of the miniGHOSTS dataset. The microGHOSTS dataset was created from manual inspection of the miniGHOSTS dataset, by isolating total 14 questions which were deemed difficult for language models.
Q25. Who was involved in data collection process (c.g., students, crowd-workers, contractors) and how were they compensated (e.g., how much were crowd- workers paid)?
Only we have been involved in the data collection process. No payment (other than one made through regular employment) in relation to creating this dataset and writing this article made.
Q26. Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please provide description of the timeframe.
The collection date matches the creation time. Tt specified in the timestamp key in each data point from GHOSTS and spans timeframe from January 9, 2023, to now. Using the timestamp, the version of ChatGPT that was used can be inferred, see Appendix C.
Q27. Were any ethical review processes conducted (e.g., by an institutional review board)? IJ so, please provide a descriplion of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation. Not applicable.
Q28. Does the dataset relate to people? If not, you may ship remaining questions in this section.
No.
Q29. Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (c.g., websites)?
Not applicable.
Q30. Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide link or other access point to, or otherwise reproduce, the exact language of the notification itself.
Not applicable.
Q31. Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent requested and provided, and provide link other access point to, otherwise reproduce, the eract language to which the individuals consented. Not applicable.
Q32. If consent was obtained, were the consenting individuals provided with mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as link or other access point to the mechanism (if appropriate).
Not applicable.
Q33. Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link other access point to any supporting documentation. Not applicable.
Q34. Any other comments?
No.
43

---

H.4 Preprocessing, Cleaning, and/or Labeling
Q35. Was any
of the data done (e.g., discretiza- tion or bucketing, tokenization, part-of-speech tagging, SIFT feature extrac- tion, removal of instances, processing of missing values)? If so, please provide description. If not, you may skip the remainder of the questions in this section.
We corrected various minor issues and inconsistencies that could arise in the process of manual evaluation, see Appendix B.5.
Q36. Was the "raw" data saved in addition to the preprocessed /leanseddd/eeee///aaaa labeled data (e.g., to support unanticipated future uses)? Iʃ so, please provide link or other access poinl to the "raw" dala.
The output key in each JSON object contains the raw output from (Chat)GPT- unless ChatGPT used rendered ETEX in which case our policy to transcribe it. In very few cases, potential copy-paste errors were noticed, which were fixed.
Q37. Is the software used to proprocoss/clcan/labe the instances available? If so, please provide link other access poinl.
The raw output of (Chat)GPTT in the output key has not been cleaned, see Q36. Cleaning of the other values has been done first using Python scripts, an automated way, and subsequently by hand, to correct any further, unforeseen mistakes, see Appendix B.5. The Python scripts are available upon request.
Q38. Any other comments?
No.
H.5 Uses
Q39. Has the dataset been used for any tasks already? If so, please provide a description.
We have used the GHOSTS dataset to evaluate and compare the mathematical capa- bilities of different LLMs, in particular, different (Chat) GPT versions; see Section
Q40. Is there a repository that links to any all papers or systems that use the dataset? If so, please provide a link or other access poind.
Future work citing the GHOSTS dataset will be listed by citation trackers such as Google Scholar and Semantic Scholar.
Q41. What (other) tasks could the dataset be used for?
Tf the dataset is growing further, we anticipate that GHOSTS can be used as training data for fine-tuning LLMs.
Q42. Is there anything about the composition of the dataset or the way it was collected and preprocessed/ aeleaned/labedd that might impact future uses? For example, there anything that future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (c.g., financial harms, legal risks) If so, please provide description. Is there anything a future user could do to mitigate these undesirable harms? No.
Q43. Are there any tasks for which the dataset should not be used? If so, please provide a description.
No.
Q44. Any other comments?
No.
44

---

H.6 Distribution
Q45. Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset created? If 80, please provide a descriplion.
Yes, the GHOSTS, miniGHOSTS, and microGHOSTS datasets will be made publicly available. It consists of three JSON files, one per ChalGPT version (9-January-2023, 30-Jamuary-2023, and GPT-4, where the latter was evaluated on miniGHOSTS, while the former two on the full GHOSTS dataset; the microGHOSTS prompts are lagged within the miniGHOSTS dataset, see the comment key). Some prompts will not be available due to copyright issues (see Appendix 133), but precise reference where the original prompt can be found will be included instead.
Q46. How will the dataset be distributed (e.g,, tarball on website, API, GitHub) Does the dalasel have digital object idenlifier (DOT)?
The dataset will be made available on GitHub the public repository github.com /rioder::/GHOSTS a collection of JSON files.
Q47. When will the dataset be distributed?
The dataset is already available.
Q48. Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide link or other access point to, or otherwise reproduce, any relevant licensing terms ToU, well as any fees associated with these restrictions.
We release the GHOSTS, miniGHOSTS, and microGHOSTS datasets under the fol- lowing Creative Commons license: Attribution-Ne Commmmrrii 4.0 International (CC BY-NC 4.0), unless we are bound by licenses of individual prompts or files from var- ious subdatasots to release those prompts or files under more restrictive licenses; see Appendix B.3 for more information.
Q49. Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevont licensing terms, as well as any fees associated with these restrictions.
IP-restrictions apply only to those prompts that were not solely created by the authors (which are under the CC BY-NC 4.0, as explained above), see Appendix B.3 for these
cases.
Q50. Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restriclions, and provide link or other access point to, or otherwise reproduce, any supporling documentation. No.
Q51. Any other comments?
No.
H.7 Maintenance
Q52. Who will be supporting/ hateiiing//oostinnnnnnnaiiii
the dataset?
The dataset will be hosted GitHuu repository; Q46. All the information about the dataset, including links to the paper and future announcements, will be written in the README file of the GitHiub repository.
Q53. How can the email address)?
of the dataset be contacted (e.g.,
The email addresses of the authors publicly available. Morcover, it is possible to raise an issue on GitHub.
45

---

Q54. Is there an erratum? If so, please provide a link or other access point.
Future changes will be documented the README file of the GitHub repository. Differences in single files can be tracked in the Git history.
Q55. Will the dataset be updated (e.g., correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicuted to users (e.g., mailing list, GitHub)?
We will continue to maintain the dataset to fix any errors that may have occurred. We will allow either direct email contact or Git pull requests this sense. Beyond fixing such errors, we consider the dataset to be frozen.
Q56. If the dataset relates people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. Not applicable.
Q57. Will older versions of the dataset continue to be supported/hoster praetedd/mii--- /main- tained? If so, please describe how. If not, please describe how its obsolescence will be communicaled to users.
Not applicable.
Q58. If others want to gatend/augmentt build on/contribute to the dataset, is there a mechanism for them to so? If so, please provide a descriplion. Will these contributions be verified? If so, please describe how. If nol, why not? Is there process for communicaiing/dissriiiii provide description.
these contributions to other users? If so, plense
Any external contribution to our dataset is strongly encouraged. Every addition the dataset will be carefully reviewed by the authors. For other details, please see Q55.
46


Reviews:
[{"review_id": "dr3h79CMzW", "reviewer": "Reviewer_9g86", "summary": "", "questions": "", "limitations": " The authors have made a good effort to discuss the limitations and potential negative impacts of their work, but there are still some areas that could be expanded on:\n\n- The dataset creation process required extensive human effort and mathematical expertise. The limitations around scaling up the dataset size are acknowledged, but more discussion could be had around potential solutions or alternative approaches. \n\n- The benchmarking is currently limited to certain models and tasks. Expanding the scope would yield more comprehensive results. The authors recognize this, but more detail on specific limitations of the current benchmark would be helpful.\n\n- There is limited discussion of potential misuse cases and steps to mitigate harm. For example, ChatGPT could produce erroneous mathematical solutions that mislead students. The authors could suggest best practices for use.\n\n- More analysis of why models fail certain tasks through ablation studies or error analysis could be suggested as future work.\n\n- Comparisons to human performance are lacking. Contrasting with professional mathematicians could better contextualize the capabilities.\n\nOverall, the authors have made a solid effort to acknowledge limitations and potential negative impacts. But the work would be strengthened by expanding the discussion on these topics and providing more constructive suggestions to address them in future research.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- It introduces a new natural language mathematics dataset, GHOSTS, to test language model capabilities on advanced mathematical comprehension. This helps fill a gap, as most existing datasets cover only elementary mathematics. \n\n- The dataset is carefully designed to cover different aspects of mathematical reasoning like stating facts, computational questions, proof completions, and solving Olympiad-style problems. This provides a more comprehensive test of language models.\n\n- The authors perform a rigorous analysis of ChatGPT and GPT-4 using their proposed methodology of ratings, error codes, and warnings. This sheds light on the models' mathematical abilities and limitations.\n\n- Benchmarking on their dataset shows ChatGPT and GPT-4 lag behind specialized models trained on math tasks. But GPT-4 shows promising improvements. This helps track progress.\n\n- The work identifies types of questions and workflows where ChatGPT could assist mathematicians versus where its capabilities are limited. This guides proper integration into mathematicians' workflows.\n\n- The dataset required extensive human effort and mathematical expertise to create. This establishes a thorough benchmark for evaluating language models on math.\n\n- The authors encourage extending the dataset and evaluation to drive progress on language models for mathematics.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "JPgp7dZXWt", "reviewer": "Reviewer_7RM4", "summary": "", "questions": "", "limitations": "The evaluation is limited to the models in ChatGPT family, while there are many open-source/commercial models that exist. Without the evaluation with those models, the new dataset/benchmark becomes a bit weak, as it is unclear how good it is to distinguish the models.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper introduces a new dataset, GHOSTS, that can be used to evaluate the mathematical performance of the existing LLMs.\n2. The paper studies the math capability of ChatGPT and GPT-4 with the new dataset, and concludes that ChatGPT is good at understanding the question, but is not good enough for answering.\n3. The authors also show that the GPT has been improved on math since the first release, and what the latest GPT-4 improves.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "vzhr4LQXKc", "reviewer": "Reviewer_Sgwr", "summary": "", "questions": "", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The paper introduces a new methodology to evaluate the mathematical capabilities of large language models.\n2. The creation of the GHOSTS dataset, covering graduate-level mathematics and distinguishing different dimensions of mathematical reasoning.\n3. Evaluation of ChatGPT and GPT-4 on various mathematical tasks, providing insights into their strengths and weaknesses.\n4. By emulating use cases that mathematicians encounter in their daily professional activities, the paper assesses the practical applicability of ChatGPT and GPT-4 as mathematical assistants.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "xxyPvLOjNC", "reviewer": "Reviewer_2boP", "summary": "", "questions": "", "limitations": "1. The datasets used to evaluate ChatGPT, while more advanced than previous benchmarks, still only cover a subset of undergraduate and beginning graduate-level mathematics. Truly testing the capabilities of ChatGPT on advanced graduate level or research mathematics would likely require even more difficult prompts.\n\n2. The datasets are limited in size, with only 709 total prompts across 6 sub-datasets. Expanding the datasets to include more prompts and a wider diversity of mathematical topics would strengthen the evaluation.\n\n3. As noted in the conclusion, the high cost of having mathematical experts manually rate each prompt response makes growing the datasets challenging for a single research group. More community participation is needed.\n\n4. The evaluation is limited to certain types of mathematical reasoning (e.g. no geometry problems). Testing a broader range of mathematical skills could reveal other strengths or weaknesses..\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Here are four key strengths of this paper:\n\n1. It provides a comprehensive evaluation of ChatGPT's mathematical capabilities across a wide range of domains, question types, and difficulty levels. The authors test ChatGPT on 6 custom-designed datasets as well as samples from existing datasets, covering topics from basic arithmetic to graduate-level proofs.\n\n2. The methodology is rigorous, with clear rating criteria and a detailed taxonomy of error codes to analyze ChatGPT's failure modes. Each response is manually rated by experts in mathematics.\n\n3. The paper introduces a new dataset called GHOSTS that focuses specifically on advanced mathematical comprehension, filling a gap compared to existing datasets that cover mainly elementary math. This dataset could become a valuable benchmark for testing future LLMs.\n\n4. The analysis provides nuanced insights into ChatGPT's abilities and limitations in mathematics. It highlights cases where ChatGPT shows impressive performance as well as common mistakes and weaknesses.\n", "weaknesses": "", "quality": null, "clarity": 1, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "5DO2rR4nqR", "reviewer": "Reviewer_3gvx", "summary": "", "questions": "", "limitations": "The evaluation process was carefully conducted, and covers a broader range of math problems. However, the evaluation highly depends on human experts. If evaluating another LLM on this set of math problems, human experts are required for making judgement on the LLM outputs. \n", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1） The paper provides a thorough evaluation of ChatGPT and GPT-4, considering their performance in different mathematical tasks.\n\n2） The exploration of mathematical capabilities in language models is a timely and relevant topic. Since many papers and studies are focusing on the mathematical reasoning ability of LLMs, which is a critical ability for humans.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Summary (key phrases borrowed from reviews)

The authors introduce GHOSTS, a new benchmark dataset for evaluating the mathematical reasoning capabilities of large language models like ChatGPT and GPT-4. The dataset comprises 709 prompts across 6 subdatasets covering a diverse range of mathematical skills, from basic arithmetic to advanced graduate-level concepts. A key contribution is the inclusion of carefully hand-crafted prompts to test abilities beyond existing math datasets.The dataset is curated by working mathematicians to benchmark capabilities on graduate-level concepts.

The authors then conduct a detailed analysis of two ChatGPT versions and GPT-4 on a novel methodology introduced by them. Performance correlates with mathematical difficulty, with models struggling on complex proofs and original problems. GPT-4 demonstrates substantially improved performance over ChatGPT. ChatGPT shows promise as a mathematical search engine, but inconsistently produces high quality solutions. They conclude that ChatGPT is not yet suitable for rigorous exam-level work, contrasting hype about its math skills.

Strengths

1. Most existing datasets cover only elementary mathematics, so this is a step in the right direction.
2. The dataset is carefully designed to cover different aspects of mathematical reasoning like stating facts, computational questions, proof completions, and solving Olympiad-style problems. This provides a more comprehensive test of language models.
3. The proposed methodology of ratings, error codes, and warnings is a novel new methodology to evaluate the mathematical capabilities of large language models
4. The work identifies types of questions and workflows where ChatGPT could assist mathematicians versus where its capabilities are limited. The dataset required extensive human effort and mathematical expertise to create. This establishes a thorough benchmark for evaluating language models on math. Also, Identifies common failure modes of ChatGPT on mathematical reasoning, such as missing proof steps, incorrect computations, and faulty logic. This can help guide the future development of LLMs.
5. Paper is clear, well written and sound.

Improvements

1. Can expand the dataset with more domains, question types, and levels of difficulty. The current number of 709 examples, while a good start, might be still very limited. This points to a difficulty of scaling the current approach of collecting examples, which might be mitigated by engaging the broader Mathematics community to build a larger dataset. 
2. Comparisons to human performance are lacking. Contrasting with professional mathematicians could better contextualize the capabilities.
3. The paper only reports 0-shot performance and not few-shot and CoT performance, which are known to improve the mathematical capabilities of GPT-style models considerably. The authors respond that they aren't able to achieve this in this paper since they have committed to a rigorous evaluation, and the number of failure modes drastically increase with the addition of few-shot examples and CoT. They leave it to future work.
Decision: Accept (Poster)