SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Simon Frieder\*,1, Luca Pinchetti1, Alexis Chevalier3, Ryan-Rhys Griffiths4, Tommaso Salvatori2,8, Thomas Lukasiewicz8,1, Philipp Petersen6,7, Julius Berner5
$^{1}$ Department of Computer Science, University of Oxford, Oxford, UK  $^{2}$ VERSES AI Research Lab, Los Angeles, US
$^{3}$ School of Mathematics, Institute for Advanced Study, Princeton, US
$^{4}$ Department of Physics, University of Cambridge, Cambridge, UK
$^{5}$ Department of Computing and Mathematical Sciences, Caltech, Pasadena, US
$^{6}$ Faculty of Mathematics, University of Vienna, Vienna, Austria
$^{7}$ Research Network Data Science, University of Vienna, Vienna, Austria
$^{8}$ Institute of Logic and Computation, Vienna University of Technology, Vienna, Austria
https://ghosts.friederrr.org
We investigate the mathematical capabilities of two versions of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel evaluation scheme. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., mathlib, the Lean Mathematical Library), current datasets of natural-language mathematics used to benchmark language models either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets test, by using 1636 human expert evaluations, whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT and GPT-4 can be used most successfully as mathematical assistants for querying facts, acting as mathematical search engines and knowledge base interfaces. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if you aim to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!
*Corresponding author: simon.frieder@cs.ox.ac.uk.
Since its release in November 2022, the language model Chat Generative Pre-trained Transformer (ChatGPT) has rapidly become a widely known question-and-answer dialogue system. ChatGPT has been referenced in mainstream media across the globe [1-4] and across all major internet platforms [5, 6]. With similar reactions, the release of ChatGPT's successor, GPT-4, followed in March 2023 [7].
The performance of ChatGPT has been analyzed in a large number of exam-related use cases, with varying degrees of scientific rigor, ranging from detailed studies to anecdotal evidence. Use cases include passing the United States Medical Licensing Examination (USMLE) [8], scoring highly on the Psychology Today Verbal-Linguistic Intelligence IQ Test [9], and answering (and generating) Operations Management exam questions that were deemed to be within the scope of a typical MBA curriculum [10], all with a performance that elicited a positive sense of surprise from the authors. In turn, the performance of GPT-4 even surpasses that of ChatGPT on a large batch of academic and professional exams [7, Table 1]. Such strong task-related performance indicates that large language models (LLMs) could be frequently used as assistants in many domains.
In this paper, we introduce a new dataset called GHOSTS, which measures the advanced mathematical abilities of LLMs. Using this dataset, we perform a detailed analysis of the mathematical capabilities of ChatGPT on two of its versions, the 9-January-2023 and the 30-January-2023 version. Note that, according to the release notes, the 30-January-2023 version should possess "improved factuality and mathematical capabilities" [11]. We further examine the performance of GPT-4 on a smaller dataset called miniGHOSTS, which exhibits statistics similar to the larger GHOSTS dataset. We further make available microGHOSTS, which itself is a subset of miniGHOSTS, and is designed to facilitate pre-screen of language models, by incurring minimal human evaluation costs. Our analysis includes but is not limited to testing how many of the skills necessary to do professional mathematics can be emulated by these models. Examples of such skills are the ability to answer computational questions, the ability to complete mathematical proofs that have gaps or missing steps, the ability to solve questions that are more focused on deep insights and original solutions, such as those of mathematical olympiads, and the ability to survey the literature and think across domains. None of the previous benchmarks (see Section 2) cover such a broad range of mathematical abilities.
To achieve the goals outlined above, GHOSTS consists of carefully composed prompts aimed at testing different aspects of LLMs related to mathematical comprehension; see Section 3. This includes both hand-crafted prompts as well as samples from existing datasets that were devised to test models specifically trained for mathematical comprehension [12, 13].
For brevity, we use the expression "(Chat)GPT" to refer collectively to both the ChatGPT and GPT-4 language models. We refer to Appendix C for further details regarding different (Chat)GPT versions.
To evaluate the output of (Chat)GPT, we designed a thorough testing methodology, including warning and error codes that represent various possible failure modes of (Chat)GPT. We score (Chat)GPT's responses, report on the results using this methodology, and compare (Chat)GPT to a selection of state-of-the-art models trained for mathematical comprehension. In summary, the contributions of this article are threefold:
- Benchmark for testing the mathematical capabilities of LLMs: We introduce a new natural-language mathematics dataset, called GHOSTS², to test the capabilities of LLMs across a range of aspects regarding advanced mathematical comprehension; see Section 3. It consists of two subdatasets derived from state-of-the-art datasets of mathematical queries for language models. Additionally, we devise four hand-crafted subdatasets covering further mathematical tasks. Parts of our dataset consist of problems that were selected to have a high probability of not being in the data on which (Chat)GPT was trained; see tags D1-D3 from Table 1.
2The GHOSTS dataset is available at github.com/friederrrr/GHOSTS, and the project is hosted at ghosts.friederrr.org.
- Insight for mathematical use of (Chat)GPT: Based on our benchmark, we show for which types of questions and which domains of mathematics, (Chat)GPT may be useful and how it could be integrated into the workflow of a mathematician. On the other hand, we identify the failure modes, as well as the limits of its capabilities. This can aid future efforts to develop LLMs that perform better in mathematics. Our analysis is akin to a mathematical model card [14] in terms of intended use, metrics, evaluation data, and qualitative analyses, as mathematical strengths and weaknesses of (Chat)GPT are summarized; see Section 4.
- Evaluation of improvements of (Chat)GPT: We can further use our benchmark to track the mathematical capabilities of (Chat)GPT variants over time. As a first step, we analyze the impact of the upgrade from the 9-January-2023 to the 30-January-2023 version of ChatGPT, which promises a better mathematical performance according to the release notes. Then, we proceed to investigate what performance increases the successor GPT-4 brings; see Section 4.1.
As a language model, (Chat)GPT can be universally employed to perform mathematical reasoning and, therefore, has to compete with technologies in this space that are sometimes decades old. Performing mathematical reasoning in an automated way has a long history and can be traced back to 1959 [15], the most focus being devoted to proving theorems [16]. According to Harrison [17], there is a realization that classical approaches, using a symbolic encoding of mathematics, have reached a "plateau".
On the other hand, there is now a growing body of literature on learning mathematical relationships directly in a supervised-learning manner [18-20] or by using LLMs to perform mathematical reasoning directly on mathematics encoded in natural language [21]. Sometimes, the distinction is blurred, because architectures of LLMs can also be used in a supervised-learning setting and have been employed successfully in learning mathematical relationships, such as between the syntactical form of a function and its integral [13, 22].
Among the supervised approaches, we mention [13], where a Transformer architecture [23] was used to generate symbolic, closed-form solutions to integrals and first and second-order differential equations, which outperformed classical solvers<sup>3</sup>, such as Mathematica, MATLAB, and Maple by at least  $14\%$  on a test set of integration problems. On the task of solving differential equations, the Transformer-based approach still exceeds the classical approach, but by a smaller margin (at least  $4\%$  in the case of first-order differential equations and with more varied results for second-order equations).
Regarding LLMs, recent ones, for instance, PaLM [24] (released in 2022), are tested only on elementary-level mathematical reasoning datasets, such as the MathQA or GSM8K datasets [25, 26]. We hypothesize that this is due to a lack of advanced-level natural language mathematics datasets. Moreover, the results obtained indicate that the models at that time had difficulty with much simpler datasets than ours. For example, the version of PaLM with 540 billion parameters only correctly solves  $58\%$  of the problems of the GSM8K dataset, even with chain-of-thought prompting and access to an external calculator [24, Table 10]. This model nonetheless outperforms GPT-3 [27], which only achieves  $54\%$  on the same dataset. Variations of BERT [28] have been shown to only solve between  $28\%$  and  $37\%$  of the problems when fine-tuned and tested on the Algebra Question Answering with Rationales (AQuA-RAT) dataset [29], which is the direct predecessor of MathQA. For some models, such as BLOOM [30] or the LaMDA model [31] (both released in 2022), an evaluation of the mathematical reasoning capability is entirely missing. An up-to-date survey on mathematical datasets and the performance of various LLMs can be found in [32].
Most similar to our dataset is the NATURALPROOFS dataset [33] and the NATURALPROOFSGEN dataset [34]. In this paragraph, we illustrate the similarities and differences between these datasets and ours. NATURALPROOFS and NATURALPROOFS-GEN are similar among
3For a given prompt, the computer algebra system is considered to have failed if it does not provide a closed-form solution or times out after 30 seconds (in case of Mathematica).
themselves and cover graduate-level mathematics by focusing on data from ProofWiki $^4$  (the latter dataset), as well as on the Stacks Project $^5$  and two open-source textbooks (the former dataset). Using the IATEX source code, which is available for all these resources, annotated theorems and their proof graphs are extracted. The annotations consist of reference graphs highlighting references to other theorems or definitions, the idea being that these references capture the "skeleton" of a proof. This task resembles the mathematical abilities that the Named Theorem Proof Completion subdataset from the GHOSTS dataset evaluates (see Table 1), although (1) we only retrieve a single reference, and (2) (Chat)GPT, as far as known, does not use training objectives that make use of information from data annotation, in contrast to models evaluated in [33, 34].
Our framework pertains to general language model evaluation, which may be presented in a black-box manner (as is the case for (Chat)GPT), and therefore does not allow to leverage any additional information, such as reference graphs. This is also reflected in the human evaluation schema introduced in [34, Table 24], which classifies common model mistakes. As reference graphs form the foundation of how the mathematical proofs are engineered, many elements of the evaluation schema are strongly tailored toward this representation of mathematical data. Our benchmark is not reference-centric and therefore allows evaluations of any type of proof (including computations, as featured in the Symbolic-Integration subdataset, which we consider to be a particular kind of proof). Therefore, our methodology includes further and more general failure modes to make for a more fine-grained evaluation that explains the nature of the errors. We refer to Appendix A for further related works.
3 The GHOSTS, miniGHOSTS, and microGHOSTS Dataset
We assess the mathematical reasoning capabilities of two ChatGPT versions, 9-January-2023 and 30-January-2023, and of GPT-4 by first creating a collection of 709 prompts from various sources, and subsequently evaluating the models on (subsets of) these data points. We rate the corresponding outputs provided by the models and collect statistics, such as error types, output lengths, or the stability of the answer under prompt engineering, see Sections 3.2 and 4 and Appendices B and D. This yields a total of 1636 ratings by human experts.
We divide our dataset, the entire collection of prompts, into six subdatasets, called
- Grad-Text,
- Symbolic-Integration,
- Holes-in-Proofs,
- MATH,
- Olympiad-Problem-Solving,
- Search-Engine-Aspects,
which, in turn, consists of multiple files, see Table 1. The boldface letters make up the GHOSTS acronym. Details on motivation, composition, collection process, and intended uses of the GHOSTS dataset are summarized in our datasheet in Appendix H, Sections H.1, H.2, H.3, and H.5, respectively.
GPT-4 was evaluated on a subset of 170 prompts, which we call the miniGHOSTS dataset. Specifically, after having created the GHOSTS dataset, we heuristically selected a subset of 10 prompts from each file of the subdatasets included in GHOSTS, having the same mean rating and the same standard deviation (of ChatGPT's output) as the original file; see also our datasheet in Appendix H for more information. In this sense, these subsets can be considered to have the most relevance by capturing the "essence" of the model performance in the respective file. The role of miniGHOSTS is thus to reduce the costly evaluation procedure of the full GHOSTS dataset on a new language model. The microGHOSTS dataset represents a further reduction of the miniGHOSTS dataset, where one question was extracted from each file making up the miniGHOSTS dataset, for a total of 14 questions. The role of microGHOSTS is to allow rapid pre-screening of a language model, where each of the 14 questions was chosen to be a question representative of the mathematical problems in that file, as well as being a problem that (Chat)GPT typically struggled with. Reference solutions, explanations, and known LLM failure modes are provided and discussed for the microGHOSTS dataset in order to aid raters who are not mathematically trained. For more information, see Appendix E.
4https://proofwiki.org/
$^{5}$ https://github.com/stacks/stacks-project
Table 1: A summary of all the files from all the subdatasets comprising our GHOSTS dataset, together with their size, i.e., the number of prompts and their associated attribute tags. The tags  $Mi$ ,  $Qi$ , and  $Di$  relate to the level of Mathematical difficulty, the Question type, and the Out-of-Distribution type from Section 3.1, respectively. For the Olympiad-Problem-Solving subdataset, 24 further prompts were created where prompt engineering was applied, see Appendix 4.2. These 24 prompts do not count towards the 709 total prompts, only towards the 1636 evaluations.
Subdataset Name	Size	Comprised of the json file(s)	Tags
Grad-Text	28	W. Rudin, Functional Analysis (ch. 1)	M3 Q4
15	W. Rudin, Functional Analysis (ch. 2)	M3 Q4
37	J. Munkres, Topology (ch. 1)	M3 Q4
29	J. Munkres, Topology (ch. 2)	M3 Q4
21	R. Durrett, Probability Theory	M3 Q4
Holes-in-Proofs	60	Proofs Collection A	M3 Q1 Q2 Q5
52	Proofs Collection B Prealgebra	M1 Q5
50	Proofs Collection B Precalculus	M1 Q5
Olympiad-Problem-Solving	101+24	Olympiad Problem Solving	M4 Q4 D2
Symbolic-Integration	100	Symbolic Integration	M2 Q3 D1
MATH	50	MATH Algebra	M1 M2 M3 Q3 Q4
50	MATH Counting and Probability	M1 M2 M3 Q3 Q4
18	MATH Prealgebra	M1 Q3 Q4
20	MATH Precalculus	M1 Q3 Q4
Search-Engine-Aspects	30	Definition Retrieval	M3 Q2 D3
30	Reverse Definition Retrieval	M3 Q1 Q2 D3
18	Named Theorem Proof Completion	M3 Q2 Q5 D3
3.1 Subdatasets
The subdatasets that make up our GHOSTS dataset are summarized in Table 1. In the following, we describe each subdataset in more detail.
Grad-Text. This subdataset consists of a collection of books (R. Durrett's Probability Theory [35], J. R. Munkres Topology [36] and W. Rudin's Functional Analysis [37]) that are widely used in universities to teach upper undergraduate or first-year graduate courses in a degree in mathematics. We have used most of the exercises from these books' first and second chapters, except for [35], where we only used exercises from the first chapter, which was longer than the other books' chapters.
Holes-in-Proofs. This subdataset consists of a number of proofs sourced from math stackexchange.com, as well as some proofs sourced from books (S. Axler's Linear Algebra Done Right [38] and W. Rudin's Principles of Mathematical Analysis [39]), and from the MATH dataset [12], where parts of the proofs were intentionally deleted and the LLM was prompted to fill in the gaps: This was done either by (1) using a MISSING token, (2) finishing the proof early and prompting the LLM to complete it, or (3) explicitly asking for certain conditions or results.
Olympiad-Problem-Solving. This subdataset consists of a selection of exercises from A. Engel's Problem-Solving Strategies [40] book, which is often used to prepare for mathematical competitions. We selected and graded the LLM outputs on one hundred exercises drawn from all chapters.
Symbolic-Integration. This subdataset consists of random samples of integrals from the test set of [13]. There are three ways in which integrals are generated in [13]: Forward generation (FWD), Backward generation (BWD), and Backward generation with integration by parts (IBP). We sample 21 integrals from FWD test set, 20 integrals from the BWD test set, and 59 integrals from the IBP test set. As these integrals are given in Polish-prefix notation, a natural-language prompt conversion of them is unlikely to be witnessed in the training dataset of (Chat)GPT. The assessment was done by verifying the correctness of the output both by using Mathematica, as well as making use of the provided solutions
(in Polish notation), which [13] generated using SymPy. In particular, we notice that all integrals in this dataset have solutions that can be expressed using elementary functions.
MATH. This subdataset consists of a random sample of problems from the MATH dataset [12]. The latter dataset attaches a level of difficulty to each problem. We focused on two domains, Algebra and Probability Theory, and sampled an equal number of problems at each level of difficulty.
Search-Engine-Aspects. This subdataset consists of problems that were not sampled from a particular source but generated by a human expert in the field. In the file Named Theorem Proof Completion, we focused on prompting the LLM to provide proof outlines of various theorems that are sufficiently well-known within Functional Analysis to have names. In the Definition Retrieval file, we asked the LLM to correctly state various definitions centered around Functional Analysis and Topology. In contrast, in the Reverse Definition Retrieval file, we verified whether the LLM was able to deduce the name of a mathematical object by describing its properties.
Because input to (Chat)GPT is purely textual (at the time of writing), certain types of questions that might be stated and solved in a non-text-based fashion (e.g., questions involving graphical diagrams, without text explaining the diagram<sup>6</sup>, as occasionally occur in [40]), have been excluded. Our subdatasets can be categorized along the following dimensions (see Appendix B.1 for more details):
- Mathematical difficulty (ascending): (M1) Elementary arithmetic problems, (M2) Symbolic problems, (M3) (Under)graduate-level exercises, (M4) Mathematical olympiad problems.
- Question type: (Q1) Stating mathematical facts, (Q2) Overview-type review questions, (Q3) Computational questions, (Q4) Theorem proofs or puzzle solutions, (Q5) Proof-completion questions.
- Types of high out-of-distribution likelihood: (D1) Nontrivial problem encoding, (D2) Succinct solution, (D3) Spoken dialogue.
The existing datasets of natural-language mathematics are far from covering all possible combinations across these dimensions. In our well-crafted GHOSTS datasets, we have striven to cover each of these aspects individually, as can be seen in Table 1. The next section specifies the format of our dataset and the methodology for analyzing (Chat)GPT's output.
3.2 Format
The format of each of the subdatasets that make up our GHOSTS dataset follows the same convention. Each subdataset consists of JSON-formatted files, and our format is similar to, e.g., the AQuA-RAT dataset [29]. A single data point<sup>7</sup> in a file has the following form:
"prompt": "Let $X$ be a topological vector space. All sets mentioned below are understood to be the subsets of $X$. Prove the following statement: If $A$ and $B$ are compact, so is $A + B$, "output": "The statement is wrong in general. Consider the example $A = [-1,1] \times times \{0\} and $B = \{0\} \times times [-1,1]$ . Then $A$ and $B$ are compact but $A + B = [-1,1] \times times [-1,1]$ is not compact." "rating": "2", "errors": ["e3", "e5_2", "e5_4"], "warningcodes": [], "comment": "The given $A + B$ actually *is* compact.", "misc": ["46A03"], "ref": "Rudin-Functional Analysis-Second-Ed. Part1-ex3/d-page38",
See, e.g., Exercise 15 in [40, Chapter 2], which asked the reader to inspect a figure on which the problem is based.
The JSON object of an output of the 30-January-2023 version of ChatGPT, as identifiable by the timestamp at which the output was generated, is shown. The prompt comes from the "W. Rudin, Functional Analysis (ch. 1)" file from the Grad-Text subdataset.
We require each data point to have the same JSON keys as in this example, some of which may be empty depending on the prompt. Among the listed keys, the rating key stands out as the most fundamental one. Its value serves as a condensed representation of the mathematical capability of the tested language model, compressed into a one-dimensional measure ranging from 1 (lowest) to 5 (highest). A more nuanced and fine-grained perspective on the mathematical capabilities is provided by the errorcodes and warningcodes keys. The msc key denotes the mathematics subject classification. We explain each JSON key in Appendix B.2. For end-users of (Chat)GPT, it is desirable to avoid having a long-winded dialogue to arrive at a solution. Therefore, we require that (Chat)GPT provides us with the correct solution given only the input prompt without any subsequent interaction.
3.3 Human Input in Dataset Creation and Mathematical Evaluation
For all data points, the values of the keys rating, errorcodes, warningcodes, comment, and confidence were manually labeled, without any automation. The msc, ref, and timestamp keys were populated in a semi-automatic way, since their values change only slightly within the same subdataset.