SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We note that it is neither possible to outsource the creation of these subdatasets to a crowdsourcing service, such as Amazon Mechanical Turk, nor is it possible to generate them automatically from code because advanced mathematical insight is required for the creation of each prompt (where applicable) and for providing the fine-grained evaluation of the mathematical capabilities. We note that already in the case of the MATH dataset, which contained less advanced mathematics than some of our subdataset contained, it was noted in [12]: "Because MATH requires a strong mathematical background to perform well on, and a long amount of time to solve problems, we were restricted to assessing six human participants and could not rely on crowdsourcing sites such as Amazon Mechanical Turk". Furthermore, unlike in the case of the MATH dataset, the answer to most of our prompts cannot be condensed into a few tokens (such as a number or a function), e.g., when the answer is a mathematical proof.
This raises the difficulty of the creation of more data since graduate-level (and in some cases, PhD-level) mathematics is required. The combined effort of devising mathematically
insightful prompts and carefully rating the output of (Chat)GPT amounts to 1636 prompt evaluations, totaling several hundreds of person-hours, see Appendix B.6. However, as a result of these efforts, our dataset goes beyond all the mentioned mathematical datasets for LLMs in Section 2 in terms of the different aspects of mathematical reasoning that are being tested.
B.7 Dataset Misuse
Our dataset does not contain personally identifiable information. Because the prompts of our dataset are compiled from a diverse collection, it does not contain systematic collections of mathematical exercises and their answers from a single type of test, which could invalidate certain school-level or university-level tests. We are not aware of any direct way of misusing our dataset, as its primary objective is the detailed evaluation of the mathematical reasoning performance of LLMs.
C Background details on (Chat)GPT
GPT-4, launched on 1st March 2023, is the latest model of the GPT lineage [7], being the successor of various versions of ChatGPT, the first of which was launched on 30 November 2022 [11]. These are all based on InstructGPT, which in turn is based on a trained GPT-3 [27], and fine-tuned using reinforcement learning with human feedback [52].
We note that already for models that predate (Chat)GPT, such as InstructGPT, where research articles and model cards [53] have been released, full reproducibility is not possible since the code and exact datasets have not been released. Furthermore, it was confirmed by OpenAI employees that for some of their models, launched prior to 30 November, a slight mismatch exists between the trained model that is accessible via the OpenAI web interface and the model referred to in the official paper [54]. This indicates how essential it is to document carefully which model our analysis pertains to and how we have accessed it. In our dataset, we have accordingly included time stamps for each prompt in order to be able to track, based on information provided by OpenAI, any changes in (Chat)GPT's version that have occurred.
We have exclusively used the GUI web interface to carry out the evaluation. This was necessary for consistency reasons since, at the beginning of our evaluation, API access was not yet widely available. At the time of writing, API access to GPT-4 is still limited, and a waitlist is employed, which made the use of the GUI web interface a necessity for GPT-4 [55]). We note that there exist no official documents that link the GUI web interface to the different model versions and possible model settings from the API. The 9-January-2023 and 30-January-2023 ChatGPT versions we evaluated are likely to be earlier instances of the newer model gpt-3.5-turbo-0301. This model itself is a "snapshot of gpt-3.5-turbo from March 1st, 2023" that will not receive further updates [56]. For GPT-4, at the time of writing, there exist four models gpt-4, gpt-4-0314, gpt-4-32k, gpt-4-32k-0314 that can be used for the chat completion API endpoint [57]. Additionally, for all models, there exist various settings when using the chat completion API endpoint, such as temperature or presence_penalty that influence the models' output, which cannot be controlled via the GUI web interface. It is also not known which values of these settings are used for the GUI web version. The only version identifier in the GUI web version is a generic "model version" link at the bottom of the page that links to the release notes [11]. The 9-January-2023 and 30-January-2023 model versions that we evaluated are the ones presented in the release notes [11] at the respective time.
D Further Results
D.1 Qualitative Analysis of Subdatasets on ChatGPT 9-January-2023
In this section, we go through common mistakes performed by ChatGPT, as well as notable observations regarding the output, one subdataset at a time. We focus on the 9-January-2023 version, see Section D.4 for more information regarding the other version. We note that the output of (Chat)GPT (and, generally, LLMs) is stochastic and therefore may differ on the same prompt. Nonetheless, clear trends can be observed, which we describe here. Individual outputs can be found in Appendix G.
Figure 3: Effect of prompt engineering on the rating (left) and the error codes (right) for the 9-January-2023 model.
Grad-Text ChatGPT, version 9-January-2023, performed best on simple set-theory and logic questions (the first chapter from the book Topology by J. Munkres [36]), which is reflected in its rating, see Figure 1. On the rest of the books, it performed substantially worse. Because of the confidence (high) with which it outputs the answer, the use of ChatGPT, version 9-January-2023, is particularly deceiving in this use-case since it may be intensively used by students studying these subjects.
Holes-in-Proofs ChatGPT, version 9-January-2023, correctly recognized most well-known results or concepts (e.g., filling in the mean-value theorem, given a proof that lacked a reference to it). However, the ability of ChatGPT to execute algebraic manipulations is surprisingly inconsistent. In some cases, ChatGPT executes complicated symbolic tasks with ease; in other cases, it fails on simple arithmetic operations or rearranging terms. The mistakes do not seem to correlate with the complexity of the algebraic expression. When ChatGPT makes an algebraic mistake, it mostly carries over this mistake reliably to the rest of the computation.
Olympiad-Problem-Solving On this subdataset, ChatGPT, version 9-January-2023, performed the poorest. From a mathematical point of view, these questions were also by far the most difficult, as they can pose difficulties even to professional mathematicians. A score of 3 was awarded when the answer started to show promise. However,  $75\%$  of the scores are 2 because the answer does not show any promise. No rating of 5 was awarded, and only one rating of 4 was achieved. This version of ChatGPT had a tendency to try and solve many questions using induction arguments. While this is not necessarily false, this was very far from the solutions given in the book, and this version's inductive proofs were easily seen to contain mistakes. In addition, ChatGPT often had difficulty understanding unconventional puzzles. For example, in the questions involving changing the color of squares on a chessboard, the solution offered by ChatGPT did not cover an  $8 \times 8$  chessboard. Sometimes it tried to solve the problem by changing only 5 squares, far from the 32 required. Similarly, the 9-January-2023 version of ChatGPT struggled to respect unusual constraints in the questions, resulting in 8 e6 errors, the highest number of e6 errors out of all subdatasets. In some cases where the problem seemed to require complicated mathematics but was actually solvable by elementary techniques, ChatGPT did not spot this but instead referred to the general theory of, e.g., diophantine equations. Interestingly, ChatGPT would sometimes say, e.g., that the question could be solved with these means but that this was hard, so the confidence score was downgraded in these cases to medium or low.
Symbolic-Integration The 9-January-2023 version of ChatGPT was dominated by systems that were trained specifically to solve integration problems [13]. In a number of instances, this version got the structure of terms right (for example, the number of summands in the output, as well as where factors had to be placed before summands), but it failed at concrete computations. Even very simple examples were not correct. For example, the antiderivative of  $x \mapsto x^2 / 2$  is evaluated to  $x \mapsto x^3 / 3 + C$ , where  $C$  is a constant of integration (the correct answer being  $x \mapsto x^3 / 6 + C$ ). For a number of prompts, this version claims there
is no closed-form solution for the integral with complete confidence when, in fact, there is a solution; only integrals that have an elementary antiderivative are in this dataset.
MATH On the questions related to Algebra and Probability theory, the 9-January-2023 version of ChatGPT got the reasoning often correctly. However, the most common type of error was e4, occurring  $36\%$  of the time (in total 62 times). This version of ChatGPT may struggle when confronted with standard operations, such as inverting fractions, least common multiples, and changing the sign of numbers when moving them from one side of the equal sign to the other. Often, in these questions, a correct solution requires performing multiple operations in sequence. In such cases, most often, at least one operation was wrong. This prevented the model from getting a rating of 5 on the output, which was only achieved for  $29\%$  of the questions.
Search-Engine-Aspects On the Search-Engine-Aspects file, the 9-Januar-2023 version of ChatGPT knew almost all the theorems that it was asked at a basic level but made mistakes when stating them. When it came to listing other results required for the proofs, this version typically requested way more than the necessary theory—occasionally even results that only follow from the theorem which was asked for (error code e5_5). On the Definition Retrieval file, this version had quite a good performance: it recited most definitions correctly. It sometimes got confused when being asked about distributions in the sense of elements of the dual space of test functions. ChatGPT, version 9-January-2023, strongly favors the notion of distributions in the stochastic sense. Similarly, for the adjective “closed”, where it chose to pick the context of algebra (instead of topology) and interpreted it to mean “algebraically closed”. On the Reverse Definition Retrieval file, this version had the strongest performance, being able to recover most definitions from their descriptions, with an average rating of 4.30 (standard deviation 1.14). This indicates the usefulness of ChatGPT as a general-purpose mathematical search engine. This subdataset is also the simplest from a mathematical point of view since no logical thinking is required, but only a name needs to be found.
D.2 (Chat)GPT's Confidence
(Chat)GPT is usually very confident, unlike other LLMs that we have experimented with. As an illustrative example, consider the following prompt testing the sensitivity to  $\mathrm{LATEX}$ -coded mathematics vs. Unicode-encoded mathematics:
Here are two statements: "Let (X, A, μ) be a measure space. Show that the signed measures λ: A -> R form a Banach space.", "Let $(X,A,\mu)$ be a measure space. Show that the signed measures  $\mathbb{S}$  lambda:A  $\backslash$  rightarrow R\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\$\
The response by ChatGPT is not phrased in order to show any nuance in terms of confidence (which is typical, even if ChatGPT is wrong):
Yes, the two statements are equivalent. They both express the same mathematical idea but use different notation and formatting conventions. The first statement uses parentheses to enclose the elements of the measure space and regular letters to represent the functions, while the second statement uses subscripted parentheses and uses the LaTeX formatting to represent the elements of the measure space and the function.
The response by Codex [58], another model that we briefly tested (but whose scope would have exceeded that of a single conference article) gives a cautions response and, unlike ChatGPT, is capable of voicing doubt:
I think they are equivalent, but I am not sure.
D.3 Figures of ChatGPT's Performance (version 9-January-2023)
In this section, we collect figures that extend the discussion in the main body and provide further views on the data and descriptive statistics.
Figure 4: Counts (annotation) and relative frequencies (color) of error and warning codes by subdatasets (bold) and files for ChatGPT 9-January-2023 on GHOSTS.
Figure 5: Average rating over mathematical fields for the 9-January-2023 version of ChatGPT on GHOSTS. The color depicts the occurrence of each MSC code, and only MSC codes that have at least 5 occurrences are shown. Note that the ranking is not indicative of the complexity of the fields since we do not use equally complicated exercises for all fields. The error bars represent  $95\%$  confidence intervals.
Figure 6: A comparison of the 9-January-2023 model, the 30-January-2023 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of percentages of ratings (right), error codes (middle), and warning codes (right).
Figure 7: A comparison of the 9-January-2023 model, the 30-January-2023 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of percentages of ratings on the different subdatasets.
Figure 8: A comparison of the 9-January-2023 model, the 30-January-2023 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of output lengths. Every interval contains  $20\%$  of the prompts, and the error bars represent  $95\%$  confidence intervals.
Figure 9: A comparison of the 9-January-2023 model, the 30-January-2023 model (both on GHOSTS), and GPT-4 (on miniGHOSTS) in terms of prompt lengths. Every interval contains  $10\%$  of the prompts of GHOSTS, and the error bars represent  $95\%$  confidence intervals.
D.4 Comparison of (Chat)GPT Versions
In this section, we collect figures which illustrate the differences and similarities between versions of (Chat)GPT. We note that even though the 30-January-2023 version performs very similarly to the 9-January-2023 version, there are some differences in the distribution of ratings, error codes, and warning codes, see Figure 6.
On the other hand, GPT-4 strictly dominates the ChatGPT versions in terms of performance. It always provides context around the question (whether that was asked for or not) and often gives useful (and correct) pointers that, for example, highlight the importance of a particular theorem. Figure 8 depicts the verosity of different (Chat)GPT versions and the achieved rating. However, we also note that the optimal level of verosity can depend on the mathematical background of the user. As a result, there have been significantly more warning codes of type w2 (i.e., rambling) for GPT-4, see Figure 6.
E The miniGHOSTS and microGHOSTS Dataset
The miniGHOSTS and microGHOSTS datasets consist of a subset of datapoints from the GHOSTS dataset. Their role is to enable faster evaluation of a given language model on advanced mathematics by shrinking the dataset size from 709 prompts (GHOSTS) to 170 prompts (miniGHOSTS), which represent about  $24\%$  of GHOSTS and 14 prompts (microGHOSTS), which represent about  $2\%$  of GHOSTS.
The miniGHOSTS dataset was created by using a heuristic to select a subset of prompts such that the mean and variance of the 30-January-2024 version of ChatGPT on this subset match the mean and variance of the 30-January-2024 version of ChatGPT on the entire GHOSTS dataset. The role of the miniGHOSTS dataset is to allow a full assessment of a model on a reduced dataset. This dataset is, therefore, not meant to assess the mathematical reasoning performance of other LLMs in absolute terms but rather to provide a relative comparison to the 30-January-2023 version of ChatGPT.
The prompts of the microGHOSTS were chosen by hand from the worst-performing questions from miniGHOSTS, or those that were deemed to pose mathematical difficulties to language models. The role of microGHOSTS is to obtain a preliminary assessment of a model by reducing the human evaluation cost to close to zero (to have initial information before potentially deciding on embarking on a more comprehensive evaluation).
Our overall evaluation experience leads us to observe that there exist the following broad categories that lead to ChatGPT performing problematically:
- instances where complex mathematical definitions are involved, which depend in turn on prior definitions (e.g., an absorbing subset of a topological vector space), and where something is asked about how these concepts interact;
- instances where the reasoning is predominantly geometric reasoning or where an insight is obtained by thinking geometrically;
- long chains of computations;
- vague definitions that require ChatGPT to understand that terms in mathematics are overloaded;
- problems when constructing examples;
- lack of ingenious ideas, such as needed for Olympiad-problem solving.
We have leveraged these insights to include prompts that specifically exploit these weaknesses.
Prompts that belong to the microGHOSTS dataset we tag by using the <microghosts> word as a value in the comment key. Further, the values of the comment also contain extensive solutions and explanations of the problem so that a good gold standard exists against which a given LLM solution can be compared by a human rater. We also include general explanations of the mathematical concepts that are involved to further aid general machine learning practitioners (who may not have prior mathematical training) if they want to test their language models' reasoning capabilities.
F Limitations and Reproducibility
In this section, we describe the limitations of what our dataset and our evaluations cover; some of these also are related to issues regarding reproducibility, which we also discuss below.
In terms of:
- model selection, we have focused on those models that, at the time of writing, are known to have the best performance among existing LLMs, either based on anecdotal evidence [3, 5, 6] or via previous benchmarks on more elementary mathematical reasoning tasks [7]. Because human evaluation has been very time-consuming, we have chosen to invest the available time solely in the evaluation of these models (and not older ones as well) to obtain a picture of their performance that is as accurate as possible.
- correctness, we have put a large number of measures in place to guarantee evaluations are as correct as possible (see Appendix B.5). Nonetheless, we cannot guarantee the absolute correctness of each evaluation. There are also some questions where, aside from human errors, fundamental evaluator disagreement occurs regarding the question of what rating should be given; this is a well-known problem in psychometrics and, in a simple instance, can be resolved by having multiple evaluators evaluate the same question. Given a fixed budget in terms of man-hours, we have chosen not to follow this approach because 1) we opted to be able to devise a larger dataset and 2) in light of the checks that we described in Appendix B.5, such an approach would result in diminishing returns on invested time.
An issue in this regard is the fact that an absolute ground truth is lacking for a mathematical dataset of this type that goes beyond simple arithmetical questions. In particular, in the case of proofs, these can be presented in myriad ways: On one hand, for the same mathematical fact, one can find conceptually distinct proofs. (See [59] for an extreme example of this, where 122 conceptually distinct proofs of the Pythagorean theorem are given.) On the other hand, one and the same proof can be presented in different ways, since the order in which the arguments are made and connected can be changed[22].
Assuming all the previous problems were, somehow, solvable, further complicating the matters is the fact that not all possible proofs are known, and new ones are being discovered for known statements (staying with the topic of the Pythagorean theorem, even millennia after its publication, new proofs have surfaced [60]). If a model were to output a proof that is not known, it would still require a human to judge correctness, as in this case, there may not be any ground truth available to compare against.
All of the above makes it very hard for the ground-truth data to be included in our dataset - and if it is included, it may not be meaningful for the outlined reasons. Some elementary datasets, such as the MATH dataset [12], have ground-truth data included in the comment. However, their ground truth is also susceptible to the issues we mentioned; human evaluation is necessary for absolute certainty.
(Chat)GPT's non-determinism, ground-truth absence, and author disagreements notwithstanding, the above shows that an effort to reproduce our results should lead to highly similar scores on each file of each subdataset. Due to the fact that our analysis is made per subdataset file and not per prompt, we are confident that a double-digit number of evaluations per file makes the conclusions that we draw robust.
- prompt engineering type, we have deliberately chosen to avoid more complex types of prompt engineering mechanics, such as using Chain-of-Thoughts [61], or Tree-of-Thoughts [62] because 1) the main focus of the paper is the standard performance of the tested language models, not their the performance augmented by various forms of in-context learning and 2) these types of prompt engineering methods require additional significant human effort to decorate the original prompts with example when carrying out few-shot prompting and to track the output. The human effort for the current dataset was already considerable; adding such prompt engineering methods would have further
$^{22}$ E.g., if proving statements  $C$  depends on three statements  $X, Y, Z$ , we could prove first  $X, Y, Z$  and then conclude  $C$ ; but we could also prove  $Y, Z, X$  and then conclude  $C$  - or construct the proof based on any other permutation of  $X, Y, Z$ . While all of these proof presentations would use the same ideas, syntactically, they would be different proof presentations
raised the evaluation cost while also increasing the complexity of our non-trivial evaluation methodology and 3) for GPT-3 and certain datasets and prompting engineering techniques, performance was degraded [63].
- datapoint-level MSC code coverage, our dataset covers 78 distinct codes spanning most areas of mathematics. Because there are 1636 distinct, MSC-classified outputs, by the pigeonhole principle, some of the MSC codes will necessarily be covered only by a small amount of MSC code (Figure 5 indicates those MSC codes that occur at least five times, although some MSC codes appear over one hundred times).
We also note that for particularly easy mathematical questions (e.g., simple arithmetical questions), no suitable MSC codes exist to classify the output since MSC codes typically classify more advanced mathematics[23]. Nonetheless, we have attempted to match them as well as possible and allow multiple MSC codes in order to classify the output as precisely as possible.
We further note that an exhaustive survey of (Chat)GPT's performance across every MSC code would necessitate a large, community-driven effort to set up an extensive database. Due to the high cost of rating each output, requiring specialized skills, this is something that no individual research group could reasonably do - but we hope that our approach is a starting point for such an effort.