SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Among LLMs prior to (Chat)GPT, Minerva [21], based on PaLM (discussed in Section 2), stands out, being trained in equal parts on websites that contain MathJax elements and arXiv preprints (additionally to general natural language data on which PaLM was trained). It achieves a score of roughly  $50\%$  on the significantly harder Mathematics Aptitude Test of Heuristics (MATH) dataset [12], which was sourced from various mathematical competitions. One distinguishing feature of the MATH dataset is that its problems admit a unique answer that can be condensed within a few characters (a number, for example). This is beneficial for the automatic evaluation of a model on such a dataset, since one can simply check the final answer, ignoring the step-by-step solution.
Finally, we would also like to mention the field of formalized mathematics, where large databases that encode advanced mathematical concepts exist, e.g., the Lean Mathematical Library [48]. Some of the ideas that we have used in this article, such as using prompts that formulate a task to fill in gaps in proofs, are echoed in [49] for datasets for formal mathematics consisting of expression trees. Yet, for the purpose of doing mathematics with large language models, these formal datasets cannot be leveraged since no straightforward way exists to convert them to natural language.
B Dataset Creation
B.1 Categorization
Our subdatasets can be categorized along the following dimensions (see Table 1):
- Mathematical difficulty (ascending):
M1 Elementary arithmetic problems, as found in the MATH dataset [12] at lower levels of difficulty.
$\boxed{M2}$  Symbolic problems (integration of functions) that can also be solved via a supervised-learning, data-driven approach to mathematics [13].
$\boxed{M3}$  (Under)graduate-level exercises from well-known textbooks [35-39] as well as questions from math.stackexchange.com, spanning diverse domains of mathematics.
$\boxed{M4}$  Exercises that are in the style of mathematical olympiad problems, such as those taken from Engel's Problem-Solving Strategies book [40].
- Question/prompt type:
<sup>11</sup>Using an unknown version of ChatGPT that predates the 9-January-2023 version.
Review questions, which ask to state or name certain mathematical facts correctly.
$\boxed{Q2}$  Overview-type review questions, which cut through an entire field of mathematics.
Q3 Computational questions.
Proof-based questions, which ask for a theorem proof or for a puzzle solution.
Proof-completion questions, where a proof that has gaps or is incomplete needs to be completed.
- Types of high out-of-distribution likelihood:
D1 Nontrivial problem encoding: The data points from the Symbolic Integration sub-dataset come from [13] and are publicly available<sup>12</sup>. Since the online training set uses Polish notation, it is very unlikely that (Chat)GPT has seen the corresponding prompts in IATEx before.
D2 Succinct solution: The solutions for the Olympiad-Problem-Solving subdataset are included in the book by Engel [40]. But the solutions are extremely concise, and simply repeating them would not show an immediate understanding of the problem.
D3 Spoken dialogue: The Search-Engine-Aspects subdataset is unlikely to be well represented in the data on which (Chat)GPT has been trained since its prompts resemble word fragments that might appear in a mathematical dialogue (e.g., an oral mathematical exam), rather than in a textbook.
B.2 Format
The dataset consists of a collection of UTF-8 encoded JSON files. We explain the JSON keys of each data point in our dataset in the following and also indicate whether its value is optional. If the value is optional, the key has to be present, but the value will be an empty array or string.
- prompt denotes the input that we provide to (Chat)GPT through its web interface at the URL chat.openai.com/chat, see also Appendix C. We use a new session for each prompt to avoid (Chat)GPT being biased by previous prompts.
- output denotes the raw output that (Chat)GTP supplies us with. In some cases, mathematical formulas were rendered in the web interface such that we copied them in  $\mathrm{LATEX}$ .
- rating is a number from 1 to 5 that shows how many points (Chat)GPT has scored, 5 being a perfect answer and 1 being the lowest rating. A detailed explanation of the rating policy that we followed is contained in Appendix B.4.
- errorcodes (optional) highlight a list of error types that illustrate the failure modes of (Chat)GPT in a more fine-grained way. Not all types of errors apply to all (sub)datasets: For example, an error code for a missing proof step would not be applicable on a dataset that tests whether (Chat)GPT can multiply numbers or find prime divisors. The detailed explanation of the error codes (and the warning codes; see below) that was provided to the annotators is contained in Appendix B.4. There, we also include a policy of how ratings and error codes have to be used together.
- warningcodes (optional) highlight any problematic aspects of (Chat)GPT; for example, (Chat)GPT might be rambling and providing the user with unrelated information or use a poor (but correct) way of solving problems.
- comment (optional) denotes any noteworthy commentary that an assessor of (Chat)GPT may make. This can be used to give a more detailed explanation of the output, provide reasoning behind awarding a certain error code or rating, or generally provide context. For some subdatasets, this key was used to indicate the difficulty level of the prompt, as well
$^{12}$ github.com/facebookresearch/SymbolicMathematics
as an official solution, if available, see Section 3.1. It was also used to indicate whether we used prompt engineering, see Appendix 4.2.
-  $\text{msc}$  denotes a list of mathematics subject classifications $^{13}$  (MSC) that pertain to the output. Note that we do not classify the prompt given to (Chat)GPT with MSC codes, as there may be no proper classification; for example, when (Chat)GPT is asked what the most important theorem in all of mathematics is $^{14}$ , it is meaningless to assign an MSC code to that prompt.
- ref (optional) indicates a reference to where the prompt was originally taken from (for some subdatasets, such as *Holes-in-Proofs*, we have used excerpts from various books or math.stackexchange.com; the original source was recorded as a value of this key). This key can have an empty value if the question was formulated by the authors and no authoritative source was plausible.
- confidence indicates how confident we have perceived (Chat)GPT to be when presenting us with its output. We allow values of high, medium, and low.
- timestamp denotes when the prompt was entered into (Chat)GPT. This can be used to track the version of (Chat)GPT; see Section 4.1.
The values of these keys within a single data point interact in nontrivial ways: If a rating of 5 is given, then it is expected that no error code is present—though there may be warning codes that are used. The error codes and warning codes are loosely in the spirit of a compiler throwing errors and warnings if it is given incorrect or sloppy code—although we have a role reversal, where the human is now the compiler, and the machine produces the code. In this sense, for some prompts, we have used multiple error and/or warning codes, which is why the corresponding values are arrays of strings. We use these codes to collect statistics on the behavior of (Chat)GPT; see Section 4.
For most of the subdatasets that make up our GHOSTS dataset, we have used  $\mathrm{IAT_{E}X}$  to encode mathematical formulas in our prompts. Our experiments have shown that (Chat)GPT can process  $\mathrm{IAT_{E}X}$ -coded mathematics well.
The usage of MSC codes can be useful for mathematicians who want to integrate (Chat)GPT in their daily workflow, as it allows them to know in which areas the model performs better and can hence be trusted more. Our dataset is very diverse, having a total of 78 MSC codes. The top short versions of these codes (first two digits) are 26 ("Real functions", 127 occurrences) followed by 00 ("General", 110 occurrences) and 46 ("Functional analysis", 77 occurrences), see also Figure 5.
B.3 Copyright and Licensing Terms
Because our dataset draws on multiple sources, which have different copyright holders, that have issued different licenses (or no licenses at all), we associate licenses on a fine-grained level to our data, to differentiate between parts where we are bound by the license of an existing dataset and the parts that we created, that we release under the following Creative Commons license: Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)<sup>15</sup>. By this license, one may not use those parts of the dataset that were created by us for commercial purposes, and one must give appropriate credit when using it non-commercially; if users are building on the GHOSTS dataset, they need to indicate the changes that were made and distribute their contributions under the same license as the original.
Licenses per datapoint: We associate licenses on a fine-grained level to our dataset in the following way:
- For each datapoint, one license applies to the value of the prompt key (which, in some cases, is governed by copyright where no license is issued, in which case we do not release it publicly; in that case, the value of the prompt key will be <copyrighted>). Which prompt values are subject to which licenses is fully detailed in Table B.1.
$^{13}$ A complete list of MSC codes can be accessed at the URL zbmath.org/static/msc2020.pdf.
<sup>14</sup>The answer is Pythagoras' theorem, according to (Chat)GPT.
<sup>15</sup>See https://creativecommons.org/licenses/by-nc/4.0/ for the detailed terms of the license.
Table B.1: A summary of all the files from all the subdatasets comprising our GHOSTS dataset, together with their license. In the case of the prompt values from the "Proofs Collection A" file, either no license was issued from the copyright holder (if the prompt value was taken from the books Linear Algebra Done Right by S. Axler or Principles of Mathematical Analysis by W. Rudin, cf. the ref key associated to that prompt value); or one of three Creative Commons Attribution-ShareAlike license, in different versions, depending on the date of the math.stackexchange.com post which served the source for the prompt value, see https://math.stackexchange.com/help/licensing.
Subdataset Name	Comprised of the json file(s)	License for value of prompt key
Grad-Text	W. Rudin, Functional Analysis (ch. 1)	No license specified17
W. Rudin, Functional Analysis (ch. 2)	No license specified17
J. Munkres, Topology (ch. 1)	No license specified17
J. Munkres, Topology (ch. 2)	No license specified17
R. Durrett, Probability Theory	No license specified18
Holes-in-Proofs	Proofs Collection A	More than one license
Proofs Collection B Prealgebra	MIT license
Proofs Collection B Precalculus	MIT license
Olympiad-Problem-Solving	Olympiad Problem Solving	No license specified17
Symbolic-Integration	Symbolic Integration	CC BY-NC 4.0
MATH	MATH Algebra	MIT license
MATH Counting and Probability	MIT license
MATH Prealgebra	MIT license
MATH Precalculus	MIT license
Search-Engine-Aspects	Definition Retrieval	CC BY-NC 4.0
Reverse Definition Retrieval	CC BY-NC 4.0
Named Theorem Proof Completion	CC BY-NC 4.0
- Another license, the CC BY-NC 4.0 license, applies to all other, remaining components of each datapoint; for the datapoints where we also created the prompt, these two licenses will coincide, as outlined by able B.1. We note that therefore, in particular, the value of the output, is also released by us under the CC BY-NC 4.0 license. This is consistent with OpenAI's terms, which at the time of writing state<sup>16</sup>: "You may provide input to the Services ("Input"), and receive output generated and returned by the Services based on the Input ("Output"). Input and Output are collectively "Content." As between the parties and to the extent permitted by applicable law, you own all Input. Subject to your compliance with these Terms, OpenAI hereby assigns to you all its right, title and interest in and to Output. This means you can use Content for any purpose, including commercial purposes such as sale or publication if you comply with these Terms."
Release policy: Some of the subdataset files contain prompts that are protected under copyright, and no specific license has been issued for them by the copyright holder: These are all the prompts from all files from the Grad-Text and Olympiad-Problem-Solving subdatasets, as well as selected prompts from the "Proofs Collection A" file. In such cases where a copyrighted prompt was used for which no license was available, we have opted not to withhold the value of the prompt key in the publicly released dataset. The ref key includes a detailed reference to the page where the original theorem or exercise was presented, so a reader can easily retrieve the original prompt.
B.4 Data Sources and Labeling Policies
Parts of the GHOSTS dataset were collected from books. These prompts were transcribed into LATEX. The output from (Chat)GPT's web interface was copied as-is, even if the output was not valid LATEX code. If the output contains rendered mathematical expressions, our
<sup>16</sup>https://openai.com/policies/terms-of-use
17Despite our best efforts to reach the copyright holder, we received no response, and, therefore, we do not have a license under which we can (re-)share the prompt.
18 Permission was given by the author to reproduce the exercises we used, but unclarities remain who the copyright owner is.
policy was to transcribe it to LATEX. Below are the policies that were followed by each assessor of (Chat)GPT's output regarding the rating, the error codes, and the warning codes:
-  $1 \to$  failure to understand the query (e.g., the user asks it something about number theory, and it responds with information about differential equations);
-  $2 \rightarrow$  query was understood, but the answer was entirely wrong (e.g., the user asks what the prime divisors of 111 are<sup>19</sup>, and it responds with 8 and 6);
-  $3 \to$  query was understood, but the answer was only partially correct (e.g., the user asks it what the prime divisors of 111 are, and it responds with 3 and 6);
-  $4 \to$  query was understood, and the answer was mostly correct (e.g., the user asks it what the prime divisors of 222 are $^{20}$  and it responds with 3 and 37);
-  $5 \rightarrow$  query was understood and answer was completely correct.
- e1  $\rightarrow$  missing examples or information (e.g., the user asks it what the prime divisors of 111 are, and it responds with 3, missing 37); this also applies, if (Chat)GPT ignores a part of the prompt (e.g., an equivalence needs to be shown, but (Chat)GPT shows only one direction);
- e2 → a few wrong/vague statements (e.g., the user asks it what the prime divisors of 30030 are $^{21}$  and it responds with 2, 3, 5, 7, 13 (wrong); or says that 2, 3, 5, and some other numbers are prime divisors (vague)); it can also denote a single statement, that is slightly vague;
- e3  $\rightarrow$  a lot of wrong/too vague statements (e.g., the user asks it what the prime divisors of 30030 are, and it responds with 2, 5, 8, 12, 13, 15 (wrong); or says that 2 and many other numbers are prime divisors (vague)); it can also denote a single statement, that is highly vague;
- e4  $\rightarrow$  wrong computations (i.e., an additional error flag to disambiguate between statements that are of computational nature or not);
- e5  $\rightarrow$  denotes wrong logic or wrong flow of arguments, which we further subdivide into specific flags, as we prohibit the use of e5 on its own (since it would be uninformative):
- e5_1  $\rightarrow$  (Chat)GPT claims that to complete a proof, statements need to be shown that are unrelated to the claim;
- e5_2  $\rightarrow$  a proof step is missing;
- e5_3  $\rightarrow$  an edge case has not been considered;
- e5_4  $\rightarrow$  an inference step is not supported (e.g., (Chat)GPT claims that from A follows B, but this claim is not true);
- e5_5  $\rightarrow$  circular logical argument (i.e., using the hypothesis to prove the hypothesis);
- e6  $\rightarrow$  the general set-up is understood, but the legal operations are not respected or misunderstood (e.g., we are given a puzzle where we are only allowed to add even integers, but (Chat)GPT changes the rules and motivates the solution by allowing the addition of odd integers; or (Chat)GPT misunderstands an adjective that has multiple mathematical meanings, such as "dual", which can mean either topological dual space or algebraic dual space).
The following policy applies for error codes: If a rating  $r$  with  $1 < r < 5$  has been given, then an error code is mandatory to explain the type of error that occurred. For a perfect score of 5, no error codes should be assigned (but warning codes can be assigned). If the
<sup>19</sup>They are 37 and 3.
20They are 2, 37, and 3.
21They are 2,3,5,7,and 11.
score is lowest, i.e., a rating of 1, error codes can be assigned, but do not have to: In the case where (Chat)GPT has not understood the prompt, there typically is no reason to further detail the type of error.
Warning codes
- w1  $\rightarrow$  (Chat)GPT is withholding essential information related to the prompt (e.g., the user asked it something about the integral  $\int_{-\infty}^{\infty} e^{-x^2} \, \mathrm{d}x$ , and it answers correctly but does not tell the user that the integral was actually a famous, named integral, i.e., the Gaussian integral);
-  $\mathsf{w2} \rightarrow (\mathsf{Chat})\mathsf{GPT}$  is rambling (i.e., after answering, correctly or incorrectly, (Chat)GPT tells the user much more details than the user wanted to know);
- w3  $\rightarrow$  (Chat)GPT is hallucinating (i.e., after answering, correctly or incorrectly, (Chat)GPT tells the user unrelated information);
- w4  $\rightarrow$  (Chat)GPT behaves weirdly (e.g., by using a weird proof structure (where applicable), using strange mathematical formulations, or by adopting a strange tone of the conversation or making opinionated statements);
-  $\mathsf{w5} \to (\mathsf{Chat})\mathsf{GPT}$  changes the notation from the prompt without being instructed to do so (e.g., the prompt contains a vector space  $\mathbf{x}$ , but (Chat)GPT calls it  $\mathbb{F}$ ).
B.5 Mitigating Human Errors
Any assessment procedure that has a human component is prone to introducing bias—in particular, a procedure involving manual work such as rating the model outputs. In the list below we describe safeguards that help to mitigate bias and human errors (such as typos) as well as streamlining procedures that we undertook:
1. Safeguards against  $\mathbf{IAT_{E}X}$  errors and typos:
Various typographical errors may appear due to incorrect LaTeX formatting. In this case, we noticed that (Chat)GPT was able to correctly infer what was intended (e.g., \ $cup\$  was correctly interpreted as \ $\$  cup\$). Similarly, we noticed that (Chat)GPT's output was stable under small "perturbations" of the prompt, such as minor typos. Thus, (Chat)GPT therefore provided its own safeguard against these types of errors.
2. Safeguards against encoding issues:
We presented clear instructions to each author who prompted (Chat)GPT on how to record and save the data in order to avoid any file encoding issues. In the end, all JSON files were inspected and streamlined to Unicode, if a different encoding was used.
3. Safeguards against unfair comparisons:
Clear instructions were given to all authors that used (Chat)GPT to ensure that the language model has, to the extent possible, an identical state and starts from a blank chat.
4. Safeguards against missing data and copy-paste errors:
Given a lack of API access in the early stages of our investigation (see Appendix C), there was a fair amount of data being copied from (Chat)GPT. To mitigate any copy-paste errors, several passes over the entire dataset, as well as automatic checks, were made to look, e.g., for potential inconsistencies, missing timestamps, and outputs not matching the prompts.
5. Safeguards against dataset misformatting:
Because the methodology authors had to adhere to is complex (see Appendix B.2 and B.4, we introduced a number of automatic checks to make sure the general format of each datapoint is consistent and in line with our rules. In particular, we have made sure that various entries are in the correct place and take the correct values (e.g., that no warning flag values are in the key for errors, that the ref key value has the correct format, etc.).
6. Safeguards against other unforeseen errors:
Random samples: Random samples  $(< 10)$  were drawn from each dataset, and a second assessor reviewed the rating. If deemed problematic, the original assessors were asked to re-evaluate.
Statistical checks: Additional statistical checks were carried out as plausibility checks to make sure no other unforeseen errors occurred: If prompts deviated from the average length on that dataset, they were flagged, the output was manually inspected, and, if deemed necessary, a re-evaluation was carried out.
We are aware that these measures are not exhaustive, but given a fixed time budget, we considered them the most feasible. We note that authors evaluated ChatGPT models in multiple stages, as the models became available, and that checks and corrections were made at the end.
Because our checks and corrections pipelines are complex and a number were made after all the evaluations on all models were obtained, we cannot exclude that the strings containing the prompts weren't altered in very minor ways (e.g., converting back to Unicode from a different, original, unknown encoding may introduce slight variations of certain special characters that slipped through inspection). However, because of the sufficiently large number of evaluations that we carried out in each file, which is the smallest unit on which we collect statistics, this will not affect any results. From the viewpoint of reproducing our dataset, this is also not an issue, since even for identical prompts, (Chat)GPT itself is non-deterministic; this continues even if its temperature parameter in the API is set to zero [50, 51]. Lastly, as noted above, (Chat)GPT's answers will almost always still express the same idea, even if minor variations of the same prompt are used, so even individual scores will stay unchanged in such a scenario.
B.6 Human Effort
The evaluation was carried out by a subset of the authors of this paper who have substantial mathematical expertise, ranging from master's degrees in mathematics to postdoc-level and professor-level positions at departments of mathematics. Assignment of prompts was done based on mathematical difficulty, with more senior mathematicians having received more difficult prompts. No third parties were involved.
Each of the 709 prompts of the GHOSTS dataset was evaluated on both the 9-January-2023 and 30-January-2023 version of ChatGPT; an additional 24 prompts were used to test the effect of prompt engineering on a single type of subdataset, see Appendix 4.2. We further evaluate GPT-4 on the 170 prompts of the miniGHOSTS dataset. This amounts to a total of 1636 prompt evaluations of advanced mathematics performed by graduate-level researchers.
We like to mention that our effort has occasionally unearthed small inconsistencies in existing datasets: For example, the "MATH Counting and Probability" file, which was sourced from the larger MATH dataset [12], contains the prompt "What is the value of  $101^3 - 3 \cdot 101^2 + 3 \cdot 101 - 1$ ?", which is neither about counting, nor about probability, but arithmetic (our MSC codes allow users to find such examples).