SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two of the subdatasets, the MATH subdataset and the Symbolic-Integration subdataset, use prompts taken from existing datasets, the MATH dataset by [12] and the dataset comprising integrals from [13], respectively. This was done to compare how (Chat)GPT performs against existing state-of-the-art models that use these datasets, see Section 4. Nonetheless, significant additional annotation effort was involved, since, in both cases, the authors rated the output. Furthermore, in the second case, the data are publicly presented in a Polish notation format, and conversion was necessary. The prompts of the other subdatasets were hand-crafted by the authors. We refer to Appendix B.6 for more information on different aspects of human effort in dataset creation.
Will ChatGPT get you through a university math class? No, you would be better off copying from your average peer—unless it is undergraduate mathematics, for which GPT-4 can offer sufficient (but not perfect) performance.
If we take a rating of 3.5, the average between the lowest and highest rating, to be the threshold between success and failure, then Figure 1 shows that for a majority of subdatasets, both versions of ChatGPT will not pass. However, for GPT-4, the situation is different, and, on miniGHOSTS, it passes (sometimes barely) on all subdatasets files, except W. Rudin, Functional Analysis (ch. 2), which tests graduate-level mathematical knowledge and the Olympiad Problem Solving file, which tests mathematical problem-solving skills. We note that, unless otherwise stated, we do not use prompt-engineered questions in the results presented here (see Appendix 4.2).
We first focus on the results of the 9-January-2023 version of ChatGPT and note that the results for the 30-January-2023 are very similar, as can be inferred from the figures. On average, the 9-January-2023 version achieves a rating of 3.20 with a standard deviation<sup>8</sup> of 1.23. It performs particularly poorly on proof-based questions in the style of graduate-level exercises or mathematical olympiads, as well as more complicated symbolic calculations. We note that prompt engineering only slightly improved the results for such complex questions; see Appendix 4.2. However, in tasks that only required filling in gaps or stating mathematical facts, ChatGPT was mostly able to achieve a score above 3.5. In particular, ChatGPT was strong at recognizing the context of the question, and the notation of the output almost always matched the one given in the prompt, see Figure 4 in the appendix. Generally, Figure 1 indicates that the ratings closely correspond to how mathematicians would rank the difficulty of the exercises. In this context, we note that the length of the prompt does not have a clear effect on the rating; see Figure 9 in the appendix. We present results for different mathematical fields in Figure 5 in the appendix. For a detailed qualitative analysis
"confidence": "high",  
"timestamp": "2023-01-31"
<sup>8</sup>We use Bessel's correction term to obtain an unbiased estimate of the variance.
Figure 1: Average rating for each file in each subdataset (bold) of GHOSTS on the 9-January-2023 and the 30-January-2023 versions of ChatGPT and for miniGHOSTS on GPT-4. Note that the maximal ranking is 5 and the minimal ranking, where the question was at least understood, is 2, see Appendix B.4; the lower rating of 1 indicates that the answer completely misses the question. Thus, a reasonable passing grade, i.e.,  $50\%$  of points, corresponds to a score of 3.5, as indicated by the vertical dotted line. The error bars represent  $95\%$  confidence intervals.
of the results on the different subdatasets, we refer to Appendix D.1. Finally, we note that (Chat)GPT almost never expressed any form of uncertainty, even if its output has been completely wrong; see Appendix D.2.
Comparing ChatGPT to the performance obtained by [13], who correctly solved nearly  $100\%$  of the integrals in a collection of 500 test equations [13, Table 3], the 9-January-2023 version of ChatGPT achieves an average rating of 2.51 (standard deviation: 0.87) on our random sample of their dataset (after conversion from Polish notation to IATEX). Specifically, a rating of 2 is dominating  $70\%$  of the time, followed by a rating of 3 and 4 for  $13\%$  of the prompts each; see also Figure 7 in the appendix. GPT-4 achieves an average of 3.50 (standard deviation: 1.43), barely a passing grade, on the corresponding subset from miniGHOSTS. These scores trail far behind the performance achieved by the model in [13]. The situation is similar when comparing ChatGPT to Minerva [21, Table 3]. Their best model achieved an accuracy of  $50\%$  on the MATH dataset [12]. However, the 9-January-2023 version of ChatGPT achieves a perfect score only on  $29\%$  of our random samples from the MATH
Figure 2: A Sankey diagram of how the ratings evolve from 9-January-2023 ChatGPT to 30-January-2023 ChatGPT to GPT-4 (from top to bottom), with all models evaluated on miniGHOST. While grades on the 9-January and 30-January models are shuffled between the ChatGPT versions, the overall performance remains approximately the same. However, we observe a significant increase in perfect ratings, i.e., a score of 5, for GPT-4.
dataset (which is above the total average of  $25\%$  of data points across all subdatasets in which this version achieves a perfect score), see Figures 6 and 7 in the appendix. In contrast, GPT-4 performs substantially better and obtains a score of 5 on  $70\%$  of the corresponding questions within the miniGHOSTS dataset, see Figure 7 in the appendix.
4.1 Quantitative Comparison of (Chat)GPT Versions
The ensuing model version, 30-January-2023, overall performed similarly with an average rating of 3.29 (standard deviation: 1.28), although performance was inconsistent across subdatasets and on some subdatasets marginally better, see Figure 1. A significant jump in performance could only be observed for GPT-4, which achieved a substantially higher average rating of 4.15 (standard deviation: 1.12). We note that the evaluation of GPT-4 is only on the miniGHOSTS dataset, i.e., a subset of GHOSTS. Nonetheless, these findings send a clear message that the performance of GPT-4 dominates the performance of ChatGPT (both versions), see Figure 1.
Figure 2 shows how the ratings change between the different versions of (Chat)GPT. Surprisingly, one can see a shuffling of the grades for the two ChatGPT versions, even though the counts in each grade bracket stay approximately the same. For instance, there are roughly the same amount of outputs that received grade 4, yet less than half of the prompts were the same between model changes. Appendix D.4 provides different perspectives on this and reinforces the mixed performance increase that the 30-January-2023 model brings. For GPT-4, we see that the percentage of perfect ratings almost doubles, while the percentage of prompts that are not understood or completely wrong (i.e., ratings of 1 or 2) approximately halves as compared to the ChatGPT versions.
Analysis of (Chat)GPT's output and our warning codes reveal that GPT-4 provides even longer ("rambling") answers, whereas ChatGPT usually answered the question without giving any additional context about the topic, see Figures 6 and 8 in the appendix. The answer style of GPT-4 was often beneficial (resulting in better overall scores) but sometimes reduced the readability of the output. Furthermore, we found the behavior of GPT-4, compared to ChatGPT, to be more opinionated. Finally, despite its better overall performance, GPT-4 still seems to be vulnerable to mistakes in seemingly simple calculations. We refer the reader to Appendix D for further results on the models' performance.
4.2 Prompt Engineering
One interesting finding of our study is related to performing prompt engineering on mathematical questions. Prompt engineering was solely carried out on questions from the Olympiad-Problem-Solving subdataset, and prompt-engineered questions consist of lists consisting of
two JSON objects. These lists contain the original question that was not prompt-engineered, as well as the prompt-engineered question. The latter question is identified, as it contains the string <prompt engineered> as the value in the comment key. These lists containing prompt-engineered questions are in the same hierarchy in the JSON file as the other questions from the subdataset.
About  $20\%$  of the questions were prompt-engineered: In these cases, ChatGPT was instructed to proceed either step-by-step, by prefixing the sentence "Let's answer this question step by step." or by adding words that formulate the mathematical task in a more explicit way, i.e., by adding "Prove that..." or "Show that..." to the prompt<sup>9</sup>. Instructing ChatGPT to proceed step-by-step in this way was shown to increase the performance of GPT-3 on datasets that test mathematical reasoning (e.g., GSM8K); furthermore, this is a type of prompt engineering that is recommended by OpenAI in their cookbook to improve reliability<sup>10</sup>.
As a result of prompt engineering, for the 9-January-2023 version of ChatGPT, the number of wrong statements and computations (i.e., error codes e2, e3, and e4) decreased, while the number of errors rooted in faulty logic (i.e., error code e5) actually increased. Overall, prompt engineering improves the average rating only slightly, see Figure 3 from the Appendix.
For the questions from Olympiad-Problem-Solving that were selected for the miniGHOSTS dataset, we allow to sample from the entire Olympiad-Problem-Solving subdataset, since the goal of miniGHOSTS is not to measure prompt-engineering effects. Therefore, some of the questions in the miniGHOSTS version of the Olympiad-Problem-Solving subdataset contain prompt-engineered questions. The <prompt engineered> string was therefore removed from the comments in the miniGHOSTS dataset.
We have examined the behavior of (Chat)GPT across various tasks that test different aspects of mathematical skill. Contrary to the media sensation that (Chat)GPT has caused, (Chat)GPT is not yet ready to deliver high-quality proofs or calculations consistently. At the same time, the quality of the answers can be positively surprising. Moreover, our evaluation of GPT-4 on the miniGHOSTS dataset reveals promising improvements over ChatGPT's performance. In Appendix G, we collect the best and worst results for a number of selected subdatasets. The best responses can be seen to justify the media sensation. It thus seems fair to say that (Chat)GPT is inconsistently bad at advanced mathematics: While its capabilities generally drop with the mathematical difficulty of a prompt, it does give insightful proofs in a few cases.
However, (Chat)GPT falls short of achieving the same performance as models specifically trained for single tasks. These models, in contrast, lack the flexibility of (Chat)GPT, which is a universal tool suitable for any area of mathematics. In fact, (Chat)GPT's ability to search for mathematical objects, given information about them, is where it shines. For a user that is already sufficiently mathematically proficient to discern the correctness of (Chat)GPT's output, (Chat)GPT can be integrated as an assistant in the user's workflow. It can function as a search engine or knowledge base to speed up various lookup tasks, as they often occur at certain stages of mathematical research.
Due to the prohibitive annotation effort, the GHOSTS dataset is not yet large enough to significantly improve the mathematical capabilities of LLMs by fine-tuning them on GHOSTS; though we believe it is sufficiently comprehensive to allow an evaluation and comparison of LLMs (and more rapid evaluation using miniGHOSTS and microGHOSTS, respectively). We encourage other researchers to mine our dataset beyond the descriptive statistics that we computed to gain a deeper understanding of how LLMs behave on mathematical tasks. Finally, we hope that our work motivates other mathematicians to contribute to this growing field, by evaluating their LLMs on micro/mini/GHOSTS in order to establish a thorough benchmark for assessing the mathematical abilities of LLMs.
<sup>9</sup>Some prompts (e.g., the ones taken from the book by Engel [40]) only contain a mathematical statement, without a clear instruction; for example, "An  $a \times b$  rectangle can be covered by  $1 \times n$  rectangles iff  $n|a$  or  $n|b$ . From the context, one must conclude that this statement is correct and should be proven."
10 github.com/openai/openai-cookbook/blob/main/techniques_to_improve_reliability.md
This work was partially supported by the AXA Research Fund. The authors would like to thank Karan Desai for helpful discussions and advise on the datasheet.
[1] Sascha Lobo. Das Ende von Google, wie wir es kannten. Der Spiegel, Retrieved 2023-01-10. https://www.spiegel.de/netzwelt/netzpolitik/bessere-treffer-durch-chatgpt-das-ende-von-google-wie-wir-es-kannten-kolumne-a-77820af6-51d7-4c03-b822-cf93094fd709.
[2] John Naughton. The ChatGPT bot is causing panic now – but it'll soon be as mundane a tool as Excel. The Guardian, Retrieved 2023-01-14. https://www.theguardian.com/commentisfree/2023/jan/07/chatgpt-bot-excel-ai-chatbot-tec.
[3] Kevon Roose. The Brilliance and Weirdness of ChatGPT. The New York Times, Retrieved 2023-01-24. https://www.nytimes.com/2022/12/05/technology/chatgpt ai-twitter.html.
[4] Joe Rogan and Bret Weinstein. What ChatGPT Could Mean for the Future of Artificial Intelligence [Podcast episode]. In *The Joe Rogan Experience*. Episode 1919, Retrieved 2023-01-05. https://www.youtube.com/watch?v=kh5dN72GTQ8.
[5] teddy [@teddynpc]. I made ChatGPT take a full SAT test. Here's how it did: [Image attached] [Tweet]. Twitter. Retrieved 2023-01-13. https://twitter.com/teddynpc/status/1598767389390573569.
[6] Timothy Gowers [@wtgowers]. It's amusing when ChatGPT makes ridiculous mathematical mistakes. But of course, it's more interesting to find out what it can do well. Here's one example that wasn't bad: I gave it a very rough outline of a proof and asked it to fill in the details [Tweet]. Twitter. Retrieved 2023-01-13. https://twitter.com/wtgowers/status/1611750773607604224.
[7] OpenAI (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[8] Tiffany H. Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, and Lorie De Leon et al. Performance of ChatGPT on USMLE: Potential for AI-Assisted Medical Education Using Large Language Models. medRxiv, 2022.
[9] David Rozado. What is the IQ of ChatGPT? Retrieved 2023-01-09. https:// davidrozado.substack.com/p/what-is-the-iq-of-chatgpt.
[10] Christian Terwiesch. Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course. Retrieved 2023-01-04. https://mackinstitute.wharton.upenn.edu/wp-content/uploads/2023/01/Christian-Terwiesch-Chat-GTP.pdf.
[11] Natalie. ChatGPT - Release Notes. Retrieved 2023-04-03. https://help.openai.com/en/articles/6825453-chatgpt-release-notes.
[12] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021.
[13] Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint arXiv:1912.01412, 2019.
[14] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and transparency, pages 220-229, 2019.
[31] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, and Apoorv Kulshreshtha et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[32] Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai-Wei Chang. A survey of deep learning for mathematical reasoning. arXiv preprint arXiv:2212.10535, 2022.
[33] Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. NaturalProofs: Mathematical theorem proving in natural language. arXiv preprint arXiv:2104.01112, 2021.
[34] Sean Welleck, Jiacheng Liu, Ximing Lu, Hannaneh Hajishirzi, and Yejin Choi. Natural-Prover: Grounded mathematical proof generation with language models. arXiv preprint arXiv:2205.12910, 2022.
[35] Rick Durrett. Probability: Theory and Examples. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019.
[36] James R. Munkres. Topology. Prentice-Hall, 2000.
[37] Walter Rudin. Functional analysis. McGraw-Hill, 1991.
[38] Sheldon Axler. Linear algebra done right. Springer, 2015.
[39] W. Rudin. Principles of Mathematical Analysis. International series in pure and applied mathematics. McGraw-Hill, 1976.
[40] Arthur Engel. Problem-Solving Strategies. Springer, 1998.
[41] Tranquil Sea Of Math. Does ChatGPT code LaTeX and write proofs? Youtube. Retrieved 2023-01-12. https://www.youtube.com/watch?v=ge2N7VI_8P0.
[42] Richard Van Noorden @richvn@mastodon/social [@Richvn]. Huh. ChatGPT confidently gives the right kind of reasoning to solve this math problem, but whiffs on the algebra in the middle and gets the answer wrong [Tweet]. Twitter. Retrieved 2023-01-09. https://twitter.com/Richvn/status/1598714487711756288.
[43] Amos Azaria. ChatGPT Usage and Limitations. Retrieved 2023-01-15. https://halscience/hal-03913837.
[44] Ernest Davis. Mathematics, word problems, common sense, and artificial intelligence. arXiv preprint arXiv:2301.09723, 2023.
[45] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023.
[46] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
[47] Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint arXiv:2210.17517, 2022.
[48] The mathlib Community. The Lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs. ACM, 2020.
[49] Markus N. Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. arXiv preprint arXiv:2006.04757v3, 2020.
[50] Shuyin Ouyang, Jie M Zhang, Mark Harman, and Meng Wang. LLM is like a box of chocolates: the non-determinism of ChatGPT in code generation. arXiv preprint arXiv:2308.02828, 2023.
[51] Sherman Chann. Non-determinism in GPT-4 is caused by sparse MoE, 2023. Accessed on August 5, 2023.
[52] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, and Carroll L. Wainwright et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
[53] Carroll Wainwright and Ryan Lowe. InstructGPT: Training Language Models to Follow Instructions with Human Feedback. *GitHub repository*, Retrieved 2023-01-09. https://github.com/openai/following-instructions-human-feedback.
[54] Sarah Wiegreffe (sigmoid.social/@sarah) [@sarahwiegreffe]. If text-davinci-001 is a rough approximate to the model reported in the NeurIPS 2020 paper, and text-davinci-002 is InstructGPT in the 2022 preprint, then what is just "davinci"? Trying to reproduce results from a time before this naming existed [Tweet]. Twitter. Retrieved 2023-01-15. https://twitter.com/sarahwiegreffe/status/1583617355678355456.
[55] OpenAI. GPT-4 API waitlist. Retrieved 2023-06-06. https://openai.com/waitlist/gpt-4-api.
[56] OpenAI. Documentation - Models. Retrieved 2023-06-06. https://platform.openai.com/docs/models/gpt-4.
[57] OpenAI. OpenAI API Reference - Chat Completion Endpoint. Retrieved 2023-06-06. https://platform.opensai.com/docs/api-reference/chat.
[58] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, and Henrique Ponde de Oliveira Pinto et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
[59] Alexander Bogomolny. Pythagorean theorem, Retrieved 2023-08-10. https://www.cut-the-knot.org/pythagoras.
[60] Benj F Yanney and James A Calderhead. New and old proofs of the Pythagorean theorem. The American Mathematical Monthly, 3(4):110-113, 1896.
[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian richter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc., 2022.
[62] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023.
[63] Xi Ye and Greg Durrett. The unreliability of explanations in few-shot in-context learning. arXiv preprint arXiv:2205.03401, 2022.
[64] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
A Further Related Works 16
B Dataset Creation 16
B.1 Categorization 16
B.2 Format 17
B.3 Copyright and Licensing Terms 18
B.4 Data Sources and Labeling Policies 19
B.5 Mitigating Human Errors 21
B.6 Human Effort 22
B.7 Dataset Misuse 23
C Background details on (Chat)GPT 23
D Further Results 23
D.1 Qualitative Analysis of Subdatasets on ChatGPT 9-January-2023 23
D.2 (Chat)GPT's Confidence 25
D.3 Figures of ChatGPT's Performance (version 9-January-2023) 25
D.4 Comparison of (Chat)GPT Versions 29
E The miniGHOSTS and microGHOSTS Dataset 29
F Limitations and Reproducibility 30
G Best-3 and Worst-3 Across Selected Subdatasets 32
G.1 Grad-Text 32
G.2 Holes-in-Proofs (Proofs Collection A) 33
G.3 Holes-in-Proofs (Proofs Collection B Prealgebra and Precalculus) 35
G.4 Olympiad-Problem-Solving 37
G.5 Symbolic-Integration 38
H Datasheet for the GHOSTS Dataset 40
H.1 Motivation 40
H.2 Composition 40
H.3 Collection Process 42
H.4 Preprocessing, Cleaning, and/or Labeling 44
H.5 Uses 44
H.6 Distribution 45
H.7 Maintenance 45
A Further Related Works
In this section, we present further related works. For (Chat)GPT, most investigations related to mathematical reasoning consist of anecdotal evidence concerning its performance and its failure modes. Notable mentions on social media can, for instance, be found in [5, 6, 41, 42]. Unfortunately, a clear methodology is missing, as most of the results are scattered on various internet platforms and cannot be easily reproduced.
To the best of our knowledge, the only investigations into the mathematical capabilities prior to the appearance of our first preprint were undertaken by [43, 44]. However, these works only report a small number of qualitative results, often on rather simple mathematical tasks and without specifying the precise versions of (Chat)GPT. The latter reference reports results only on a few selected examples, while the former reference investigates ChatGPT's $^{11}$  ability to compute irrational numbers as well as to solve some elementary math word problems.
Recently, the dataset by [45] contains a systematic evaluation of ChatGPT on the GSM8K dataset [26], the MATH dataset [12], and the MMMLU-STEM dataset [46]. These datasets allow for an automatic evaluation using only accuracy as an evaluation metric. The LILA dataset [47] also consists of a set of diverse mathematical problems, spanning different dimensions and problem types. The problems are formulated in such a way that solutions can be given by Python programs, which precludes problems that involve more advanced forms of mathematical proofs. Further anecdotal examples of mathematical performance are presented in [45].