TheoremQA: A Theorem-driven Question Answering Dataset
Authors: Wenhu Chen
Ming Yin
Max Ku
Pan Lu
Yixin Wan
Xueguang Ma
Jianyu Xu
Xinyi Wang
Tony Xia
Venue: EMNLP 2023 Main (2023)
Keywords: Language Model
Question Answering
Math
Theorem
TL;DR: We collect the first theorem-driven question answering dataset to investigate the capabilities of large language models to understand and integrate STEM theorems to solve real-world problems.

Abstract: The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving fundamental math problems like GSM8K by achieving over 90% accuracy. However, their capabilities to solve more challenging math problems which require domain-specific knowledge (i.e. theorem) have yet to be investigated. In this paper, we introduce TheoremQA, the first theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems from Math, Physics, EE&CS, and Finance. We evaluate a wide spectrum of 16 large language and code models with different prompting strategies like Chain-of-Thoughts and Program-of-Thoughts. We found that GPT-4's capabilities to solve these problems are unparalleled, achieving an accuracy of 51% with Program-of-Thoughts Prompting. All the existing open-sourced models are below 15%, barely surpassing the random-guess baseline. Given the diversity and broad coverage of TheoremQA, we believe it can be used as a better benchmark to evaluate LLMs' capabilities to solve challenging science problems.

TheoremQA: A Theorem-driven Question Answering Dataset
Wenhu Chen, Ming Yin, "Max Ku, 'Pan Lu, "Yxin Wan,
Xueguang Ma, Jianyu Xu, 'iinyi Wang, "Tony Xia University of Waterloo, Canada
University of California, Santa Barbara, United States
University of California, Los Angeles, United States
Abstract
narrow subject. On the other hand, these datasets do not involve much Nomain-sspecife knowledge, aka theorem. Due to these two deficiencies, we be- lieve that these datasets are not ideal to benchmark the existing powerful LLMs (Brown et al., 2020; Tamkin et al., 2022; Chen et al., 202 Chowd- hery et al., 2022; Hoffmann et al., 2022; Taylor et al., 2022) due to their simplicity. In fact, the popular GSM8K dataset (Cobbe al., 2021), GPT- 4 (OpenAI, 2023) and PaLM-2 (Google, 2023) both already achieved 92% accuracy. Similarly, we tested GPT-4 (OpenAI, 2023) on the subsets of several other listed datasets in Table and observed 90+% accuracy in most cases. The only exception is MATH (Hendrycks et al., 2021) containing high- school math competition problems with SoTA per- formance around 50% (Zheng et al., 2023). How- ever, MATH (Hendrycks al., 2021) is focused on math skills rather than theorem.
The recent LLMs like GPT-4 and PaLM-2 have made tremendous progress in solving funda- mental math problems like GSM8K by achiev- ing over 90% accuracy. However, their capa- bilities to solve lems which require domain-speciffe knowledge (i.e theorem) have yet to investigated. In this paper, introduce TheoremQA, the first theorem-driven weession-answering dataset de- signed evaluate Al models' capabilities to apply theorems to solve challenging science problems. TheoremQA is curated by domain experts containing 800 high-quality questions covering 350 theorems' from Math, Physics, EE&CS, and Finance. We evaluate a wide spec- trum of 16 large language and code models with different prompting strategies like Chain-of- Thoughts and Program-of-Thouue that GPT-4's capabilities to solve these prob- lems are unparalleled, achieving an accuracy 51% with Trogram-of-Thougge Prompting. All the existing open-sourced models below 15%, barely surpassing the random-guss gollom-geessss base- line. Given the diversity and broad coverage of TheoremOA, we believe it can be used as bet- ter benchmark to evaluate LLMs' capabilities solve challenging science problems.
challenging math prob-
We found
In this paper, we propose the first theorem- driven QA dataset built on university-level theo- rems across Math, Physics, EE&CS, and Finance. The whole collection process takes two steps: (1) we first enumerate roughly 400 theorems in differ- ent subfields like algehra, number theory, graph theory, information theory, etc, (2) we ask domain experts to search for questions regarding these the- orems from different sources like Internet and Text- books. The domain experts will adjust these ques- tions to ensure the answers follow the desired for- mat for the ease of automatic evaluation. Through the careful construction process, we collected 800 high-quality question-theorrm-answee triples our final release version.
1 Introduction
A long-standing goal of AI systems is help hu- man beings solve challenging problems, especially more domain-specific problems. To benchmark the progress towards this goal, researchers pro- pose to evaluate AI systems' performance on dif- ferent math word problem (WMP) datasets. In recent years, there has been plethora of WMP datasets (Lu et al., 2023c), which we include in Ta- ble Most of these datasets are meant for funda- mental questions aimed Grade 1-12 students on
We evaluate wide spectrum of instruction- finetuned language and code models including GPT (Brown et al., 2020), Claude (Bai al., 2022), LLaMA (Touvron et al., 2023), Pythia (Bi- derman et al., 2023), CodeGen (Nijkamp et al., 2022), GLM (Zeng et al., 2022). StarCoder (Li et al., 2023), and CodeT5+ (Wang et al., 2023)
Authors ordered by contribution. Corresponding author email: crahhchen@uwaterlooo '.g. Taylor's theorem, Lagrange's theorem, Huffian coding, Quantum Theorem, Elasticity Theorem, etc

---

Taylor Expansion
Stoke's Theorem
Language
x=?
Lebesgue Theorem
Huffman Coding
Prompt
A
Jensen's Alpha
Program
Questions
Language Models
Theorem
Thought
Answer Verilier
Figure I: The overview of TheoremQA and the prompting strategies adopted.
Dataset
Domain
Level
Source
Theorem
DRAW (Upadhyay and Chang, 2015)
Algebra
Elementary School
Generated Generated
MAWPS (Koneel-Kedziorsee ct al., 2016)
Arithmetic
Elementary School
DRAWIK (Upadhyay and Chang, 2017)
Algebra
Elementary School
Generated
ASDiv (Miao al., 2020)
Arithm/Algebra
Elementary School
Internet
SVAMP (Patel et al., 2021a)
Arithm/Algeber
Elementary School
ASDiv
Math23K (Wang et al., 2017)
Algebra
Elementary School
Internet
TabMWP (Lu et al., 2023b)
Arithm/Algebra
Elem Middde School
Textbooks
NO
GSMSK (Cobbe et al., 2021)
Arithm/Algebh Algee/
Middle School
Annotated
NO
GEOS (Seo et al., 2015)
Geometry
Middle School
SAT
NO
GKometry3K (Lu et al., 2021a)
Geometry
Middle/High School
Textbooks
NO
GeoQA (Chen et al., 2021a)
Geometry
Middle/High School
Exam
NO
UniGeo (Chen et al., 2022a)
Geometry
Middle/High School
Textbooks
NO
ScienceQA (La ct al., 2022)
Science
Middle/High School
Texthooks
NO
MATH (Hendrycks et al., 2021)
Math
High School
Competition
YES
AQuA (Lingg al., 2017)
Arithm/Algebra
University University University
GMAT/GRF.
NO
MathQA (Amini et al., 2019)
Arithm/AAgebra
AQuA
NO
MathQA-Python (Austin et al., 2021)
Arithm/Algebra
AQuA
NO
FinQA (Chen et al., 2021c)
Finance
University
CrowdSource
NO
TAT-QA (Zhu et al., 2021)
Finance
University
CrowdSource
NO
TheoremQA (Ours)
STEM
University
Internet+Expert
350+
Table
List of existing Math and STEM QA datasets.
our dataset. We adopt two prompting methods: Chain-of-Thoughhe (CoT) (Wei al., 2022b) and Program-of-Thouggh (PoT) (Chen et al., 2022b) to prompt the large language models. We also inves- tigate how infuse the theorem into the thought process of LLMs and how to present the multi- modal inputs to the LLMs.
instruction-tuned language and code models scored below 15% in accuracy, barely exceeding the ran- dom guess baseline of 10%. Our human evalua- tion reveals that open-source models like Alpaca are making errors mainly due to their ignorance of the theorem, where 90% of the errors are not rectifiable. This stark gap between GPT and open- source models suggests that further enhancement strategies, such as science- focused pre-training fine-tuning, should be considered to narrow the per- formance disparity. Thirdly, explored the poten- tial do theorem- augorrem-agmmented generation. However, the simple strategy of concatenation did not yield significant improvement. We conjecture that a more complex integration strategy may be needed to achieve more gains. Lastly, we examined the per- formance of various multi-moda instruction tuned models on the multimodal subset of the TheoremQA dataset. Surprisingly, these models did not demon- strate significant performance gains over their text- only counterparts. This is mainly due the un- naturalness of the image, which consists of lots of diagrams and text. Such images are not well
In the course of our experiments, several no- table observations were made. First, GPT-4 (Ope- nAI, 2023) significantly outperformed all existing models, reaching an accuracy level of 51% when combined with ofgraam-o--Thuugeee Trailing behind GPT-4, the second most effective model was ChatGPT, achieving an accuracy of 35% through the same prompting method. Additionally, our human evaluation determined that half of GPT- 4's errors are caused by minor mistakes like calcu- lation errors, rounding errrrs, etc. We believe these errors could be easily rectified with more deliber- ate prompting strategy human intervention. This suggests that there still significant headroom for GPT-4 to achieve with more deliberate prompting strategies. Secondly, we found that all open-source,
prompting.

---

captured by existing visual encoder models. To sum up, our contributions are three folds:
Scaling model size, data, and computing are cru- cial to enable this learning ability. Later, Rac et al. (2021); Chowdhery al. (2022); Zhang et al. (2022); Touvron et al. (2023); Chen et al. (2021b) have proposed to train different types of LLMs with different training recipes. The capability to follow few-shot exemplars to solve unseen tasks is not existent smaller LMs, but only emerges the model scales up (Wei al., 2022a). More recently, GPT-4 (OpenAI, 2023) shows tremendous progress on lots of complex reasoning tasks spanning math- ematics, coding, vision, medicine, law, psychology. and more. Bubeck et al. (2023) shows that GPT-4 is already demonstrating more general intelligence than previous Al models. To further validate GPT- 4's capability to solve challenging reasoning tasks, propose TheoremQA the new benchmark further understand LLMs' upper limit.
We propose the first thcorem-driven question- answering dataset understand LLMs' capa- bilities to apply science theorems.
We comprehensively evaluate wide spectrum of 16 LLMs on TheoremQA.
We perform different analyses in the theorem integration and multimodal understanding as- pects provide detailed insights.
2 Related Work
2.1 Math Word Problems
Mathematical reasoning skills are crucial for pereral- intelligent systems, garnering sig- nificant interest from the research community. In the past, studies have explored the ability of NLP models solve arithmetic and algebraic prob- lems (Hosseini al., 2014; Koncel-Kedie Kedziorsk et al., 2015; Roy and Roth, 2015; Ling et al., 2017). More recently, researchers have introduced increas- ingly challenging datasets (Saxton et al., 2019; Miao et al., 2020; Amini et al., 2019; Hendrycks et al., 2021; Lu et al., 2021b; Patel et al.. 2021b) aimed at enhancing difficulty, diversity, and adver- sarial robustness. LiLA (Mishra al., 2022) pro- poses to assemble vast collection of mathematical datasets into single, unified dataset. LiLA also an- notates Python programs as target outputs for solv- ing mathematical problems. However, the existing datasets were mostly focused on grade school sim- ple mathematics. To further investigate the LLMs' capabilities to assist humans to solve challenging math problems, propose Theorem@A the first benchmark to enable research in this direction.
2.3 Reasoning with Large Language Model
To better unleash large language models' capabil- itics to solve complex reasoning tasks. Chain-of- Thought Prompting (Wei et al., 2022b; Kojima et al., 2022: Wang et al., 2022) was proposed, which aims at prompting the large language models to generate the 'thought process' before outputting the answer. Later on, several other works (Drozdov et al., 2022: Zhou al., 2022; Nye et al., 2021) also propose different approaches to utilize LLMs to solve reasoning tasks by allowing intermediate steps. Our method can be seen as an extension to CoT by leveraging an extra step of symbolic ex- ecution. Another line of work (Gao et al., 2022; Chen et al., 2022b) was proposed to adopt Python programs as the demonstration for the 'thought process' solve different reasoning tasks.
Dataset
Our dataset collection pipeline contains two steps:
2.2 Large Language Models
Theorem Enumeration Our aim was encom- pass a wide range of theorems. To this end, we be- gan by prompting Large I Language Models (LLMs), specifically GPT-4 (OpenAI, 2023), to enumerate popular subfields in Mathematics, Physics, Finance, and Electrical Engineering & Computer Science. The covered subfields are listed in Figure 4. Sub- sequently, we prompted GPT-4 propose plausi- ble university-level theorems relevant to these sub- fields. For instance, within the 'Calculus' subfield, GPT-4 might suggest the "Intermediate Value The- orem', 'Rolle's Theorem', and so on. After gath-
recent years, there has been a surge of research and development the area of large language models (LLMs) that have significantly advanced the field of natural language processing. GPT- 3 (Brown et al., 2020) demonstrated strong ca- pability to perform few-shot predictions, where the model is given description of the task in natural language with few examples. By using human-feedhack reinforcement learning, Instruct- GPT (Ouyang et al., 2022) has shown its unprece- dented capabilities to follow human instructions.

---

ering an extensive list of theorems, we assembled team of domain experts (holders of Masters and PhDs in Statistics, Electrical Engineering, Com- puter Science, and Finance) to refine the theorem inventory and supplement any omitted theorems. Ultimately, we collected upproximately 400 theo- rems, encapsulating a diverse range of topics within these fields. We then delegated these theorems to nine domain experts, instructing them to locate question/answer pairs from varied sources. During the annotation process, small number of theorems were discarded due to their evaluation complexity.
Question: Please the Stoke's therorem to evaluate [Js curl. dr' where F 2 3zyj + z and S is the part of = 5 x² - 3² above the plane z-1. S is oriented upwards.
oriented smooth surface that is bounded by a simple, closed, smooth boundary curve C with positive orientation. Also let F be a vector. We can compute the integral as follows:
Stoke's Theorem:
JO
FF.dt df /|. curlF ds
Answer: 0, Type: Float
Question Annotation Our problems were sourced from websites, books, or devised by the experts themselves. One challenge we encountered was the potential for questions found online to have been included in the training data. To mitigate this 'data contamination' issue, we encouraged domain experts to modify these questions. Another challenge arose from questions with answers in symbolic form, matrix form, figure form, etc. These presented significant obstacles for our automatic evaluation. To overcome this, instructed domain experts to alter the question so the answer would be limited to the following forms: (I) integer, (2) float, (3) list of integers/Bloass, (4) boolean, and (5) multiple-choice options. For instance, if the original question concerned matrix, we would revise it to ask about the trace of the answer matrix. This modification significantly streamlined the evaluation process. An example of this can be found in Figure 2.
Quesiton: Let W(t) be the standard Brownian motion. Find the probability of ()2))) 2).
Winer's Process The Wiener process is characterised by the following properties: W has independent increment. For every 0, the future increment Weta W are independent from the past W. has Gaussian increments, We+ W, has Gaussian distribution A/O0, u).
Answer: 0.186, Type: Float
Figure 2: Examples from TheoremQA. The first question requires the usage of Stoke's theorem tramsform the double integral into a line integral. The second question requires knowing the properties of Wiener's process.
Human-Level Performance To provide a rough but informative estimate of human-level perfor- mance. We randomly select 20 questions and as- sign these questions to the 4 Math&CS undergrad- uate students (average GPA) who have taken the required courses regarding these questions. The participants are given 24 hours with internet access to solve these questions. The four undergraduate students achieve 12/20, 15/20, 18/20, and 19/20 scores on these randomly sampled questions. From this experiment, we are more confident that an expert-level performance should be 100%.
Dataset Statistics Finally, we collected total of 800 questions over 354 theorems. Specifically, there 199 Math theorems, 52 Physics theorems, 55 Finance theorems, and 48 CS&EE theorems. There are 442 Math questions, 146 CS&EE ques- tions, 131 physics questions, and 81 Finance ques- tions. show the answer-type distribution Fig- ure 3. To further enhance the multimodality aspect of TheoremQA, we also include 51 questions with image input (diagrams), where the model needs to understand the visual input to answer questions.
4 Method
Our method for addressing these demanding ques- tions in the Theorem@A is comprised several dis- tinct modules, as outlined in Figure 1:
The majority of the questions in TheoremQA have float and integer the answers, which is more realistic than the existing multi-choice datasets like ScienceQA (Lu al., 2022) or AQuA QA (Ling et al., 2017). Therefore, the models are unlikely to take shortcuts to achieve high accuracy.
Prompting strategies:
We utilize two established prompting
Chain-of-Thought Prompting (Wei et al.,

---

float
Multimodal Input A small portion of our data (50 instances) includes images, such as diagrams, as supplemental input, particularly geometry questions. Since current LLMs don't support such multimodal inputs, we propose a solution: to em- ploy captions like Chameleon (Lu et al., 2023a). These captions describe the image and are then ap- pended to the LLMs' output an additional signal.
47%
option
27% %
9%
list
15%
integer
Experiments
bool
5.1 Model Descriptions
Figure 3: Answer type distribution in TheoremQA.
In our experiments, we mainly investigate the fol- lowing models:
2022b): This strategy prompts the language model to initially generate step-by-step thought process, eventually leading to the final answer.
GP73//.5CChaGGP7/GGT7
These are
instruction-tuned models from OpenAl"
PaLM-2: This is the instruction-tuner model from Google (Google, 2023).
Program-of-Thoughh Prompting (Chen et al.,
2022b; Gao et al., 2022): This strategy prompts the language model progressively generate program. The final answer is then derived by executing this program.
Claude-v Haaude-v/Clludd-insttan instruction-tuned models from AnthropicAl".
These are
Alpaca-13B: This model is based the LLaMA (Touvron et al., 2023). Alapaca is instruction-tuned by the 52K data generated from GPT-4.
By delegating computational tasks to external ex- ecutor, the problem-solving process is considerably enhanced in its reliability. This improvement re- sults in remarkable advancements existing math datasets being reported in (Chen et al., 2022b).
Vicuna- This model is based on the LLaMA (Touvron et al., 2023). Vicuna instruction-tuned by the 100K ShareGPT data generated by different GPT-based models.
Answer Extraction We observed that parsing the output from Large Language Models (LLMs) can be challenging due to two main issues: (1) The answer is often embedded within sentence, mak- ing it difficult to extract using regular expressions, and (2) The answer may not be normalized, such as 'pi/3' 33 12"00-e', e', which complicates comparison with the ground truth. To tackle these problems, we initially employ ChatGPT to identify the answer span within the model's output, then forward this string to WolframAlpha (Inc.) for normalization into a float, integer, or list.
OpenAssistant- 12B: This model is based on Pythia (Biderman et al., 2023). The model is instruction-tuned by OpenAssistant data*.
MOSS-instruct-i This model is based on CodeGen (Nijkamp et al., 2022), which is fur- ther instruction-tums tuned with instruction follow- ing dataset distilled from GPT.5.
Theorem Augmentation We explored the po- tential of enhancing large language models with retrieved theorem descriptions to assess their effect performance. One approach is to retrieve de- scriptions of the given theorems from the Internet supplement the LLMs' output. Another exper- iment involved prompting GPT-4 to generate text descriptions of the thcorem, which are then used as an additional augmentation signal.
StarChat-16B: This model based on Star- Coder (Li et al., 2023). StartChat is being instruction-tuned on OpenAssistant data and ShareGPT data.
hhtps://opeaai.com
https://open--assitaat..t
httpss//txxuu199/// http:///tssmm199./ggggu..o/oooo/ githhb..o/blogg/moss...m
https:///openaasissaat.ttt

---

Mathematics
Physics
Finance
Calculus
103
Kinetics
30
Economics
22
Combinatorics
57
Electromagnetism
21
Quantitive methods
14
Algebra
52
Atomic physics
11
Derivatives
14
Mathematical analysis
42
Wave
8
Fixed Income
11
Number theory
29
Optics
8
management
10
Geometry
27
Condensed matter
8
Investments
10
Numerical analysis
Particle
6
Total
Statistics
Statistical physics
6
Complex analysis
21
Relativity
7
CS EE
Probability theory
19
Celestial mechanics
6
Signal processing
47
Stochastic process
16
Thermodynamics
5
Graph theory
34
Group theory
11
Quantum
5
Information theory
29
Functional analysis
10
Classic mechanics
5
Computer networking
23
Real analysis
7
Fluid mechanics
5
Machine learning
13
Total
442
Total
131
Total
146
Figure 4: Subfields TheoremQA under Math, Physics, Engineering, Finance.
InstructCodeT5:: This model is based on CodeT5+ (Wang et al., 2023). Instruct- CodeT5+ is further instruction-tune tundd on Code Alpaca data' to follow instructions.
Open-source Models For the open-source mod- els, we found that their performance is much be- hind. To better understand their accuracy, also provide the random-guess baseline of 10%. We test both prompting strategies, however, their re- sults consistently lie in the range of 10- 14%. The results indicate that these open-source LMs are still struggling with more complex mathematical rea- soning tasks in TheoremQA. Given that ChatGPT of similar size is able to achieve much higher per- formance, we believe the parameter size is not the only cause. There is still significant amount of ef- fort during pre-training or supervised fine-tuning instill enough scientific knowledge into the models' parameters to close the gap.
5.2 Main Results
We demonstrate our main results Table 2. We will summarize different findings in the following:
Closed-source Models For GPT-3 (text-davinci- 002) and GPT-3.5 model, since these two models are not Chat-based models, we need to demonstrate one example ensure to help them generate outputs of the desired format. With CoT prompting, GPT-3 ((ext-davinci-002) and GPT-3.5 models are only achieving 16.6% and 22.8% accuracy. By adopting the program the intermediate reasoning form, both models gain reasonable improvements. For Claude-vl, we found that it is matching the performance of GPT-3.5. ChatGPT outperforms GPT-3.5 and Claude-v1 significantly by 8%, which indicates ChatGPT's capabilities to perform com- plex numerical reasoning. GPT-4 is the strongest model being evaluated, which beats all the rest models by huge margin. With Chain-of- Thooughsss prompting, GPT-4 can outperform ChatGPT by 13%. With Program-of-Thoughhs prompting, GPT- can outperform ChatGPT by 16%. Though some other models have shown to match GPT-4 on sim- ple tasks, GPT-4's capability to solve challenging tasks seems unparalleled.
Program of Thoughts Analysis From Table 2, we observe that PoT brings consistent improvement over CoT on GPT_* models. Different GPT-* mod- els can normally yield gain of 5-8% accuracy. In contrast, Claude-vl and StarChat are almost obtain- ing the same accuracy. To better analyze where the gains are coming from, we plot Figure under- stand how many of generated Python programs are actually 'executable'. As can be seen, both Star- Chat and CodeT5+ are having trouble generating 'runnable' programs with only 40% programs being executable. Claude-vl 1 is able to increase the valid- ity of the generated programs to 60%. In contrast, GPT3.5 and ChatGPT can further increase the ratio to around 80%. GPT-4 is extremely accurate in generating programs, where 92% of the generated programs are runnable. Such high executable ra- tio explains why the gain brought to GPT.* model
"https//githubbcom//shi2211 w4t0:/0000114/////// 14/codcalpaca

---

0.92
0.72
Oriit
0.5
0.4
0.399
Fig
Fig. Qlb
GPT3
GPT3.5
CharGPT
GPT4
Claude
StarChat
Code
Question: Are the circuits shown in Fig. Qla and Fig. Q1b are identical in terms of the Tranfer functions.
Figure 5: Ratio of Executable Python Program of differ- ent models with PoT prompting.
BLIP Caption: diagram of a block diagram with
block diagram a block diagram.
Figure 6: An example of Multimodal question.
is much higher than Claude-v1 and StarChat.
9.35
5.3 Additional Result
7.6
Theorem Augmentation We also investigate whether feeding theorem as an additional text con- dition would help the model better solve the prob- lem. Specifically, we ask GPT-4 to generate paragraph describe the theorem, which post- processed ensure correctness. We feed the the- orem in the prompt different language models to see the performance change and plot our find- ings Table 3. For all the evaluated scenarios, we found that the improvement is limited to within 1 %. Unlike the Text or KB knowledge, theorem knowl- edge is more abstract and symbolic, simply con- catenating the theorem definition is not enough. We believe a more sophisticated augmentation scheme is needed to truly help the model understand and apply the theorems solve problems.
3.7
Alpaca
VisualGLM
LLaVA
Claude
GPT-3
CharGiT
GPT4
Figure 7: Accuracy on the Multimodal Question Subset
equivalent, Alpaca, with the addition of a visual encoder not significantly enhancing the results. We hypothesize that the current visual encoding mod- ules may not be suited for representing these dia- grammatic images, resulting in these less than ideal outcomes. We believe these multimodal questions remain a challenge for the research community, and we eagerly anticipate further advancements in addressing these multimodal scientific questions.
Multimodal Questions Our aim was to assess how effcetively the current method could tackle multimodal questions (those with image inputs) in the TheoremQA dataset. An example is illustrated in Figure 6, where image is converted into 'cap- tions' by BLIP (Li et al., 2022). We graphed the results from over 50 multimodal question subsets in Figure 7. Notably, this subset posed substantial challenges; none of the models were able achieve an accuracy rate 10%. This primarily due to information loss during the captioning process.
Error Analysis We conduct detailed error analy- sis on 200 erroneous cases from different models to analyze their error distribution. Specifically, we pick GPT4, ChatGPT and Alpaca to understand their error sources. We include the following er- ror types: (E1) the model does not even know this theorem, (E2) the model does know the theorem, but uses the wrong formula algorithm, (E3) the model knows the theorem and the formula, the er- ror only caused by minor calculation mistakes. The severity of the error decrease as the error num- ber increases. We plot our findings in Figure 8, where the bar indicate the percentage of specific error types. We can observe that almost half of the errors made by GPT4 are non-critical with caused by minor calculation mistakes. This error analysis suggests that there is still significant headroom for GPT4 to improve with more deliberate prompt- ing strategies or human intervention to mitigate these minor errors. In contrast, Alpaca's errors are mainly caused by not knowing the theorem at all.
In light of this, we conducted further evalua- tions on two multimodal instruction-tunee models, LLaVA-13B (Liu al., 2023) and VisualGLM- 6B (Zeng al., 2 These models utilize visual encoder (either CLIP (Radford et al., 2021) or BLIP (Li et al., 2022)) to encode image input, which is then integrated with language models for multimodal conversation. However, these models demonstrated performance similar to their text-only

---

Model
Integer
Float
Option
List
Math
CS&EE
Physics
Finance
Random Guess
0
38.9
65.5
10.0
24.7
4.9
10.5
Chain of Thoughts (CoT)
GPT-3
11.7
27.8
6.8
15.8
2.3
12.3
16.6
GPT-3.5
13.0
14.3
50.0
13.7
22.6
23.5
22.8
ChatGPT
32.4
22.3
50.0
20.5
31.0
41.1
16.8
28.4
30.2
GPT-4
36.7
66.7
37.0
74.6
43,9
30.5
51.4
43.8
PaI.M-2
22.8
61.1
23.3
71.6
31.0
47.3
19.8
27.2
31.8
Claude-vI
18.1
19.4
27.8
15.1
61.2
21.7
42.5
13.7
28.4
24,9
Cluade-instant
19.9
16.7
44.4
17.8
53.4
21.5
36.3
14.5
27.2
23.6
Alpaca (13B)
11.1
6.9
27.8
2.7
45.7
12,9
27.4
3.8
9.9
13.5
Vieuna (13B)
6.9
16.7
2.7
12.2
24.0
12.3
12.9
OpenAssistant (12B)
5.0
22.2
1.4
4.9
MOSS (16B)
5.4
24.2
2.4
11.3
28.4
1.6
8.9
12.2
StarChat (16B)
4.9
22.3
1.9
10.7
6.8
Program of Thoughts (PoT)
GPT-3
17.1
15.9
22.2
9.6
49.1
23.3
25.4
8.4
17.3
20.6
GPT-3.5
23.6
19.9
50.0
21.9
61.2
26.7
41.1
14.5
30.9
27.8
ChatGPT
31.0
35.0
38.9
21.9
54.3
35.7
35.6
26.7
49.4
35.6
GPT-4
44.4
50.7
66.7
39.7
78.4
$2,0
51.4
45.8
66.7
52.4
Claude-v1
17.1
21.8
33.3
6.9
62.5
23.1
37.5
17.1
28.4
25.9
StarChat (16B)
0.0
3.0
13.6
17.6
11.3
InstmccCodeTT5+ (16B)
8.9
6.3
0.0
6.9
45.2
13.8
17.9
4.2
5.1
11.6
Table 2: Results for CoT and PoT prompting on TheoremQA. We report the accuracy over different fine-grained question types and scientific fields.
Model
Method
Thenrem
All
I GPT4 Iu ChatGPTIL Alpaca
ChatGPT
CoT
30.2
0.75
ChatGPT
CoT
30.8
Claude-v1
CoT
24.9
Claude-v1
CoT
0.38
0.24
ChatGPT
PoT
35.6
0.15
ChatGPT
PoT
35.8
Alpaca-I 13B
CoT
13.5
F1
F2
E3
Apaca-13B
CoT
14.2
Figure 8: Error Analysis for GPT4, ChatGPT and Al- paca. Error severity: E1 > E2
E3
Table Results for CoT and PoT prompting with addi- tional theorem conditions.
Case Study We list few successful and failed examples generated by GPT-4 in Figure 9 to do side-by-side comparison between chain- of-thoughts prompting and program-of-thouge prompting. the first example, the question is regarding 'orthogonal projection theorem'. As can be seen, Chain-of- Thh-ugghsss prompting requires very long paragraph to generate the results. We prompted GPT-4 few times with the same in- put and the results seem unstable. Sometimes the model will make tiny computation mistakes in the middle to derive the wrong answer. In contrast, the program solution brief and concise, which leads to rather stable outputs. For the second ex-
ample, the computation requires 'for loop" to it- cratively compute delta values for Riemann Sum. We found that such problems also more natural for programs solve. Through these examples, can see GPT-4's unprecedented capabilities solve these difficult math problems even without any demonstration or hints.
We also show some examples in Figure 9 to com- pare the results of CoT and PoT prompting. We can see that the PoT can significantly shorten the output sequence length. By leveraging the addi- tional tool, PoT is able to significantly lower the task difficulty.

---

Question: Find the orthogonal projection of [9, o, O] onto the subspace of RM4 spanned by [2, 2, 1, 0J and [-2, 2, 0, 1].
The orthogonal projection of a vector onto a subspace W spanned by a set vectors (v1, v2, ... vn) is given by the sum of the projections onto each of these vectors, In case, the vector is v 9e
O, O, 0] and the subspace W is spanned by = [2, 2, 1, 0] and w =0-2 (-2, 2, 0, 1]. The projection v onto u given by: proj_u(v) = ((v"u) 2
def solve():
Import numpy as np
# Define the given vectors
where "v' denotes the dot product, and Iluil is the norm of u.
np.array[[1 0, 0, O)) np.array(2, 2, 1, Oj)
And the projection of v onto w is given by:
proi_w(v) ((vww) (lwll"2)
The total projection onto the subspace W then given the of
v2 np.array(-22 2, 0, 1D
these individual projections:
proj [(v) = proj_urvW) u(V) + proj_w(v)
# Calculate the orthogonal projection
Now let's compute these values:
A np.column_ stack((v1, proj_ vojjmmtrix A @ np.linalg.inv/A. @ A) @ A.T
np.coluum_sttck((11,, v2))
The dot product
(9
(0 2) (0
The norm llull/2 = (2 2 1 0*2) 9
proj_e1 @e proj_matris @ et)
So, proj_u(v) u(v) = ((18) /9) [2, 2, 1, 0] 4, 2| Similarly, the dot product v-w
-2) 2)) (0 +) (0 (0-1)= -18
return proj_e1 vo._lstt
The norm lIwil (-212 2 0*2 1*2) =
So, proj_w(v), ((-18) [-2, 1] = -4, 0.0,2. -2]
The total projection onto the subspace W is then given by the sum of these individual projections:
print(solve()
proj W(v) proj_u(v) proj) proj_((v) [4, 4, 0] + -4, 0, 2 [8, 0,
So, the orthogonal projection of 9e 1 onto the subspace of RM4 spanned by [2, 2, 1, 0] and (-2, 2, 0, 1] is [8, 0, 2, -2).
Chain of Thoughts
Program of Thoughts
Question: Let rectangle R= [1, 2.5] * [1, 2]. Calculate the Riemann Sum S $3(3 for Vint \int_(R) xy dA for the integral, using the lower-left vertex of rectangles as sample points.
The given region R is rectangle from [1, 2.5] in x and [1, 2) in y. are asked find the Riemann sum S_(3.2) which means equal subintervals and
def solve(:
# Define the limits of integration
will partition the interval the interval in into equal subintervals.
x_min, x_max 1, 2.5
y_min, y_max 1,
# Define the number of rectangles in each direction num_recttee cuu__recangiee_..
The width of each subinterval x, Σx, (2.5 1) 0.5. The height each subinterval in 4y, (2 1 0.5. Now, are sampling from the lower left corner of each subrectangle. This means we'll be evaluating our function f(x, y) = xy points (x, y) where ranges over (1. 1.5, 2). and 1.5).
num_ num_rectangies_J =
# Calculate width and height each rectangle
delta_ x ___maxx---_ -__min) num rectangles
delta_y = (y_max y__mn)/ / /nummnnrreee rrt_rrrttnngee_y
ranges
# Initialize the Riemann Sum fiemann_sum _umm_ =
The Riemann sum is given by:
-_(3,2) Ax by f(x_i, y_j) where the vertices of the subrectangles.
# Iterate through the rectangles for in gange(rum_rectangles x):
is all pairs (x_i, y_) from the lower left
for j in range(num http:///unnurettngessssse
00. 0.5 [(11 + 11.5 1.51 1.51.5) + (1.51 + 1.51.5
Calculate the lower-left vertex the rectangle
21.5)]
x_min delta_x
=0.25 53
00.5 =.5 2.5
y y_min j delta_y
Calculate the value the function at
So, the Riemann sum S (3.2) for the double integral over R f(x, y)= = with respect area, using lower-left vertex of each subrectangle as the sample point, is 2.125.
xxy = x "y
# Add the contribution to the Riemann Sum
riemann_sun Samaan_ssmm_ao += f_xy delta_x" delta_y
return riemann_sum
Chain of Thoughts
Program of Thoughts
Figure 9: Case Study of GPT-4 generation with both prompting strategies.
6 Conclusion
tial to leverage more science-related pre-training or fine-tuning close the gap. On the hand, we found that the multimodal science questions are still extremely challenging for the existing visual LLMs. We believe more specialized visual encod- ing models are needed to better represent diagrams in these science questions.
In this paper, propose the first -riven science question-ansswring dataset and evaluate different LLMs on it. Though GPT-4 can achieve strong performance on our new dataset, the existing open-source LLMs are still struggling to achieve reasonable performance. We conjecture it is essen-

---

Limitations
Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, et al. 2023. Sparks of artificial general intelli- gence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12711
In this work, we explore the possibilities to utilize different large language models solve challeng- ing theorem-driven questions. There are still some limitations: (1) our answer extraction is still not perfect. There are some cases where our answer extractor is not able to locate the answer. Thus the final accuracy is still an approximate lower bound. (2) in our dataset collection, we specifically avoid the hard-to-evaluate cases where the answer is formula, figure,
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. 2022a. UniGeo: Unifying geometry logical reasoning via reformu- lating mathematical expression. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing. pages 5313-3323, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
matrix. Our choice of the questions can be biased in terms of evaluating the overall ability. (3) in the multimodal questions in TheoremQA, we have investigated different exist- ing models but none of them succeed achieving reasonable performance.
Jiaqi Chen, Jianheng Tang. Jinghui Qin, Xiaodan Liang, Lingho Liu, Eric Xing, and Liang Lin. 202 Geoqa: A geometric question answering henchmark towards multimodal numerical reasoning. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 202 pages 513-523.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 202 Evaluating large lan- guage models trained on code. arXiv preprint arXiv:2107.03374.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Yejin Choi, and Hannaneh Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-bart based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 2357-2367.
Wenhu Chen, Xueguang Ma. Xinyi Wang. and William W Cohen. 2022b. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks. arXiv preprint MXXv:2211.125881
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, lana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R Routledge, et al. 2021c. Finqa: A dataset of numerical reasoning over financial data. In Proceedings of the 202/ Con- ference on Empirical Methods in Natural Language Processing, pages 3697-3711.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. 2021. Program synthesis with large language models. arXiv preprint 1xXix:2108.0777
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Michoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from feedback. arXiv preprint arXiv:2212.08073.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv.2244.02311
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mo- hammad Aflah Khan. Shivanshu Purohit. USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. arXiv preprint ar 234.0:230.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:21 (1.:2/10.11166. 14168.
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. arXiv preprint arXiv:2209. 15003.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1991.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2022. Pal: Program-aided language models. arXiv preprint arXiv:2211.10433
Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar,

---

Google, 2023. Palm 2 technical report.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. 2021a. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 6774- 6786, Online. Association for Computational Lin- guistics.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. Conference on Neural Information Processing Systems.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford. Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models, arXiv preprint arXiv: 15556.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan, 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. EMNLP, pages 523-533.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023a. Chameleon: Plug-and and--aad compositional reasoning with large language models. arXiv preprint arXiv:2304.09N8
Wolfram Research, Inc. Mathematica, Version 13.2. Champaign, IL, 2022.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Pan Lu, Liang Qiu, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. 2023b. Dynamic prompt learn- ing via policy gradient for semi-structured mathe- matical reasoning. International Conference on Learning Representations (ICLR).
Rik Koncel-Kedzionski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equa- tions. Transactions of the Association for Computa- tional Linguistics, 3:585-597.
Pan Lu, Liang Qiu. Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Chunn Zhu. 2021b. Iconqa: A new benchmark for abstract diagram understanding and visual language reason- ing. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks.
Rik Koncel-Kedzzorge Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps:
A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152-1157.
Pan Lu, Liang Qiu, Wenhao Yu, Sean Welleck, and Kai- Wei Chang. 2023c. A survey of deep learning for mathematical reasoning. In The 61st Annual Meet- ing of the Association for Computational Linguistics (ACL).
Junnan Li. Dongxu Li, Caiming Xiong, and Steven
Hoi. 2022. Blip: Bootstrapping language-image pre- training for unified vision- languae understanding and generation. In International Conference on Ma- chine Learning, pages 12888-12900 PMLR
Shen- Yun Miao, Chao- Chon Liang, and Keh- Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984.
Raymond Li, Loubna Ben Allal, Yangtian Zi Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Mare Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpuro- hit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. Lila: A unified bench- mark for mathematical reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167.
Erik Nijkamp, Bo Pang. Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingho Zhou, Silvio Savarese, and Caiming Xiong, 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203, 2003.1344.4
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.0848

---

Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, al. 2021. Show your work: Scratch- pads for intermediate computation with language models. In Deep Learning for Code Workshop.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshor, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnie. 2022. Galactica: A large language model for science. arXiv preprimt arXiv:2211.0008
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971.
OpenAI. 2023, Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-2774 27744.
Shyam Upadhyay and Ming-Wos Wei Chang. 2015, Draw: A challenging and diverse algebra word problem set. Technical report, Citeseer.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021a. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computaatonaa Linguistics: Human Language Technologies, pages 20944
Shyam Upadhyay and Ming-Was Wei Chang. 2017. An- notating derivations: A evaluation strategy and dataset for algebra word problems. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Vol- ume 1, Long Papers, pages 494-504, Valencia, Spain. Association for Computational Linguistics.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021b. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094, Online. Association for Computational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le. Fdd Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. urXiv preprint 2:XXi:2203311177
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 conference on empirical meth- ods in natural language processing, pages 845-854,
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748 8763. PMLR.
Yue Wang. Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint 2005.222....792..
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, al. 2022a. Emergent abilities large language models. Transactions on Machine Learning Research.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1743-1752.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le. Denny Zhou, et al. 2022b. Chain-of-thougg soning in large language models. In Advances Neural Information Processing Systems.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical rea- soning abilities of neural models. arXiv preprint arXiv:1904.0155..
prompting elicits rea-
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. 2015. Solving geome- try problems: Combining text and diagram interpre- tation. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1466-1476, Lisbon, Portugal. Association for Computational Linguistics.
Aohan Zeng. Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu. Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b- An open bilingual pre-trained model. arXiv preprint 10..02221.002414.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2005.0006
Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. 2022. Task ambiguity in humans and language models. arXiv preprint 2XXi:22112.10711

---

Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Li. 2023. Progressive-hin prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Fdd Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.10625
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang. Shuo Zhang. Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. Tat-qa: A question answering benchmark on hybrid of tabular and textual content finance. Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3277-3287.


Reviews:
[{"review_id": "HwbpzpDy1A", "reviewer": "Reviewer_quPx", "summary": "", "questions": "It is mentioned that some hard-to-calculate questions and answers are omitted to build the dataset. What was the hypothesis of omitting those questions? Was it assumed that it would be more difficult for LLMs to solve those difficult questions?\n\nIs there any implicit bias present in LLMs due to the corresponding training dataset and strategy used that affects the overall performance? Is there a way to determine that?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for providing the constructive review.\n\n- \"Although this paper proposed quality dataset questions along with the theorems, the distribution of the questions is somewhat imbalanced. Out of 800 questions, more than half of the questions are from Mathematics only. So, distribution is not even.\"\n\nThis is an insightful observation. Currently, the dataset is driven by theorems, especially those requiring calculation, etc. It turns out that mathematics, including all its subareas, is the easiest area to find such theorems. Moving forward, we plan to expand our dataset continuously, either from our author's or the community's contributions. We will keep that in mind to balance the distribution across different STEM subjects.\n\n-  It is mentioned that some hard-to-calculate questions and answers are omitted to build the dataset. What was the hypothesis of omitting those questions? Was it assumed that it would be more difficult for LLMs to solve those difficult questions?\n\nIn Line 2, we wrote, \"During the annotation process, a small number of theorems were discarded due to their evaluation complexity\". It's actually the hard-to-evaluate questions, rather than hard-to-calculate questions. For example, some theorem outputs are matrices, curves, functions, figures, etc. We don't have good tools to automatically evaluate the correctness of these outputs. That's why we have relatively fewer problems in functional analysis, complex analysis, graph theorems, etc. \n\n- What was the hypothesis of omitting those questions? \n\nThe hypothesis is that the easy-to-evaluate questions are diversified enough to reflect LLMs' general capabilities to solve theorem-driven STEM questions. \n\n- Was it assumed that it would be more difficult for LLMs to solve those difficult questions?\n\nThe hard-to-evaluate questions are not necessarily more difficult. Some of them are not very difficult, like SVD (Singular value decomposition), which outputs matrices as output. Some of them could be very difficult, like the ones from functional analysis.\n\n- Is there any implicit bias present in LLMs due to the corresponding training dataset and strategy used that affects the overall performance? Is there a way to determine that?\n\nYes, due to the fact that almost all LLMs are based on text-only pre-training and instruction-tuning corpus (the multimodal parts are omitted during preprocessing). The model is not able to capture lots of non-textual signals in STEM theorems, like figures, graphs, tables, charts, or other imagery. The lack of multimodality in the pre-training corpus presents an implicit bias in the model to handle certain types of theorems better than others. For example, the numerical analysis theorems mostly reside in text format, so LLMs are generally good at them. In contrast, graph theorems contain lots of figures, which the models cannot handle very well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The paper contributes a new dataset for the QA research community, which consists of university-level questions backed with corresponding Theorems\n(2) It performs rigorous experiments with current SoTA LLMs with this dataset to check the performance measures of those LLMs\n(3) Also, it gives insights into why LLMs fail in some particular cases to perform QA and shows a detailed error analysis and its limitations.", "reasons_to_reject": "Although this paper proposed quality dataset questions along with the theorems, the distribution of the questions is somewhat imbalanced. Out of 800 questions, more than half of the questions are from Mathematics only. So, distribution is not even.", "questions_to_authors": "It is mentioned that some hard-to-calculate questions and answers are omitted to build the dataset. What was the hypothesis of omitting those questions? Was it assumed that it would be more difficult for LLMs to solve those difficult questions?\n\nIs there any implicit bias present in LLMs due to the corresponding training dataset and strategy used that affects the overall performance? Is there a way to determine that?", "paper_topic_and_main_contributions": "This work is aimed at evaluating the capabilities of LLMs to solve challenging math problems which may require domain-specific knowledge. Toward this objective, this work introduces TheoremQA. TheoremQA is a Question-Answering dataset where answering the question requires the application of appropriate theorems. The dataset consists of 800 high-quality questions, relying on 350 theorems across Maths, Physics, EE and CS, and Finance. Extensive experiments include comparisons among 16 LLMs with different sets of prompting strategies like Chain-of-Thought and Program-of-Thoughts. While GPT-4 was found to be significantly better than all the existing models, open-source models' performance was very poor. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Suggest putting extra effort into improving the technical presentation. Needs significant effort."}, "extra_text": {}}, {"review_id": "akwS4mnXN0", "reviewer": "Reviewer_CivJ", "summary": "", "questions": "1. An open question is: what are the possible reasons for the poor performance of current open-source models? Furthermore, what are the directions that have already been identified for potential improvements?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the recognition of our paper.\n\n- \"An open question is: what are the possible reasons for the poor performance of current open-source models? Furthermore, what are the directions that have already been identified for potential improvements?\"\n\nWe have some hypotheses, which are not completely verified. \n1.  The STEM data mostly comes from textbooks and websites with a severe amount of markup language (HTML, Markdown, Latex, etc) and structures. Therefore, it requires a significant amount of effort to preprocess and normalize this type of corpus. The open LLMs haven't spent enough effort in dealing with this type of corpus.\n2. The math-related problem-solving can benefit significantly from fine-grained human feedback. For example, OpenAI has released its human-feedback dataset at https://cdn.openai.com/improving-mathematical-reasoning-with-process-supervision/Lets_Verify_Step_by_Step.pdf. The open LLMs have yet to be benefited from such RL training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Motivation:\n\n- Addresses an important open problem - evaluating scientific reasoning capabilities of AI systems, which requires applying knowledge.\n- Proposes a new benchmark dataset to advance progress in this direction.\n\nExperiments:\n\n- Thorough comparative evaluation of 16 state-of-the-art models with different prompting strategies.\n- Rigorous analysis of results reveals capabilities and limitations of current models.\n\nWriting:\n\n- Clearly presented. Does a good job explaining the dataset, experiments, results and analyses.\n- Provides useful insights that can guide future research to overcome limitations.\n- Shares data to enable further analysis.", "reasons_to_reject": "This is a resource paper, and it excels in all aspects. I believe it deserves acceptance.", "questions_to_authors": "1. An open question is: what are the possible reasons for the poor performance of current open-source models? Furthermore, what are the directions that have already been identified for potential improvements?", "paper_topic_and_main_contributions": "This paper introduces TheoremQA, a new question answering dataset for evaluating AI models' ability to apply theorems to solve challenging science problems. The main contributions are:\n\n1. Proposes the first theorem-driven QA dataset spanning math, physics, engineering, and finance with 800 questions covering over 350 theorems. \n2. Comprehensively evaluates 16 large language models with different prompting strategies. \n3. Analyzes errors and suggests room for improvement.\n\nOverall, the paper introduces a valuable new benchmark for testing knowledge application in scientific reasoning, while providing insights into current model capabilities and limitations. The dataset and analyses should spur further research.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "None"}, "extra_text": {}}, {"review_id": "Hx9Qngoyte", "reviewer": "Reviewer_aN4a", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for the recognition of our paper.\n\n- The paper does not provide detailed examples of the prompts used in the experiments. Including such examples, particularly in the appendix, would have been beneficial for readers to better evaluate the appropriateness and effectiveness of the prompting strategies employed by the authors. Without these examples, it is challenging to fully assess the validity of the authors' claims.\n\nThanks for the reminder. We will update our paper to include the used prompts and exemplars in our revision. The prompts and exemplars will also be released in our Github repo to help other researchers replicate our results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The proposed dataset, TheoremQA, is the first of its kind to focus on theorem-driven question-answering. It covers a wide range of theorems from various fields such as Math, Physics, Electrical Engineering & Computer Science, and Finance. This dataset can serve as a valuable resource for researchers in the field.\n2. The authors have conducted a thorough evaluation of a wide spectrum of 16 LLMs on TheoremQA. Their comprehensive analysis provides valuable insights into the performance of these models and their ability to integrate theorems and understand multimodal inputs.\n3. The methodology adopted by the authors is sound and well-justified. They have used innovative prompting strategies, including Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT), to guide the LLMs. ", "reasons_to_reject": "The paper does not provide detailed examples of the prompts used in the experiments. Including such examples, particularly in the appendix, would have been beneficial for readers to better evaluate the appropriateness and effectiveness of the prompting strategies employed by the authors. Without these examples, it is challenging to fully assess the validity of the authors' claims.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces TheoremQA, a theorem-driven question-answering dataset designed to evaluate AI models' capabilities to apply theorems to solve challenging science problems. The dataset covers a wide range of theorems from various fields such as Math, Physics, Electrical Engineering & Computer Science, and Finance.\nMain Contributions:\n1. The authors propose a theorem-driven question-answering dataset to understand Large Language Models' (LLMs) capabilities to apply science theorems.\n2. They comprehensively evaluate a wide spectrum of 16 LLMs on TheoremQA.\n3. They perform different analyses in the theorem integration and multimodal understanding aspects to provide detailed insights.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper focusses on the description and evaluation of TheoremQA, a novel dataset designed to assess the capabilities of Large Language Models (LLMs) in applying theorems to solve complex scientific problems. TheoremQA is unique in its approach, as it requires the application of appropriate theorems to answer questions, making it a challenging benchmark for AI models. The dataset comprises 800 high-quality questions that are based on over 350 theorems spanning diverse fields such as Mathematics, Physics, Electrical Engineering & Computer Science, and Finance. The papers highlight extensive experiments conducted on 16 different LLMs using various prompting strategies. Among the models evaluated, GPT-4 outperformed others, while open-source models lagged in performance. The research provides valuable insights into the strengths and limitations of current AI models in scientific reasoning. Overall, TheoremQA serves as a significant benchmark that promises to drive further research in the domain.

The paper's introduction of TheoremQA, comprehensive evaluations, clear presentation, and valuable insights make it a significant contribution to the AI research community. In particular: a)  The paper introduces TheoremQA, a novel dataset designed for theorem-driven question-answering. It's the first dataset of its kind, covering a diverse range of theorems from fields like Mathematics, Physics, Electrical Engineering & Computer Science, and Finance. This dataset is poised to be a valuable resource for the research community. b) The authors have conducted an extensive evaluation of 16 state-of-the-art Large Language Models (LLMs) using TheoremQA. Their analysis is thorough, revealing both the strengths and limitations of current models in applying theorems and understanding multimodal inputs. c) The research employs innovative prompting strategies, such as Chain-of-Thoughts (CoT) and Program-of-Thoughts (PoT), to guide the LLMs in their evaluations.

However, one of the primary criticisms is the imbalanced distribution of questions within the dataset. Out of the 800 questions presented, a significant majority are from the Mathematics domain. This uneven distribution could limit the dataset's applicability and versatility across diverse fields, potentially skewing results or insights derived from it.
Decision: Accept-Main