Measuring Vision-Language STEM Skills of Neural Models
Authors: Jianhao Shen
Ye Yuan
Srbuhi Mirzoyan
Ming Zhang
Chenguang Wang
Venue: ICLR 2024 poster (2024)
Keywords: Benchmark
STEM
Multimodal
Vision-language models
Language models
Primary area: datasets and benchmarks
TL;DR: We introduce a new challenge to test the STEM skills of neural models.

Abstract: We introduce a new challenge to test the STEM skills of neural models. The problems in the real world often require solutions, combining knowledge from STEM (science, technology, engineering, and math). Unlike existing datasets, our dataset requires the understanding of multimodal vision-language information of STEM. Our dataset features one of the largest and most comprehensive datasets for the challenge. It includes $448$ skills and $1,073,146$ questions spanning all STEM subjects. Compared to existing datasets that often focus on examining expert-level ability, our dataset includes fundamental skills and questions designed based on the K-12 curriculum. We also add state-of-the-art foundation models such as CLIP and GPT-3.5-Turbo to our benchmark. Results show that the recent model advances only help master a very limited number of lower grade-level skills ($2.5$% in the third grade) in our dataset. In fact, these models are still well below (averaging $54.7$%) the performance of elementary students, not to mention near expert-level performance. To understand and increase the performance on our dataset, we teach the models on a training split of our dataset.
Even though we observe improved performance, the model performance remains relatively low compared to average elementary students. To solve STEM problems, we will need novel algorithmic innovations from the community.

Published as a conference paper at ICLR 2024
MEASURING VISION-LANGUAGE STEM SKILLS OF NEURAL MODELS
,rrbuhi Jianhao Shen Ye Yuan Srbuhi Mirzoyan' 123, Ming Zhang Chenguang Wang*:
School of Computer Science, Peking University
2 National Key Laboratory for Multimedia Information Processing, Peking University
PPeking University-Anie Anker Embodied AI Lab
*Washington University in St. Louis
{jhshen, yuanye _pku, mzhang_cs Jēpku. edu.cn, srbuhimir pku. edu. cn
chenguangwangewust... edu
ABSTRACT
We introduce new challenge to test the STEM skills of neural models. The problems the real world often require solutions, combining knowledge from STEM (science, technology, engineering. and math). Unlike existing datasets, our dataset requires the understanding of multimodal vision-language information of STEM. Our dataset features one of the largest and most comprehensive datasets for the challenge. It includes 448 skills and 1, 073, 146 questions spanning all STEM subjects. Compared existing datasets that often focus on examining expert-level ability, our dataset includes fundamental skills and questions designed based on the K-12 curriculum. We also add ofate-of-the-art foundation models such as CLIP and GPT-335-TTure to our benchmark. Results show that the recent model advances only help master a very limited number of lower grade-level skills (2.5% the third grade) in our dataset. In fact, these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance. To understand and increase the performance the models on a training split of our dataset. Even though we observe improved performance, the model performance remains relatively low compared to average elementary students. To solve STEM problems, we will need novel algorithmic innovations from the community,
dataset, teach
INTRODUCTION
STEM, namely, science, technology, engineering, and math, the basis of solving a wide set of real-world problems. This helps solve hard problems to better understand the world and universe, such as modeling gravitational waves and protein structures, proving mathematics theorem, designing new principles for quantum computing, and engineering the James Webb telescope. Mirroring real-world scenarios, understanding multimodal vision-language information is vital great variety of STEM skills. For example, are asked to compute the magnetic force given a diagram in physics. Geometry problems often require mathematical reasoning based on diagrams.
The challenges of the real world often require solutions that combine knowledge from STEM. Existing vision- Lasion-lnguuage benchmarks, however, often concentrate on evaluating one of the STEM subjects. For example, IconQA (Lu et al. bb2 and Gcometry3K (Lu ct al. 202 focus
evaluating mathematics understanding, while ScienceQA (Lu et al. 2022) examines science related skills. Other multimodal datasets such as VQA (Antol et al. 2015) and CLEVR (Johnson et al.) 2017) not specifically designed for STEM. Another set of benchmarks often includes textual STEM skill sets, where images are converted LaTeX or formal languages (Hendrycks al. 202 b).
Equal contribution.
Corresponding authors.
'The dataset and leaderboard available http:////wgggiiee..ooo//ttttttts

---

Published as a conference paper at ICLR 2024
magnetic betwee magnets each pair.
What compuier
Vick woodere would tub wate: and other five into with voter miced with vinearr Which
Menilfy object.
componenr do you see?
Which is tnie?
(A) Display Adipter/Video
of opprinani anoup!
It smaller in Pair 1.
Card
(R) it the semo both pairs
(R) CPU Socket
(A) These siiked in water
It soulller Pair
SATA Bus
(B) Those seaked salt water and vinegar
(A)
(B)
(C)
(D)
(i) Science
(ii) Technology
(iii) Enginecring
(iv) Math
STEM
Q How many bikes are
What color the small cube?
Which picture shows the oven!? (A)
Which of force from
there!
opens the cabinet
(B) The right
(A) Pall (8) Push
VQA
CLEVR
IcenQA
ScienceQA
*Questions
Muhiinodal
#Answers
ASalls
Subjects
Grades
Image
Difiiclity
014.103
61
CLEV
999,968
100,000
15.4
Natuni
Tou&Nanker
MATH
13.5003
64.
Marh
9712
Number
Advanod
MMLU
15,908
$2.6
STEM
Mult-chokce
Advazeed
Gonmeny
AL km
2,342
16.1
6113
Daggamm
Mobb-chiia
Medium
25
Pardanestul
21,208
12.1
1712
DiugalDDugran
Muls
BYER
DO5.56
13911228
17.
23
68
"STER"
MGB chille
Fardanena
(a) Comparison between STEM and existing datasets. Upper: examples of STEM and other datasets. Lower: key statistics STEM and other datasets. "#Questions". "#Images", "#Answers". "#Skills" denote number questions, images, answers, skills. "Multimodal" indicates whether every question of dataset contains both text image. "Q Length" means average question length.
GPT-3.5-Turbo
Zero-Shot
CLIP Finetuning
100
100
80
Accuracy (%)
Exam Score
60-
60
20
(i) Average accuracy of all subjects.
(ii) Average exam scores of all subjects.
(b) Neural model performance on STEM dataset.
Figure 1: Summary of our dataset and results.
In this paper, create a new challenge test the STEM skills of neural models. We collect a large-scale multimodal dataset, called STEM, consisting of 448 skills and 1, 073, 146 questions spanning across all four STEM subjects. STEM provides the largest set of both skills and questions among existing datasets. Figure [aa shows the comparison of its key statistics with other datasets. The dataset consists of multi-choice questions, and Figure (a) shows an example for each subject. STEM is multimodal
we exclude question if both the question and its answers are text. Each question consists of question text with an optional image context. The corresponding answers the question are either in text (Figure I [a) image (Figure [a)(v)). The design of skills in STEM is important: we focus on fundamental skills based on the K-12 curriculum. This enables us to present diverse and comprehensive STEM skill set. More importantly, this facilitates the understanding of neural models from different perspectives such as at skill level. We use IXL Learning (Learning 2019) as main data source to create STEM it aligns best with our design principle.
The STEM dataset challenging. Although our dataset focuses on the fundamentals of STEM, its multimodal nature makes it very difficult for modern neural models. Different from previous
2

---

Published as a conference paper at ICLR 2024
multimodal benchmarks, we include foundation models such as the state-of-the-ie rision-language model, CLIP (Radford et al., 2021). and the large language model, GPT-3.5- (Ouyang et al.) 2022). While these models are able to advance the model performance compared to the near random- chance performance of previous neural models, they still drop the performance by averaging 54.7% compared to that of average elementary students. For example, the models are only capable of understanding 2.5% third-grade skills. Notably, our model results are evaluated quantitatively under the same real-world exam environment as humans. Instead of manual evaluation which expensive, we simulate the conditions of IXL's online exams and use their scoring system to grade the model results. Compared to accuracy, this score (Bashkov et al. 202I) aims measure humans' true understanding of skills by integrating the learning progress into the final score calculation. While the majority of existing benchmarks do not yet provide detailed meta information for analysis, the design of STEM supports deep performance analysis at different granularities, e.g., at particular subject, skill, or grade level. For example, we show that basic math skills are still challenging for existing models. This is often due to the models failing to parse the images that are of great importance to mastering multimodal skills (e.g. geometry). To understand and increase the model performance on STEM, teach models large-scale training split of STEM. However, the model performance still remains relatively low compared to general elementary students, not to mention near expert-level performance.
Our contributions are as follows. (i) We create a new dataset, called STEM, to benchmark the multimodal STEM skills of neural models. STEM is the largest dataset among existing datasets. Its design focuses on fundamental skills in the K-12 curriculum. This enables diverse and comprehensive tests across all STEM subjects. To facilitate future research, we also contribute large-scale training set in STEM. STEM is challenging and useful to help advance models to solve more real-world problems. (ii) We benchmark wide set of neural models including foundation models such as GPT-3.5- Turbo and CLIP on STEM. The meta information in STEM (e.g.. skills and grades) supports a deeper understanding of model performance, and helps point out important shortcomings of existing models. (iii) We show current neural model performances students in terms of STEM problem solving. We conclude important insights that suggest new algorithmic advancements from the community necessary for understanding STEM skills.
still far behind that of average elementary
2
THE STEM BENCHMARK
2.1 DATASET
We create massive dataset, called STEM to test the STEM problem solving abilities. Unlike existing benchmarks, STEM features a large-scale multimodal dataset covering all STEM subjects spanning science, technology, engincering, and mathematics. We split the dataset into a train set, a validation set, and a test set for model development and evaluation. The overall dataset statistics are included in Table I More details of STEM dataset are described in the appendix.
Attributes Our dataset includes the following key attributes to support deep analysis of model performances. (i) Subjects. There are four subjects in STEM, namely science, technology, engi- neering, and math. We follow this high-level concept to create our dataset. (ii) Skills. We design skills according to the U.S. National Education and California Common Core Content Standards. This design also aligns with the skill categorization of our data resources (details are below) and closely follows recent studies (Hendrycks et al. 2021b: Lu et al.) (2211b). (iii) Grades. We use the grade information of our dataset resources in STEM. STEM does not contain grade information for the technology subset its raw data does not provide the grade-level information. (iv) Questions. Each question in STEM is a multi-choice question and is multimodal. We exclude a question if both the question and its answers are text. Each question belongs to a particular skill, hence subject.
Science Science includes branches of domain knowledge focusing on testing reasoning abilities. Subject areas include biology, chemistry, physics and so on. Science tests specific domain knowledge, e.g.. physics tests understanding of fundamental physics principles. It includes skills examining basics of science such as identifying properties of an object or calculating density. For example, to test the skill of comparing magnitudes of magnetic forces, an example question in Figure Πa)() will be asked. We collect questions from IXL. Science. Its skills and questions are designed based
3

---

Published as a conference paper at ICLR 2024
Table 1: STEM dataset statistics.
Subject
#Skills
#Questions
Average #A
#Train
#Valid
#Test
Science
82
186,740
2.8
112,120
37,343
37,277
Technology
8,566
4.0
5,140
1,713
1,713
Engineering
18,981
2.5
12,055
3,440
3,486
Math
858,859
2.8
515,482
171,776
171,601
Total
448
1,073,146
2.8
644,797
214,272
214,077
Science
Technology
Math
oijects
loors
logo
ideniffy trapetaids
ionidy apdi-a16
perichwnnis
parts
rASS Sruce
80
pteditt
caliles
photo
machhandie and dool
555115
60
Shermal-
papillipee pend ben
ciinate
Engineering
orinnggsss
40
changee
feverteeeeeratte undrgg tiersllss
Hentincthe saleey equppment
20
Grades
Figure A summary of skills.
Figure 3: #Skills per grade.
on U.S. National Education and California Common Core Content Standards. It includes questions from second grade to eighth grade. We also processed the data such as deduplicating questions and randomly shuffling the order of answers each question. We exclude question if both the question and its answers are text.
Technology Technology includes principles that test the knowledge of empirical methods. This subject mainly includes computer science. An example is included in Figure I (i)) It includes fundamental skills such as identifying parts of a computer or the basics of programming languages. We collect the questions from Triviaplaza Computer, which includes questions for tech interviews. To the best our knowledge, STEM provides the first technology problem set for the multimodal test.
Engineering This engineering subset includes skill set that covers fundamental engineering practices ranging from solving problems using magnets to exploring the design of spaceships. Figure aiii) illustrates example. The dataset is constructed based on the engineering portion of IXL. The skills and questions are ranging from third grade to eighth grade. To our knowledge, this subset considered an early exploration on testing multimodal practical knowledge engineering.
Mathematics Mathematics often requires reasoning and abstract knowledge. For example, solving math tests algebra generalization abilities. For example, the addition of numbers obeys the same rules everywhere. This subset includes fundamental math skills such addition, algebra, comparison, counting, geometry and spatial reasoning. An example shown in Figure [a)(vv). The questions are from IXL Math spanning from P eighth grade. To encode mathematical expressions, LaTeX to avoid unusual symbols or cumbersome formal languages.
Comparison with Existing Datasets STEM is the first large-scale mulitmodal STEM dataset. As shown in FigureTaa), STEM provides the largest number of questions and skills among existing STEM related datasets. Compared to the previous largest multimodal STEM datasets, STEM is about 10 times larger in terms of the number of questions. STEM offers the most thorough fundamental skill and question set ranging from pre-K K to eighth grade. Compared to datasets of a particular subject, STEM covers all STEM subjects and is at least competitive in terms of the number of questions and skills. For example, STEM's s math subset has 27 times more skills compared to the recent math benchmark (Lu et 2021b).
4

---

Published as a conference paper at ICLR 2024
Which property matches this object? Select the Silty
Which property matches Oi objeat? Sclect the Salty
Escoder
Sour
Question: Whish propeity
bateer anower
GPT-3.5-Taro (B) Sour
Concext: A grzen apple.
Amswrr: (A) Sully (3) Sour
Encoder
Caprion
apple.
grom
(a) CLIP.
(b) GPT-3.5-Turbo.
Figure Zero-shot model setups.
2.2 ANALYSIS
To provide more insights into our dataset, we conduct the below analysis with a focus on the unique perspectives of STEM including skills and grades. Other dataset details such question analysis shown in the appendix.
Skills The design of STEM emphasizes diverse skills spanning all STEM subjects. Figure 2 presents brief summary of the skills (a complete skill set is included in the appendix). STEM contains the largest skill set among existing datasets (Figure T [)). Each skill contains 2, 395 questions on average. A large number of new skills introduced to STEM that
yet covered by existing datasets, e.g.. skills in technology and engineering. Besides, understanding multimodal information is crucial to these skills. For example, solving the geometry problem in Figure I (iv()) challenging since both the image and text contribute to the problem solving. Through this design, STEM helps to recognize important shortcomings of machine learning models by referring to difficult skills for these models.
Grades STEM is designed with comprehensive 12 currieulum examine fundamentals of STEM. This leads to another unique feature of testing on STEM: we are able to obtain the grade-level performance of models. The majority of existing datasets aim compare models with human experts e.g., solving competition-level questions (Hendrycks et al.) 2021b Zheng et al. 2022). However, thanks to the grade-level information provided by STEM find that models are only competitive with first graders in understanding certain STEM skills. Figure 3 shows the total number of skills per grade of subjects.
2.3 MODELS
We benchmark both state-of-thee ahr and foundation models on STEM including: multimodal (vision- language) models such as CLIP and language models such as GPT-3.5-Turbo.
Vision-Language Models
(i) Zero-Shot. We use CLIP (Radford et al. 2021), ViLBERT Lu et al. 2019), 12-in-1 (Lu et al. 2020). UNITER (Chen (020b)) and Virtex (Desai & Johnson 2021 for the zero-shot evaluation of multimodal models. Multimodal models generally include two modules: image encoder and text encoder. CLIP considered one of the srate-of-the-e multimodal models. For zero-shot CLIP, we follow its original setup in Radford et al. (2021). Figure 4 a) illustrates example. Other models follow the same zero-shot setup.
(ii) Finetuning. To test the learning ability of the models, also finetune CLIP. We follow the linear probe setup presented in Radford et al. (202T). For each subject, we train the model on its entire training the evaluation is the same as the zero-shot setup.
shown in Table I and select the best model on the validation set. At test time,
Language Models
(i) Zero-Shot. We use Glo Ve (Pennington et al. 2014). UnifiedQA (Khashabi et al. 2020), GPT- 3 (Chen et al., 2020a) and GPT-3.5-Turbe (Ouyang et al. 2022) zero-shot for the language model evaluation. We formalize the task as a question answering task. We use the OpenAl API "text- davinci-002* and "gpt-3.5-turbo" corresponding to the best-performing GPT-3 and GPTr-35-Tuuroo
5

---

Published as a conference paper at ICLR 2024
Table 2: Results on STEM dataset. All evaluation scores are higher the better.
Model
Science 38.6
Technology
Engineering
Math
Average
Random Guesses
25.0
44.9
39.1
36,9
Language Models
GloVe PPeniintton et al. 2014
38.0
25.2
48.1
39.0
37.6
UnSnal (Kaasaab et al.
2020
39.6
27.2
58.0
39.6
41.1
UnifiedQArga Khashabi
2020
42.6
28.8
55.4
40.0
41.7
GPT-3 Brown al. 2020
47.1
22.1
73.5
44.0
"46.7
GPT-3.5- Turbo
50.1
26.3
74.6
45.0
49.0
Vislon-Languuage Models
Virtex (Desat & Johnson
2021
37.5
24.0
48.1
38.9
37.1
12-in-1 et. 2020
39.4
27.5
44.2
41.9
38.3
ViLBERT (Lu at. 2019
39.0
32.1
44.2
42.7
39.5
UNITER
(Chen al. 20ZUb
50.8
34.6
55.1
43.2
45.9
47.8
43.6
52.9
RN101
50.3
46.7
43.7
51.5
RN50x4
48.8
69.2
49,4
44.1
52.9
RN50x16
49.8
66.1
51.4
44.3
52.9
CLIP
RN50x64
50.9
70.0
55.5
43.2
54.9
Rudford et al. 2021
ViT-B/32
48.3
63.7
59.5
42.8
53.6
ViT-B/16
48.6
65.9
43.6
51.3
ViT-L/14
49.8
68.6
54.3
43.1
54.0
ViT-L/14 336px
50.3
68.7
55.1
43.6
54.4
+Finctuning
87.0
71.9
67.7
78.4
76.3
respectively. We convert images to visual context text based captioning model following Lu et al. (2022). Figure shows an example. All language models follow the same setup.
2.4 METRICS AND HUMAN PERFORMANCE
We report accuracy on the questions in our dataset are multiple-choice questions. We also compute macro average accuracy across the test sets of all subjects. Unlike the micro evaluation setting, this score relieves data or class imbalance issues. In addition, we focus on two kinds of evaluations for human performance comparison purposes. (i) Exam score. In particular, for science, engineering, and math, we use the IXL SmartScore (Learning) 2019). Different from accuracy, SmartScore considers the progress of learning and is designed to measure how well humans understand a STEM skill (Bashkov al.) 2021 It starts increases students answer questions correctly, and decreases if questions answered incorrectly. We simulate the conditions of its real online exams. The final score is graded by IXL's SmartScore system. According to IXL (IXL b a, score higher than 90.0 is considered excellent for a mastered skill. Therefore, we use this score as reference to human performance, For technology,
of each subject. We use accuracy as the evaluation metric since all
the average human accuracy available at Triviaplaza. The average accuracy is 68.6. (ii) Accuracy. We sampled 80 questions from our test sets (20 questions for each subject) and collected the responses from seven university students. They attained an average accuracy of 83.0 on all subjects. All evaluation scores are higher the better.
3 EXPERIMENTS
In this section, we show the performance a wide set of neural models as well as humans on STEM. The results show that state-of-the-art foundation models like CLIP and GPT-3.5-Turbo still underperform general elementary students. The details of the experimental setup, additional results and analysis are described in the appendix.
3.1 MAIN RESULTS
Zero-Shot The results are shown in Table We first test language models to see whether models that only understand text are proficient at the multimodal skills in STEM. GloVe has near random- chance accuracy. This means that STEM cannot be solved by simply matching the text semantic similarity between questions and answers. UnifiedQA does slightly better than GloVe with an improvement of averaging 4.1% points. GPT-3.5-Turbo performs the best among these language models, reaching 49.0% accuracy on average. Both foundation models (GPT-3.5- T-rbbbo and GPT-3) perform well in engineering. This is mainly because engineering practices mainly described in the

---

Published as a conference paper at ICLR 2024
text (see Figure I (ii)). Recent advancements in large language models help dramatically improve text understanding capabilities. However, large language models still struggle in other subjects. This implies that the understanding of both vision and language information is essential to STEM skills.
Next, we examine vision-language models. We find that the performance of Virtex, 12-in-1, and ViLBERT is near- ing the performance of random guesses. These models capture very limited knowledge of STEM subjects. On the other hand, UNITER and CLIP show significant im- provements over the random-chaee chance accuracy. Specifi- cally, CLIP-RN50x64 achieves the best result on STEM. It achieves 18.0% points improvements over random guesses. Notably, ELIP-RNSOXE outperforms HEPT-3355TTrrbe by 5.9% points. This shows that CLIP has a basic understand- ing of multimodal STEM skills. Its vision understanding ability certainly contributes to this performance. Among all subjects, see only marginal improvements in math. This applies to all foundation models. In addition, the result implies that math is the most challenging subject for current neural models. Novel algorithm advancements that can enable strong reasoning ability are necessary to solve math problems.
cirdee
cones
faces-of-DD-figures
describe-rrnstommaiin
match-clocks
classify fruits
dassify-animals mals
vertebrates
m maageticcffrcess
cables
font
Icons
Finetuning The results are shown in Table 2 It is en- couraging as finetuning CLIP ViT-L/14 (TTl//443366 336px able to significantly boost the performance on science and math by averaging 30% points over its zero-shot setting. The performance improvements on other subjects are 7.9% points, which is much smaller. While having large amount of training data helps to some extent, the fine- tuning performance is still far behind that of an average elementary student (the human-level performance is pre- sented in Sec. Sec.B.3). This indicates that more fundamental advancements are required to solve STEM questions in the STEM dataset. For simplicity, we use CLIP to represent CLIP ViT-L/14@336px in the rest of this section.
logo
peripherals
enginearing
control groups
de dependent variables
experimental-
safet sauuty-e
20
40
60
80
100
Accuracy (%)
Random FPT333
CLIP Finstringg
CLIP Zern-Shot
3.2 RESULTS ANALYSIS
Figure 5: Results categorized by sampled skills of cach subject. M: math. S: science. T: technology. E: engineering. Full results are in the appendix.
Skills As STEM provides massive skills, analyzing mod- els' performance the skill level helps understand models better. We show the performance of foundation models (GPT-3, G and CLIP) on an uncurated set of skills of each subject in Figure 5 We find that these foundation models are able to perform well zero-shot on skills focusing on identifying common objects (e.g., classifying fruits). However, zero-shot and finetuned foundation models all fail in challenging skills that require abstract knowledge and complex reasoning (e.g., describing transformation).
Randam
CLP Beuttuting
100-
Grades Intuitively. questions for higher graders are more difficult than those for lower graders. We illustrate the grade-level model performance to investigate if the same trend holds for neural models as well. We show the exam scores of models along each grade in Figure 6 Surprisingly, there is no obvious performance drop as the increase in grade levels. This implies the learning curve for neural models may be different from that of humans. reason is that neural models are trained on data includ- ing all grade-level questions simultaneously while humans
80-
Exon Store
40-
0-
Grsces
Figure 6: Average grade-level exam scores.
7

---

Published as a conference paper at ICLR 2024
95
CLIP Zera-Shot
RN50x64
CLIP Finetuning
85
75
Accuracy (%)
54
VIT-L/44
VTT-8/32
Accuracy
65
55
53
RN50 RNSOKA RN5OX16
45
35
52
VIT--/116
25
35
75 Confidence (%)
65
200
400 # Parameters (M)
600
Figure 7: CLIP calibration results.
Figure 8: Zero-shot CLIP model scaling results.
Randem
GPT-3
GPT-3.5-1B 5--Tubo
CLIP Zero-Snet
CLIP Finetuning
Human
100
100
80
Accuracy
20
20
0-
0-
Science
Technology Engineering
Math
Science
Technology Engineering
Math
(a) Exam scores on each subject.
(b) Accuracies a real-world test on subset of STEM.
Figure 9: Comparison between models and humans.
gradually learn from lower to higher grade-level questions. Also, the average scores of elementary grades (grades 1-6) equals 40.8, which is 54.7% lower than human reference (i.c., 90).
Calibration A trustworthy model should be calibrated. This means that its confidence should approximately match the actual probability of the prediction being correct (Guo al. (017a). However modern neural networks are often not well calibrated (Nguyen et al. 2015 Guo et al. 2017b). We show the relationship between the confidence of CLIP and the corresponding accuracy in Figure 7 We use the softmax probability as the confidence. We observe that the zero-shot CLIP model not well calibrated. In fact, is overconfident about its predictions and is only loosely related to its actual accuracy. After finetuning, CLIP is more calibrated. The results suggest that further improving calibration on STEM is another promising direction.
Scaling Laws Figure shows the average accuracy zero-shot CLIP with different model sizes. As expected, the performance improves models grow larger. But the performance also saturates. This implies that other than increasing model scales, new advancements in model design or training schema required improve the performance STEM.
3.3 COMPARISON WITH HUMAN
In this section, we explore whether the hest-performing foundation models namely CLIP, GPT-3, and GPT-3.5-Turbe are nearing human-level performance.
Figure (a) shows the exam scores (Sec. 2.4p models and humans on each subject. A score of 90 means student is proficient in the subject. The zero-shot performances of all tested neural models are well below that bar. In technology, CLIP finetuning achieves human-level performance. This is mainly because most technology skills are about specific empirical knowledge, which is learnable for neural models after finetuning. Overall, there is still a large performance gap between general neural models and average elementary students even in understanding the fundamental skills in STEM. In
8

---

Published as a conference paper at ICLR 2024
addition, the offline real-world test-takers (Sec. 2.4) produce similar outputs with the above online setup on a subset of questions in the STEM. The results are shown in Figure b).
3.4 CASE STUDY
Which cicck?
diepe
We show examples of GPT-3.5-Turbo predic- tions in Figure T0 We show example of correct and incorrect predictions respectively. For the correct ones, the corresponding skills are mainly about the basics, such as names of objects (e.g., shapes or animals). The incorrect predictions are mainly due to the complex nature of skills. These skills are often about abstract concepts such symmetry and the direction of force. They are also more relevant to logical reasoning, such as finding patterns or inferring the function of animal adaption.
Select the vettffate. Hiat Mannals binds, rqycles, and anghitiass are vertatraio.
Which sniraal's moxth sxlepial inwocts out of bummnod
(a) Correct predictions
(b) Incorrect predictions
Figure 10: Examples of GPT-3.5-Turi predictions.
RELATED WORK
There are various types of vision- vision-language tasks, such as reference resolution (aazemzadeh et al. 2014), image captioning or tagging (Thomee et al., 2016; Shhrmma et al.] 2018). image-text re- trieval (Lin et al. 2014 Plummer et al. 2015), visual question answering (Antol et al. 2013; Goyal
al. Z Zhang al. 2016 Zhu al. (016), and visual reasoning (Suhr al. 2017 Johnson et al. 2017). Our STEM differs from the previous datasets in that it covers diverse fundamentals of STEM and requires both multimodal understanding and domain knowledge in STEM. This makes STEM natural testbed to evaluate the real-world problem solving abilities of machine learning models.
Existing STEM related benchmarks do not cover all STEM skills for multimodal understanding. There are benchmarks targeting math (Saxton et al. 2019 Hendrycks et al.) [202bb] Zheng et al. 2022 Lu et al. 2021a|b] Xiong et al., (0236). PIQA (Bisk et al. (0200) is benchmark for physical commonsense understanding. ScienceQA (Lu et al. 2022) is multimodal dataset for general science. MMLU (Hendrycks et al. 2021a) contains 57 tasks including STEM but is only restricted to single text modality. Our STEM is the first to include all STEM subjects for vision-language understanding.
Pretrained foundation models help achieve state-of-the- performance in both NLP and computer vision tasks. Pretrained language models (Radford et al. 2018 2019
Devlin et al. 2019), especially
the recent large language models (Chen et al.] 2020a, Wang et al. 2020 2022a] Ouyang et al. have significantly advanced
2022, Crispino al. 2023 OpenAl| 2023 Chowdhery et al. 2022)
the performance in general natural language understanding tasks. Based on these models, various
techniques (Shen et al. 2022a by Imani et al.
2023 Jiang et al., [023) Wang al., 2023 Xiong et al. 2023a: Pan et al. 2024b a) have been developed to address specific challenges in a domain such as math. We focus on testing the basic STEM ability of state-of-the-arr models in zero-shot setting and
identifying room for improvement by referring to our finetuning results. CLIP (Radford et al. 1021
one of the state-of-the-art pretrained vision-language models (Lu et al. 2019) Krishna et al. 2017
Chen et al. 2020b; Desai & Johnson 2021 Lu et al. 2020). Other similar models include GLIP
al. 2022b), GLIDE (Nichol et 2022). OFA (Wang et al. (2022)), and BLIP (Li et al. 2022a]
2023
We use CLIP in our test while the majority of existing benchmarks have not explored it yet.
CONCLUSION
We introduce STEM, new challenge to examine the STEM skills of neural models. STEM the largest multimodal benchmark for this challenge. It consists of a large number of multi-choice questions and skills spanning all STEM subjects. STEM focuses on fundamentals of STEM based on the K-12 curriculum. We also include state-of-the-art foundation models such GPT-3.5-Turbe and CLIP for evaluations. The benchmark results suggest that current neural model performances are still far behind that of elementary students. STEM poses unique challenges for the research community to develop fundamental algorithmic advancements. We hope our benchmark will foster future research in multimodal understanding.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
We hereby acknowledge that all of the co-authors of this work are aware of the provided ICLR Code of Ethics and honor the code of conduct. We collected data from several sources, and we cited the data creators. The copyright belongs to the original data owners. The STEM dataset is under the CC BY-NC-SA 4.0 license (Creative Commons 4.0 International) and is used for non-commercial research purposes. The collected data does not contain any personally identifiable information or offensive content. Our dataset is mainly built upon instances from real-world exam data. Therefore was less likely to contain sensitive data. We evaluate foundation models, for which the risks and potential harms are discussed (Brown et al. 2020 Radford et al. 2021
ACKNOWLEDGEMENNT
This paper is partially supported by the National Key Research and Development Program China with Grant No. 2023YFC334120$ well as the National Natural Science Foundation of China with Grant No.62276002.
REFERENCES
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pp. 6077-6086, 2018
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In ICCV, pp. 1425-2433 2015.
Bozhidar M Bashkov, Kate Mattison, and Lara Hochstein. Ixl design principles. 2021.
Yonatan Bisk, Rowan Zellers, Ronan LeBras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In AAAI, pp. 7432-7439, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
2020.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In NeurIPS, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, pp. 104-120, 2020b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski. Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellster Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. CoRR, abs/2204.02311, 2022.
Nicholas Crispino, Kyle Montgomery, Fankun Zeng, Dawn Song, and Chenguang Wang. Agent instructs large language models to be general zero-shot reasoners, arXiv preprint arXiv:2310.03710. 2023.
10

---

Published as a conference paper at ICLR 2024
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR, pp. 11162-11173, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171-4186, 2019.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x words: Transformers for image recognition at scale. In ICLR, 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Stayy Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In CVPR, pp. 2325-6334, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, pp. 1321-1330, 2017a.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778, 2016.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021a.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS, 2021b.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. CoRR, abs/2303.05398, 2023.
IXL. Understanding the ixl smartscore. http::///Aaggg.....oooooooorrrrr wertttSsoorriiiidd..
IXI. How does the smartscore work? b.
Albert Qiaochu Jiang, Sean Welleck, Jin Peng Zhou, Timothée Lacroix, Jiacheng Liu, Wenda Li, Mateja Jamnik, Guillaume Lample, and Yuhuai Wu. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. In ICLR, 2023.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pp. 1988-1997, 2017.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara L. Berg. Referitgame: Referring to objects in photographs of natural scenes. ACL, pp. 787-798, 2014.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single QA system. In Findings of EMNLP, pp. 1896-1907, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li. David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, pp. 32-73, 2017.
IXL Learning. The impact of ixl math and ixl ela on student achievement grades pre-k to 12 (pp. 1-27), 2019.
11

---

Published as a conference paper at ICLR 2024
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp.
PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman image pre- training with frozen image encoders and large language models. arXiv preprint arX1v:2301.12597 2023.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Li- juan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-iman pre-training. In CVPR. pp.
2022b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, pp. 740-755 2014.
Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, Ming Zhang, and Qun Liu. Fimo: A challenge formal dataset for automated theorem proving, 2023.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for lainn-and-language tasks. In NeurlPS, pp. 13-23, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chur Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL-IJNNLP, pp. 6774 6786, 2021a.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: new benchmark for abstract diagram understanding and visual language reasoning. In NeurlPS, 2021b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks easily fooled: High confidence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-436, 2015.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: towards photorealistic image generation and editing with text-guided diffusion models. In ICML, pp. (6784-16804 2022.
OpenAl. GPT-4 technical report. CoRR, abs//203.08874,, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Yu Pan, Ye Yuan, Yichun Yin, Jiaxin Shi, Zenglin Xu, Ming Zhang, Lifeng Shang, Xin Jiang, and Qun Liu. preparing lessons for progressive training on language models. arXiv preprint arXiv:2401.09192 2024a.
Yu Pan, Ye Yuan, Yichun Yin, Zenglin Xu, Lifeng Shang, Xin Jiang, and Qun Liu. Reusing pretrained models by multi-linear operators for efficient training. Advances in Neural Information Processing Systems, 36, 2024b.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world, 2023.
12

---

Published as a conference paper at ICLR 2024
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, pp. 1532-1544 1543, 2014.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, pp. 2641-2649, 2015.
Alec Radford, Karthik Narasimhan, Tim Salimans, lly Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 7763, 2021.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. Analysing mathematical reasoning abilities of neural models. In ICLR, 2019.
Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, pp. 2556-2565, 2018.
Da Shen, Xinyun Chen, Chenguang Wang, Koushik Sen, and Dawn Song. Benchmarking language models for code syntax understanding. In EMNLP, 2022a.
Jianhao Shen, Chenguang Wang, Ye Yuan, Jiawei Han, Heng Ji, Koushik Sen, Ming Zhang, and Dawn Song. Palt: Parameter-lic transfer of language models for knowledge graph completion. In EMNLP, 2022b.
Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. corpus of natural language for visual reasoning. In ACL, pp. 217-223, 2017.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale, 2023.
Bart Thomee, David A. Shamma. Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: the new data in multimedia research. Commun. ACM, pp. 64-73, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2010. 11967, 2020.
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. Deepstruct:
Pretraining of language models for structure prediction. In ACL, 2022a.
Haiming Wang. Ye Yuan, Zhengying Liu, Jianhao Shen, Yichun Yin, Jing Xiong, Enze Xie, Han Shi, Yujun Li, Lin Li, Jian Yin. Zhenguo Li, and Xiaodan Liang. DT-solver: Automated theorem proving with dynamic-tree tree sampling guided by proof-level value function. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 12632- Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 00.18653/v1/2022 h/20222acc..22ongg... acI-long..7000 706. URL https::/aclannthol
orgooog//// 2023. acl 1001--1op0 1ong.. 706
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through learning framework. In International Conference on Machine Learning, pp. 2331--33340. PMLR, 2022b.
a simple soquence-to-sequeee
Jing Xiong, Zixuan Li, Chuanyang Zheng, Zhijiang Guo, Yichun Yin, Enze Xie, Zhicheng Yang, Qingxing Cao, Haiming Wang, Xiongwei Han. Jing Tang. Chengming Li, and Xiaodan Liang. Dq-lore: Dual queries with low rank approximation re-ranking for in-context learning, 2023a.
13

---

Published as a conference paper at ICLR 2024
Jing Xiong, Jianhao Shen, Ye Yuan, Haiming Wang, Yichun Yin, Zhengying Liu, Lin Li, Zhijiang Guo, Qingxing Cao, Yinya Huang. Chuamyang Zheng, Xiaodan Liang, Ming Zhang, and Qun Liu. TRIGO: Benchmarking formal mathematical proof reduction for generative language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11594-11632, Singapore, December 2023b. Association for Computational Linguistics. doi: 1/2666///122223mmmmmmmmaa 1. URL https://aclanthonn
org/ wwm..mmn.p..mmmmmmmmmmmmmmm.mmmm emn1p-main.711 711
Peng Zhang, Yash Goyal, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Yin and yang: Balancing and answering binary visual questions. In CVPR, pp. 5014-5022, 2016.
Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. minif2f: cross-system benchmark for formal olympiad-leve mathematics. In ICLR, 2022.
Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, pp. 4995-5004, 2016.
14

---

Published as a conference paper at ICLR 2024
A MORE DETAILS ON STEM
In this section, we provide more details on STEM, including dataset analysis, models, evaluation settings, and dataset collection.
A.1 ANALYSIS
Questions and Answers STEM contains multi-choice questions (Appendix provides question example for each skill). The question contains a textual description with an optional image context. Answer options in text or in image. We further analyze the questions from the following aspects. (i) The number of answers. STEM has averaging 2.8 answer options for each question. The distribution presented in Figure [1] In practice, the more answer options one question has, the more difficult it is. (ii) Question type. We categorize questions based on the first three words of the question text shown Figure [2] STEM mostly includes factoid questions that start with words such as "which" and "what". We also show the word cloud of our STEM in Figure [3] We can see the most common words like "shape" and "number". This indicates the questions require joint reasoning of the text and images. (iii) Question distribution. Figure 14 depicts the distribution of question lengths. We can see subjects generally follow a long-tail distribution, while math distribution is most steep and science distribution is flatter. Heuristically, longer questions more difficult to solve. Figure [5 shows the number of questions in each grade. While pre-K has more questions, the number of questions in other grades is approximately evenly distributed.
2
3
27.0%
46.8%
26.2%
Figure 11: #Answers distribution.
Figure 12: Question type distribution.
THE SIDE
GRAPH
FACTHEE
HOT ANIMAL
TRIANGLE
RIBURA
PAIN
GINEAR
GROUP
TUJ
BOOK
MODEL
BY
TRUE
SOIL
CLOCK
MAGNE
TIME
ATEMENT
NUMBER
THW
INGIIVIO
TRSTIO WITO THE
LINE
BEST
ST
TEEMPERATTOO
MAKET
FUNCTION
ARARS
PICTURE
CHANDE
BLOCK
ONE
S
ANOVER
FOL
ANGLE
FIRST
COUNT
PLACE
USE
HEAT
SAMRLE
DIFFERENT
MANY
TAY
UIE PLANT SMAIE
Figure 13: Word cloud of question texts in STEM.
15

---

Published as a conference paper at ICLR 2024
200k
Technology Engineering Math
80
150k
Percenlage (96)
#Questions
100k
40-
50k
10
20
50
60
90
100
Pre-K
2
#Question Tokens
Grades
Figure 14: Question length distribution.
Figure 15: #Questions per grade.
Table 3: Skill comparison between STEM and existing datasets (IconQA and ScienceQA).
(a) Number skills.
(b) Skill comparison between STEM and IconQA
Subject
IconQA
ScienceQA
STEM
IconQA
STEM
Counting
Count Count shapes Count sides
Science
167
82
Technology
0
Goometry
Classify trangles Idemtiyy symmelty. Idcntify
shapes
Engineering
0
Time
Match times, Identify A.M./MM,, Read calendar
Math
13
0
351
Total
167
Science
Compare consenirations solutions
Not
Teoknology
Mentify peripherals Identify laboratory tools
Engincering
exponential functions.. ...
Skill Comparison We compare the skills of STEM with other related datasets in Table STEM contains the largest skill set among existing datasets, with a great number of new skills introduced to STEM that are not yet covered by existing datasets, e.g., skills technology and engineering.
A.2 MODELS
In this section, we introduce the foundation models we benchmark in detail.
Vision-Language Models
CLIP (Radford et al., 2021). CLIP is pretrained sufficiently large dataset of 400 million text-image pairs across the Internet. It uses a Transformer as the text encoder, and has several variants of image encoder, including ResNet (RN) backbones and Vision Transformers (ViT) (Dosovitskiy ct al. 2020). CLIP aligns the text and image representation by training on in-batch contrastive loss, and is able to zero-shot transfer to downstream vision language tasks. To align with CLIP pretraining. we formulate question answering as matching text and images. We use the cosine similarity between the text and image embeddings as the matching function, the same as the original zero-shot image-text retrieval settings in CLIP (Radford et al. 2021).
ViLBERT and 12-in-1 (Lu et al., 2019: 2020). ViLBERT adopts two parallel streams to process image regions and text segments separately, with co-attentional transformer layers connecting them. There is also a multi-task version called 12-in-1 (Lu et al.) (2020) that trains 12 different tasks with individual task-specific heads sharing 1 "trunk" ViLBERT model. Its multi-modal alignment prediction serves as the matching score.
UNITER (Chen et al. 2020b). UNITER consists of Image Embedder with Faster R-CNN (nndrrson et al. 2018), a Text Embedder with Transformer /Vaswani et al.) (017). as well as a multi- layer Transformer get cross-modality representation. During inference STEM, the matching
16

---

Published as a conference paper at ICLR 2024
score function is the same CLIP, i.e., the cosine similarity between the text and image embeddings (Chen et al. 2020b).
Virtex (Desai & Johnson 2021). Virtex first extracts visual features with ResNet-50 (He et al. 2016) backbone. The visual features are then fed into a text head, which consists of two unidirectional Transformers, to predict captions. We extract the image feature with the image encoder, then feed text into the textual head and use the sum of bidirectional generation logits as the matching score.
Language Models
GPT-3 (Chen et al. 2020a) and GPT-3.5-Turbo (Ouyang et al., 2022). These foundation language models are generation models pretrained large corpus of text. We use the OpenAl API "text- davinci-002" and "gpt-3.5-turbo" corresponding to the best-performing GPT-3 and GPr-b.5-TTrbo respectively. We formalize the evaluation task as a question-answering task. The input GPT-3 and GPT-3.5-Turbo is the concatenation of the question text, the context text, and multiple answer options. The output is predict final answer from answer options. For images in questions, we follow] Lu et al. (2022) to convert them to visual context text based on a captioning model consisting of ViT Dosovitskiy et al. 2020) and GPT-2 (Radford et al. 2019).
UnifiedQA (Khashabi et al., 2020). UnifiedQA is pretrained question- weestion-answering model. We use both its base and small versions. Its evaluation setup the same as that of GPT-3 and GPT-3.5-Turbo.
GloVe (Pennington et al. 2)14). GloVe is pretrained word embedding model. We use the similarity between the average embedding of the concatenation of the question and context and the average embedding of each answer option. The answer option with the largest similarity score is the answer output. We use average pooling based on the 300-dimensional word embeddings. The images are also converted to text using the same method GPT-3 and GPT-3.5-Turbo.
A.3 EVALUATION SETTINGS
We benchmark of-th-of-thheart foundation models STEM under different settings, including zero- shot, fow-shot, finetuning, and multi-task.
(i) Zero-Shot. We use CLIP (Radford al. 2021 ViLBERT (u ct al. 2019). 12-in-1 (Lu et al. 2020). UNITER (Chen et al. 2020b), and Virtex (Desai & Johnson 2021 for the zero-shot evaluation of foundation multimodal models. CLIP is the tfte-of-the-ass multimodal model. For zero-shot CLIP, we follow its original setup in Radford et al. (202T). The input to the text encoder is the concatenation of the question text and an answer option. The input the image encoder is the image context. The output is the cosine similarity scores between the text embeddings and image embedding. Then the answer option with the largest similarity score serves as an answer. For questions with image answer options, the input to the image encoder will also add the image answer options.
(ii) Few-Shot. We also use CLIP to benchmark the multimodal few-shot results. For k-shot setup, we randomly select i questions for each skill from the training set as a meta training set. For each STEM subject, we train the model on the meta training set and select the best model on the validation set. At test time, the evaluation is the same as the zero-shot setup.
(iii) Finetuning, We also finetune CLIP on the entire training set for each subject. The remaining setup is the same the few-shot setting.
(iv) Multi-Task. Under this setting, we train CLIP on the mixture of training sets of four subjects to produce single model for all subjects.
A.4 DATASET COLLECTION
We collect science, engineering and math problems from 1X1_ and technology problems from ProProfs Quizzer and Triviaplaz4" We first collect multi-choice problems that have at least one image in either question context or answers. We collect at most 2,000 problems for each skill and remove duplicated problems. There are many formulas embedded in math problems that are not
httpps://ww.. ixl. com/
https: http:///wwww.oopoffsoooo www.ppofs.cco//uiiioooro cww./qui-school
https://ww.trrivvaalaaaaacoo wwtp:wwwwiiiiilla.....
17

---

Published as a conference paper at ICLR 2024
Table 4: Results CLIP with different training schemes.
Method
Science
Technology
Engineering
Math
Average
Zero-Shot 50.3
68.7
55.1
43.6
54,4
Few-Shot
75.2
70.9
61.9
63.2
67.8
CLIP
Finetuning
87.0
71.9
67.7
78.4
76.3
Multi- Task
86.3
60.4
73.4
77.7
74.5
represented in the text. We use the MathpixOOCe API to convert these math formulas into the latex format.
B MORE DETAILS ON EXPERIMENTS
B. EXPERIMENTAL SETUP
For the zero-shot setting, we evaluate all models on the test set. For the few-shot, finetuning, and multi- task setting, we train on the valid set, and finally evaluate on the test set. We use AdamW for optimization and tune hyperparameters as follows: batch size chosen from { (16, 32, 64, 128), and set 16 for few-shot learning, 128 for finetuning and multi-task learning after hyperparameter tuning. The learning rate chosen between [5e-6, 5e-5] and set 1e-5 for all training. We set the warm-up ratio to 0. and set weight decay as 0.2. We set the maximum of training samples to 100k for finetuning, 200k for multitask training, and 10 epochs for few-shot training, all with early stopping on the valid set. We use NVIDIA GeForce RTX 3090 GPUs for training.
336px on the corresponding train set, tune hyperparameters
B.2 DETAILED EXPERIMENTAL ANALYSIS
Few-Shot In the few-shot setting, we sample different number of samples in each grade see how the learning performance varies. Specifically, we sample 16 samples per skill and train CLIP on the sampled data. The results are shown in Table We observe that CLIP gains much improvement in all subjects after few-shot learning. This implies that CLIP has already stored STEM-relates knowledge and few samples are able trigger such knowledge. We also show performance varies when number of samples of each skill changes (Figure T6). The overall performance improves with more samples, but 1-shot and 2-shot in technology are worse than zero-shot. Since there are only 9 skills in technology, 1-shot and 2-shot learning in technology might lead to overfitting.
Multi-Task We show the results in Table Multi-task learning improves in engineering but performs worse in other subjects compared with individual finetuned models. The reason for the great drop in technology is mainly because its data is much less than other subjects. Multi-task training actually improves performance in engineering. This implies that data from one subject may be beneficial for another when the knowledge is transferable. For example, science shares many common topics with engineering like chemical experiments.
Number of Answers We also analyze how model performance changes with the number of answers. The results are shown in Figure [7 We find that for GPT-3, GPT-3.5-Turbo. CLIP zero-shot, and few-shot, the accuracy drops as the number of answers increases, but the accuracy of CLIP finetuning and multi-task does not drop. This implies that models after full training are actually solving the problem rather than guessing, so the number of choices does not affect the performance much.
Question Lengths Figure 18 shows how the question length affects model accuracy. For GPT-3, GPT-3.5-Turbo and CLIP zero-shot, the accuracy decreases slightly as the question becomes longer. For tuned models, the same trend holds for questions less than 70 tokens, but the accuracy starts to increase for longer questions. We think this may be caused by some bias in longer questions and the
https://mathpix.com
18

---

Published as a conference paper at ICLR 2024
62
36-
55-
52-
Sdoree Hanh
Engincaring Boorelogy
2
Randon
CLT Tee-hoot
CUP NUBSTA
of snarples
Figure 16: Result of few-shot CLIP.
Figure 17: Results guestions with different numbers answers.
Gamtin (lwuuig
2
Figure 18: Results on questions with dif- ferent lengths.
Figure 19: Zero-shot CLIP performance on differ- question types.
86-
02
62"
00
50
Figure 20: The correlation graphs of exam scores with model accuracy (left) and human accuracy (right).
tuned models learn such bias and achieve higher accuracy, Since there are only small proportion of questions that are longer than 70 tokens, such bias will not affect the whole dataset much.
Question Type We mark the types of problems the first word in the question request of each problem. Figure 19 we show the accuracy of the top 10 frequent types. Questions starting with "What" and "How" have relatively low accuracy, these questions more difficult to answer.
19

---

Published as a conference paper at ICLR 2024
handen
OrTOCiTato CLI Zero diat
Teedte: CUP Testiing
GFT-3
Zeen Shatt
Rostuning
100
Acouriyy (8)
80
Accuracy Score
40
2
Science
Technology
Engineering
Math
Figure 21: Average accuracies on each grade.
Figure 22: Accuracy on sampled STEM for human performance.
Table 5: Error analysis of CLIP math and science subsets of STEM.
Subject
Reason
Ratio (%)
Commonsense
36
Numerical calculation
24
Math
Counting
16
Read table/graph
12
Transformation
12
Comparison
40
Commonsense
32
Science
Direction
20
Read table/graph
8
Grades We show the model accuracy on each grade in Figure 21 There is no obvious performance drop as the increase grade levels, which is similar to the trend of exam scores. This implies the learning curve for neural models may be different from that of humans.
Correlation Between Exam Scores and Accuracy We evaluate exam scores' correlation with model accuracy and human accuracy(Figure 20). They in general positively correlated to each other. Even though exam score is different from accuracy, it overall captures accuracy as an important factor.
B.3 ERROR ANALYSIS
To better understand the errors made by CLIP zero-shot, we sample 25 error cases of CLIP zero-shot on math and science. We manually check the reasons for these errors. Table shows the analysis results. For math, 36% eтrors are caused by a lack of mathematical commonsense, such area formulas and symmetry. Other errors include failure of calculation (24%). counting objects (16%). reading tables or graphs (12%, e.g., graphs of functions), and transformation (12% e.g., rotation of a 3D object). For science, comparison causes the most errors with a ratio of 40%. Most these questions only require straightforward comparison like the distance between two pairs of magnets. However, CLIP fails on such basic problems. This indicates that it not good at comparing objects and properties yet. Lacking science commonsense also leads to good number of errors (32%), followed by identifying directions (20% e.g., the directions push and pull, towards and away) and reading tables graphs (8%).
Moreover, we show the top-5 skills with the most errors fine-tuned models on math and science subsets in Table and Table 7 respectively.
20

---

Published as a conference paper at ICLR 2024
Skall
Krror Rale
Esample
Descrpticn: The schion didratt compared how masy swaups cas ckenattrr setood Ban. Whish school wallgg?
lenst-ttttteeeee word- wondd-roblems-u---oe. 76.8%
Cholces [Sherline Eleceerary, Hilbito Benemay. Valley Blemeraary Lineoto Blenettry
Asswer
On which day Pus practic the leadt
grestest-and-least wtwpe/deeeeeetttddeeeedeooeeeeee htt.://sw.ooo...oooeoem.-.--o-oo 760%
Torts
Wodnonday,
Asewer trakoc
Prediction: 2
Deariptan: Lock the fotlownng scheotee muniing erdea 12:00 PM?
reading-schaakee schailles
75.0%
police council rreeting, contracion permi meeting. parking seter meeting, Asener tnalo:
Prodiction:: 2
DEarquan What macii the ago
Pkenui::
angles-of-90 1o--aa--------------1--- 180-270-and- 360-deeesss
73.8%
Prodiction: 5
Deargaan: What
points-lines lines segments-rayy rays angles
Choisre lin segrere alne a Asswet indos
Piemer:
Prediction:
Table 6: Error analysis of top-5 skills with most errors on math.
B.4 COMPARISON WITH HUMAN
Exam Score We test exam scores on all skills in engineering and technology, and randomly choose 40 skills from math, and 30 skills from science due to technical and time constraints. We compare neural models with humans using the exam score, and the results are shown in Table The detailed scores and skills are listed in Table [0
Accuracy We randomly sample 20 problems for each subject and ask 7 Ph.D. students to answer these questions, and calculate the average accuracy for each subject. To evaluate neural models on these questions, we use the corresponding skill accuracy for each sampled problem the models' score on this problem and average all accuracy together as the final score. We do not evaluate models on these sampled data directly since the small number of samples will lead large variance, and skill accuracy can avoid such variance. The comparison results are shown in Table and Figure 22 All sampled problems listed in Table to 17
B.5 ZARO-SHO SHOT PROMPT SENSITIVITY
We study the effect of prompts CLIP zero-shot. We design types of prompts and demonstrate them with an example problem. The example question is "Which property matches this object?" and the answer "Rough". Examples of different prompt types and the corresponding accuracies shown in T We observe that "Q+A results in the best performance on average, but the difference only marginal, meaning that CLIP zero-shot is not very sensitive the format of prompts.
B.6 DETAILED PERFORMANCE ON SKILLS
We show the accuracy of neural models on all 448 skills Figure 23 to 28
We can see that the zero-shot performance is generally better than random guesses most skills and achieves near 100% on some skills (e.g., "circles" and "cones"), After finetuning, accuracy improves on most skills and becomes near 100% many skills.
B.7 VQA RESULTS
21

---

Published as a conference paper at ICLR 2024
Discrptan: descrber acuunal fis oppoar aficts
tlan Noat lops Seil head bussdd!
the coll the By's biod farn earzed attorus that like leg. la group of frsi fies, wera lare
aldo. allek seetral atcome groups gone ataenaan This Paaner opuns chose vllegeing
slick
use puntec. calculate rallos ottipring types
68.10%
nato.
Puture
Ao
Choice: 33.22, 13.48.1 rndeo
Prolictie
Diserpuoor
(rs.. thi gauap devinann perlciilty plant dows caree ferteeee
fesl tual alkdo. The aliie le souech inai mory fait
raceaiie du da inai kasuee goad!
us-pumaett-suuue calculute grobubilziis off.ttlppiingggeeoooooooooo types
Choikon: [iok, 14, 2%, 344 44.1 Proliction:
Innonaar pepreeeanee plont eneryy ascapes fome the persa aurk livi rnsoaaaed Which ofterpprnnns
puoiit-icnge strreeclnges
55 0000
Assrerisdee
Pesteciion
magees
ripel pechippe
idemiify magnets that ontroct oe repel
21.10%
Bazzzz Choike: [areet rpell Prstcciion
Deverpton: blockcure widffecet lorpenares, boching Wiktipiarr dues will Phaaec Noee
predict flow
16.20%
Ansver indoo
Prstecionn
Table 7: Error analysis of top-5 skills with most errors on science.
22

---

Published as a conference paper at ICLR 2024
Table 8: Comparison between models and humans.
Exam Score
Accuracy
Method
Science
Engineering
Math
Technology
Science
Technology
Engineering
Math
Human
90.0
90.0
90.0
68.6
90.7
62.9
86.4
92.1
Random
26.7
16.1
51.1
25.0
38.3
25.0
40.0
36.8
GPT-3
45.7
50.2
51.4
22.1
48.4
21.3
65.2
42.4
GPT-3.5- Turbo
48.9
58.7
53.5
26.3
48.5
27.4
62.5
40.6
Zero-Shot
33.9
19.0
52.9
53.8
60.7
65.5
44.3
Fow-Shot
39.1
43.9
67.6
70.9
77.3
59.7
55.5
67.8
CLIP
Finctuning
57.8
37.4
75.7
71.9
91.9
62.6
60.3
83.5
Multi-Tase Task
61.9
50.3
72.0
60.4
90.9
50.6
70.2
82,5
Table 9: Examples for different prompts and their zero-shot accuracy.
Prompt F'onnot
Esumple
Science
Technology
Engincering
Math
Average
Q+A
Which property matches this objezt" Rough
50.3
68.7
55.1
43.5
$4.4
A+Q
Rough, Which pouperty matches this object?
50.0
66.0
49.6
43.2
52.2
"Choose the best inswer" A
Which property mutches this object? Chosse the best answers Rough.
50.1
70.7
49.7
44.2
53.7
"Aaswer the quesiion"" Q
Answer the question: Which property munches this objectt Rough.
49.4
67.6
51.0
43.6
52.9
"hest answrr the guestion"
Rough best answers the question: Which propery matches this object?
49.7
69.5
50.8
43.8
$1.4
We evaluate the zero-shot CLIP model and models finetuned on each subject on the VQA (Antol et al., 2013) dataset. Results are shown Table [T The average increase of the finetuned models over the zero-shot setting is 1.2%.
Model
Accuracy
Zero-Shot CLIP Finctuning with Science
27.3%
Finetuning with Technology
26.5%
Finctuning with Engincering
24.8%
Finetuning with Math
24.9%
C ADDITIONAL RELATED WORK
Table 11: Results on the
VQA (Antol et al. 2015) dataset.
In addition to vision-language foundation models included in the main text, we expand the discussion to some recent models, in- cluding BLIP-2 (Li et al. 2023), EVA-CIIP (Sun et al. 2023), and KOSMOS-2 2 (Peng et al. 2023). BLIP-2 provides a versatile and efficient strategy for pre-traine training. This strategy enhances the vision- language pre-training process by utilizing frozen pre-trained image encoders and frozen large language models, while EVA-CLIP proposes series of methods increase the training efficiency of the CLIP model. KOSMOS-2 enables new capabilities for perceiving object descriptions. This work focuses on the creation of dataset evaluate the multimodal STEM understanding and we chose the foundation models like CLIP for a pilot study on our dataset. There are more benchmarks targeting formal math reasoning (Zheng et al. 2022 Liu et al. 2023 Xiong et al.) (023b), however, they are all restricted to single text modality and they can not evaluate fundamental skills.
D SUMMARY OF SKILLS
We list all skills in STEM in Table 18 20 and show some examples in Table 21 27
23

---

Published as a conference paper at ICLR 2024
Subject
GradeSkill
Rondeet
Finctune
pracoo-lellssily - hosp:sssslooottttaa--llllt--o htwpit/aaaa--lll-lli------oooooe
gndde-//dentityy- www.ttttty--iill--uuiiie wttp://lwwwwoondddneieeeeeee
gradle-2 2fidentily mammals hirds reptiles und und-umphiiianss
grade 2identity materials in-objeces
21
100
grade22/ideti 2aiei/////nilyeeeeeee Sapooniiy-ppppiiiss-eee pfppppprii-------eeeeee
35
65
65
godd-3/eampure-aie cooompre--atnnnee satnee--tnngtt----neee cetoooo--mn-itttriiiiii
18
63
grade Vilesmibe ecosystems
65
100
grade 3rtidd evidence of changes cartassss surface
17
Videntify ecosystems
gade-Videniiyy-miin miitiil--iiiill-uiigg--eo p
grale-4fompare-mo prpprterppppete--ooe apperties-ofccgeet
20
grade 4/descibe ecosysiems
74
100
100
grade sndk--iientily minerals using properties
35
16
35
grade 4ltse--viddnnne wow.ncc-oo-lllsty--m--.. manymmlm----bee bads fish-hh-hhhht eputics-anddmmmhhlaa
26
35
35
Science
Stclusslfy elementury substances compounds using models
snde-Stanimil-nkkge cetiiiimll--iiiattiiionn oupptationn beaks-monhh monos--moos--n---nne -nn--e---
75
17
75
35
grade Sicompare
onpanisms use-oheervatioss support poythessi
grade Sidaiiit-dddieiiiianiee
35
grade Siduntity che.-hhhttegnhhhiceeeeeg
grade Stpredict semperature changes
grade Stuse evidence to classify animals
35
35
35
grade Sluse evidence to-classiyyy masmall buns fish coptiles-ann and -nd--aapaamaaan
18
35
35
godee Socuthei-nnd-llmmi awwund-he-uoort
60
36
60
grade 6ecompare concentrations solution
15
100
óldescribe- whee--hee-ell--ff--eeee-- gene mutations on-orammioms
mode---iiffsiinnee across membrranes
grode-TMeeerrlle hete---ee-ffocc-----em- htt.:-ooo.oo--mmtttsts--------
grade-illlssafy www.atte- mnniiiiiiiittntnaaiiinn
25
36
45
grade-&tdifliiinn &riiffusiin--ccee membranes
0
18
35
grade Wimoss and forn life cycles
12
0
ólevaluate
engineeting design solutions
100
grade- bedd--ideemffy--- cotfit--cooooaannn -n-eep-eem---emmmmii..... groups
0
grade oldemiity independent dependent vistubles
gradb-Sridett wetett---eeeeeeeeeeeeee queettl-uuuiiiioo
grade "Nevalutte lests of engineeting design solutions
grace- Tidenidy zooh old experiental groups
0
40
grade "Fhdenityy independens and dependent varuubles
0
30
grade- Tiid--- feetft----eeeeeeeee cete--eprieettllgiee rimentllqqussiin Shdeniify control esperiemial gnoups grade Sidentify
40
40
Engineer
question
grade Sidentify laboraory
21
31
godd- Sadl--/iiittfyyyyaoomeeo babo gride 5'laburatory saley equipmen.
21
24
52
7hi-777httyyaaaeeee sools
10
28
21
grade 7/labrratory safety equipmens
9
58
52
geade "'identiyy-lahhnttray sahhnntar--ooois
49
21
unale Wlaburalury quipmen
eouipmen
58
algehra 2louthiets scatter plots
calculuskktemine continaty using gropts
36
80
caleales fenttta//nnddd find--mmis-----ee lamit vervecaal aeyoalllaypppooe snyoppoes--ssiggg seeeees-sinngggaaeee.. gropts
60
63
85
grade- Futhtracioon cetences-up--oe
whichhhhmmm hich-makl-muthhhe
50
30
99
grade 2hdemiffy haives thirds and fourths
65
75
97
grade 2fidentily lines peee-of--sse of-m-oofmmmmryy
70
99
Wunterpes graphs-ii
rmke--heeiif-pailllllee parailelggrams
51
70
grade-Slis polygon
71
98
70
66
45
grade wrads-4nntt-off of-fts-of-tbres.. tine--imaasioenlll
68
40
99
graie Sfnets mime-ddmmaiinal-ffie
53
99
grade d/changes
median
range
38
grude-fleissiff
grade-6ddeeiiify -olyyhere
gndde 6lmean mediun- mode-anl--aeeee mal-nnne-finnl miig---i----ee---iinnn--e numbibeere
Math
grade Glmodel and solve equations using algebra
grade-6lrtioon annall-naas hind------------ii
36
31
78
57
99
grade paraanooll-yymmrrr
62
78
grade d/which mon--cogrrentt-ie ginent-ligunes
grik monk--Asimilranee and aee--nnddd-gentnt
34
46
described
36
86
find sign
grade Strotitional-- semdl-yymerrry....mm awcount-o--oooooo of-rotation
Kinderganiniivwnttooo frmees-up-oo up----
kinderannnn/eweer and more up to
80
97
tindergantee//bbrrrttioi sentences to- which model matches
41
30
96
k/addition sentences to 10-which model matehes
60
55
96
we-Minunt-ee to-ti--------ete fromes---eenn 1n-3
84
51
pre klewer
compare matching
63
90
pre-
piclures-up-te
pee- come--more--ee with pictures-up-to up-to
pre-//dlpes-of--n tote---o--oeddd-yyyyorttt pre kopperre
96
67
96
pre- Aitrinngles
57
75
75
s.whit comes next
75
70
Mokooniinll-uumhor.ooo wap-herr------eetttt kindergamen/ ore there enough
27
82
96
Table 10: Exam scores for each skill.
24

---

Published as a conference paper at ICLR 2024
20
1.00
Figure 23: Accuracy per skill on math (part 1).
Figure 24: Accuracy per skill math (part 2).
25

---

Published as a conference paper at ICLR 2024
Figure 26: Accuracy per skill on science.
Figure 27: Accuracy per skill on technology.
Figure 28: Accuracy per skill engineering.
Figure 25: Accuracy per skill math (part 3).
26

---

Published as a conference paper at ICLR 2024
Desurgtinn: This ivata slich Sanceo puggant
Deserptor:: What comppeent hers?
Cad, Midichoadl Cad.)
Patasc
Atanzi itdes.
Cloico: Offec Ostik,, Onditine, OfficSulte Opens inke:
lectastayy
Desoption This lugar (fumer) loge whiih coupaae wiserd
Sukjes: Technokegy
A
Desemptorn Thi (ermar logo witchs compaacr brand!
0
Raao: Chikes (Pajtu Samnang, Tyres, Brofke] Asowel Indes:
Cheeced AASlods Arig irc. Artts, All Techesiopies
Anerer
This (homet) comppaee wilerd brand!
Desetption What conpoeeet hwar?
Picter::
Hemas:
Clatcoo (Xiausi, Clan ändk, Wik.]
Cheioe:: (Muptay AdgereVidoo Cart, Cant Sapply thin
Douiittan What incening in fassisuri in saally coniiaad vilk dis web inaakeee sydlad?
Sinteer Technology
Devriptoo This sin-44 lego which fannes appur progiont
Piraner
Choica: (Pait, Scadd, Tiplics CdladaaEEvstt
Playrr,
Anoxer
Sttaett Techastogy
Discrpttn:: what compurrntaadd play port do you hame?
Destipton blnnlly tha foe
ActionQuiz
Cheios aidd Nous Gobllc MT. Couder Aaoveriedes:
Pictor type Cyblg, plag pope plop. Cllele
tectnntry
Dosc Mesiff dés foot tpe
Destipton Mesttl this feettpe
Baae: ActionQuiz
Pieaee
Clacco: (Couanniaa Scritt BT. Boudi Saijt Visadi D. Widaky BT.]
Cletao: (Gananeed Timo Nee Rosnan Craina New, Gesgias
Arawci
Desergtia:: The (par a (fennen of shich compaier relane hoad??
Subrec: blotti
Frane ActionQuiz
(Weldings Impart Indes:
Pettor
Cheraee (Hent), Lecrans, Crnite Tecreciuy, Leaos.)
Subyee Techenloyy
Desetptorn What trranng frecnon in aouraaly aroucanud with dhi wob murthor syrthee?
Vernity
ActionOuiz
Calsoo
Arawer ankox C
Cheios:: (Pinittio cmmettng dirky, Surnge drivnt file, Optens/rattgs Prire grostens)
Sanget Techaslogy
Subres: Technolety
Descrptia:: What menniu functin sually soccused vat tha vob isaritce symbbo?
Desctptor:: What rreasin lascioe in ueually aussiand wit dhs neb meefase-orrhed
C'atzan Etorn lidp, Like soeratinng wlect]
Chetzar LAppts Optamo/atttges Rekustttttra,, Hinetioss
Techeology
Desctptoon What maaiing faxaton lourally proutasd wits dhis wit intrtiin syerha??
Doviigtan Whaa lype videe gans onnok do kock
Pictzz
Chiiess (Marel brefisiion, Sega Mate Sysen, Magesvos Ody wyy 3, Nar 5300.
Chetems (Prad, Atacineni, Cakeatattive,,
Table 12: Human evaluation problem set (part
27

---

Published as a conference paper at ICLR 2024
Sikree laybour
wlich
bakon dasillo expeciaannt Lusy and Erik taking san colonniing Duing loe
Sukjett Expisea
ther inomesor they woeld beer apater was indemidecet seoettaant
glowes
ursiht applind rovord
Pietarm Neme
sansbvant emaggh down the hill E-A, lined low bag cadi rike sook. Lass sppetaa thoe rile os foo, other loards alhanding als roce wilbust
Cheicoc
Anerze rudeet
Chokose: szoehwond with carebronk with wiliec)
Asave: index:
Sarpor trggnes
Supper Fipracr
Picturs Neaz
Honue None
Chicas Araneer
Chiias: Ancwertedees:
sulces Esprace expartmere, otich
expertnoncal The posage
deenntes ixpormment Kinteety gaee mor fhe a frwee shop. Oni diy, che noticed timve the Six
Ergines
gracticen on strme exee
plane bad emon yat
bastria Kinseely caeaed
betin fack
Pietre Noss
speping las plaro wall julco would prorest laznes foom fornnng un les plaass Csez himtary sprundd gatle jacce infocted plans unweter misccee parts unspraed compared
groepe.
Claicos
Aranci
Plmeer Chetoos (ike noceepeacd with paile sioo. dee rome that wars gpnardd)
Asovcr index:
Deseiptans enioe: Which following codi Kendna's danx? Word te tats siad poveer to postene aoo kity Kendl maaotal engineer who drdgged wnal tuftrees sented roonkity So konds crased computr medid of tartas makk irom lgdaveiati matetta Thet ahe und be medello to calcalae lase much dlettictt the an protten Roal the pass Then anosee the gueatn hullow
Suljes. Expisoa
dndge
produce
Dosiptiee geises la too cue ineet, wlick were yast supminectall netttal group" The ponsage woedors have tal for anistes. Thes, le uttddal fox doo-lbog ana Dew the ldfe fox live moee siaille dlcanaing whellas sesttedd fines
nufine froe lightoiiht monvia soald
radly
tail world ather
dex
del
dosrthes
could compard
solution
miriio. bax reponell Belle
Pistec madd
Pempe: Cleiso: withost talls tiis.
would weigh, whalles dhe tariia produce 108 riveniitty the berbier could
Anesur rike:
Saggen Erginse
Desr prie:: Hesttyy queeice
experteent the intractins talow.
baloe docrbo Boyast plazed a piaz peny ball catiual, pulled the catapel'ss am back ss a suth and laaxted
Head the
time puliin caleral's ann and the plice whem the hit bi the proad Bryan and Tarne repeare the benches with ping hour dentical crapedic. They oxipered dacunaze traveed Landdeii 10r
Bryas branch anstue pinz pons
Subree: Eusierr Destistos Edkussya
Cheios:
Ansrerirdes:
anpk u4 aujic, Do pluz prop balla lnctac when hanahed insm 30 snak ouspered Auaner ankox
Chvice (D's ping peeg bolls copiniinn sling the ground oaner after heing bunched from
Table 13: Human evaluation problem set (part 2).
28

---

Published as a conference paper at ICLR 2024
Deceqpice: yore Which or the sattoring coels hark the shoor hon sox landsope archsst Sampeer trngnser
space shoul he koe 150 muner trom the noal so triti noes dat''s insereee the maer. ite diceloped froe preatle design for the park with the cononn ana ditinere iccance doays. Then bekin Josciho law lhe onginening gegeeenneonnggggnggg
loige noor cay pack. clecly cousil the
les
slakklly
Subjec: bngner
Desenpoutt Seaci the beater.
ascil to tost codlatios to prokcer, Roai
the passage Than saoves the quesder belke..
Anoun 1nde::
Closces (if lowa 35% of park would be staded by tas in eadh dosign, which doigg would lave the greatos distnce betwoos the wnacnt srea and the red, binggg wodld have the bust tulll sosse cereelt avas)
Subres Enginss
Sirgect Tirginer
Descrptiee gytor thee An independerrs vortalle vontatle shoue effect you are intraigenng drpendent
spteer Ideasty gansron desniles experinass Real the and faos fielow the instre lea bacon Zoue diraked 40 unipe beassa cools userg calk paus legs and sodee the leas. He pokel 20 saull lote- line sope
natadle this cxpeiient! The paragg befow dociles caperiaans Besd tee incepenzore dovul the varidlls that uee destlled. Tyke designsl skotric ciraai to koi low well dilecrus tr axtal condldd clettiitt shest
task
and creraed the nanther hoow spotices irmpereers compuned
circil indesie fetiey, light
wiso, clp
Them Zelo openad
meral coeduted elsoneryy
weald
the mor nerherat brom eysts en hanarce trve hage with belee to the starage sorther of brown spato henasi bage wilhout
apporder. the ruts conkured eiemery well the light but woets opear begs: lyler collaced nin cualls sinel slocs mucti: daee shesh coges. frve slocta inoe,
sleci Taka tsed rnder cansare lase rach iiht the ball prokucdd.
Pieauee
Ohicze (1in honazoe denclip more boown specc they are keps in hage with hoker companed in lops wilhoal Iodes"; lexanas Jevelop sooe brows sott whaci fayy and kept al noon insperatue tefrigeatee?)
Asawer ankox. C
Cheiose tohe aesoust of light prodased the light heth, the type munst thant noud the eiecss.
Anoxer tndes:
Teyrawy
Suiject Ergisos
Desctirina
Which fivluiing could tadte's tua shownt all bost corice melpp thor mudi with thick curiiie for bo portimn crunchty cosicies Lake nout thr soing riffnee typec sgga lase lins cookin Ho sesipe ane both wiite boon sagas he desiled would
ypee below drecnbes expenteen. Reat paaced fuur wasos of wake inks of in plases Desos disccled yee tabloyoerofo observel
espeneean the tnorantoee beloo, Desce
glno and with the pluniss meceting esca time the eng treied Deven repoard this sun with nso mor aggs and counied mumber times the epgs thared in frosh water companed ealty waner
glases
She ohhe floee Desus placel is oga of aal in nad- dival
ossicie teing his nepe, her ha latt net the brown sngir unt donrree the amoran of white eage Hi babed conite the one ancere of as bic mepe After cookens finiied ased solition pooblere Rond the paisgg. Then urowec
the
Lake
oxgincciig devigs dnc qacdkaa Indor.
Chaices (Dees arcoee woter glass allict whetler oggs sink Bout is the water?, Are
Raps mure loudy that fred waser or valy sesert))
Chelow: Ti cookins mode with white cagar if hakiing colliie longer mode therr sanic crancio. i cookio made with dosblc amscunt broor saza ware cuuniry ]
Anonez inden:
Subject Ergineer
Desctptons trare: Maniis quesize dhs Meza's coperiment cas bot sncver. The paoage age below cenn In archhard don hous of repul doe She galend right of heurs back and right the beues shitr Myn made envil hale dide each
Which ot the belkow my coets and boddye' tue steee' zoe and loyety muking backo af ccecute her creamatin presat maae ta sesoets duyy mese together dr bush
Desemptier
Real
odlll powder greed, warer they
eesugh using
the auck rhvrroeee partally naslight buckyazi Two houp late, ale memured the Sompersarr insid each Mors complane and doc avernac toaperatase tukke the kads bonzs to the avange iomponuaare insds dhe while
kett the concetic the meni oone for socrads. Then, they llted the cons soe the cinarete daped in come dupee collapueri It conernte bash calligeed, they would know to Inst solation to pritken Roal the pessgge Then ansvee quesiet beltww
Pictare
[Do seepenatees inside brues depend
boued,,
Choioec [If the coneueee fire nuch boach nork the came smosent of time sovtyy It's new boach
beoes
lasides
the
orsevve wee fimm eanegh me..
Azower onkec
Sanget Bratnoer
Descuptoo: rssoe: Manrahy the quosioe dea Bilk's couriiaan cas bot awoerr The russage bdbeer dbamtber an experimens, belos pomane and then foloo the antraciins added dele plansel fentler to theo of the iis gets. She ploved the pous in plant growth dhveeber when of the sends repe revedd the cime temper monn ararers of fight, and hamiliyy berrl After two works Bello fenileas be namior scadlingo pots withori fertiier.
Subrece basterr
cournd The lilkww descihes on cxgeliment Afler seere winter shemn Sandeg's di drieway was covercl with loc. He read thas makes mak ata liwe temperanee tettve covernng his eatire drsiwwyy with vak. and the other hull anuked. Be place all ke colesoussle One foor like 5 Sandep spicilly dliel each xee cahe and sneeighed
griier expeitser. which
hep mick has
scadingo
Sandeer
cabes. He qeoiieed
Picturr
Peane Auaves indio:
wil wileas fostlurr'. Docs humbity levd whize sanuto sond pastal aflec namberr
caher,
lossao sooditgg thai gowe..
Table 14: Human evaluation problem set (part 3).
29

---

Published as a conference paper at ICLR 2024
Subee: Steas
Natjee Screse
Desorpaant oee rahezzor that mitumd
Dociphan uno
Naze
Ouicac
Chekoue
ling Hege meee FA0
Greed mem
Discrpiie: inten: This crzatoms is spot poncperatob. to accetticc rure Doddos bysttt coumsnan vame aenau spet percstazcisn.
Subrec: Scieace Desolptann Sclent Biyil
Chiccoo Ateazt indes
Akpiis progocs
Deas bown
Choize:
Aneverirdae
maki 8.
beac
choosia 2e10
Sulgpe idenfica
place
Suiject Sciene
Oie brack eads oflee. Host can flw foon bock to anutle but carect osrge fives Neckt sempuratnoe each Hock mezerol agarz. par tumperatans
Destiptinn the cepprtoon Scleet the anmal the to all the mas liaed shour Bloos pelicina thee alony was
parp Thes
Nert Amenes, with ao shill botics feollwatr abouns. Salone thic jowcifal fim toswit, cat eonnjant jant stall Falk!
undernaer
boacs Bhenn
oggs
per't
Berck
CT DUALE
Choican
Desetpturr Select
Hatus: Nav:
Desctption ipsiez Two unid bankk koaee irspenterrs akonns The badd placed
Bey loach. Which diarras alowa the dircctie had will dow?
Chilret
Cheiza:
Sutget Scenze
Desrpetan patee: Uo dao anouer guetion helon fellwding sturnerr about syrry
plase,
tallvring dacrens
Decetptorn ipiin: Think aborr the magaetz frece beteore the magnete each pai: Winch the
Ncamor
tmuget noo pains maggess magnets is dittener ingaats
fenr. dillgett *
Cleco: [D0e lnc.]
Chekow
rooe couater ther The anggeteee Erim litth--ime cime both pain.. The inginnse of the magante nooe enuller Pain I..)
Arower anoos
Table 15: Human evaluation problem set (part 4).
30

---

Published as a conference paper at ICLR 2024
Subsec Saaae
Subject Sacaze
ga Prane: None
pair dfforr
follwanng exterranti mann
parsds for pehor All magree steen made the one mavnaa
Doizae
Phi
Arceplh the seggecic Auce the sanse in bodk patt..)
Cheios:
Sciaace
Desciptins Which tha objoztt belect the beteer
Pirnnr Nane
Fower
vib of
Cheeze: anoeth.]
Aoowel Index:
Senjer
badiiunc.
not lave bucklenes.
Sakyee: (pice:
repersete
palick sellic Wiich wiiti ligler oncettratie pariikk??
Cheiker:
Sahatto Sclutee
Anowirtrdrs::
Anniur mikes
Sangee Scraze
Jewn mone? Piannr Nose
Desorqtinn:
immperatress noxhing
Subiec: Scieact
Destiplou Sceci bee chonscal fouzala for meboode.
Peaae:
Cheicar: (112, IR: IKE.
Chitco:
Atower snke:
Whidh dosarbes clanat Nurhaa Herisplere wax in lux. July and Augad, Winten in the Nerlhan Hestsplerz oau is Deccalee. Trnurry, and Eduasy. Barges, Mans, is eilty in Uhitel Stzces las vam
Summen de
Sukjee. Sciaue
Descnpporr imperatee
Pistec
Choizo. (13E.6.(333. 63F.556..
higher veppratims..o dightly pocipioaiie
Asove: inden.
Arpner mkkoc
Table 16: Human evaluation problem set (part 5).
31

---

Published as a conference paper at ICLR 2024
Docaatiin This sdde danns Jaaz's Janaary bulge. What could Jasu balanc ins Nengre?
Subees Nath
Destipbou la addingg das tringie vinii hov mast sou list!
Cheize:: Cosane, Simus
the artads spend S31 mom on siteo girnss
Answertrrs::
Anawerr mides
Descrppion: Look caher saauEcCc cloas do voluase will wue?
than which dollnwiig
Doci pthn this asgle wote right, obture, strigle?
Choccss
right)
Feaner
Anower sakes
dhe solune The cato of volomm vilome will 361, rilloa dhe con volure to the vourm will 2713.
Descrptin:: Whech
Subreen fanciio
Picten Nost
Chikes [1o, po. Arowel Index.
Antwer mule:
Staset Nall
ratiral member
dagrars hnl quection Sillwiig
Sahype: Nith Descrptort thippalggea rapores
Cloiso: po.. Asowei index:
Chocent 8-133, 194,
AnDwer mnke:
Sugeet Mal:
Sulgpe: stape
por shape
Pheturm Neaz
Przame None
Cheizos
Sanget Matk
Desctptom izioes Heaaly dee secice of lbs disect Amsrac ebjects av pererddeaka i tay appear
Subrec: Klak
soore siedee viasalo?
Hoare 8 10000
Choloe: cidtis mangles Arcuriirdes:0
Chiizan
Decotptoo &s the nrezice continaeee
stage Whick sloee reffecioo?
Chicss A.T.. Abower mboe
Hase::
Cleiso: yo.] 1ndes11)
Dewtittonn clgge bava stingle fev?
Duscrptoon As croue sandar cook ddlltarand 11 crets, Be hake eesegh eansyy to bey i??
Pickez:
Clatcoo Eecu, mo.)
Cheiza:
Mah
Descrppton graph bebox sbooe faxctine, alo fonctves!
Discrptoo:: ppices lunii at agan: Whee the stage of howe??
Claico: Monggn, integee. coteglloott
Arower anoss
Cleiso: [ro. so.
Desergtion: What this cxponntial fanction??
Dessiptin Look thi grggle edadion funstion?
Pistaz:
Chetoer: (no. yo.1
Table 17: Human evaluation problem (part 6).
32

---

Published as a conference paper at ICLR 2024
Sehect
Gode
dialee botto sesecc mitec ali digrrtmn foe
bost overigg clestiy futoored oggn para dh eatee
wn sat uas page
and os
opuriio decdd vile ankead duiloo cloa wo dele qlllo part, deady
mols mou piddet bua dow and support
somma sopaledeesion depion andti sicit anim antgaatione aocc uniraled musoraa cnneett clactff tatioado regrahis wagppppeeaaaaaeeeeo paaappraa dio deeoddde op
Ero en comit und feee festly te his by poits
arees and fimm klee netto fnvmnc plaiug
uula wush wi selon we for ane Acpe
als di thees
a eoods doceooofoooeeee chllii dyvettitt
hoe besto and arche, grinec segrotiee
oeet cleutfy domeniing onforree compuns enep moith, avetyy
compan desudl fermo crmpure orpprrrctantt cuetrrcettee foottoo prpprn sxphing rechuae comprimgno pnyettactinn choeeeeeeee mite cytte with
an alll andeon
toral sieet by ade dema
any
saub io didd-a-cceppai ao dusity corcerannn potacc andee denepeeeeoooerrreeeoe ceeeedeeeece dharffenccionn uapiegooood menan aeceraaaeeennnttii...... couttyrrtiiioocciieee compar no, looele peplan tim and mvny morter wnker plan
dew de deciociii
seeeees in Bowning Atretrigtoon
reach five lile emporneee
dieeety lim qreceet wee ayys like
wire
cauly on das apus po of aasts and aupespore andes ioppllas duikty blea ahea semeooa soparaa ko mand cle rade ippone aalaaaaa sapaarapii
uhie sumis codeeaoo Merility friind overga oit ands lento-ciciinn cecoocccniiiieeee fabell sed modc kima kinatyycen und cadiiiiaa to euros Memttyy copoecooneen donttt aon---naaaiitaaaaee in dlage won builee for
enily dai to appeiiiiaal anth mnginpore an leke Eii-qp clant'y ele
undog clas woite voucii uaer of dlocks
Satra comaaaaouuaad comppieee eue dist conetttttttkkeeee elep botatatt-aaeeeeeeee de & Soothoo coce--tffccceeeeeoooo coctteteeeigggoooaae sito-----rrppeeeee eus, cits the aches eontissoo Henal denalli-aa did comeesscceeaaae seave konebyyoddo qutcad, kiseady doa--ohhcoooegggeoooooo wegllo- racoscalo boc ciii- ailieeiiiiieea darcs prodic poinpic parppcaan cangee
une
endlaie bn worki denige with Sinn de dy wartd und enceeuniin and aloud' inleee de anl daye de de dayy Like baln, dowaly de experces qunticl
meviny Amiga
expartmen praps Vtealy snreee
Table 18: Full skill summary (part 1), including science, technology and engineering skills.
34ps Gooe
In dent aiiiaddeen coooeddaag elcppeeciss www.-eeeneegggg......... donatiiip penpuiiiiiooocco dusc soright Bicenngddee depii-iii scleee die Mind Sondty-ihoporraaae ieeeddd tenencrratitte idrelly
idloquulina. dusatty lacan has chee accdo quaccaloon cae saiche cren guadts dumaly Lacanguu aedoocac-ooo oppeeraea cociiii aobos dondy plopart ong eneg-lee diniado wite ane de
porrectn price
angn you
chastf-rzzstmm doct to (ndd Auccon an dustturdi Brald mrettc scttios tad
and doap deman congoot aboultoo deceeaaaii gogpt denu to and el ttttaaa dematanddragge une lisiity diwdiiiddeeddeedddd
2
Table 19: Full skill summary (part 2), including math skills for (igebra-(1,2) and calculus.
33

---

Published as a conference paper at ICLR 2024
Ai compar dinlls ceceerenn canagrate cha dioo angleas
and
how se aed wlide it
day
of
reperaade de
gre
watk to iaaoocalii 12
placor tastorthho whg.oetthhttttt.....
acaoo saacobtaaaaddegee weghhrrdraaaaaaiiie is. atrenges and grouannd www.eeeeeoppooooooooo wetppppooootttrrneeeeee. conal prrieni andtprenn dhowe aoprrrettopprrrooooooooo eoco anpeetgee pvepac-cno andy
food claualy
680
was ran
slaudly live
chestty chovefy- vot
yon
wtg.---ngi-cctteeeeeee ton in andinacte andeatccccinniii
andin kout Boooeaadttstt hacolias
tecrpmn de-
the wale
age pes
to Of what class shon
kow for aoddd lii unsosss d-chidranti motu
dinos cotos over coetics consocitioo
comars catos ditinn cpai
at
to wodds ali--g oua deteeppee to Sipuadi tall dis day inde
soald dhe Sulido
(il did by chiltrng arooend oo contonpgraneet carnsetlppooaae wttaeaaotttttttttttt distt
oppatoo crack comeemmeenn capacctcioo
anch
finer reperesier
une loo de aito
Table 20: Full skill summary (part 3), including math skills for grade 1-8 and pre-k, kindergarten and pre-calculus.
34

---

Published as a conference paper at ICLR 2024
S2IKS Brgness
Sibre lnrioee eupcieett
dosigns
Dusce quiae: whee of milloring could Uloro se dhee tilo kos saking pan her wheol's the corperaiie, scaded haid ne pepeict etck bragge the
Destiption to lii cxperiannt wiich nete oc's control your pooegg lelow douilo ousiness Maddes havu babble uadise and wouts to kane Jow wo maki the bubkées koi lange when licade
enginaeing compentoon
ilea. Sle lui is pick use ofe sisiges only cospettions led vilud Joiga vurstoigg Blaze bell two protttppes cads will 200 possale aicks, She thes added le vegglt io eub pewettpe solation grablen. Read passage Then aeptaornig hellow.
coeld make nyttles leaga codlay the lubile wtathss Madcon cuelod sis lsciko bubble solittoo to 30' lelon rou losperslar.. uadlay bubiiko
So, serddeed
bebble
ieapostur, Thes, ale losad.
Peeter
Cheices Blw neight biilge bell puppick wicks orald hold which design could
Cheizen: bonto that cocked doem, bostex thet somperantes Anowertrdes:
Antwer sukes
Stoget Bramns
independer conaate varaale whose offect insoatyxing dependers contata iia varhhi tha you messem Which the tollwing was rependert coriatle is lis experianne The paovaas Grasii
thout homone De Borh derigned ornking enow thet contrned a ther mooee Gurdis from wore De Keth wastet fron it lorger therigg one world masose mow Guntt, perpored oec liee hakdoo waks, cads conaanity i0,00 Grauis. Thes, Roda prosal leach waks throggh cadh stune. Allis cads beids provel Encoah fhe stow, she woel exkonoope to cuesa the Gradia that ceneated in saudl suaspl wales.
deaclles cppciannt
peasaee
Subrect Burierr
made
Desripton Select mendbbtterr fask Rarre Now
Chetza:
Pistac moket
lregh shonag stose
"aryer Frggese
inaasry ettpppeeeeeetttttspppp... qpettto
dewnibes experimoos Real lellee''
The below
and thes follos instuciisaa betee. Jellies spacd becknta into aatian-rich Besid whac the becaate could srow. He poured fog casscs the He alnoed bazwis the
Sdies, Exjiser Skill (eomarr--ziyyyesurp
is 1PP tnugass luttiy noud micnssspe to soust der ouiler budiai is amaal sample from cacs dlok, He coupace and th: amsontt badenta liqaid with cinsaato azsuunt of brateni lagid
hraze Naze
Claiso: Ancxiriedes:
Pister
liqaid smpporitno affict how much harris on goo Rigadt))
Anesor rnder
absolee whe-cediinggrr pobllers
Doscipoorr Detbic ldes vnictiee the show tngeneenn Marcess tait nights epeonds, the eestezenny wan vasued doperaper dess mns. hurner botion acreen sf
Skili s2ze obtune cades--ndd-iipteeeee unaplle Destittas What land inaanic das?
A Cliias, iglt, Atowel index. 0
Chocest Bftton nine, skpccager)
Anower make: C
Szipet Mal:
Suipec Kill
sesengghobeeeerrdee attuu, tmright?
Deserptor Which seatoon
sheey?
Pateic
Raue::
Cleico: blaige, obtte oastc righ.)
Cleics: (w1227)
Strget NZZl
Surcec KEll
Skill malisoo-caaaaceeaooo sdich anstikaaadoo
Skill Swtiiieeaatnncepoooooeeeeee decoposkl dooe
Diserpior: Nows
plate
Bann
Claico:
Cletos: (4) 3:144,
Auaner
Asonci
Skill sadiion Sinteeesssssintntiiaeeeeeee madiiiaaaa ensiiirradoo
Subrec:
Doocgpton:: Whech showo 2rd?
aifiio mando lines Destiptonn Which wddiinn sentene does this modld Khow!
Reaze
S-des.]
Lhvene:
Anever
Strgett MEZl
Skill uijutebbudget
Sabyaer
Desciptoar This sines Angic' Fehrary bodgst could Angic belans badget?
Destiption: Farner Kerran getling tilk lts coes, li nai before mambee His vutch SaaneWaa lazc io
Pateo
Chisco booo opeadion only 990 mill, spood decerat cuiken haiing soct hating das for 530, cecncer moni cusem ccokie in cvm anomher $18, and spendl 530 mone ne the
Cloizo. (4.30 PM. A.M.] Asovei index.
baling
Sunget Mat
Skill aeglc www.-rrittiiieeeeooo. foe-inila vrianlo
Deserption: PCH and lKL. stone bekow Whith statement zuc?
skilt snglo aw-tacctoo--ttaaaomo
Suljec:
Descrptor:: What Inseoon ef cirlee angle out??
Peaner
18)
Dhviae (FTal is vlvitamo IKI, roti eimiarm IKT., There anough information to sicilas.
Cheiss, Atowei inalx. 0
Arawer andox 2
Table 21: Question examples for each skill (part 1).
35

---

Published as a conference paper at ICLR 2024
Subrec: Xaah
ganteeeeenie cheough Fllw
Stall possic and and-aadett
Descrptoa:: How crach moces thei??
Hemaz:
Abower ankoz
Chekoon (18C M6e 196.
Still cest-oneee frenecs sp-ae-1D Dosorption New tany (ots frane?
Skill couss ap-o0
Destition How eany dot- flonet
Pitten Chaiee H4
Hoaa:
Cheioe:
Skill count peasies
Descripten: manncy
(eknr Now sqares the firre??
Chatcoo (,233) Asower ankos
Ansuar Indea:
St.ll. couce plenaes sp-ko
Sahtee Moth
coens www.ee---ee-------
1-1.4
441-4
Reae WWW Anewer
Auaner aako.
Subrec: Xlal
Stul po 1-5
Desceptor: How may sooee ther?
Pieaee
Chicce 12 3.4.3.3
Arawer
Andouriedee::
Mall
couns www.eeattddddde dup.---pppee-- 1p--1
comer-dopee www.deeeeeeeeeepppooo.. tp-
Desorpiint mossglls
Cleicos (2.4..33.1.
Claico: 2. 2...1.2..2.....
Anenzr
Asovei index.
Desurptian: eaary leass lillere there?
Skill depessinneesspiio.
Destipton How slepes no fano?
000
Chiccs 15, 7,7. 3.6.6.3.66
Anowerr
Claizo-[33 (..2.....
Anowir Indes:1)
stalt
Suiject Mal
Desempporr: Which ctape isde??
St.ll. count do
Nome
trngle
Platez:
Aneazr
Choikoo
Anowar
Skil cere-cdranddoome
resorppion: Which comens Neas
Skili cosni-ip-00
Destipton How saap dob daze?
Chiiss. Anowirtrdrss
Aooee anles
Table 22: Question examples for each skill (part 2).
36

---

Published as a conference paper at ICLR 2024
Minpprr Meah
Dosegtton:
[UW[UW) ur puslll
laa und
auilo incrice angle??
skili isenttyyoorpleeeti apprenooarryeeeeeeee wttpraattteeeeeeeeeee cregnacre angarr--ngioo
Deseppour ongle vential anglo(3)3
Peeamm
Cherze (ongee (39) and insle(uno)), axgle(15i)) and segle angle(tV)) and
Choizo: (ongle()) anale sxale(4), [mpie
sxpkj[RV) ITVV sxgle
Asane: indes:
Strget Naal
Skill compeettntt sapknenttay-eniiill and eljacai-apioo
Desciptinn agics adpacal
Subrezt bbab
Desolption: sliges congraett
Skill Mestify-oegnaae figurs
Pettee
segled)) (inggee) (xple7) ( angle (engle(()) eegle(8)
Annwer veee
Heeettyddiect
Descrpuon whech
Pietere: Nosz
Skill wwwpppggggiiiiiieneeeeeee
Desecpeine: This mobil riprosrts the soprastion sec-led Whtch eqgrsso appholen s/ad
erip-medel.
X
34
-34
Cheiber (ix, 7ewh, Teed, 2oed,
Chaicin
Stall www.ety-oppaakeeeeeeee.oo stppraarre-mccceee--ot careeereeeeee Iaaee Deserpton:: aquroien 1/!
Mkill isentty-eguraiier trattree-ausg carteerntttt---etttteeee tame
Desepporn opurakee
Choccst
Chikos: Answer indea 0
Ansser vkbo
Sutee: NDll
ilently-facod dfrreeedieeessred figucs Deserptor Prane Nome
Skill ideatiy eaaialaaaa laata sning sladiracioo
cedo
Doccithon These ilc aqucion de exprovioe e exprossou is oquinkan to baiSs!
Pickez
Cheea: Roe2.]
Atanci inden:
Sunget Math
Subrec: Xlak
Sassttty-lae-----eeee stapo
Skill
Deseiptoon Look orkond pan Which deras foum? Planer Now
Plaan Navs
Anawer
Anorar
Nurger wtth
Sucoe: Malk
fanciionn
Skill slantl--aandieessiaee
fanctio??
sclaioos
Przan: Nome
Pieaee
Chvicac tym,
Cheico:
Anerer
Senl invatty-taative contcol-lineee
Subrec: Malh
Doscaptu: Whech of these eclasien Austics? Near
Deseipton fesciee continanee??
Reazs
Cheica- bro. 2o.
Atower mbke:
Suhve: Matk
Skill hamntt--hlo--ant-teee foutls?
Look colord of Which Patus Nost
Destipton: Plaxec Neve
Dricac
Cholown Assoan Indes:
Antoer meke:
Table 23: Question examples for each skill (part 3).
37

---

Published as a conference paper at ICLR 2024
Suiject Mah
Descrgttan:: What tha?
Subces: Math
Skill madie podiiiiiioo with aaster plots Desorptoon
prodicioo
Petter
segroo, nn.1
Asone: indes:
Cheioe::
Skill candbasaaty digila--nl--illa--eeee
Still mash cubooittonn www.oooooooooooaaaaoeo
Desciption asing dightl
Dosrrpion plinn Which shows 2e2ud?
Neas
Checco
7-25
6:25
4:35
Asower
Cheics:
Asanci inde..
Strget Nash
Subrec: Xlak
maacs chado What clock seou?
clockennd-iins
What thecleck show?
Pizamer
Rran::
Chaice: 470.
Chekoer: (4ight oove thty, rine tvny.
Sakyer
Skill manb-ecppeectii furaionn and and-amplla
Descrptta:: which deck shows thery-dre!
Suil mashh wow.hdgggtiidd Migltii-dddoonndtiiie
Destiptes ienada doseas
Noae
5:30
6:35
6 6:45
Cheico:
Suieect Mal
maads coneomtiiilaatiii--aaaaeeee Doworpies sermiatteeppg
Skil madd-mnatan modddd sabves-irca--ad kozzto
Desrptoon Look oskood pan of stape. Whish stpe dacasone furm?
Pieam Noss
Chotoan
Ancoertedes:
Sarth meshh
masch-mtted rasedd sembersto-medek
Mkilt mon--editte meditte-rrsteepaannee
Diociptios: Which ussed inailees bolknn??
Destipton Sous billoning dasalf de axsa 25, whic mamber could be?
Anener mike:
Anowertrdrs:1
Surpet Mat
Subse: Nllk
and-nage- masin
Decriptoo tathi
Pictes
Chiice: [.1.3.]
Ptane:
Choice: inght, chrees,
Skill medias deditic cnnden
Skill mocdl-andootooeeeoo uning slgele
Doocipton Daase leo indowwing desull undias waich widdh suraher colld
Destipton Which equaiont does Blosd slackra tike reproient
Chokae: tt-1-- fo-ls Ba-lo soll--iit
Anenur mkke
Anowrirder:
Suttett Mab
Still model ondee spaiieeeoaaiiiiiiieoee abyteasties
Destipton: Which lea monc?
Desorptonn opetios digeter oepossett
Hanus: Nax
Checod Bleecn Sends, Zerdi, 2er44.
Chiios: Antverirdees
Table 24: Question examples for each skill (part 4).
38

---

Published as a conference paper at ICLR 2024
Still rellcton-ntaa mntiio and tsaedatin
Discrptor: migmn hean raaoommalth
saill mepeeaelmmmeeieeeeoeeo
Deotipton: is shissohppe inghirpetygnn" pelygun?
Pletere
rnflcted mexees
Cheizer: (ye 25.1
Rrane:
Mah
Seall rppeeree zumttereap-o-11
Desciptinn goop tikegles?
Nase
Skill prmeataarreeesppp--33
Destition Which picsee dasesdddd ddos?
d0scasee 000000 uaseaaosa Anoun Inder:))
Chaican
Sa-ll mpporre cumberoaathgiccam--ppp
saili srpowat-rrnterr-eee pwt.ppeeespspa
Descrphua: sbowo
Destiptons sbons
Cheicin
Chikos:
Saill mpporte rumbereeeettateee thastapppe sp-in-3
skall srreua--srteeeee oith cuth-ccups--pp-i
Deserpter: group fue
Descriparn: groug tox brropos?
Claicoo
Cheioos
Qail rivelosas
Skill nextinal syrreerry
Doccpptut: Which stape dhaecbes??
Desorption Don this pieure sonatinal syreert??
Neas
Chicoo
Chokos
Arower indes
Antverirdes:
Suipet Mall
Decomppure truge to rotale the langl for look the sase?
cralice frution fall
Subrec: Nath
Cheize: no. Asover indes:
tam 44ufu of fullttn))
Aneoer miket
Strpee Nath
Which tacas cetis?
Piaame Nose
Suljpe: Malk
Desenptor Oran what trstion thes
Choicac
Sarper xran
Satppr weal
suil udee weweddeeeeieiiddleeeoooo depes
Desorpton shaps
clque monangggtt prinn?
Patuan Nesz
Paaue: Now
Nill
Nalll
Still sctec soc.-bo-diinninnn--eee
skill staesslleeyylg
Descition: shapeis licngge??
Destiption Which degeci Ikensolindas? Nome
Neas
Chiicao
Cheizas
Table 25: Question examples for each skill (part 5).
39

---

Published as a conference paper at ICLR 2024
Spaksi Scisace
Sakyer Sclince
Desc qqiee Sureerad mile an forrd in many pans North Ameriic. They line hemras cornoud mei
Descipiie: lnnendd te Seath Amertos The acupted he
modecest contworn mas whiic they and stqpod trrigggin?
cansulaged srscsa precn learo, Wlach ssinal is abo alagted to be conoodlee amou preee
fandigiig
Pickor
Phicce
Choizo: Asswat
Antaar ruker
andi-aalleeeeee wegelieeeeoeeeeee auplaat parts
Dosepptor: Popp
plano fead. mualty the plant makes
Desorptoon Complex the sieenser Nioggen is modil befow regineess einnggs Nimrgee get makes nearly SMS broothe.
of bood foe the plan. list. A plant's loves make focd, plant's soets cas growe iato nnw plant pant lstace pazz asadly ear?
Pettor:
compooast
Arswer Inbo. C
Choize: ltaves, reeds,
Assverirdes:
Strper so-ent---lps---ee claccly-rraaee
Subyeer
actidd
Deserption
Pistes
Chiicos a liquil )
Cleiko: avlll
mide:
Anoourtedees:
"arype soress
wod.-m/dd-nddmeeellyyyy..o coternlllttooooostseeem -ope
chootfy-met meboonddriiennn cokmodape sotery Pistasi Nexc
Pituus Noaz
Ancuuriedes:
Still claily undo sonoooeooooeennnnnyyy.o.o oe-ndtamephi
Dostittin Dicike typ mock. Whet aediad codaa tokon the curll' saifao, iees foom
goune type
rule that make arend line prrtection thas ssy other sale is not when catemilla minio ast der asto feed and prited calemila inacad other ant nea..
tell live caspulas furt with Myrrica eas. When casoppli tos with urss riiencs Qaeen
prowech
seatHeed
Then
qaoian, Alom bos batortho
copy mg fiod hever
Clicc: balinettss ]
Arowe ando.
Fizmee Anourt Indra:
persdrn))
comper--po--d dhe dra
Iollonine boels odar? select liady
Skill compac-arpliades and wondeegghs of Descriptrs with prance arplinad..
Choizo:
Asane: index:
Auawer ankoz
Suijec Scieno
Subrec: Scatice
cuppurt-rrllidds snd wew.ghtoolllllooooooooooooooo flopancdend seaes Descrptin::
Skill: corpppa-nccieee... cet-e-eptterraaaeee cberrster capparo- Homan bakckreis hase sitile aleiiuties servivo unilervates. The Ametican loboer and Baronn biideeds stiller adqutaton to survice underwatee
lhe
Pister Nozs
Puzae:
Chotze:: (itenuga indecenns and
secervates The Arreicae abwier unkevster.
Amsrican loboor
Atoner mkee
Aoower indes:
Sulper ScEae
Skill conpasoangtinalo Coeaagncic fome beteeer the
Sail conertration-a of wuintiess
Which of follvinn estement is tn The mmags belww even non pf magrets The magres in dittones pans do aftect other. megnat shren made matural
Descrppine: aboet
Dosipthan The dagens Indlo modk too whitions Ead grons aquooato one palick
lights consrnttaiin pooppenidoo
Piraner
Razus:
Chaice Bsteeoo Sohrtin nether conertmation some.)
Choizo: angustade mannciic foec
assostade
sendle
Ancwirirdes:
Sprpee Scrace
compereggueee perpprrgggpfeeeoottaaaae
Pister Nose
Dosorption Which
Skill companmmppptttooeeee
Dostigtou Which possaly de dese four oijoct lave cdde deet lest unwwe.
Rinne: Cleios: (sicks, oox,
Arants andou.
Table 26: Question examples for each skill (part 6).
40

---

Published as a conference paper at ICLR 2024
Scrjet Scraae
eeeeeddleeette toosil sninass
Which sxaement ben docthee chmaae Sara te Mar: Sumren Narbhmn
Deuripice: Placend reateral aeeal
cnirals with similartat The frllwing rais Oiberne
Ronisghee cour Jas. Jal, and Auges Whrees Nacher Honkighere occer Deverbee hoully
sund idernty placens munemal
otspree they tore
Fehraay
lsred alove See orkess hake vey thik Thair teljs then ware coll water Fonsir se orees plve Nitt Sia offeee lng the wasrr Red ulleeedee derss have langs! They on hrrute hully nod untamussee anclervake.
poads They lay
Pickuz
abdla unake
Chekoss (Wrnees hove moch bover ireperatres fhon serrses, Whnter hore les preceptatie tptatie dlun
Henus: Noms
Cheicas
Assweer mnkoz
st.ll unc losls crarondalio folksoytlovaal anl anplidisas grio itensty Saer tase lisho. They make with shall. Ubwere zereal
group
Skill Lacipaaddi spuace da www.ooolllloonniiiioooooo wogpepttiiiiieeeeeo seleeeeellliaa pugiiig
Dowtiplios lu prog casdalaan dogs same andoidnab love nough olhers lane
the deceppinns sekei one untral tue his al of trass bend abouz Lhrohe dueka basc moo qpe wits thiie They tane
tix, They
ducherand What alidd farm This Ponnen showe
a the protaastity that dichidesd dig prodhcr the mose willl he hemsrryaea dentruer for
They thet uring bodk mole cuskehiee lodes lrces. The cooopetioo las froe likes anss Charemo
shels.
Homas:
Cheioo- 13A, 24, 16, (4.44)
Arower indes
Sumper Screcce
ue-maaeet--qpaae inctillatae saace tyippen--tpppe slvn and lane for
grog for leegth ouk has dliees The alicke for lien fa (F) is dosiinnt In lis grou. the thet Phinen squar stoea cro berwer Syrun hammers espected ottopting
Dessiption: This rrganimn anarlel Jewies scianiiic Aleentia pullitta Sdect organiess ame specin huanteddhwsee
octppting
Chaices (H1, 20.
Anenzr
Ancwiriedre:
Sahject Science
Skill caller-anl--iin arcendlllc world
Dosistins
lie pauage dovstis the weather the clisaic-Miat, Weather
What Medd ofcerpaa glag here?
the swathr
cleat frees meverton acceusim that in homs as wide varety of specise The wis mostl zbor bur ourr cload foneet Flzndder
Chaieen twnaher, diroas
Chiias: (USB tyge-A yart HDMI pla. VOA purt type
saal loen
Desoppoom This an ekd) lego of which braees apper progree
Still for
Descrptor: Memnly type
Actionuutz
P
Patez:
Clatco: (Tia Asckst Rosea, Maises Hassa021 BT, Booiana Old ]
Aranci
Chetzar [Amoter Rrader, Google Pry, Microont Fnecrtione, GaltndMee
Tnckesiogy
Scloer Tezsnalryy
Still lago
Skill oden
poo This iefre o (fonei ligo
which compptte hoae??
Destiptin
key!
Patez
Chocco: (luatics Cluco, Vaitain]]
Cleteo: (Caxs, Undo, Deiic Park;
Iole:
Skill pons
Dosrytion wha compercooppeet de you heve?
Skill periskzaas
Destipton What lind of comppttt paripkerl de yeu lex??
Peaaee
Pistus
(Plegy Spiede, Anoun
Suppl Cropste Fut Coneecto,
Seill plom
lntiuttyy
Subrec: technology
Discppion: rype gone you
Destitton What moaiing faxeioo is uuuellly aooiiand win dis web inerfice
Pictur
Raaax: Cleico: (Sangg fo dddead lies Cospuer gares, RdoadiRciokk Seule-anil
Arawer makes
Table 27: Question examples for each skill (part 7).
41


Reviews:
[{"review_id": "kXdw8aKvTP", "reviewer": "Reviewer_psfv", "summary": "- The paper proposes a new dataset, STEM, to test the STEM skills of neural models. Focusing on fundamental skills in the K-12 curriculum, STEM features a high coverage of skills in all four subjects, a large number of questions, and its multi-modal property. \n\n- The paper benchmarks a wide range of neural models, including baselines and SOTA models for both language models and vision-language models. Results show that neural models are still behind human performance.\n\n- The paper breaks down its result analysis into different granularities, such as skills, subjects and grades, and identifies the shortcomings of models and challenges for future research.", "questions": "1. The claims that the paper concluded seem inconsistent with the results it presented. In the abstract, the paper claims that \"these models are still well below (averaging 54.7%) the performance of elementary students, not to mention near expert-level performance.\" My questions and concerns are:\n  - How do we define \"the performance of elementary students\"? I see that the paper compares the model performance with a score of 90, which \"according to IXL, is considered excellent for a mastered skill\", but it does not make sense to take a score of excellence as \"the performance of elementary students\", because it is an \"expert-level\" human performance.\n  - Where is the number 54.7% from?\n  - Besides, it doesn't seem fair to use Ph.D. students' test scores for the human evaluation of these elementary school questions.\n2. What is the reason of not experimenting on GPT-4? It would be beneficial to test it since its performance has largely increased compared to GPT-3.5", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel and challenging benchmark for multi-modal STEM skills. Compared to existing benchmarks, STEM covers all four subjects, contains more skills and questions, and strictly excludes single-modal questions. Therefore, it significantly contributes to the advancement of neural models' multi-modal skills.\n\n2. The paper benchmarks a wide set of neural models, including the SOTA models such as ChatGPT and CLIP. The paper also analyzes the results at different granularities and in detail.\n\n3. The paper presents illustrative figures and charts. For example, Figure 1(a) serves as a good demonstration of STEM questions that give the readers a good sense on what the dataset looks like; Figure 4 clearly shows the different mechanisms of CLIP model and ChatGPT model.", "weaknesses": "1. The paper does not present strong enough evidence for its claims. The human evaluation part is not clear enough. Therefore, I am not convinced that current neural models fall far behind the human performance on the STEM dataset.\n2. When benchmarking on SOTA models, the paper does not use the newest language model, i.e., GPT-4. If GPT-4 is tested, we might see quite significantly different results.\n3. The paper touched the bad case analysis, but not deep enough. As part of its biggest contribution, it will be better to make a more thorough analysis on the examples and patterns where the models fail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jblF7egyOJ", "reviewer": "Reviewer_FcAF", "summary": "This paper propose a new dataset, named STEM, consisting of science, technology, engineering and math subjects. The proposed STEM includes 448 skills and 107M+ questions, which is a large-scale multimodal dataset. Upon the new dataset, this paper analyzes some existing foundation models, e.g., CLIP and ChatGP. Experiments show that these models are still below the human although the model is finetuned. Code and dataset are available.", "questions": "Please refer to the Weakness Section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The new dataset STEM is valuable, which consists a large-scale skills and questions.\n2. The analyses of some foundation models on STEM are detailed and interesting.", "weaknesses": "1. Despite the analyses of foundation models, e.g., CLIP and ChatGPT, there are still some new and better models missing, such as EVA-CLIP, Kosmos-2, BLIP-2 and etc. What are the results of these models?\n2. Does the quality of the caption model influence the performance of zero-shot performance on vision-language models. How about changing the caption model to a more accurate one, e.g., BLIP-2, GPT4 (I understand that the API may be not available now)?\n3. What is the performance on other datasets after finetuning on STEM, e.g., VQA (2015)? Which is better when comparing with the model before finetuning? It would be important to verify that STEM is important and finetuning on STEM can maintain or improve the generalization of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0WpY7XISDV", "reviewer": "Reviewer_aT3Q", "summary": "The paper proposes a new dataset to test the STEM skills of neural models. This dataset is one of the largest and most comprehensive datasets for the STEM challenge, especially on fundamental skills in the K-12 curriculum. They test state-of-the-art foundation models on this challenge, but found limited performance. Even when training these models on a training split, the performance is still not satisfactory, which indicates the difficulty of this challenge and needs novel algorithms.", "questions": "- Could the authors show many examples for better understanding the dataset?\n\n- Could the authors provide some directions for improving the performance according to your expertise?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "${\\bf Strengths:}$\n\n[$\\textbf{New Dataset for STEM}$] This work introduces one of the largest and most comprehensive datasets for the STEM challenge.\n\n[$\\textbf{Presentation Quality}$] The presentation is clear and easy to follow.", "weaknesses": "${\\bf Weaknesses:}$\n\n[$\\textbf{Missing Comparison}$] This work only shows the results of foundation models on the proposed dataset. It would be nice to also see the performance of the same foundation models on other datasets, which helps understand the difficulty of different datasets.\n\n[$\\textbf{Missing Limitation and Potential Impact}$] It would be nice to show the limitations and potential impacts of the proposed dataset.\n\n[$\\textbf{Confusions in Figure 9}$] In Figure 9, for RN50, RN101 and ViT-B/16, the performance decreases when the model size becomes larger. Could the authors explain this a bit more?\n\n[$\\textbf{Unconvincing Human Study}$] In the human study, the inclusion of only seven university students may not be fully representative, potentially limiting the generalizability of the findings to the broader population and raising considerations about the robustness of the observed human performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)