Automatically Answering and Generating Machine Learning Final Exams
Authors: Sarah Zhang
Reece S Shuttleworth
Zad Chin
Pedro Lantigua
Saisamrit Surbehera
Gregory Hunter
Derek Austin
Yann Hicke
Leonard Tang
Sathwik Karnik
Darnell Granberry
Iddo Drori
Venue: Submitted to ICLR 2023 (2023)

Abstract: Can a machine learn machine learning? We propose to answer this question using the same criteria we use to answer a similar question: can a human learn machine learning? We automatically answer final exams in MIT's recent large machine learning course and generate new questions at a human level. Recently, program synthesis and few-shot learning solved university-level problem set questions in mathematics and STEM courses at a human level. In this work, we solve questions from final exams that differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span a broader set of topics. We provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions. To make our dataset a reproducible benchmark, we use automatic checkers for multiple choice questions, questions with numeric answers, and questions with expression answers, and evaluate a large free language model, Meta’s OPT, and compare the results with Open AI’s GPT-3 and Codex. A student survey comparing the quality, appropriateness, and difficulty of machine-generated questions with human-written questions shows that across multiple aspects, machine-generated questions are indistinguishable from human-generated questions and are suitable for final exams. We perform ablation studies comparing zero-shot learning with few-shot learning, chain-of-thought prompting, GPT-3 and OPT pre-trained on text and Codex fine-tuned on code on a range of machine learning topics and find that few-shot learning methods perform best. We make our data and code publicly available for the machine learning community.

Under review as a conference paper at ICLR 2023
AUTOMATICALLY ANSWERING AND GENERATING MACHINE LEARNING FINAL EXAMS
Anonymous authors Paper under double-blind review
ABSTRACT
Can machine learn machine learning? We propose to answer this question using the same criteria we use to answer similar question: can human learn machine learning? We automatically answer final exams in MIT's, Harvard's and Cornell's large machine learning courses and generate new questions at human level. Recently, program synthesis and few-shot learning solved university- problem set questions in mathematics and STEM courses at a human level. In this work, we solve questions from final exams that differ from problem sets in several ways: the questions are longer, have multiple parts, are more complicated, and span broader set of topies. We provide new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions. To make our dataset reproducible benchmark, we use automatic checkers for multiple choice questions, questions with numeric answers, and questions with expression answers, and evaluate a large free language model, Meta's 's OPT, and compare the results with Open Al's GPT-3 and Codex. student survey comparing the quality, appropriateness, and difficulty of machine- machine-generated questions with human-written questions shows that across multiple aspects, machine-generated questions are indistinguishable from human- generated questions and are suitable for final exams. We perform ablation studies comparing zero-shot learning with few-shot learning, chain-of-thought prompting, GPT-3 and OPT pre-trained on text and Codex fine-tuned on code on range of machine learning topics and find that few-shot learning methods perform best. We make our data and code publicly available for the machine learning community.
INTRODUCTION
Can machine learn machine learning? This work presents new dataset of machine learning final exams with 646 question parts and a benchmark of baselines using transformers and their respective grade performance, demonstrating that the best baseline performs at human level. In university-level STEM courses, students complete assignments (including problem sets and labs) and exams throughout the course. Recent work has opened the door for machine to solve course problem sets (Drori et al. 2022) using language models and few-shot learning. However, final exams remain challenging, and this work is the first present structured dataset of machine learning finals and a benchmark of baseline methods for answering them. Final exams differ from problem sets because they serve as benchmark of cumulative understanding of material learned over a semester and evaluate the students' depth and breadth of expertise. Further, questions on final exams are longer, have multiple parts, span a broader set of topics, and are more complicated and nuanced.
All the above holds for MIT's and Cornell's Introduction to Machine Learning classes and Harvard's Machine Learning course. These are undergraduate courses with hundreds of students each semester, making them the largest undergraduate courses offered. Introduction to Machine Learning is core class in the computer science program. The prerequisites for the course are Python Programming and Multivariate Calculus, with Introduction to Algorithms and Linear Algebra recommended. The class typically consists of weekly exercises, labs, quizzes, homework, a midterm, and a final exam. There were no final exams in at MIT for Fall 2020 and Spring 2020 due to COVID-19.
Introduction to Machine Learning final exams differ from problem sets in several ways, and the experience of solving each varies. First, finals are long, containing around nine questions with

---

Under review as a conference paper at ICLR 2023
around seven parts each. Final exam questions are also multifaceted and multi-stepped: different parts of a single question require applying different concepts and problem-solving skills, and parts may build upon each other. While weekly problem sets focus single topic. finals span topics from the entire semester. Further, final questions are often kory-bassed problems that may require mathematical modeling. Due the time constraint of these exams, finals are also designed to test core understanding and application of course material over rote calculations. Thus, asking a machine answer questions from finals allows for testing whether the model is able learn breadth and depth of topics beyond problem sets.
In this work, we present new dataset curated from final exams of MIT's and Cornell's Introduction to Machine Learning course and Harvard's 's Machine Learning class, totaling 646 question parts spanning the broad range of topics in machine learning: regression, classifiers, logistic regression, features, loss functions, neural networks, convolutional neural networks (CNNs), Markov decision processes (MDPs), recurrent neural networks (RNNs), reinforcement learning, clustering, decision trees, model selection, ensemble methods, Bayesian networks, hidden Markov models (HMMs), and optimization. Our dataset covers the finals given at MIT for semesters of Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019, Spring 2021, Fall 2021, and Spring 2022, Harvard Spring 2015 and Spring 2021, and Cornell Spring 2017 and Fall 2018. Due to the COVID-19 pandemic, no finals in the MIT course during 2020.
Table 1: MIT Spring 2021, Question 10.a: question part, solution, Codex Zero-Shot answer, program output, GPT-3 Zero-Shot answer, and GPT-3 Zero-Shot with CoT prompt answer.
Topic
Classifiers
Question Part
Given set of data Drrain ((x(1) y())), weighted nearest neighbor regressor has the form
h(x;; 0) ((0)) [ x :))
Σ (/(0), f(x x:10;
A typical choice for f
fxx 00 -010--2-211
where 0 is a scalar and || z'12 EE= (xy x/)2. Assume our training data Dirain = (), (3. ()) What is h(10; 0)? That is, letting 0 = 0, what is prediction for 10?
Solution
3
Codex Zero-Shot Answer
i import numpy np
def f(x, x_i, theta)
return np.exp (-theta * np. ww.linalg..orm. - x_))++2)
def h (x, theta):
np.array 1), (2, 2), (3, 6)
numerator =
denominator = 0
for x_i, y_1 in D: numerator f x_i, theta) y_i
denominator +- fx, x_i, theta)
return numerator / denominator
print (h (10, 0))
Program Output
3.0
2

---

Under review as a conference paper at ICLR 2023
GPT-3 Cero-Shot Answer
h(10;0) = 110.1100 4000000y""//0000000000000
1-1112+46
((00100+1110200000000300
141+1
can 3
GPT-3 ZS with CoT Answer
f(10, (1, 10---]10-0000 (1.1)1 -0 E 0-81
f(10, (2,2), = - (2.))l = = e -0-1
(3, -1|100 (3,6)] = =e -01711 = -0-49
(, Ma,z! (00,000,0000 ,00'
h(10,0) =
111112--16
210(9 Priin Pinin Ma
1+1+1
We verify that our results are not due to overfitting by including MIT finals from 2021 and 2022 that unavailable online. Also, the MIT Spring 2022 final exam given after GPT-3 and Codex were last updated, which means that the models were not trained on this data. The final exam questions have many parts, each posing new problem, and each question in the dataset corresponds one part. The questions in the finals are varied in topics and solution types. Solutions are primarily open-ended questions with some true/false and multiple-choice questions on theory, math, and code implementations.
We make the dataset publicly available and welcome others use to aid in developing and assessing new language models and methods. Due to the diversity of Intro to ML final questions, our dataset uniquely assesses advanced problem-solving and reasoning skills in machine learning, math, and natural language processing. This dataset opens the door to achieving breakthroughs in machine learning performance in machine learning final exams. In addition to the dataset, we present a benchmark using several baseline methods. We apply zero-shot and few-shot learning GPT-3 and Codex, adding tf-hhuh prompting for GPT-3. We find that few-shot learning methods perform best. As shown in Table the best performing methods pass the final exams, and their grade is comparable with human grades of MIT students on the same machine learning finals evaluated by the same human graders. We generate new final exam questions that are indistinguishable from human-written questions.
1.1 RELATED WORK
There is often thought that humans generalists, whereas machines specialists. However, large language models based on transformers such as GPT-3 (Brown et al. 2020). Gopher (Rac et al.) 2021), and PaLM (Chowdhery al. 2022). , also called foundation models, Specifically, in our setting, while humans care about the number of topics in an exam and therefore find finals more difficult than problem sets, foundation models effortlessly scale to many topics without re-training. Language models may be pre-trained on text and fine-tuned on specific datasets such code, for example OpenAI's Codex (Chen al. (02T). which allows generating programs from text. There are several ways to improve the mathematical reasoning ability of language models: (1) using chain-of-the of-thoughtt (CoT) prompting (Kojima et al. 2022 Wei et al., 2022). (2) using the top-k
generalist learners.
ranking solutions (Li et al. 2022) and merging them by voting (Wang et al. 20022) or least-to-most prompting (Zhou et al. al.] (022), and (3) using program synthesis and few-shot learning to generate
code that answers questions (Drori et al. 2022).
Grading:
Human Human
Human Human Non Mengemage
Homan Machine Non- Inuge
Machine Machine Non- Nou-Image Non- Opn-Open
Answers:
Questions:
All
MIT Spring 2021
75 84
80.77
62.09
MIT Fall 2021
74.38
60.88
58.94
$1.33
MIT Spring 2022
69.07
70.82
73.53
Mean
73.10
70.82
63.29
62.95
Table Human and machine grading of human and machine solved final exams. Mean human and machine grades on Introduction to Machine Learning final exams by semester. MIT Spring 2021, MIT Fall 2021, and MIT Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that our results are not due to overfitting. Non-image grades consider question parts that do not contain images that are required for solving the question.
3

---

Under review as a conference paper at ICLR 2023
Semnester
Questions
Parts
Topic
Questions
Parts
MIT Fall 2017
10
61
Regression
62
MIT Spring 2018
9
42
Classiliers
24
85
MIT Fall 2018
10
60
Logistic Regression
10
MIT Spring 2019
9
58
Features
3.5
21
MIT Fall 2019
8
61
Neural Networks
19.5
87
MTT Spring 2021
13
71
Loss Functions
4
16
MIT Fall 2021
8
86
CNNs
9
65
MIT Spring
5y
77
Harvard Spring 2015
8
12
RNNS
7
33
Harvard Spring 2021
6
32
Reinforvement Learning
11
60
Camell Spring 2017
30
48
Clustering
5
17
Comell Fall 2018
29
56
Decision Trees
14
48
Model Sclection
16
Mean
12.42
53.83
Ensemble Methods
9
20
Total
149
646
Buyessan Networks
HMMs
4
Optimization
10
16
Bonus/Name
3
8.28
35.89
Total
149
646
Table 3: The number of questions and parts in the final for each semester and topic of Introduction Machine Learning. MIT Spring 2020 and Fall 2020 did not have final exams due COVID-19. Topics can have half-questions attributed to them if question has some parts under one topic and the other parts under another topic.
Much of the prior work focuses on high school or middle school level material (Qu al. 202T). ). The first work to tackle university-level machine learning course problem set questions (Tran et al. 2021 used a transformer and GNN architecture and heavily relied on data augmentation. This resulted in overfitting and did not scale up to other types of questions or courses. Probability and statistics course problem-set questions have been answered (Tang et al., 2022) by probabilistic program synthesis with human performance. Problem-set questions from the core university math courses (Drori et al.) 2022) have been automatically solved using few-shot learning and program synthesis at a human level. Other work considers university-level course questions across variety of domains (Hendrycks et al. 2021) and identifying theorems (Srivastava et al., 2022). Prior work on question generation includes question- answer pair generation based on text passage (Qu et al. 2021 and question text generation based on other questions (Drori et al. 2022).
DATASET
We present new dataset of 646 question parts from a dozen recent final exams of MIT's and Cornell's Introduction to Machine Learning courses and Harvard's Machine Learning class. The dataset spans questions on the 17 machine learning topics covered in the courses: (1) regression, (2) classifiers, (3) logistic regression, (4) features, (5) loss functions, (6) neural networks, (7) convolutional neural networks (CNNs), (8) Markov decision processes (MDPs), (9) recurrent neural networks (RNNs), (10) reinforcement learning, (11) clustering, (12) decision trees, (13) model selection, (14) ensemble methods, (15) Bayesian networks, (16) hidden Markov models (HMMs), and (17) optimization. We make our data and code publicly available
The breakdown of questions, parts, points, and non-image points by each semester and topic are shown in Table 2 Each question in a final exam consists of multiple parts. Questions are written by providing set-up and context information first, followed by the question parts (which may come with additional information). Set-up and context information may contain (1) story elements (cx., character names, and motivations). (2) relevant definitions and equations, and (3) data points. We format questions in the dataset by concatenating the question context, any context or solutions from prior parts of the question required for answering the part, and the part's 's context and question. We split the questions into their corresponding parts. Questions consist of English text, mathematical notation, and images. Mathematical notation is represented in the dataset by LaTeX and images by screenshots from pdfs files. The types of question answers are diverse. A few are multiple-choice or
Data and code in the Supplementary Material.
4

---

Under review as a conference paper at ICLR 2023
true/false questions. Most open-ended, for which the evaluation requires modeling the problem, mathematical manipulation, or code writing. Many questions require providing an explanation.
We used twelve final exams from different semesters for data curation. We had access to the Latex version for the three most recent semesters of MIT Spring 2021, Fall 2021, and Spring 2022, and therefore did not require transcription. For the nine remaining exams, MIT Fall 2017, Spring 2018, Fall 2018, Spring 2019, Fall 2019, Harvard Spring 2015 and Spring 2021, and Cornell Spring 2018 and Fall 2018, we had access to the pdf versions. In these cases, we used mathpix m ththii.... com for an initial transcription, and curators then evaluated and manually corrected the input questions and verified the correctness of each input question.
We extract questions and solutions for all parts of all types of questions, including those that rely on images. We curated nine exams from publicly available pdf files. MIT Spring 2020 and Fall 2020 do not have final exams due to COVID-19. The three MIT exams between 2021 and 2022 were unavailable online; therefore, the model does not overfit their solutions. The aggregate average grades were available to the students and did not contain any personally identifiable information. Three duplicate questions were originally on the final exam of MIT Fall 2017 (questions 1, 3, 6) and appeared again in the final exam of MIT Spring 2022.
3 BENCHMARK
3.1 BASELINES
We provide a benchmark by comparing six baselines for answering the final exam questions: (1) GPT-3 with zero-shot learning, (2) GPT-3 with few-shot learning, (3) GPT-3 with zero-shot learning and chain-of-thouggh (CoTT) prompting, (4) GPT-3 with few-shot learning and Mhain-of-thougle (CoT)
prompting, (5) Codex with zero-shot learning, and (6) Codex with few-shot learning.
Table|[show the prompt used for each approach. GPT-3 zero-shot uses the question as-is, whereas GPT-3 zero-shot with CoT uses the suffix "Let's think step by step." after the question to encourage multi-step output. Codex zero-shot uses the prefix "Write a program that answers" before the question within Python comments denoted by triple quotes .." to encourage Codex to write code. GPT-3 few-shot finds the closest questions in the embedding space, measured by cosine similarity, and uses them and their corresponding answers before the new question as examples in the prompt. Codex few-shot finds the closest questions in the embedding space also as measured by cosine similarity and uses these questions and their corresponding code as examples.
For students, good study technique is use previous final exams review and practice for their upcoming final. We model this method by few-shot learning using the question guestion-answer answer pairs (for GPT-3) or question-code (for Codex) with the closest question embeddings from previous finals. We implement this by considering all the exam questions, marking each question by its semester and year, and using only previous semesters' questions for few-shot learning. The MIT Fall 2017 and Spring 2022 exams contain three duplicate questions, and we handle these same questions the same way humans do by allowing few-shot learning in MIT Spring 2022 based on successful Fall 2017 zero-shot answers. It is reasonable that if student studies all previous exams, there may be 8.5% of repeated question points. Since MIT Fall 2017, Harvard Spring 2015, and Cornell Spring 2017 are the first final exams in the corresponding universities, we do not perform few-shot learning on these.
3.1.1 COMPARISON WITH OPEN LANGUAGE MODELS
We also evaluated our dataset on an open-source language model, Meta's OPT-175B. 0PT-175B is model consisting of 175 billion parameters. Our dataset consists of final exam questions from machine learning courses and fit to be used by OPT-175B. Tables 5 and compare the results of OpenAl GPT-3, OpenAl Codex, and Meta OPT. We evaluated OPT on only 163 question parts, since OPT was limited to handling questions under 256 characters in length. We implement the inference for the OPT-175B model using Alpa. Alpa is a particular framework designed for training and inference of large models. For the hardware, we use an 8x A100 PCIE cluster. The model requires about 560 GB of VRAM in our run case and each example takes nine minutes for inference.
5

---

Under review as a conference paper at ICLR 2023
Method
Prompi
GPT-3 Zero-Shex
<question>>
GPT-3 Few-Shol
Q: <similar question>
A: csimilar question's
Q equestion>
GPT-3 Zero-Shot with CoT
Q: <question>
A: Let's think step by step.
GPT-3 with
Q: esimilar question>
A: <similar question's answer>
Q: <question>
A: Let's think step by slep.
Codex 7ero-Shor
Write program that answers the following question: <questions
Codex Few-Shot
peogram that answers following question: csimilar question>
<similar question's conrect code
Wrile a program that answers the following question: <question>>
Table 4: Input prompt for each of six baseline methods (1) GPT-3 Zero-Shot, (2) GPT-3 Few-Shot, (3) GPT-3 Zero-Shot with CoT, (4) GPT-3 Few-Shot with CoT, (5) Codex Zero-Shot, and (6) Codex Few-Shot. Similar questions (as measured by cosine similarity) drawn from previous finals only.
3.2
GRADING
3.2.1
HUMAN GRADING
The questions are of different types: multiple-choice, numerical, expressions, and open-text. We grade answers and aim to keep all factors equal in grading human and machine answers. Human and machine answers full, partial, or no credit for each answer. We approximate partial credit by assigning half-credit. The course staff graded student final exams, which included graduate TAs and instructors. Two of the same graduate TAs and the instructor that graded the student answers also graded the machine answers. Grading instructions the same for grading student answers grading machine answers.
graded based on the number of points allocated each question part, giving
3.2.2 AUTOMATIC GRADING
We label each question's 's answer type into one or two categories out of four options- multiple choice (MC), numerical, expression, or open. We consider answers multiple choice if the test-taker is presented with an enumerated list of choices, numerical if the answer is a number, expression if the answer includes variables or other notation, and open if the answer calls for free-response text. We categorize questions that have additional questions nested within them by the multiple relevant categories. Most often, this is the case when question with one of MC, numerical, or expression, followed by a follow-up question asking the student to explain their previous answer. The breakdown of the questions is: 98 are multiple-choice, 84 numerical, "Non-Oper Points' column of Tables 7 and 8 show the answer type breakdown by number of points. Tshows the number of question parts that do not rely on images, the number of points that do rely images, and the number of non-open question points in Introduction to Machine Learning finals for each semester. MIT Spring 2020 and Fall 2020 did not have final exams due to COVID-19 19.. Table shows the breakdown by topic. Our automatic grading uses string matching and regular expressions. In the case of multiple-choice results, we check that the output of the code is equal to the solution. In the case of numerical answers, we look for matching integer or real number.
expressions, and 204 are open. The
3.3 PERFORMANCE
Table shows the machine grades by semester and Table 6 shows the machine grades by topic, excluding question parts that rely images. We compare the average grade of GPT-3 with zero-shot

---

Under review as a conference paper at ICLR 2023
Semester
GPT
GS
GPT33 ZS Col
GS CaT
Codex ZS
Codex FS
OPT ZS
MIT 2017
NA
21.43
NA
NA
MIT Spring 2018
44.35
60.48
38.71
70.97
32.26
67.74
33.33
MIT 2018
51.99
52.18
61.63
64.17
49.78
$4,00
47.54
MIT Spring 2019
43.45
54.23
41.07
58.81
15.54
41.55
34.64
MIT Fall 2019
$4,92
77.05
29.92
58.20
26.23
61.48
NA
MIT Spring 2021
44.33
55.81
53.45
60.21
33,62
62.09
33.77
MIT Fall 2021
58.94
69.44
30.35
54.90
18.11
42.00
24.44
MIT Spring
51.01
65.46
Harvard Spring 2015
85.71
NA
85.71
NA
50.00
NA
21.43
Harvard Spring 2021
47.73
86.36
47.73
81.82
43.18
86.36
45.45
Corell Spring 2017
78.91
NA
80.86
NA
51.30
NA
21.88
Comell Fall 2018
36.45
53.27
44.39
61.21
42.52
56.07
28.97
Table We benchmark different baselines for each semester, excluding question parts that rely on images. We compare the average grade of GPT-3 with zero-shot (ZS), GPT-3 with few-shot (FS) learning, GPT-3 with ZS, and chain-of-thoughh (CoT) prompting, GPT-3 with FS and CoT prompting, Codex with ZS, Codex with FS, and OPT with ZS. MIT Fall 2017, Cornell Spring 2017, and Harvard Spring 2015 were the first semester for each university, so few-shot learning results based on previous semesters unavailable (NA). MIT Spring 2020 and MIT Fall 2020 did not have final exams due to COVID-19. MIT Spring 2021, MIT Fall 2021, and MIT Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that the model is not overfitting. The result of the best best-perffrming method for each semester is marked in bold.
Topic
GPT-3 ZS
GS
GPT-3 ZS CUT
GPT-3 ES Col
Codex Z8
Codex FS
OPT ZS
Regression
31.71
50.00
25.61
40.85
40.24
50.00
50.00
Classifiers
18
46.21
26.28
42.35
18.88
53.74
50.00
Logistic Reg.
50.00
77.50
77.50
55.00
70.00
16.67
Features
58.65
75.96
53.85
77.31
68.85
81.54
10.00
Functions
Neural Networks
48.34
60,23
44.54
68.42
37.82
63.45
27.27
47.
13.38
36.77
23.83
MDPs
49.19
52,01
46.03
54.23
24.38
38.03
28.32
RNNS
61.46
71.88
57.29
66.
12.50
40.63
39
RL
36,67
50.11
45.11
24.28
Clustering
100,00
100.00
100.00
100.00
50.00
50.00
63.33
Decision
Model Selection
82.93
83.74
72.76
95.12
67.48
69.92
21.95
Ensemble Methods
27.89
50,00
22.12
66.35
32.69
50.00
13.46
Bayesian Networks
100.00
100.00
100.00
100.00
0.00
0.00
100,00
HMMs
100.00
100.00
50.00
100.00
100.00
100.00
100.00
Optimization
Table 6: We benchmark different baselines for each course topic, excluding question parts that rely on images. We compare the grade of GPT-3 with zero-shot (ZS). GPT-3 with few-shot (FS) learning, GPT-3 with zero-shot and chain-of-thought (CoT) prompting, GPT-3 with FS and COT, Codex with zero-shot, Codex with few-shot learning, and OPT with ZS. The question parts on loss functions rely on image information and therefore unavailable (marked NA). The result of the best-performing method for each semester is marked in bold.
(ZS), GPT-3 with ZS and chain-of-thought (CoT) prompting, GPT-3 with few-shot (FS) learning, GPT-3 with FS and CoT prompting, Codex with ZSS Codex with FS, and OPT with Z.S. Fall 2017 the first semester, so few-shot learning results based on previous semesters are unavailable (NA). Spring 2020 and Fall 2020 did have final exams due COVID-19, Spring 2021, Fall 2021, and Spring 2022 final exams were unavailable online when GPT-3 and Codex were trained, ensuring that the model is not overfitting content has seen previously. The results consistently demonstrate that few-shot learning methods perform best across semesters and topics, as marked in bold.
3.4 LIMITATIONS
Our dataset consists of all question parts and their solutions, including images. However, our baseline methods do not handle questions that rely on an image containing the information required to solve the question since GPT-3 and Codex do not handle images. Tables and @ show the breakdown of the number of question parts and points of questions that do not rely on image information for answering the question. On average, 27.55% of the question parts, which make up 30.32% of the points in final
7

---

Under review as a conference paper at ICLR 2023
Semester
Non Image Parts / All
Non Image Points
Non Open Points
MIT 2017
49/61
70/ 100
69/70
MIT Spring 2018
27 44/
62/ 100
59/62
MIT 2018
30/60
62/ 100
37.5/ 62
MIT Spring 2019
41 /58
70/ 100
48/70
MIT Fall 2019
46/61
61 100
50/61
MIT Spring 2021
/1/
62/ 100
43/61
MIT Fall 2021
56/ 86
48 100
43/48
Spring
46/59
ITrvardd Spring 2015
8/12
70/90
35/70
Harvard Spring 2021
15/32
22/
11//2
Cornell Spring 2017
48/48
128 / 128
74/ 128
Comell Fall 2018
/.56
107 / 1220
71/107
OPT Total
418/585
759 1091
525/759
Total
468/664
830 1191
594/830
Table 7:
The number of question parts that do not rely on images, the number of points that do not rely on images, and the number of non-open question points, in finals for each semester. MIT Spring 2020 and MIT Fall 2020 did not have final exams due to COVID-19.
Topic
Non-Image Parts
Non-Image Points
Non-Open Points
Regression
35.5
Classifiess
72 85
169 200
126.5 169
Logistic Regression
10/ 10
14
7/14
Features
18/21
26 38
22/26
Loss Functions
2/
3/21
3/3
Neural Networks
153
CNNs
$4/65
61 80
57 67
39
RNNs
27 33
36 48
Reinforcement Learning
52/60
87 /
$7
Clustering
5/
30/55
15/30
Decision Trees
33 48
76/ 113
39
Model
16/16
Ensemble Methods
20/20
52
31/
Bayesian Nerworks
0/2
HIMMs
1/4
117
0/1
Optimization
13/16
20/32
15/20
Bonus/Name
3/3
4/4
3/3
830/
/830
Table 8: The number of questions parts that do not rely on images, number of points that do not rely on images, and number of non-open question points in the finals for each topic of the course.
exams, questions that rely image information. The points attributed the non-image parts tallied, recorded, and used to calculate non-image percentage grades.
3.5 GENERATING NEW QUESTIONS
The creation of new, high-quality questions by course instructors and TA often consumonn high-effort process. These new questions must be varied from past questions while still testing the same core concepts. We explore the potential of using GPT-3 to write exam content efficiently by using the dataset of exam questions to generate new questions automatically. We use questions from our dataset as prompts to create new high-quality questions not present in our dataset. We create a list of various questions in curated dataset and use the resulting list to prompt GPT-3 to create new question. The supplementary material demonstrates the results of this process for each topic in the course. The Appendix consists of new generated questions and the closest question from dataset as measured by the cosine similarity of the embedding of each question. These new questions are diverse and qualitatively similar to questions on previous MIT final exams. This provides an efficient way for course TAs and instructors to generate new final questions.
3.5.1 STUDENT SURVEY
To evaluate the machine- machine-generated questions, we conducted an anonymous online student survey comparing them with the human-written questions in terms of quality, appropriateness relative to
8

---

Under review as a conference paper at ICLR 2023
the course, and question difficulty. We surveyed 15 students who have taken the Introduction to Machine Learning course or its equivalent. The survey was optional and included informed consent, with the following description: "We are conducting survey to assess the quality and difficulty of automatically generated questions for an introductory machine learning course final. You will be presented with series of questions, either human-written (taken from an actual course final exam) or machine generated, but you will not be told the source of a given question. For each question, you will be asked (a) whether you think the question is aoman-written or sachine-generated, (b) whether the question is appropriate for the given course final, and finally (c) how you would rate the difficulty of the question. Please carefully read each question and answer to the best of your ability".
We randomly sampled one generated question and its closest (measured by cosine similarity) original, human-written question for each of the twelve machine learning topies. Students were asked to read these 24 questions in the survey, mixed and presented randomly, and then answer three questions for or machine geneiinteee"r""d""" (2) "Is the question appropriate or
each: (1) "Is the question not appropriate for the specific course final?", and (3) "What is the question's difficulty level scale between (easiest) and 5 (hardest)??' We ask the students to provide ratings and not to solve the questions. The results of our survey identified 56. 11% of them correctly as human- -ormtnnwitttnn and 43.89% incorrectly as machine generated. Of the machine-generatee questions, students identified 45% of them correctly as machine-generater and 55% of them incorrectly as human- The difficulty ratings were between (the easiest) and 5 (the hardest). Students rated machine- -eeerrttd questions with a difficulty level of 2.55 with a 1.11 standard deviation and rated buman-writter questions with a difficulty level of 2.85 with a 1.12 standard deviation. Students rated machine- gennertee questions as appropriate 82.6% of the time and questions as appropriate 85.0% of the time.
follows: Out of the
mirttnw questions, students machine-generated.
The conclusions we draw from the survey are that (1) survey participants considered human- -writen questions to be as likely to be human human-written -writenn or machine- generated, machine-generated, and similarly, machine- -eeeennteee questions were considered equally likely to be machine- generated as human-written, (2) survey participants considered the machine-generann and (3) survey participants considered machine-generater questions as appropriate as human-write -wrtten questions. Based on these results, we conclude that across multiple aspects, the machine-gene machine-generated questions are highly similar to human-generate questions and can be adapted to generate questions for machine learning courses.
questions slightly easier than human-written questions,
3.6 IMPLEMENTATION DETAILS
We use the latest OpenAl GPT-3 and Codex models and do not re-train these very large language models. We fix all the hyperparameters of the models so that the answers are deterministic and reproducible. Specifically, we set both top which controls diversity, and sampling temperature, which controls randomness, to O. The frequency and presence penalties are also set to 0, and we do not halt on any stop sequences. We allow diversity for generating new questions by setting the top P and temperature to 0.1. We run Codex with an upper bound of generating programs with 1024 tokens. We use the OpenAl text-davinci-002 and code-davinci-002 engines for generating text and programs. For new-shot-learning and question generation, we use the engine to embed the questions and find the closest questions in the dataset by cosine similarity. The running time for answering or generating each question part is a few seconds.
similarity-babbage
CONCLUSIONS
We present dataset and benchmark for answering and generating university-level final exams in machine learning. Machine performance and human performance are evaluated by the same graders and grading instructions, as well as by automatic checkers. A comparison of baselines shows that few-shot learning methods perform best across semesters and topics. A limitation of our work is that our benchmark does not consider questions that rely on images for their solution. This work may result in improving students learning for final exams, help course staff generate questions for finals, and compare levels of difficulty of exams across semesters and schools. This work includes final exams from MIT's and Cornell's 's Introduction to Machine Learning classes and Harvard's Machine Learning course. We hope this dataset and benchmark serve the machine learning community and advance the state-of-the art in the field.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 1877-1901, 2020.
Mark Chen al. Evaluating large language models trained on code. arXiv preprint arXlv:2107.03374. 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023. 2022.
Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng. Roman Wang. Nikhil Singh, Taylor Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level. Proceedings of the National Academy of Sciences, 1 119(32), 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In Proceedings of the International Conference Learning Representations (ICLR), 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916 2022.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, al. Competition-level code generation with alphacode. arXiv preprint 20Xiv:2003.07814, 2022.
Fanyi Qu, Xin Jia, and Yunfang Wu. Asking questions like educational experts: Automatically generating question-answer pairs on real-world examination data. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446, 2021.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alo Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04611 046115 2022.
Leonard Tang, Elizabeth Ke, Nikhil Singh, Bo Feng, Derek Austin, Nakul Verma, and Iddo Drori. Solving probability and statisties problems by probabilistic program synthesis human level and predicting solvability. In Proceedings of the International Conference Artificial Inielligence in Education (AIED). 2022.
Sunny Tran, Pranav Krishna, Ishan Pakuwal, Prabhakar Kafle, Nikhil Singh, Jayson Lynch, and Iddo Drori. Solving machine learning problems. In Proceedings of the Asian Conference Machine Learning (ACML), pp. 477,488,,20 2021.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 2022.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. arXiv preprint 1.Xiv:2011./1900 2022.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-mos prompting enables complex reasoning in large language models. arXiv preprint ar05v:2205. 20lie.22............22 2022.
10

---

Under review as a conference paper at ICLR 2023
A APPENDIX
Table 9: Generating new questions: example new question for each topic automatically generated and the closest question in the dataset based cosine similarity of the questions embeddings.
Topic
Question
Similarity
Regression
)==1" where 2(0) Rd and y E R. Let X be d n matrix in which the x11) are the columns and let Y be a x n vector containing the values of ( Using the ordinary least-squares formula, we can compute Wots (XXT) XYT Using ridge regression, we compute W ridge (XX* + =1) XY We decide to try to use these methods to initialize single-unit neural network with a linear activation function. Assume that XX" is neither singular equal the identity matrix, and that neither Wols nor Widge is equal to (0, O, ...... .). Consider a neuron initialized with W Wddee Provide objective function J(W) that depends on the data, such that batch gradient descent to minimize will have no effect on the weights, argue that one does exist."
Generated Question: "We're given a data set D { {x(()) y())
0.945
Closest Question: "We're re given a data set D = {(x(0) .y)) }²= where x66) Rd and y( R. Let be d matrix in which the (0) are the columns and let Y be 1 x n vector containing the values of y Using the analytical regression (ordinary least-squares) formula, we can compute
Wols = (xx") XY
Using ridge regression, we can compute Wridge (xx* + XYT We decide to try initialize single-unit "neural network" with a linear activation function and no offset: h(x;; W) WT'. Assume that X X" is invertible and not equal to the identity matrix, and that neither Wols nor Widge is equal not using explicit offset/bias
these methods to
to (0, o......), 0). Note also that term. Rory has solved many problems from this particular domain before and the solution has typically been close W = (1,...,) 1)T. Define an objective function J (W) that we could minimize in order to obtain good estimates for Rory's next problem, even with very little data."
Classifiers
Generated Question: "Consider a binary classification problem with two classes, +1 and -1. Assume that the training data is linearly separable. Consider the following two models:
0.782
Model 1: g(x) sgn (w x:)
Model 2: g(æ) sgn w x x b)
Assume that the training data is linearly separable. Which model is more likely to overfit?
Closest Question: "In some cases, we will have a validation set in addition to training and test sets. Assume the validation set is approximately the same size as the test set. This validation set is often used to tune hyperparameters such as X. Imagine classifier using regularization, with À chosen based on performance on the training set. Which will have the highest accuracy the training set, the validation set or the test set?"
have trained
11

---

Under review as a conference paper at ICLR 2023
Table 9: Generating new questions: example of new question for each topic automatically generated and the closest question in the dataset based on cosine similarity of the questions embeddings.
Topic
Question
Similarity
Logistic Regression
Generated Question: "In logistic regression, we use the sigmoid function 0.810 to "squish" the output of our linear function to be between 0 and 1. What is the output of the sigmoid function when the input is 0?"
Closest Question: "Consider the simplest of all neural networks, consisting of a single unit with sigmoid activation function: h(2; owo + wjz) where a(z) (1 exp(- (of(-2))- Let's start with a classifier defined by -1 and = 1. Which range of input values are classified as positive? Which as negative?"
Features
Generated Question: "You are working
project to predict how successful a given Kickstarter campaign will be, based on information about the campaign (such the amount of money pledged, the number of backers, the category of the project, the country where the project is based, and forth). One of the features you want use is the project's title. What is good way to encode the project's s title for use in machine-learnin predictor?"
0.761
Closest Question: "Mac O'Larnin is considering selling app on Frugal Play. You have friend with inside info at Frugal, and they 're able share data how previous apps have performed Mac decides that he will learn neural network with no hidden layer (i.e., consisting only of the output layer). He needs help in figuring the precise formulation for machine learning. What is the best way to encode the app characteristic 'Suitable for people ages (2-4, 5-10, 11-15, 16 and over)' as a feature for an input to the neural network? Choose from among the following: multiple unary features (one-hot encoding), multiple binary features (thermometer encoding), an integer or real-valued feature. Also give the exact function that maps each input to its corresponding feature(s)."
the store.
12

---

Under review as a conference paper at ICLR 2023
Table 9: Generating new questions: example of new question for each topic automatically generated and the closest question in the dataset based on cosine similarity of the questions embeddings.
Topic
Question
Similarity
Loss Functions
Generated Question:
"We have data of images, each of which is either picture of dog or a picture of cat. We want to train binary classifier to distinguish between the two classes. Which the following is better choice of loss function for this task:
0.790
(a) The 0-1 loss, which gives loss of if the classifier predicts the wrong class and a loss of O otherwise.
(b) The cross-entropy loss, which given by -log(p) if the correct class is predicted and -log(I-p) otherwise, where p is the predicted probability of the correct class.
(c) The hinge loss, which is given by max(0, 1-s) if the correct class is predicted and max(O, 1+8) otherwise, where s is the predicted score for the correct class."
Closest Question: "Consider the following 2D dataset in (x.y) format: ((I,-), +1), ((1.1), +1), (11,2.5)++1), ((2,-2).-1), (223), (1,1). Break ties in distance by choosing the point with smaller Iı coordinate, and if still tied, by smaller 22 coordi- nate. Compute the leave-one-out cross validation accuracy of the 3- 3-nearest-neighher learning algorithm on this dataset."
Neural Networks
Generated Question: "Consider a neural network with two hidden layers, each with two units. The input layer has two units, and the output layer has one unit. The hidden layers use the sigmoid activation function, and the output layer uses the linear activation function. The weights from the input layer to the first hidden layer are w w11 1, w w22 = 1. The weights from the first hidden layer to the second hidden layer are wll = 1, w = I, w21 1, and w22 = 1. The weights from the second hidden layer to the output layer w 1, w21 = 1. The bias terms are all zero.
0.880
w21 = 1, and
What is the output of the neural network for the input x1 1, x2 1?"
Closest Question: "A neural network is given as Zt = X W1 A' f1(Z'). Z W A', ý fz²) Specifically, the input X is 4 X 1 column vector, ÿ is 1 x scalar. W is a 3 x 1 vector. We also know that, z (W1)7> X and 722 = (W2)T A What are the dimensions of 22n"
CNNs
Generated Question: "Suppose we have a 3x3 image and we use a 2x2 filter with stride What are the dimensions of the output image?"
0.895
Closest Question: "A neural network is given as Z' = X * W
A f'(()), 7 W2 A ú = (Z2). There only one data
point which is: X = [1, 1, 1, 1] and y [1). If W' and 2 are both matrices/vectors of all ones, what is the resulting Loss where the Loss (y 2m
13

---

Under review as a conference paper at ICLR 2023
Table 9: Generating new questions: example new question for each topic automatically generated and the closest question in the dataset based on cosine similarity of the questions embeddings.
Topic MDPs
Question
Similarity
Generated Question: "Consider MDP with four states, called A, B, C, and D, and with two actions called Move and Stay. The discount factor ? 0.9. Here is reminder of the Q-learning update formula, based on experience tuple (s,a, 8')
0.988
Q(s,a) = (1 (0l-a)11 ())(s, (r+ max Q (s', (s'''a))
the following ftate-action-eeeer sequence: A, Move, B. Move, 0 C, Move, A, Move, 0 B. Move, C. Move, With Q-values all starting at O, run the Q-learning algorithm that state-action sequence. Provide the q-learning value for Q(C, Move)."
Let 1. Assume
Closest Question: "Consider an MDP with four states, called A. B,C, and D, and with two actions called Move and Stay. The discount factor y 0.9. Here reminder of the Q-learning update formula, based on experience tuple (s,
Q(s,a) = (1-a))(s,,))) -a (++ y (s',
the following state state-action-reward sequence: A, Move, B, Move, 0 C, Move, A, Move, With Q-values all starting at we run the Q-Jearning algorithm on that state-action sequence. Provide the q-learning value for Q(A, move)."
Let 1. Assume
RNNs
Generated Question: "Consider the following RNN: s = 0.907
(anh(w)2; W28t-1 + b) 3t Wys, b2 Assume sn 0 and b2 = 0. What values of w1, W2, We and b would generate output sequence [0, 0, 1, 1] given input sequence [0, 0, 0, 0, 1,
Closest Question: "Ronnie makes simple RNN with state dimension and a step function for fi. so that 8, step [1 W28 + b) where step(z) (2) = if > 0.0 and equals 0 otherwise, and where the output
8
Assuming 80 = we want to generate output sequence [0, 0, 1, given input sequence [0, 0, 0, 1, O, 1, 0). Remnie thinks this is not possible using Ronnie's architecture. Rennie makes an argument based on the relationships in the table above. Is Rennie right?"
Reinforcement Learning
Generated Question: "What is the tabular Q-learning update equation, 0.796 based on experience tuple (s, a, r, s
Closest Question: "b is the bias for classifier network. What are dimensions for multi k-class classification?"
14

---

Under review as a conference paper at ICLR 2023
Table 9: Generating new questions: example of new question for each topic automatically generated and the closest question in the dataset based on cosine similarity of the questions embeddings.
Topic
Question
Similarity
Clustering
Generated Question: "Suppose that we have a dataset with data points, k clusters, and d features. After running the k-means algorithm, the within-cluster sum of squared errors (WCSS) is given by:
0.767
R Σ
Pag 1|
where y/ is the cluster label of the ith data point, and My, is the cluster center associated with the ith data point. The within-cluster sum of squared errors (WCSS) is a measure of how well the clusters fit the data. Suppose that we have two datasets, X1 and Xg, where X1 has n1 data points and X₂ has n2 data points. We run the k-means algorithm on both datasets. We find that the WCSS for X1 is smaller than the WCSS for X2. Does this imply that the clusters for X1 are better than the clusters for X2? Why or why not?"
Closest Question: "Consider the following 2D dataset in (x.y) format: ((I,-), +1), ((1.1), +1), ((1,2.5)++1), ((2,-2),-1), (223)) (1-1). We will construct tree using greedy algorithm that recursively minimizes weighted average entropy. Recall that the weighted average entropy of a split into subsets A and B is: (fraction of points in A) H (R.,)) + fraction of points in B) H (R where the entropy (Rm) of data in a region Rm is given by H Ex Pmnk log, Ponk The P.m is the empirical probability, which is in this the fraction of items in region that of class k. Some facts that might be useful to you: H(O) = O, H(3/5) = 0.97, H(3/8) 0.95, H(3/4) = 0.81, H(5/6) = 0.65, H(1) O. Draw the decision tree that would be constructed by our tree algorithm for this dataset. Clearly label the test in each node, which case (yes or no) each branch corresponds to, and the prediction that will be made at each leaf. Assume there no pruning and that the algorithm runs until each leaf has only members of single class." "
Decision Trees
Generated Question: "The Gini score is measure of how often 0.867 randomly chosen element would be incorrectly labeled if it was randomly labeled according to the distribution of labels in the subset. It ranges from to with meaning that there are no mislabeled elements and I meaning that the subset is perfectly mixed. Explain whether or not would be a good idea to use the Gini score as a scoring function for pruning decision trees."
Closest Question: "There are different strategies for pruning de- cision trees. We assume that we grow decision tree until there is small number of elements in each leaf. Then, we prune by deleting individual leaves of the tree until the score of the tree starts to get worse. The question is how to score each possible pruning of the tree. Here is definition of the score: The score the percentage correct of the tree on separate validation set. Explain whether or not it would be good idea and give a reason why or why not."
15


Reviews:
[{"review_id": "i9YagTgC8z8", "reviewer": "Reviewer_FkVg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a new university exam dataset spanning questions on 12 machine learning topics covered at MIT's Introduction to Machine Learning course. This joins a thin body of work of exam datasets that have mostly focused on high school and lower levels. The dataset and code are open-sourced and uploaded as part of the Supplemental Material. The exams span multiple quarters of this class. The experiments include benchmarking LMs on this dataset, as well as generating new exam questions. The results show that the generated questions are of equal quality as the human-written ones, have the same average difficulty rating, and get recognized as human-written with the same probability.\n\nUPDATE: After discussion with my fellow reviewers and area chair, I agree that there are valid concerns about the size of the dataset. While we all praise the authors for the originality of the work and consider it valuable to the community, the dataset as presented may be too small for the community to use. We recommend trying to expand the dataset and resubmitting the paper.", "summary_of_the_review": "This work introduces a novel dataset of university level machine learning exam questions and answers. This is a much needed resource for the community, and the results of the benchmarking is valuable, as well as the ones for exam question generation.", "strength_and_weaknesses": "Strengths:\n- Much needed dataset for the community and interesting topic\n- Strong GPT- and Codex-based baselines, as well as baselines with Open LMs\n- Extensive experiments provide benchmarking results to start off of\n- Great experiments and promising results with generated questions\n\nWeaknesses:\n- No major weakness\n\nNotes:\n- The references should be in parentheses\n- In Table 5, there is the exact same result between GPT3-ZS and GPT-3 FS in Fall 2021, is that a typo?\n- \"We extended this work to include final exams from Harvard’s and Cornell’s machine learning classes as well. \" Are these questions included or to be included in the future?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dhXGC_TUJXU", "reviewer": "Reviewer_bhJL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presented a new dataset that contains final exam questions, and compares the performance of popular transformer models for answering them. The dataset includes questions that arise from a wide range of topics in machine learning including loss functions, RNNs, reinforcement learning, and etc. ", "summary_of_the_review": "This paper presents an appealing and novel approach by generating a new dataset of machine learning final exams. This work also provided a benchmark of models for answering the final exam questions. ", "strength_and_weaknesses": "Strength: This is the first study to present a dataset of machine learning final exams\nWeakness 1: A lot of details are missing with regards to how the dataset was curated. An illustration/figure to show the method used in this work would be advantageous.\n Weakness 2: The recommended models do not consider questions that rely on images for their automated solution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "T6a7-DSncS", "reviewer": "Reviewer_ffgb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions. They build baselines by applying zero-shot and few-shot learning to GPT-3 and Codex, adding chain-of-thought prompting for GPT-3. ", "summary_of_the_review": "I think this is an interesting dataset, but it is like one dataset from BIG-bench. I think we don't need every dataset from BIG-bench to be published.", "strength_and_weaknesses": "Strength:\n1. A quite interesting dataset\n2. Solid baselines\n\nWeakness:\n1. Seems like one more dataset from BIG-bench, https://github.com/google/BIG-bench . Not too much novelty\n2. The paper is not well-written. Not sure why Figure 1 is shown in the paper. It only shows students taking the final exam which is not related to the dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject