Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors provide a new dataset and benchmark of questions from machine learning final exams and code for automatically answering these questions and generating new questions. They build baselines by applying zero-shot and few-shot learning to GPT-3 and Codex, adding chain-of-thought prompting for GPT-3. 
strength_and_weaknesses: Strength:
1. A quite interesting dataset
2. Solid baselines

Weakness:
1. Seems like one more dataset from BIG-bench, https://github.com/google/BIG-bench . Not too much novelty
2. The paper is not well-written. Not sure why Figure 1 is shown in the paper. It only shows students taking the final exam which is not related to the dataset.
clarity,_quality,_novelty_and_reproducibility: What's the difference between the proposed dataset and the BIG-bench datasets?
summary_of_the_review: I think this is an interesting dataset, but it is like one dataset from BIG-bench. I think we don't need every dataset from BIG-bench to be published.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a new university exam dataset spanning questions on 12 machine learning topics covered at MIT's Introduction to Machine Learning course. This joins a thin body of work of exam datasets that have mostly focused on high school and lower levels. The dataset and code are open-sourced and uploaded as part of the Supplemental Material. The exams span multiple quarters of this class. The experiments include benchmarking LMs on this dataset, as well as generating new exam questions. The results show that the generated questions are of equal quality as the human-written ones, have the same average difficulty rating, and get recognized as human-written with the same probability.

UPDATE: After discussion with my fellow reviewers and area chair, I agree that there are valid concerns about the size of the dataset. While we all praise the authors for the originality of the work and consider it valuable to the community, the dataset as presented may be too small for the community to use. We recommend trying to expand the dataset and resubmitting the paper.
strength_and_weaknesses: Strengths:
- Much needed dataset for the community and interesting topic
- Strong GPT- and Codex-based baselines, as well as baselines with Open LMs
- Extensive experiments provide benchmarking results to start off of
- Great experiments and promising results with generated questions

Weaknesses:
- No major weakness

Notes:
- The references should be in parentheses
- In Table 5, there is the exact same result between GPT3-ZS and GPT-3 FS in Fall 2021, is that a typo?
- "We extended this work to include final exams from Harvard’s and Cornell’s machine learning classes as well. " Are these questions included or to be included in the future?
clarity,_quality,_novelty_and_reproducibility: The work is clear, well evaluated, novel and reproducible with the supplemental material.
summary_of_the_review: This work introduces a novel dataset of university level machine learning exam questions and answers. This is a much needed resource for the community, and the results of the benchmarking is valuable, as well as the ones for exam question generation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work presented a new dataset that contains final exam questions, and compares the performance of popular transformer models for answering them. The dataset includes questions that arise from a wide range of topics in machine learning including loss functions, RNNs, reinforcement learning, and etc. 
strength_and_weaknesses: Strength: This is the first study to present a dataset of machine learning final exams
Weakness 1: A lot of details are missing with regards to how the dataset was curated. An illustration/figure to show the method used in this work would be advantageous.
 Weakness 2: The recommended models do not consider questions that rely on images for their automated solution. 
clarity,_quality,_novelty_and_reproducibility: 
This work is of high quality since it presents a novel approach by being the first study to curate a machine learning dataset of final exams. This work is also reproducible since the authors made their data and code publicly available for the machine learning community. 
summary_of_the_review: This paper presents an appealing and novel approach by generating a new dataset of machine learning final exams. This work also provided a benchmark of models for answering the final exam questions.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The submission presents a new dataset of machine learning exam questions. The paper initially had a wide divergence in reviewer opinion, reflecting disagreements over the usefulness of an additional dataset. Machine learning exams comprise an interesting and challenging domain for evaluating models. However, the proposed dataset is quite small, which reviewers felt would limit interest. Given the vast number of existing datasets, new datasets need to be compelling to make an impact. It would significantly improve the submission to expand the dataset, perhaps to a wider range of subjects than machine learning. However, the idea has much potential, and I would encourage the authors to continue working in this direction and resubmit.
justification_for_why_not_higher_score: After discussion, reviewers felt that the paper needs further work to be accepted. Particularly, expanding the dataset to be larger or cover exams in other subjects would substantially strengthen the paper.
justification_for_why_not_lower_score: n/a
Decision: Reject