Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper presents the MATH dataset, consisting of problems from high school math contests. Unlike other math-related (or language) datasets, the MATH problems are currently unsolvable by large-scale transformer models, and scaling analyses indicate that they would not be solvable at any reasonable scale. Furthermore, mathematical problem solving is a key skill for artificial intelligence to solve. This paper additionally provides a supplementary dataset for pretraining, which they show can significantly improve performance on the MATH dataset.

In sum, this dataset presents an exciting challenge in that it could force creative innovation in ML architectures, while testing models on important dimensions like problem solving, reasoning, natural language understanding, explanation and interpretability, conditional computing, "working memory" / "scratch space" processing, and curriculum learning. 

strengths: * well motivated as an important problem for AI to solve
* provides different levels of difficulty as well as a separate pre-training dataset, which enable investigating questions about bootstrapping, transfer learning, and curriculum learning
* well-formatted and easy-to-use dataset, being both easily human readable (mathematics is expressed in LaTeX), normalized (a unique answer per question, enabling automatic evaluation), and a language for diagram/figure representation (so that sequence models can parse them as well)
* interesting results already, such as the 
  - large improvements on MATH dataset after pretraining on the AMPS dataset
  - decreases performance when models were forced to generate intermediate computations before generating a final answer
  - scaling analyses on the predicted performance of large-scale transformer models

weaknesses: - Perhaps it is a bit unfair to evaluate existing language models out-of-the-box on this dataset, especially using pre-existing tokenizers that are not designed for the particular statistics of the mathematics data (nor the graphics representations). As the paper mentions, for example, the T5 tokenizer removes many LaTeX symbols.
- It would be great to have some suggestions on how to do evaluation of the intermediate computations, even if this must be done on a manual basis (it seems that this was done to some extent)
- What is the accuracy of the language models on the AMPS dataset? This would give some sense of the difference in difficulty between the two datasets.

correctness: Correctness does not appear to be an issue.
clarity: This paper is very clearly written.
relation to prior work: This paper provides a challenge for large-scale transformer models that are much harder to solve than existing benchmarks on either language or mathematics tasks.
additional feedback: --
review_title: Well motivated and well designed dataset that is an exciting and unsolved challenge

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces the MATH dataset that includes challenging competition mathematics problems. The authors argue they measure the mathematical problem-solving ability of machine learning models using this dataset. Each problem in the dataset has a step-by-step solution, which enables to perform evaluations in multiple levels by showing different fractions of the solution to the model. The benchmark also includes a large auxiliary pretraining dataset. Experiments have done using different GPT-2 and GPT-3 models and results suggest that accuracy of those models are low on the dataset.
strengths: The paper introduces two large datasets. The MATH dataset for benchmarking mathematical problem-solving of ability and the AMPS dataset for pretraining models. The problems in the datasets can also be categorized to different subject areas such as calculus, statistics, and algebra. Authors have shown that the dataset covers several subtopics from each of the subject areas. Moreover, the paper also uses a standard difficulty level of these math problems. 
A comprehensive evaluation protocol that enables to evaluate models step by step. Each problem in the MATH dataset consists of a step-by-step solution and a final answer. This setup helps in evaluating models by varying the fraction of the solution seen by the model. Moreover, having different subject areas, difficulty level for each math problem enables a thorough analysis of model performance. 

weaknesses: 1.	The claims made by the paper seems too optimistic. Authors main goal is to introduce the MATH dataset to measure problem-solving ability of machine learning models. However, solving the problems of the dataset, does it really measure the ability of the models. In line 40 authors further mention that despite the low accuracies produced by the models, they clearly possess some mathematical knowledge. However, it’s not clear what evidence authors use to make these claims. In my opinion, the paper should restate the claims made. 

2.	The design of the dataset is not entirely clear to me. Authors have identified different subject areas for the problems in the dataset, which is a plus point. However, it is not clear how the problems are distributed in the train and test sets. For example, if statistics problems are included in the test sets, are statistics problems included in the training set. If it cannot be ensured that problems from the same subject area is not included in the training, it is a major drawback in the dataset. Even human would not be able to solve problems in a certain subject area without learning the subject. 

3.	It is not clear if the GPT-3 model is pretrained or not. In line 239 authors mention that models are pre-trained on AMPS and in Table 2 indicates that GPT-3 models do not pretrain. Please adjust this confusion in the paper. If the GPT-3 models are not pre-trained, could the authors explain why two different setting are used for GPT-2 and GPT-3 models?

4.	Authors have conducted an experiment with humans have an informative comparison between models and human performance, which is an interesting comparison to me. Authors have not mentioned how many human participants were present in this study. 

correctness: Minor typos. 
Line 221: accuracy is increasingly very slowly -> increasing
Caption in Table 2: on AMPS.Model a -> AMPS. Model

clarity: I think the paper could have been written in a more clear way. For example, the claims that authors made, the design of the dataset, and details of the models could have been discussed and justified in a comprehensible way.
relation to prior work: The papers discusses related work in detail from different areas. 
additional feedback: no
review_title: Measuring Mathematical Problem Solving With the MATH Dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper presents the MATH dataset that poses mathematical problem-solving challenges with high-school competition-level questions. It also curates a larger math problem-solving corpus called AMPS with which we can do secondary pretraining. 

Benchmarks show that MATH is extremely challenging for large language models, and that if we extrapolate the scaling trends, it would not be feasible to solve MATH by simply increasing the model size, thus encouraging algorithmic innovations in this domain. 
strengths: The MATH dataset is curated such that it contains not only question-answer pairs, but also a step-by-step solution for each question, expressed in natural language and symbolic formula. 

The dataset is carefully formatted and normalized. All answers are expressed in a unique format so that evaluation doesn’t need to resort to fuzzy matching. Diagrams are also expressed in a unified scripting language. 
weaknesses: The authors may consider setting up some benchmarks with non-fully-E2E-neural-models. For example, with methods that predict a numerical program and then use a symbolic calculator to give the final answer. 
correctness: Per the description in the paper, the MATH and AMPS dataset are legitimately collected, parsed or generated. Experimental settings also look correct. 

The MATH dataset is divided into a train set (7.5k examples) and a test set (5k examples), but there seems to be no development set. 
clarity: The paper is clear and easy to follow. 
relation to prior work: I like the extensive discussion of prior work on mathematical reasoning through theorem proving, and how the authors present MATH as providing an easy and objective way to measure progress in mathematical reasoning. 

Yet MATH is a QA-style dataset, and there is not too much focus on its distinctiveness from other mathematical QA datasets, such as MAWPS, MathQA, ASDiv, SVAMP, etc. I would love to see more discussion in this area. 
additional feedback: It’d be great if the authors can release the mathematica scripts that are used to generate part of the AMPS pretraining dataset.
review_title: Really meaningful and challenging dataset for mathematical problem solving

Decision: Accept