OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text
Authors: Keiran Paster
Marco Dos Santos
Zhangir Azerbayev
Jimmy Ba
Venue: ICLR 2024 poster (2024)
Keywords: web-scale dataset
natural language processing
large language model
reasoning
AI for math
Primary area: datasets and benchmarks
TL;DR: We open-source a large-scale mathematical dataset extracted from the web.

Abstract: There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, open-sourced and released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.

Published as a conference paper at ICLR 2024
OPEN AN OPEN DATASET OF HIGH-QUALITY MATHEMATICAL WEB TEXT
Keiran Paster; Marco Dos Santos, Zhangir Azerbayev,
Jimmy Ba
University of Toronto; Vector Institute for Artificial Intelligence i University of Cambridge, Princeton University keirgg heirpecs. seirpecs.toronto. edu, mjad30 cam ac.uk
ABSTRACT
There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the rea- soning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quan- titative reasoning. However, because all known publicly released web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce WathWehMath, open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail method for extracting text and 18TІ content and removing boilerplate from HTML documents, as well as our methods for qual- ity filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on Open WebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abil- ities of large language models.
INTRODUCTION
Advances in large language models have opened up new opportunities in numerous fields, providing transformative shift in our approach to wide range of complex problems (Brown et al., 2020; Raffel et al., 2020). Among these problems, mathematical reasoning has drawn the attention of several researchers in recent years, becoming both a common benchmark to judge the performance of large language models and inspiring new approaches to improve their reasoning capabilities in the hope that they will one day be able to solve complex mathematical problems. One of the biggest advancements in mathematical reasoning in recent years has been the Minerva model (Lewkowycz al., 2022), which achieved state-of-the-ard results on quantitative reasoning benchmarks such as MATH (Hendrycks et al., 1021). Minerva was trained by finetuning Pal.M (Chowdhery et al., 2022) on curated dataset consisting of billions of tokens of high quality technical content sourced from both scientific papers and the web.
Minerva and the datasets used for its training were not released publicly and the current capabilities of open-source models (e.g.. Touvron et al. (2023b;c;a): Geng & Liu (2023): Biderman et al. (2023)) in quantitative reasoning lags behind. We believe that there are important research directions that can only be enabled through open-access to such models and datasets, such as work on memorization and generalization, reinforcement learning, the development of new reasoning benchmarks, and advancement in the reasoning capabilities of language models.
In our work, we produce an open alternative to the Math Web Pages dataset used to train Minerva (Lewkowyez et al., 2022). We extract documents from Common Crawl', applying our pipeline to
"Keiran and Marco created the dataset and Zhangir led model training and evaluation. 'htpp:: wommoncraw .org/

---

Published as a conference paper at ICLR 2024
Language ID
MathSSore Filter
Perplexity Filter
Deduplication
Manual Filter
Common
Prefilter
Crawl
1B
336M
66M
7.8M
OpenWaebttth
237B
6.3M
14.78 takens
HTML
documents
Figure I:
The pipeline for constructing OettWhhhath involves aggressive filtering so that the final dataset only contains high quality, English, and mathematical content.
extract text while preserving mathematical content in the form of ВТRX equations. We then filter the documents, ensuring that only high-quality English mathematical documents are kept. Finally, we deduplicate the dataset, resulting in 14.7B tokens of high-quality mathematical content suitable for both pretraining and finetuning large language models. The key contributions of this work are as follows:
We publically release OpenWehMath a dataset of 14.7B tokens of high-quality mathe- matical web text. Our dataset can found at http://wwgnnnaace...oo///dttt/// math/open-web-mmth on the Hugging Face Hub.
We extensively document our pipeline, sharing our findings with the NLP community. We open-source the code needed to reproduce our results.
We analyze the quality of Open WehMaWh First, we analyze the contents of our dataset, providing statistics on the types of webpages, subjects, and top domains. Then, we train several language models on our dataset to show that per-token, is more effective than existing mathematical pretraining datasets, and is most effective when combined with other datasets.
RELATED WORK
2.1 MATHEMATICS DATASETS AND BENCHMARKS
Mathematics datasets Over the past couple of years, several datasets of mathematies have been introduced. AMPS, a dataset of informal mathematics, was introduced alongside the MATH dataset (Hendrycks al., 2021). AMPS includes by-step solutions in LaTeX and over 5 million problems generated using Mathematica scripts. In total, AMPS contains 23GB of problems and solutions. Another notable example is NaturalProofs (Welleck et al., 2021), which encompasses 32,000 theorem statements and proofs, 14,000 defini- tions, and 2,000 other types of pages (e.g. axioms, corollaries) derived from ProofWiki. the Stacks project and data from mathematics textbooks. Proof-Pile (Azerbayev et al., 2023) is a dataset of mathematical
than 100,000 Khan Academy problems with step-
that contains more than 14.5GB of informal mathematics texts obtained from arXiv, Stack Exchange, ProofWiki, Wikipedia, openly licensed books, and the MATH dataset. There are also many proprietary datasets for mathematics. WebMath a large-scale dataset mentioned by OpenAl rescarchers (Polu & Sutskever, 2020) that contains 35B token mix of content from Github, arXiv, and Math StackExchange, adding up to 35GB informal mathematics. MathMix is another OpenAl dataset used finetune GPT-4 (Lightman al., 2023) that contains 1B high quality math- ematical tokens containing both natural and synthetic data. The proprietary web dataset used to train Minerva, called Math Web Pages (Lewkowycz et al., 2022), was compiled by collecting 17.5B tokens from web pages that contain 18TК code.
Mathematics benchmarks Several popular benchmarks have been used by researchers to assess the capabilities of language models on both formal and informal mathematics. The MATH dataset (Hendrycks et al., 2021) is comprised of 12,500 challenging competition problems in informal lan- guage. Each problem is also accompanied by step-by-stepy informal proof. Answers by the bboxed environment, allowing for easier answer verification. GSM8k (Cobbe et al., 2021) another popular multi-step informal mathematics reasoning benchmark. contains 8,500 grade
delimited
2

---

Published as a conference paper at ICLR 2024
Other Blog
Other
Eoonomics Chemistry
5%
6%
Paper
12%
Forum
3%
Statistios
12%
40%
Mathematics
50%
12%
Comp Sci
16%
Reference
21%
Physics
Educational
Types of OpenWWebMath documents
Subjects of OpenWebMath documents
Figure 2: Left:
The documents in OpenWebMaate are sourced from forum posts, educational content, reference pages, scientific papers, blogs, and more. Most content comes from Q&A A forums where users discuss how to solve problems. Right: The majority of the content OpenWebMath is related mathematics, but large part is related other technical subjects like Physies, Computer Science, Statistics, and more.
school math problems that are intended to solvable by bright middle school student. Lewkowyez et al. (2022) also introduce a benchmark based on OpenCourse Ware. OCWCourses includes a set of 272 automatically-veriin
solutions at the undergraduate level, covering chemistry, information theory, differential equations, special relativity, and more. Lewkowycz et al. (2022) also evaluate on subset MMLU (Hendrycks et al., 2020) called MMLU-STEM, which focuses on science, technology, engineering, and mathematics.
2.2
WEB DATA PROCESSING PIPELINES
The pretraining of large language models requires large, diverse datasets. Data scraped from the web is one of the primary sources for such data. However, sources such as Common Crawl, which contains over 200 billion web pages, are known to have significant amounts of low-quality and duplicate content, requiring extensive filtering and deduplication to be suitable for training. Prior works such as C4 (Raffel et al., 2020), RefinedWeb (Penedo et al., 2023), CCNet (Wenzek et al., 2019), The Pile (Gao et al., 2020), and GPT-3 (Brown al., 2020) introduce various pipelines for extracting quality data from Common Crawl for the purposes of language model training. These pipelines typically consist of three primary steps: text extraction, filtering, and deduplication.
Text extraction Extracting plain text from HTML files is critical step the creation Common Crawl-based datasets. The easiest way to extract text from Common Crawl documents is to use the WET corresponding to each webpage, which contains pre-extracted plain text of the webpage. CCNet and C4 both use Common Crawl's WET files. However, the text extracted in WET files may contain too much boilerplate or miss out on important content such as LTTX equations. is also possible to extract text directly from the HTML found in Common Crawl WARC files. The Pile uses an open source library called jusText (Endrédy & Novák, 2013) to extract text from HTML while RefinedWeb uses library called Trafilatura (Barbaresi, 2021). These text extraction approaches differ in terms of extraction speed, customization, and their precision and recall for removing boilerplate content.
The first layer of filtering often involves language identification (Wenzek et al., 2019). Language filtering is used because certain other parts of the pipeline only work for specific lan- guages, and is often done with simple linear classifiers such as from fastText (Joulin et al., 2016). Quality filtering
Filtering
be done with combination of perplexity, classifier, and rule-based methods. the KenLM library (Heafield, 2011) trained on the target domain. The documents in the dataset then sorted and filtered by their perplexity under this model. Other datasets such as the one used to train GPT-3 (Brown et al., 2020) classifier-based approach. This involves training classifier on known-high-quality
CCNet uses a 5-gram Kneser-Ney language model implemented
3

---

Published as a conference paper at ICLR 2024
documents, such as those from Wikipedia, as positive examples and unfiltered documents from Common Crawl as negative examples. The classifier scores are used to filter low-quality documents from the dataset. Finally, rule-based approaches such as those used in C4 (Raffel et al., 2020) and MassiveWeb (Rae et al., 2021) involve removing pages with certain characters, too many or too few characters, too high proportion of symbols, those with abnormal average word length. OpenMathWeb uses a mixture of these three approaches.
Deduplication Given the periodic nature of Common Crawl snapshots and a general redun- dancy in web-sourced text, deduplication is an important processing step. Document-level near- deduplication (c.g., in (Brown et al., 2020; Penedo et al., 2023)) often employs MinHashLSH. an efficient algorithm for estimating the Jaccard similarity of documents. CCNet (Wenzek et al., 2019) uses paragraph-Ievel deduplication, which can help to remove common boilerplate content found in WET text-extractions.
BUILDING OPEN
3.1 OBJECTIVES
Our aim with Open OpenWebMath is to build dataset of as many mathematical documents sourced from the web as possible while preserving the formatting of mathematical content such as 15T₃ equations as in Lewkowycz et al. (2022). For the purposes of this work, we define a mathematical document as a document containing either core mathematical contents such as theorems, definitions, proofs, questions and answers, formal mathematics, interdisciplinary documents featuring mathematical formulas within fields like physics, chemistry, biology, economics, and finance. We source our doc- uments from Common Crawl, which is large open-access crawl of the web containing petabytes of raw HTML files. Due to the high variance in the quality of documents from Common Crawl, we additionally use several methods for filtering and boilerplate reduction. Throughout the creation of Open WebMath, we iteratively refined these methods to ensure that we do not remove too many rele- vant documents, optimizing for high recall whenever possible. Since expect that Open WebMath will be used primarily as an additional source of pretraining data for large language models, we pre- fer having a small percentage of non-mathematical but high quality documents in the dataset rather than removing them and potentially losing relevant mathematical content. Finally, due to the limited number of mathematical data available on the web, we use significantly more manual inspection and tuning of our processing pipeline than other web-based datasets. We document our processing choices and pipeline in the section that follows.
3.2 HIGH-LEVEL OVERVIEW OF THE PIPELINE
As shown in Figure 1, the processing pipeline for Open WebMath falls into five stages. First, we ap- ply a prefilter to all HTML documents in Common Crawl to quickly judge whether they have math- ematical content, skipping those that do not before doing the extensive processing needed to extract text and equations and remove boilerplate. Second, we extract the text, including mathematical con- tent, from the HTML documents. Third, we apply language identification filters, perplexity-bon quality filtering, and a mathematical content classifier filter. Fourth, we deduplicate the dataset using SimHash (Manku et al., 2007). Finally, we manually inspect the documents gathered in the previous steps and view documents from the most popular domains by document-count and character-count, removing domains that are not high quality. We describe each of these steps in detail in the following sections.
3.3 PREFILTERING
Since there are over 200B HTML documents in Common Crawl, applying our processing over each document would require a significant amount of compute. To improve the efficiency of the pipeline, we first apply stack of pre-filters optimized for high recall to reduce the number of documents that need to be processed. Our first filters check for common mathematical strings as in Lewkowycz et al. (2022), such as the presence of tex classes, <math> tags, and the word "mathjax". See Table 11 for a full list of terms. If none of these terms are present, we search for the presence of the top 100 most-popular 197EX symbols in the text. This is done by first filtering for documents
4

---

Published as a conference paper at ICLR 2024
Thia paper wpp./// <ing axce "rcc"tttta////.//
the quantity
Supposs
map
Gra hh>
(tooon \nathbe
<senantica>
Latex.pte wtt.ppppaaatxxttaeeeeeee.. 218:::2225.. alt-* [EIX))" S>, defined as the Length of the longest aubsequence the numbers tron
Lonarightarzov 5 [/cx]] I:
identity [tex] (nathbb ( [/tex]
<annotation ....
[tex] 3"3
('diagi aysty_e Amathen [MA]
100, 0,19, [/cox] via
[1 10:14F (811))
stereogrephie projection
semanntics
c/motho
Image Equations
Special Tage
Figure 3: I formulas can be embedded in HTML documents in many ways, including in images, within arbitrary delimiters, and within special tags. Most common text-extraction pipelines do not extract BTF- code properly.
containing backslash command using a simple regular expression and then searching specifically for these BTEX symbols in the plain text from the HTMI. document. none of these symbols are found, we run the plain text through our MathScore classifier (see section 3.5.1) and keep documents that exceed a confidence threshold of 0.8. By tuning these filters and using hierarchical layers of progressively more accurate but more expensive filters, we were able to reduce the compute needed to process the dataset by several times while retaining a high recall of relevant documents.
3.4 TEXT EXTRACTION
In contrast with prior works that extract text from Common Crawl such C4 (Collins al., 2023), The Pile (Gao et al., 2020), and RefinedWeb (Penedo et al., 2023), we chose to make a mostly custom pipeline for extracting the main content from HTML documents. This is because we found that while other tools get decent performance on average over many documents on the internet, they do not work optimally on many of the most common sources of mathematical content on the web. We instead opted to build on top of Resiliparse (Bevendorff et al., 2018; 2021), a fast and efficient library built in Cython that includes performant tools for parsing HTML pages, processing their DOMs, and extracting the main content. As shown in Table 8 in the appendix, Resiliparse significantly more efficient than alternative libraries such as jusText. Another notable part of our text extraction pipeline is that randomize the parameters of the extraction to add diversity to the dataset. This includes randomizing whether we use a plain text or Markdown format for the documents and randomizing the amount of boilerplate terms required trigger line being removed.
Our text extraction pipeline consists of four stages: 18TXX extraction, text extraction, DOM process- ing, and line processing.
ETEX Extraction Lewkowyez et al. (2022) employ a relatively simple ВTę ex- traction pipeline that extracts equations from <script type="math/latexx <script encoding=""pplcccatitooo//
and <math> blocks with <annotation blocks within them and replaces these tags with the extracted equations. When we applied these filters documents from Common Crawl, noticed an extremely low number of these tags compared to what was reported. We suspect that this is due to a difference between the HTML files available within Google (Lewkowyez et al., 2022) and those available on Common Crawl. The majority of the ЕТTX on the internet is written using MathJax, where developers write equations delimited by dollar signs or other delimiters in their HTML pages and then the included javaseript code replaces these equations with properly rendered KΛTę equations within the above script tags when the page loaded. HTML documents on Common Crawl do not include the changes to the HTMI. that result from running javascript, requiring that we instead extract the 1KTІІ equations by finding delimiters ourselves. This significant challenge since we need to detect whether the page contains the required MathJax javascript code, which delimiters were chosen by the user to denote equations, and then match and extract the equations from the text the page. See Appendix D for more detailed discussion.
In order to extract MathJax, we first determine whether the page is importing the MathJax javascript code by searching for the word MathJax the page. If it is not found, additionally search for common ыTTX symbols, and if they are found, we treat the page as though is running MathJax. We use regular expressions search for code that calls the configuration function for MathJax to
5

---

Published as a conference paper at ICLR 2024
extract the delimiters used for equations. We add these delimiters to extensive list of default delimiters and treat any content between these delimiters as ETEX equations.
In addition extracting equations from MathJax, found several more ways that LTEX is encoded on the internet. These methods were discovered by filtering small portions of Common Crawl for documents that contain \frac, of the most popular ITEX commands, and making sure that our processing code supports all the different ways that math could encoded. We found that 18TiX on the internet is encoded in the following ways:
1. equation and align environments.
2. The alttext elements with special classes like tex.
3. Images from domains like latex. codecogs com often include equations encoded in the URL.
4. Special wordpress plugins.
5. <math> tags with <annotation encoding="applleettionn=s-ssss within them.
blocks
6. <math> tags with MathMI. content. We use a style sheet to convert these equations into ВТRX.
7. MathJax equations encoded in the text of the page.
The relative frequencies of the different ways math encoded can found in Table 9 in the ap- pendix.
DOM Processing After extracting the BTEX equations from the HTML, we several processing steps the DOM-tree of the HTML document. This includes removing invisible elements based on their styles, removing buttons and link clusters, annotating code, tables, and headers, and removing known problematic elements based on class or ID.
Text Extraction
We use the extract.ppl. pplalplannnn plain.text netext (ma ihaanncoo www.innnnenttrrnnemeee method Re- siliparse (Bevendorff al., 2018) to extract the main content text from the DOM following several preprocessing steps to get around common issues with their specific implementation that cause it to be overly sensitive when removing boilerplate.
Line Processing After extracting the plain text on the page using Resiliparse, we apply our own processing remove boilerplate lines based iteratively-refined set of common boilerplate phrases, remove empty headers, and escape dollar signs that are not part of 18Tμ equations.
3.5 FILTERING
We apply filtering with the goal of removing non- non-English documents (since our filters pipeline is optimized for English), removing documents that not mathematical, and removing low-quality documents that would be harmful to train a language model on. We apply the following filters in order:
1. We use a FastText language identification model (Joulin et al., 2016) to remove documents that are English.
2. We use our MathScore classifier (see section 3.5.1) to get probability that the document is mathematical. If our previous extraction step found 1dEX equations, we keep documents with probability of over 0.17. If no ЕTęX equations were found, we keep documents with a probability over 0.8.
3. We use KenLM language model (Heafield. 2011) trained on ProofPile (Azerbayev al., 2023) to get a perplexity score for each document. We remove documents with a perplexity score of more than 15,000.

---

Published as a conference paper at ICLR 2024
Classifier Input
explicit example, on Tuesday, anewer for that day will be 8$1 \timea 3+2 \tines 213 Atimes 1-1058. problem waa adopted tron a similar problem giver to me by
.. ar: explicit example, on Tuesday, our anawer for that day will be. This problem adopted from a similar problam given by
LaTeX equations
has LaTex Commands?
Text Corpus
MathScore Classifier
predict if LaTeX
present based on text
Hla LaTeX Commanda: Yes (Vtimes)
Classifier Label
Figure 4: The MathScore classifier used filtering OpenWebMath is trained to predict whether text has any of the most popular ыТТК commands based only surrounding words. This lets us include documents on the web that do not include extractable ЕTEX but still contain technical content.
3.5.1 MATII SCORE
During our filtering process, we train model to predict the probability document mathematical, which we call MathScore. We first gather dataset of hundreds of thousands documents extracted from our pipeline from an early stage of the project, and label them depending on whether they contain one of the top-100 most common DTT₃ commands. We then remove any ITTFX code from the documents and train a classifier to predict whether the documents contain one these common LTTX commands. The training process for MathScore is depicted in Figure 4. Since we remove all 18TEX code from the features fed into the model, the model needs to learn the words and phrases most commonly associated with иЯЕ content. We FastText (Joulin al., 2016) train this model, and find based on manual inspection that content with a score of under 0.2 is very unlikely contain useful mathematical content.
3.6 DEDUPLICATION
Due to the large amount of duplicate documents in Common Crawl, we apply a deduplication step to remove dear-duplicate documents. We the SimHash implementation from text-dedup (Mou et al., 2023) to deduplicate the dataset using a threshold of 0.7. We find that this threshold is high enough to remove most duplicate documents even if they have slight differences in their texts.
3.7 MANUAL. INSPECTION
Finally, we manually inspect the top domains by document count, the top domains by character count, and the longest documents in the dataset ensure that the documents high quality. We remove domains that are not high quality or clearly not mathematical by adding domains to a black- list and adding domain filters such as removing Lewkowycz et al. (2022), and removing search result pages.
profile pages, abstract-hosting websites in
DATASET ANALYSIS
Token count At 14.7B tokens, WehMaah is just below the size of Minerva's Math Web Pages (17.5B tokens) Lewkowycz et al. (2022) and significantly larger than the web part of any other dataset. OpenWebMath has around the same number LLaMA tokens as ProofPile (14.2B) (Azer- bayev et al., 2023), but note that there is very little overlap between the two datasets. Table shows the token compositions of ProofPile (Azerbayev et al., 2023) and WebMath relation to the entire Minerva (Lewkowyez al., 2022) training dataset. MettWhh brings a large number of new mathematical tokens that were previously unavailable to the open source community. Due to differences in data curation strategies, it is hard compare these datasets other than by training

---

Published as a conference paper at ICLR 2024
models them. Since not much is known about how properly filter a dataset, opted to keep as much relevant content as possible. However, future work could explore filtering Open WebMath more aggressively to further improve its quality.
Data Composition We measured the distribution of domains in Open OpenWebMathh both by document and by character count. Table and Table 3 show the top twenty most common domains by doc- ument and character count respectively. The most common sources of data tend to discussion forums, blog posts, and scientific papers. We find that the distribution of characters in the dataset is distributed over 131,206 domains, with 46% of the characters appearing in the top 100 domains.
In order get sense of the types of documents found in the dataset, analyzed 100,000 randomly sampled documents. First, we created embeddings of this data using a 1--Miiin Miniilin MInLLMLL1. 212 (Wang al., 2020) in Transffrrrssss (Reimers Gurevych, 2019). Then, we clustered these embeddings using k-Means with k = 128. Finally, we took the five closest documents to each cluster center and asked gpt-3. 5-turbo (httss//platform.oop) www.orrmmmpppiii.o.oo www.//ocs/apiirrfeeeenee
to classify each cluster Math, Physics, Statistics, Chemistry. Economics, Computer Science, or Other. We then aggregated these statistics, using the size each cluster to get an estimate of the fi- nal number of documents in each category. We note several potential issues with this methodology, including inaccuracies stemming from using an LLM for classification, and the potential that not every document within cluster belongs to the predicted category. Figure 2 shows the results of this analysis. The majority the documents in the dataset are directly related to mathematics, while the rest are spread out throughout physics, computer science, statistics, chemistry, and economics, with 12% documents not falling neatly into any of these categories.
We also used GPT to analyze the types of websites found in Open WebMath. To do this, took sample of 200 documents and asked gpt 3. 5-turbo to classify each as Forum, Paper, Blog, Reference, Educational, Reference, or other. We also gave the document URL as feature, since
Openly Available
Dataset
Tokens (in billions)
arXiv
Formal math
Web Pages
Minerva (Google)
21.0
0.0
17.5
X
ProofPile
13.0
0.2
0.8
Open Matthh
0.0
0.0
14.7
Table I:
We used the LLaMA tokenizer to count ProofPile and GetMaah tokens. A small amount of ProofPile tokens related to textbooks not included in this table.
Domain
Documents
% Documents
Domain
# Characters
% Characters
stackexchange.co
1,136,407
17.99%
stackexchange.co
1.6,5,132,784
9.55%
physiesforums.co com
300,044
4.75%
nature.com
1.529,935.838
3.14%
mathhelpforum.com
170,721
2,70%
wordpress.com
1,294,166,933
2.66%
socratic.org
133,983
2.12%
physiesforums.com
1,160,
2.38%
mathoverflow.ne
120,755
1.91%
github.io
1.49%
gradesaver.com
96,
1.52%
zbmath.org
620,019.503
1.27%
zbmath.org
91,939
1.46%
wikipedia.org
618,024,754
1.27%
wordpress.com
87,876
1.39%
groundai.com com
545,214,990
1.12%
github.io
81.
1.28%
blogspot.com
520,392.333
1.07%
brilliant.org
68,573
1.0%%
mathoverflow.net
199,102,560
1.02%
gamedev.net
50,560
0.80%
gmatclub.com
442.611,169
0.91%
openstudy.com
49,041
0.78%
gamedev.net
426,478.461
0.88%
gmatclub.com
48,812
0.77%
ac.uk
402.111.665
0.83%
blogspol.com
48,036
0.76%
aimsciences.org
344,716,386
0.71%
wikipedia.org
46,606
0.74%
mathhelpforum.com
319,215,756
0.65%
ac.uk
41,342
0.65%
deepai.org
313,512,520
0.64%
com
37,403
0.59%
ubretexts.org
282.014.149
0.58%
aimscicnces.org
36,368
0.58%
readthedocs.io
269.816,413
0.55%
libretexts.org
32,216
0.51%
tib.eu
199,7 14,017
0.41%
readthedocs.io
31,455
0.50%
mit.edu
198,487,30 7,2362
0.41%
Table Most Common Domains by Document Count.
Table 3: Most Common Domains by Character Count.
8

---

Published as a conference paper at ICLR 2024
Training Dataset
GSMBk
MATH Counting & Probability
Prealgebra
Algobra
Intermediate Algebra
Number Theory
Preculculus
Goometry
The ( (14.7B tokens)
2,20332
1.9127
1.9751
L8420
1.8193
1.9227
168477
1.9499
ProofPile (14.78 tokens)
2,2350
1,7370
1.7214
1,5799
1.6462
1
1,4838
1.7229
OpenWehMath (14.78 tokens)
1.9075
1.6285
1.6503
1.5949
1.6022
1.8894
1,4522
1.5748
Mixture (14.7B tokems)
1.8968
1,055
1.6190
1,5301
1.5719
1,6077
1.4119
1,5999
The Pile (300B tokens; Pythia 1.4B)
1.9430
1.7117
1,7560
1.6358
1.6359
1.7460
1.5191
1.7252
Table 4: We trained 1.4B parameter models for 14.7B tokens on various datasets and measured their perplexity on different mathematics benchmarks. Both DpenWehMath and a 50/50 mixture of ProofPile Azerbayev et al. (2023) and Open WehMath perform well outperforming Pythia 1.4B (Biderman et al., 2023) trained on 300B tokens of The Pile (Gao et al., 2020).
MATH Algebra-Easy maj@
Training Dataset
MATH Algebra-Easy
LILA multiarith
The Pile (14.7B tokens)
2.81%
3,93%
9.77%
ProofPile (14.7B tokens)
2.81%
3.93%
8.04%
Math (14.7B tokens)
5.62%
9.55%
16.67 %
Mixture (14.7B tokens)
5.06%
10.11 %
13.22%
The Pile (300B tokens; Pythia 1.4B)
3.93%
5.62%
21.80 %
Table 5: Accuracy on Different Math Benchmarks.
we found GPT is often able to judge the topic from the URL alone. We validated our analysis by asking GPT' to do this classification on the top 100 domain names and got similar results. Figure shows the results. The highest proportion of documents are forum pages, where users ask and answer questions related mathematical subjects. There also large proportion of educational and reference content.
Downstream Performance We experiments to find out how dataset compares to other lan- guage modeling datasets. We compare models trained on MetMahh for a single epoch (14.7B tokens) with models trained for the same number of tokens The Pile (Gao et al., 2020), general langauge modeling dataset, and ProofPile (Azerbayev et al., 2023), a dataset of both formal and in- formal mathematics. We also train 50/50 mixture of ProofPile and OpenWebMath to evaluate the performance of Open WatMath when included in mixture of other datasets, as would be common practice.
We train randomly initialized models with the same architecture as Pythia 1.4B (Biderman et al., 2023). We use batch size of IM tokens and the same hyperparameters as Pythia otherwise. These models are evaluated on collection of mathematics benchmarks which show signal on models of this size. This includes the subset of level-1 I algebra questions from MATH, coding ability, and GSM8k and MATH perplexities, which scale more smoothly than accuracies. We also compare to Pythia I.4B (Biderman al., 2023), which was trained on 300B tokens of The Pile (Gao et al., 2020) with the same architecture.
to test
Table 4 shows the results for our perplexity evaluations. There is clear performance lead for models trained with Open WebMath and the mixture seems to perform best. We suspect that since Open Web- Math contains many more examples of step-by-step thinking and discussion of basic mathematics relative to ProofPile (Azerbayev et al., 2023) (which primarily contains arXiv data), models trained on excel at mathematical problem solving. Despite Pythia being trained on over 20x the number of tokens, the performance of our models the perplexity benchmarks far exceeds its performance, showing the potential of domain-specific models for mathematics. Similarly, Table 5 shows the performance of the models MATTH-Alebrra-Paa and LII.A-multarrit. (Mishra al., 2022). Open WebMath models outperform models that were not trained on it by a significant margin.
CONCLUSION
In this paper, we describe Open WebMath, an open dataset 14.7B high quality mathematical docu- ments from the web. We extensively document our pipeline, including several novel methodologies

---

Published as a conference paper at ICLR 2024
for extracting LTEX formulas, reducing boilerplate, and filtering the dataset. Open OpenWebMath consists of high quality Q&A forum posts, educational documents, blogs, and more spread across mathemat- ics, physics, computer science, and other technical domains. We also train several models on Open- WehMath and other language modeling datasets to compare the downstream performance achievable dataset. Notably, find that models trained OpenWeb Wab outperform mod- els trained on 20x more general-domain tokens in mathematics. We hope that Open WebMath can lead to the creation of language models with improved mathematical reasoning capabilities.
by training
10

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
JB is supported by NSERC Grant (2020-06904), CIFAR Al Chairs program, Google Research Scholar Program, and Amazon Research Award. KP is supported by an NSERC PGS-D award. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, Fujitsu Limited, and companies sponsoring the Vector In- stitute for Artificial Intelligence (www. sources for model training were provided by EleutherAI and Brigham Young University. We thank Finn Paster for the graphic design for the logo. We additionally thank Ziming Chen, Yuhuai Wu, Stella Biderman, Aviya Skowron, Hailey Schoelkopf, and Sean Welleck for their helpful comments.
al/ sslparrtness) Computing re-

---

Published as a conference paper at ICLR 2024
REFERENCES
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Ha- jishirzi. Mathqa: Towards interpretable math word problem solving with operation-basee for- malisms, arXiv preprint arXiv: 13319, 2019.
Alex Andonian, Quentin Anthony, Stella Biderman, Sid Black, Preetham Gali, Leo Gao, Eric Hallahan, Josh KrameKr Connor Leahy, Lucas Nestler. Kip Parker, Michael Pieler, Jason Phang. Shivanshu Purohit, Hailey Schoelkopf, Dashiell Stander, Tri Songz, Curt Tigges, Ben- jamin Thérien, Phil Wang, and Samuel Weinbach. GPT-NeoX: Large scale autoregressive lan- guage modeling in PyTorch. GitHub Repo, 2023. URL https: / //www//wwwwwww////////////// gi github. com/ e eleutherai/ gpt- neox.
Zhangir Azerbayev, Bartosz Piotrowski, Hailey Schoelkopf, Edward W Ayers, Dragomir Radev, and Jeremy Avigad. Proofnet: Autoformalizing and formally proving undergraduate-level mathemat- ics. arXiv preprint arXiv:2302. 12433, 2023.
Adrien Barbaresi. Trafilatura: A Web Scraping Library and Command-Ling Tool for Text Discov- ery and Extraction. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing: System Demonstrations, pp. 122-131. Association for Computational Linguistics, 2021. URL
http:///aaaaaatoto////// org 2021 demo 15.
Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. Elastic ChatNoir: Search En- gine for the Web and the Common Crawl. In Leif Azzopardi, Allan Hanbury, Gabriella Pasi, and Benjamin Piwowarski (eds.), Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018), Lecture Notes in Computer Science, Berlin Heidelberg New York, March 2018 Springer.
Janek Bevendorff, Martin Potthast, and Benno Stein. FastWARC: Optimizing Large-Scale Web Archive Analytics. In Andreas Wagner, Christian Guetl, Michael Granitzer, and Stefan Voigt (eds.), 3rd International Symposium Open Search Technology (OSSYM 202/). International Open Search Symposium, October 2021.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 2023. URL https:/ proceedin ngs m.lr.pres//ve v202. /t.////iioo httbbbiderraam 3a. html.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, lla Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:::::////pppppppppsoooo neurips. wccppappe///2///// 14 1457c0d6bffc4490 wtt.:/effcc44441111oo6o0o88 atw.a4/ooooaaa11224aaaaaaaaaa Abst html.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard. Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David I Luan, Hyeontaek Lim, Barret
12

---

Published as a conference paper at ICLR 2024
Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/2204.02311, 2022. doi: 10.48.50/aXxi.224442311 URL https: http:://doi.orrree wooiioorg/0... 0011//11.8855000 att/variiivv..////v 2204. 02311.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14166 2021.
Katherine M Collins, Albert Q Jiang, Simon Frieder, Lionel Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Joshua B Tenenbaum, William Hart, al. Evaluating lan- guage models for mathematics through interactions. arXiv preprint arXiv:2306.01694 2023.
István Endrédy and Attila Novák. More effective boilerplate removal-the goldminer algorithm. Polibits, 48:79-83, 12 2013. doi: 10.17562/PB-48-UU 10.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang. Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020.
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III au2, and Kate Crawford. Datasheets for datasets, 2021.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. URL https: //github. com/ openonm open @pma__lama..
Kenneth Heafield. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshop on statistical machine translation, pp. 187-197, 2011.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. CoRR, abs/2103.033 03.77744 2021. URL htt.s:ttp/sooo //arx//aavvvvv//aa///aa org worg//aass.............o.ssb.sse 2103. 03874.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. In In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. arXiv preprint 2016.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. CoRR, abs/2305.20050, 2023. doi: 10.48550kaxiv.2205.20000 4 48550/arXiv
URL https: / http://oooo.og/...//// orq/10.
Gurmeet Singh Manku, Arvind Jain, and Anish Das Sarma. Detecting near-duplicates for web crawling. In Proceedings of the 16th International Conference on World Wide Web, WWW '07. pp. 141-150, New York, NY, USA. 2007. Association for Computing Machinery. ISBN 9781599936547. doi: 10.1145/12495221122292. URL 1242572. 1242592.
13

---

Published as a conference paper at ICLR 2024
Shen- Yun Miao, Chao-Chur Liang, and Keh- Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 975-984, 2020.
Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, et al. Lila: A unified benchmark for mathematical reasoning. arXiv preprint 2777770077, 2022.
Chenghao Mou, Chris Ha, Kenneth Enevoldsen, and Peiyuan Liu. Chenghaomoo/lxx--eedn: Refer- ence snapshot, September 2023. URL https: http::::/oooooooo..o...oo 5281122n0000.0.... 8364980.
OpenAl. Gpt-4 technical report, 2023.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080- 2094, Online, June 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/2011. naacl-main.168 URL https::////c https/:://aaaatthhoooooo wwtp:/thwwoooo..../222222212 wtg//022112 naacl-main 168.
Guilherme Penedo, Quentin Malartie, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, doi: 10448550/rx1w.23360001 (8550/arXiv 2306. 01116.
abs/2306.01 116, 2023.
URL https: /tw./iiioo// org/10.
Stanislas Polu and Πyy Sutskever. Generative language modeling for automated theorem proving. arXiv preprint arXiv:2009.0333 2020.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lor- raine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Ange- liki Lazaridou, Arthur Mensch, Jean- Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cy- prien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew J. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language mod- els: Methods, analysis & insights from training gopher. CoRR, abs/2112.11444 2021. URL https::/aarat
org/ abs 2212.111111 11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 248555545--5 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arra
org/abs 1908. 10084.
Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Transac- fions of the Association for Computational Linguisticx, 3:1-13, 2015.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, 0bs/2302.13971, 2023a. doi: 10.488550arx1v.2332.11 13971. URL petps://doo wwa.oo.oog//..... 48.50.88555aaaaa 2302. 13971.
14

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXhv:2302.1397,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307 000/2337.00288, 2023c. doi: 10,488550/arXxx.2207..22222 https: /tt::///do... wwt.o............oooo/ 48550/ arxiv 2307 09288.
URL
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers, 2020.
Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving natural language. arXiv preprint arXiv:2104.01112. 2021.
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data, arXiv preprint
2019.
15

---

Published as a conference paper at ICLR 2024
Problem Statements with 30-gram Overlap
Total Problems
Benchmark
MATH Test
348
5000
GSM8k
1319
LILA (all numerical tasks)
1310
Table 6: We measured the 30-gram overlap between OpenWehMath and common test sets. We observe that there is little overlap.
A LIMITATIONS AND FUTURE WORK
Despite the high quality of OpenWebMath, we note several limitations and avenues for future works. First, due to the high cost of extracting data from all shards on Common Crawl, we were only able run our pipeline once. Therefore, many of our choices are without empirical justification and we provide no ablation study. We also note that the nature of this particular type of dataset means that there are many subjective choices be made. For instance, what counts as a mathematical docu- ment? What is high-quality document? How do we choose the threshold for near-deduplication? For each of these, we chose several values and manually inspected few examples to choose. Due to the cost constraints, there are also practical challenges with balancing cost with accuracy when filtering and extracting text. For instance, our prefilter reduces the number of HTML documents processed to under 1% of the documents in Common Crawl, which may be too aggressive. We also note that OppeneeMaath is an English-only dataset, which limits its applications for researchers and users who speak other languages. Finally, we that Open WebMath only contains the text from math on the web, not associated figures, which can be important for solving mathematical problems (OpenAI, 2023). Future work should focus finding empirical answers the questions of what constitutes good data, creating new, efficient filtering methodologies, and extracting images inline with math text.
B OVERLAP WITH COMMON TEST SETS
We measured the n-gram overlap between OpenWebMath and a set of common benchmarks for language model reasoning. We use similar methodology to that introduced in GPT-3 (Brown et al., 2020). We used 30-gram overlap since 20-gram had large amount of false positives due com- mon mathematical formulas and sequences. Table 6 shows the overlap statistics for the MATH (Hendrycks et al., 2021), GSM8k (Cobbe al., 2021), and LILA (Mishra al., 2022) test sets. We find that there is very little overlap between any of these benchmarks. Due to observed false positives during the overlap analysis, we opt to not perform any decontamination on Open WebMath and leave memorization analysis those who training models on it.
C DOWNSTREAM PERFORMANCE
The Pile (14.7B tokens)
ProofTile (14.7B tokens)
OpenWebMath (14.7B tokens)
Misture (14.7B tokens)
The Pile
Benchmark
(300B tokens)
JJI.A-multiarith (Roy al., 2015)
9.77%
8.04%
16.67%
13.22%
21.80%
LIL.-mmatha
MILA-mmahhsppoobabill probubiiity (Amini el al., 2019)
0.00%
0.00%
0.00%
0.00%
4.17%
LILA-mathga-geon genigggggggeaaaaaaaao (Amini al. 2019)
0.13%
0.13%
0.51%
0.39%
0.00%
LIL.A-mathaa guin (Amini et al.,, 2019)
1.02%
1.0%%
0.26%
0.26%
0.26%
(Amini al., 2019)
0.85%
0.85%
2.56%
0.85%
2.56%
LILA-mathgp-other (Amini al., 2019)
0.00%
0.00%
1.10%
0.00%
0.00%
L. A-math-ma-ppyyae physics (Amini et al., 2019)
0.61%
0.82%
1.23%
1.02%
1.43%
A...---SSM-kkssttttta (Cobbe al., 2021)
1.30%
0.84%
2.21%
1.68%
0.92%
A-d. suh (Hosseini et al., 2014)
33.945
26.61%
44.95%
46.79%
53.21 %
(Miao al., 2020)
LIL LLLA-svamp-ttracuure (Patel et al., 2021)
5.35%
7.69%
10.70%
14.72%
11.71%
Table 7: Accuracy on LIIA benchmarks with numerical answers.
We tested our models trained in section 4 on a variety downstream benchmarks to evaluate their mathematical proficiency. GSM8k (Cobbe et al., 2021) is popular math benchmark that contains
16

---

Published as a conference paper at ICLR 2024
Method
Runtime (s)
Source Code Link
Resiliparse
3.99
https://githbb.cc
HTML-Text
10.75
Inscripts
19.14
http://giiibbboomm/elyyyaa//nnnt
BoilerPy
24.94
https//jtttbbbomm/mmmbbbbbbbbllt
jusText
31.17
http://jitubboommmhiibbbbeeliiiu
HTML.2Tex,
37.17
http://gghubbbccm///Aiiisstttttt
BeautifulSoup
38.42
Trafilatura
63.90
htts://ggthbbbcommmmbb///laaaaat
ExtractNet
299.67
https//githbb.omm///urraaaaaaaa
Table 8: We measured the performance of various HTML text extraction tools dataset of 1k documents. Resiliparse was by far the most efficient, leading us to choose it for use in our pipeline.
grade school math problems that are intended be solvable by a bright middle school student. We also evaluate our models on the MATH dataset (Hendrycks et al., 2021), which contains challenging mathematics problems in informal language. Finally, we evaluated on several benchmarks which have numerical answers from the LILA (Mishra et al., 2022) evaluation suite. LILA is a collection of tasks where models are tasked to write Python code to solve word problems. These problems test the ability to do mathematical problem solving while offloading arithmetic and computation to the Python interpreter.
We evaluated GSM8k and MATH by measuring the perplexity of included thain-of-thought answers in Table 4. We evaluate the accuracy of our models on the MATH Algebra-Eass subset of the MATH benchmark, which contains only Algebra questions with difficulty rating of level 1. We also evaluated on the full suite of LILA tasks which have numerical answers, making it simple to automatically evaluate the performance of our models. The task accuracies of models can be found in Table and Table 7. Note that models of this scale only get a few questions right on several of these benchmarks, leading to high variance.
D TEXT EXTRACTION
Choice Base Text Extractor When considering which HTML text-extraction library to use, we considered the efficiency, customization, and existing boilerplate reduction methods for each option. The most commonly used option, using WET files extracted by Common Crawl, was not an option since they do not deal with L5TEX correctly and offer no customization. Other options such as jusText (Endrédy& & Novák, 2013), used in The Pile Gao et al. (2020), removed boilerplate too aggressively, leading sections containing math to be discarded. Likewise, Trafilatura (Barbaresi, 2021), which was used in Ref (Penedo et al., 2023), had poor efficiency. We decided to go with Resiliparse (Bevendorff al., 2018) due its balanced boilerplate removal, f'at runtime, and efficient Common Crawl parsing tools. Table 8 shows the full results for our comparison.
KTEX Extraction EЯEX code comes in many forms throughout Common Crawl HTML files. We employed an iterative process to refine our extraction rules. First, we filtered shards of Common Crawl for documents that contain the string \frac. Then, we filtered those documents to find those which our extraction code found no extractable ISTEX. Then, we refined our code to include additional sources of math until we were confident that we had reasonable support for all formats of LTTEX in HTML documents. Table 9 shows the breakdown of different common types of LTTEX found in HTML documents.
We note that most of the ЕТTХ in WebMahhttth and across the internet is encoded using MathJax, which presents a challenge. The majority of MathJax documents use dollar sign delimiters, but most dollar signs on the web do not delimit 18IiX equations. This leaves us with a few options:
Detect the use of the MathJax script in the HTML file. If the script is imported, treat dollar signs BTTX code.
17

---

Published as a conference paper at ICLR 2024
Math Format
Percentage of Documents
Found at least one instance of math
91.42%
MathJax with delimiters (inline)
50.27%
MathJax with delimiters (display) Math found in images
23.37%
6,96%
.math-container
3.94%
MathML code
3.28%
<annotat on> withing <math> tags
2.35%
<mathjax> tags
2.24%
align environments equation environments
1.72%
1.18%
within <script> tags alttext property of <math> tags
1.01% %
0.24%
Table 9: Frequencies of different types of ETEX found in Open WebMath. The most common format of 1TEX found in Common Crawl MathJax, which uses user-defined delimiters to denote math equations. Second most common is 18TEX code within either the URL or a alt text of an img tag.
Model Size
Layers
Model Dim
Heads
Learning Rate
Batch Size
1.4 B
24
2048
16
2.0 x 10-4
1M
Table 10: Model Hyperparameters. We use the same architecture and hyperparameters, other than batch size, as Pythia 1.4B (Biderman et al., 2023).
Table 11: List of Math Keywords used in the prefiltering stage.
Math Keywords
MathJax
mathjax
<math
math-container
katex.min.com css
latex.php
codecogs
tex.cgi
class="tex"
class= tex'
Detect common ІТкХ commands in between dollar signs. If they are present, treat dollar signs as 15T code.
Use the MathScore classifier to determine whether the page looks like it is talking about math. If so, treat dollar signs as LTTX code.
The first option is not always accurate since the MathJax javascript code may be nested inside of another import or named differently depending on the website. The latter two options make up for many of these cases, but can fail to detect edge cases where math equations are present but the surrounding text does not indicate that the document mathematical. We suspect Minerva (Lewkowyez et al., 2022) gets around this issue by using HTML documents where javascript code has already been executed, in which case MathJax is converted from delimited text to explicit HTML tags that are easy detect.
18

---

Published as a conference paper at ICLR 2024
E INTERPLAY BETWEEN EXTRACTION AND FILTERING
In prior works, we noticed many cases where suboptimal HTML text extractors were used and yet text quality remains high in the dataset. This due to the interplay between extraction and filtering. Specifically, if a text extractor fails to extract the main text, gets the formatting wrong, or includes too much boilerplate in the extraction, then both the classification and perplexity filters can filter out such examples. This can lead to subtle biases in the dataset, where specific poorly-extracte websites are excluded entirely even though they do contain high quality content. In the case of making mathematical dataset, failure to extract and deal with inline 18T code properly can hurt perplexity scores and lead to these documents being filtered out. We suggest practitioners tune their text extraction pipeline diverse set of documents before applying filtering to avoid this bias.
F MODEL HYPERPARAMETERS
We trained models on 14.7B tokens using the LLaMA (Touvron et al., 2023c) tokenizer and the architecture described in Pythia (Biderman et al., 2023). We train the model using the GPT-NeoX library (Andonian et al., 2023) 8 A100 80GB GPUs. Exact hyperparameters be found in Table 10.
G EXAMPLE DOCUMENT FROM OPEN
19

---

Published as a conference paper at ICLR 2024
Pinite and Infinite-Dimensionaa Vectoo Spaces Examples 1
Pinite and Infinite-Ciment onal Vector Spaces Examples
Recall from the Finite and Infinite-Dimanaional Vector Spaces page that vector space gVş is sald be #inite-dimenslonal if there exists vectors SM v_l, V_2, .... .... V.n VI5 in (cahrm/ppan (V_i, v_2, .... V.n).. the vector space aVş cannot spanned by finite set of vectors from svs, then SVş said to be infinite -dimensional.
of
spans $vs, that
We will now look at some examples regarding firite and infinite-dimensiomal vector spaces.
ff Example 1
Show that $\wp_2 (\mathbb(R))$ is finite-dimensional vector space by finding set of three polynomials $\1 p_0(x), p_l(x), p_2(x) V that spans $\wp_2 (lmathbb(R))$. can slpp 2 (\mathbb(R))5 be spanned set of twe polyncmials? Can s'ep_2 ( be spanned by set of feur polynomials
2x Vin \wp_2 (nnathbb polynemials $p_ (x), F_)(x), 2_2 Ain \wp_2 (R))t such that for scalars $b_0, b_i, b_2 Vin (mathbb(RJ5 we have that (x) a linear combination the polynomials in SVf p_0 (x), pxlxx, p_2(x) M1, that is:
8p (x)
need find
(1)
\begin/align) Aquad a_0 a_ix + 2_2x'2 2 b_0p_0(x) b_ip_1 (x) + b_2p_2 (x) \end(align)
Noce that if we let (x) 16, $p_1 (x) - x8, and $p_ - x then:
(2)
Nbegin/alight \quad a_ a_ix + a_2x²2 0_ b_1x 2_2x"2 lend
Thus for any polynomial (p(x) a_ + a_ix a_2x"2 Ain \up_2 (\/) we have that $b_0 - e_08, $b_ a_15, and $b_ s52 Thus SY 1, x
\) spans $1mp_2 (\mathbb(R)1$.
exist any of polynomials $V Q_0)x), g_1 where $q_0 (x) e_ + e_lx + e_2x'26 and $q_1 (x) f_o f_1x f_2x that spans $ $ww__2 (\natbbb(R))s, To show this, let $c_0, e_l Vin [Rathbb[R]S. Then we'd have that:
does
(3)
(bliin \quad a_0 a_ix + a_2x²2 Oxx) a_lq__ (x) W \quad a_o
+ a_ix + a_2x'2 c_0(e_0 + e_ix e_2x22) c_l f f_2x W
quad a_c + a_ix + a_2x - e_ + e_1__ + (c_0e_1 e_ 2e_0e_2
e_ ==_2)x²2 (align)
Figure 5; An example of document from OpenWebMath. Documents in OpenWebMath use Mark- down formatting for elements like headings, tables, and lists, and preserve equation formatting in 1.T. Original source: hww.://wwwww.............aaoaaae spaces-examples-I
20

---

Published as a conference paper at ICLR 2024
H DATASHEET
We provide a datasheet for Open following the framework in Gebru et al. (2021).
MOTIVATION
For what purpose was the dataset cre- ated?
The dataset created to enable the train- ing of large language models on mathemat- ical texts, in order to improve their mathe- matical reasoning capabilities.
Who created the dataset and on behalf of which entity?
The dataset was created by the authors of this work.
Who funded the creation of the dataset?
Resources used in preparing this research were provided, in part, by the Province Canada
of Ontario, the Government through CIFAR, Fujitsu Limited, and companies sponsoring the Vector Insti- tute for Artificial Intelligence (www. vectorinstitute a ai/partners) Computing resources for model training were provided by EleutherAl and Brigham Young University.
Any other comment?
None.
COMPOSITION
What do the instances that comprise the dataset represent?
The instances are text documents extracted from mathematics-related wehpages from Common Crawl.
How many instances there in total?
In total, OpenWebMath contains 6.3 mil- lion documents.
Does the dataset contain all possible in- stances is it a sample (not necessarily random) of instances from larger set?
MettWhh doesn't contain all instances of text extracted from mathematics-related webpages from Common Crawl, as our filters can miss non-zero proportion of such webpages. However, we expect OpenWebMath to contain most of them.
What data does each instance consist of?
Each instance consists of plain text and metadata including the source URL, the snapshot date, and other extraction param- eters.
Is there label or target associated with each instance?
No.
Is any information missing from individ- ual instances?
No.
Are relationships between individual in- stances made explicit?
No.
Are there recommended data splits?
No.
Are there any errors, sources of noise, redundancies in the dataset?
Yes, small portion of the documents from OpenWebMath are not related to mathe- matics, or contain bad quality content.
21

---

Published as a conference paper at ICLR 2024
Is the dataset self-contained, does it link to or otherwise rely on external re- sources?
The dataset is entirely self- self-contained.
Does the dataset contain data that might be considered confidential?
No.
Does the dataset contain data that, if viewed directly, might be offensive, in- sulting, threatening, or might otherwise cause anxiety?
The data is filtered for quality and we do not expect that this content will be offen- sive, but since filters may be imperfect we make no guarantees.
COLLECTION
How was the data associated with each instance acquired?
The data was acquired by processing data from Common Crawl.
What mechanisms or procedures were used to collect the data?
We refer to the CommonCrawl website (commoncrawl.org) for details how they collect data.
If the dataset is a sample from larger set, what was the sampling strategy?
We use all data from Common Crawl that was available before May 2023.
Who was involved in the data collec- tion process and how were they compen- sated?
Keiran Paster and Marco Dos Santos col- lected the data and were compensated by their respective graduate programs.
Over what timeframe was the data col- lected?
OpenWebMath uses shards of Common- Crawl gathered between 2013 and 2023.
Were any ethical review processes con- ducted?
No.
PREPROCESSING
Was any preproessing/ddaainngllllll ing of the data done?
Yes. See section 3.5 for details.
Was the "raw" data saved in addition to the preprocssed//ceanel//bbee
Yes.
data?
Is the software that was used to prepro- cess/clean/label the data available?
Yes. See supplementary materials.
USES
Has the dataset been used for any tasks already?
Yes, the data was used to train 1.4B param- eter language models in section
Is there repository that links to any or all papers or systems that use the dataset?
No.
What (other) tasks could the dataset be used for?
We primarily envision that MebMathh could be useful for language model pre- training, finetuning, and evaluation.
Is there anything about the composition of the dataset or the way it was col- lected and prprocessedddlenne/lbbee that might impact future uses?
is possible that the filtering stage of the project discarded valuable documents, such as those not written in English. This makes MatMath suboptimal for cre- ating mathematical models in other lan- guages.
Are there tasks for which the dataset should not be used?
Any tasks which may considered irrespon- sible harmful.
DISTRIBUTION
22

---

Published as a conference paper at ICLR 2024
Will the dataset be distributed to third parties outside of the entity on behalf of which the dataset was created?
Yes, the dataset will be available the
Hugging Face Hub for NLP practitioners.
How will the dataset will be distributed?
We will distribute the dataset on the Hug- ging Face Hub
When will the dataset be distributed?
The dataset will be available when the pa- per is made public.
Will the dataset be distributed under copyright other intellectual prop- erty (IP) license, and/or under applica- ble terms of use (ToU)?
The public extract is made available un- der an ODC-By 1.0 license; users should also abide to the CommonCrawl ToU: http::/commonnrawlorr//mmm
Have any third parties imposed IP- based or other restrictions on the data associated with the instances?
Not to our knowledge.
Do any export controls or other regula- tory restrictions apply to the dataset or to individual instances?
Not to our knowledge.
MAINTENANCE
Who will be supportingghosting/mman- taining the dataset?
The dataset will be hosted on the Hugging Face Hub.
How can the owner/cuurator/manager of the dataset be contacted?
keirpêcs www.nto..tttooto.ooooooooooooo.. edu
Is there an erratum?
No.
Will the dataset be updated?
No.
If others want to extend/auggmet//uuii on/contribute the dataset, is there mechanism for them to do so?
No.
Table 12: Datasheet for Open Mab,Mat,,,,, following the framework in- troduced by Gebru et al. (2021).
23


Reviews:
[{"review_id": "BFOvOabszN", "reviewer": "Reviewer_oFxt", "summary": "The paper presents a large scale dataset of mathematical text (14.7B tokens, 6.3M documents) filtered from the Common Crawl dataset: OpenWebMath. The paper primarily describes the extensive pre-processing applied to obtain this dataset. To indicate the value of the dataset the paper trains a 1.4B Pythia model on the gathered data and reports perplexity on GSM8k and MATH datasets and task accuracy on MATH and LILA-multiarch. The results indicate that a model trained on OpenWebMath sees improved perplexity and better task accuracy.", "questions": "- It seems like the MATH dataset was gathered from aops.com/community/c3158_usa_contests. Is this a part of Common Crawl? What is the overlap between OpenWebMath and MATH? This is important given concerns of dataset contamination with web scale datasets: https://arxiv.org/abs/2310.10628\n- How does OpenWebMath differ from ProofPile? Are there obvious reasons why using OpenWebMath results in significantly better performance than ProofPile?\n\t- The citation for ProofPile (\"Proofnet: Autoformalizing and formally proving undergraduate-level mathematics.\") seems incorrect. Please consider adding a note of what the dataset is and its source.\n- What exactly is LILA-multiarith? It seems the LILA benchmark contains multiple different datasets, why did the evaluation here only use this one dataset in the benchmark? \n\t- In similar vein to the above comments, does the data here overlap with OpenWebMath?\n\t- Please consider citing the original source of the multiarith dataset in addition to the benchmark, its not clear what the original data is. The citation chain to the original dataset seems to be: https://arxiv.org/pdf/2210.17517.pdf (LILA) -> https://arxiv.org/pdf/1608.01413.pdf (methodological paper using the data?) -> https://aclanthology.org/Q15-1001.pdf (original data) - please verify this.\n- Please consider describing the tasks of Table 2 in more detail.\n- Please place a table or figure closer to the texts discussing it.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a well documented dataset.\n- The dataset seems to be useful for training LLMs of small-medium scale.", "weaknesses": "- The paper presents no special insight on the dataset or the effect of the processing steps applied - it only describes the pre-processing pipeline. A key aspect that would strengthen this paper is a description of the overlap (computed in some apt way: eg overlap of urls, text overlap, others) between OpenWebMath and the benchmark datasets it evaluates on - computing overlap other popular reasoning benchmarks would also be a welcome addition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NLQnyjDdZh", "reviewer": "Reviewer_2Z7u", "summary": "This paper proposes OpenWebMath, an open dataset of 14.7B high-quality mathematical documents from web text. The authors highlight the importance of pretraining on mathematical content to improve the reasoning abilities of large language models. They mention the success of the Minerva model, which was trained on a curated dataset of mathematical documents. However, existing open-source web datasets do not preserve mathematical notation accurately, limiting their usefulness. OpenWebMath aims to address this gap by providing a dataset of 14.7 billion tokens of mathematical web pages extracted from Common Crawl. The authors describe their method for extracting and filtering web pages for high-quality English mathematical documents. They also conduct experiments showing that models trained on OpenWebMath outperform models trained on larger general language datasets.", "questions": "Why there are some invisible Unicode characters and other language text in sample_dataset.jsonl? For instance the  40th and 43th lines of sample_dataset.jsonl.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes an open high-quality mathematical dataset, which can let models get a good reasoning ability in a lower computation.\n2. This paper proposes a new method for extracting and filtering mathematical text from web pages. This method is worth deeper research.", "weaknesses": "1. The authors should provide an example of a dataset in the paper.\n2. The order in which the table appears is inconsistent with the logic of the text.\n3. There are invisible Unicode characters and some text in other languages in the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SXqlalFtHf", "reviewer": "Reviewer_aKVX", "summary": "In this paper, the author introduces OpenWebMath, a new large-scale dataset for language model mathematical problem-solving. A comprehensive illustration of the dataset construction pipeline is provided and some further analysis of the dataset is conducted.", "questions": "What is the advance of OpenWebMath compared with existing datasets such as Proof-Pile?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides an opensource large-scale mathematical web text dataset which can benefit the following research.\n2. The detailed dataset construction pipeline is provided.", "weaknesses": "1. The advance of OpenWebMath compared with existing datasets such as Proof-Pile is not provided.\n2. My main concern here is that the paper is a dataset construction paper without novel technique contribution provided. I’m not very sure if this kind of paper is suitable for ICLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)