InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning

Authors: xiaotian  han
yiren  jian
xuefeng  hu
haogeng  liu
yiqi  wang
qihang  fan
yuang  ai
huaibo  huang
ran  he
zhenheng  yang
quanzeng  you
  bytedance
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-23 22:40:53

Abstract: Pre-training on large, high-quality datasets is essential for improving the reasoning abilities of Large Language Models (LLMs), particularly in specialized fields like mathematics. 
However, the field of Multimodal LLMs (MLLMs) lacks a comprehensive, open-source dataset for mathematical reasoning.
To fill this gap, we present InfiMM-WebMath-40B, a high-quality dataset of interleaved image-text documents. 
It consists of 24 million web pages, 85 million image URLs, and 40 billion text tokens, all carefully extracted and filtered from CommonCrawl. We outline our data collection and processing pipeline in detail.
Models trained on InfiMM-WebMath-40B demonstrate strong performance in both text-only and multimodal settings, setting a new state-of-the-art on multimodal math benchmarks such as MathVerse and We-Math.
We release our data at https://huggingface.co/datasets/Infi-MM/InfiMM-WebMath-40B.

Main paper: 
 related work: 
 LLMs have demonstrated potential in mathematical reasoning across various studies. To evaluate and enhance their capabilities, several math-specific benchmarks [11, 20, 18, 4, 40, 32, 67] and training datasets, both proprietary [45, 31, 27] and open-source [19, 55, 41, 53, 60] , have been introduced.The rise of Multimodal LLMs (MLLMs) has sparked interest in enhancing their multimodal reasoning capabilities. To support this, various evaluation benchmarks [62, 35, 24, 56, 38, 57, 34, 64, 46] and training datasets [7, 15, 51, 68, 26, 3, 30] have been developed to assess and enhance MLLMs' mathematical reasoning skills. 
dataset construction: 
 In this section, we detail the methodology used to construct InfiMM-WebMath-40B, a large-scale multimodal math dataset integrating interleaved text and image data, following approaches used in prior works [44, 29, 43] . We enhance the methodology used in the OBELICS dataset [26] by incorporating both text and corresponding image URLs. Text Extraction and Language Filtering We chose Trafilatura, a Python library widely used to extract text from web pages. While effective for text extraction, Trafilatura omits mathematical symbols and equations. Therefore, the subsequent section will outline our development of a specialized extraction tool tailored for math-related content.Following DeepSeekMath [49] , we focus on retaining only Chinese and English content when constructing our dataset. To achieve this, we apply language filtering to the CommonCrawl repositories with approximately 122 billion webpages, as shown in Figure 1 . For language detection, we employ a fastText language identification model [22] . This language filtering process significantly reduces the dataset size, lowering the number of pages from 122 billion to 57.2 billion.Extracting mathematical content from HTML presents unique challenges, as standard tools often fail to accurately capture LaTeX equations and image URLs. After evaluating various tools, we chose Resiliparse as the foundation for our development. Figure 2 shows a comparison of extraction results between Trafilatura and our enhanced version of Resiliparse.High-Recall Filtering for Mathematical Content Inspired by DeepSeekMath [49] , we trained a fastText classifier to filter mathematical content, using half a million positive samples from OpenWeb-Math [42] and negative samples from our earlier extracted content. This filtering reduced the dataset from 57.2 billion to 9.5 billion samples, prioritizing recall with a probability threshold set at 0.4.We applied MinHash [6] for content deduplication, following FineWeb's methodology [43] . Deduplication was performed within each snapshot and neighboring snapshot pairs, reducing the dataset by 43%, from 9.5 billion to 5.4 billion samples. URL deduplication further reduced the sample size to 3.9 billion.We applied a few essential filtering rules, such as removing "lorem ipsum" content, applying a punctuation ratio rule for English, filtering NSFW content, and excluding documents with Unicode errors. This step eliminated 3% of the samples, resulting in 3.8B samples.To enhance the accuracy of our labeling process, we utilized the LLaMA3-70B-Instruct model [12] , using prompt formats inspired by the FineWeb-Edu dataset [33] . This approach allowed us to score the mathematical quality of each sample on a scale from 0 to 10. The full prompt is displayed in Table 3 of Appendix.From the data remaining after rule-based filtering, we randomly sampled approximately one million entries. We assigned math quality scores and applied a threshold of 6 to select 640,000 positive samples for training our updated fastText classifier, alongside an equivalent number of 640,000 randomly selected negative samples from prior filtering steps. These positive and negative samples were combined to train the new fastText classifier. 3During fastText training, we applied data cleaning rules to optimize the model's performance for mathematical content (see Appendix D for details). For evaluation, we used all samples in the Geometry3K [35] benchmark as positive examples of mathematical content. With these refined preprocessing techniques, fastText's accuracy improved from 48.74% to 72.15Text-Only Filtering Evaluation We pretrained a deepseek-coder-1.3b-base model on the filtered text dataset and evaluated its performance on GSM8K [10] and the MMLU (STEM) [18] . Our model outperformed both OpenWebMath and DeepSeekMath, highlighting the quality of our dataset (results are shown in Appendix E).After filtering, 24 million documents with 85 million image URLs remained. We extracted image URLs from each webpage and paired them with the corresponding text, following the OBELICS format [26] . Deduplication reduced the image URLs to 23 million. Further filtering based on keyword analysis (e.g., "log", "banner", "avatar", "icon") left us with 22 million URLs, from which we successfully downloaded 14 million unique images. These images were reintegrated into the documents, resulting in 24 million records with a total of 28 million images. 
experiments: 
 Model Architectures We employ the SigLip model siglip-so400m-patch14-384 to extract visual features, a 3-layer Perceiver Resampler [21] with 64 latents to reduce the number of tokens/features per image to 64. These visual token/feature embeddings are then concatenated with text embeddings before being fed into the LLMs (DeepSeek-Coder [17] : deepseek-coder-1.3b-base and deepseek-coder-7b-v1.5).Training Details Our training data and processes involve a three-stage approach: modality alignment, continued pre-training using InfiMM-WebMath-40B, and instruction fine-tuning. Detailed training procedures are provided in the Appendix F. We refer to our resulting model as InfiMM-Math. Evaluations on We-Math Here, we compare models on the We-Math benchmarks, consisting of 6.5K visual math questions. We report results on the testmini set using four metrics: Insufficient Knowledge (IK), Inadequate Generalization (IG), Complete Mastery (CM), and Rote Memorization (RM). As shown in Table 2 , our model, InfiMM-Math, surpasses all open-source models. 
conclusions: 
 In this work, we introduced InfiMM-WebMath-40B, the first large-scale multimodal pretraining dataset for mathematical reasoning, filling a crucial gap in open-source research. Our dataset significantly enhances models' performances on key benchmarks. Integral form\n[edit]Gauss's law may be expressed as: [6] \nwhere ΦE is the electric flux through a closed surface S enclosing any volume V, Q is the total charge enclosed within V, and ε0 is the electric constant. The electric flux ΦE is defined as a surface integral of the electric field:\nwhere E is the electric field, dA is a vector representing an infinitesimal element of area of the surface,[note 2] and • represents the dot product of two vectors.  
: 
 LLMs (MLLMs). We validate the effectiveness of InfiMM-WebMath-40B through experiments on benchmarks like MathVerse [64] and WeMath [46] , showing improved performance in multimodal mathematical reasoning.Our contributions include: (1) We introduce InfiMM-WebMath-40B, the first large-scale, multimodal math dataset for pre-training, filling a critical gap in open-source research. (2) We provide a detailed preprocessing pipeline for filtering relevant content from CommonCrawl to ensure high-quality, relevant data. (3) We demonstrate the impact of InfiMM-WebMath-40B through experiments, where our models excel on multimodal mathematical benchmarks, showcasing the dataset's potential for advancing MLLM research. 
 
 Appendix: We display the full prompt used in High-Precision Filtering for Mathematical Content in Table 3 .Table 3 : Prompt for evaluating mathematical content using Llama-3-70B following FineWeb-Edu [33] .Below is an extract from a web page. Evaluate the mathematical value of the extract and its potential utility as a teaching resource in a mathematical context using the additive 10-point scoring system described below. Points accumulate based on the satisfaction of each criterion, with special attention to the presence and quality of mathematical equations: -0 points if the extract includes no mathematical content, such as only provides historical context, summarizes an article's abstract, or exclusively features a person's resume.-1-2 points if the extract offers rudimentary information on mathematical subjects, even if interspersed with irrelevant material such as advertisements or non-academic content.-2-4 points if the extract touches upon mathematical topics without rigorous adherence to academic standards and contains a mix of mathematical and non-mathematical content, or if the presentation is haphazard and the writing lacks clarity.-4-6 points if the extract presents key concepts pertinent to educational curricula and includes mathematical equations, albeit potentially non-comprehensive or alongside superfluous information. It should resemble a mathematical text, such as an introductory section of a textbook or a basic tutorial.-6-8 points if the extract is highly relevant to mathematics, is well-structured, and offers a clear exposition, including a significant number of mathematical equations and solutions. It should be akin to an in-depth textbook chapter or tutorial, with a strong focus on mathematical content and minimal unrelated information.-8-10 points if the extract exhibits exceptional mathematical merit, characterized by detailed explanations, a comprehensive array of mathematical equations, and a coherent, accessible writing style that provides profound insights into mathematical theories and applications. The extract: <EXAMPLE>. After examining the extract: -Briefly justify your total score. -Conclude with the score using the format: "mathematical score: <total points>"In this section, we examine the efficacy of two classifiers-LLM-based and fastText-based-focusing on high-precision mathematical content filtering. The comparison utilizes the DeepSeek-Coder 1.3B model, which we trained on a dataset previously introduced in Sec. High-Recall Filtering for Mathematical Content with a sequence length of 4096. This model was trained to score documents based on their relevance to mathematical content on a scale from 0 to 10.We conduct the continue pretraining of the DeepSeekCoder 1.3B model using datasets filtered by both the LLM-and fastText-based classifiers. Table 4 shows the performance results. The results highlight a length bias in the LLM-based method, which tends to favor longer documents, averaging 2,500 tokens, compared to 1,700 tokens for the FastText filter. The length bias associated with the LLM-based classifier has adversely impacted the dataset's performance on the GSM8K dataset. As indicated in the table, the LLM-filtered dataset achieved lower accuracy (17.5%) on the GSM8K dataset compared to the fastText-filtered dataset (20.2%). This decrease in performance indicates that the LLM's preference for longer documents may not align well with the requirements of datasets like GSM8K, which demand concise and precise mathematical descriptions.Given these insights, we have decided to continue utilizing the fastText classifier for high-precision filtering in our ongoing research. Nonetheless, the implications of the LLM-based classifier require further investigation to fully understand and address its biases. During fastText training, we implement data cleaning rules to optimize the model's performance for mathematical content. Mathematical texts pose unique challenges due to specialized terminology, symbols, formulas, and numeric data, which differ from typical natural language and require more refined preprocessing techniques.Our goal is to standardize and simplify the input training data while preserving essential mathematical information. Key considerations include maintaining consistency in token representation, minimizing noise from extraneous characters, and standardizing numeric values. The following steps reflect this approach:• Utilizing the SpaCy English language model (en_core_web_sm), we preprocess the input text, tokenize it, and process each token by converting it to its lowercase and lemmatized form. Common placeholders are replaced, certain non-alphanumeric characters are removed, and patterns of special characters like dashes and underscores are normalized. We also strip any unnecessary whitespace, ensuring the text is well-prepared for downstream processing.• All numeric values are replaced with the <NUM> placeholder to standardize the representation, and line breaks along with carriage returns are removed. Tokens exceeding 100 characters in English are discarded.To provide a preliminary evaluation of the quality of our filtered dataset, we continue pretraining a deepseek-coder-1.3b-base model for one epoch using the filtered mathematical content in Sec. High-Precision Filtering for Mathematical Content, excluding image URLs. We validate the effectiveness of our math-related filtering with a few-shot evaluation using the GSM8K [10] and the STEM sections of the MMLU [18] benchmark. Training is conducted for one epoch using DeepSpeed Zero2, with the AdamW optimizer, configured with a cosine learning rate scheduler, a maximum learning rate of 1e -4 , betas of (0.9, 0.95), and a weight decay of 0.1.We further continue pre-training our models using the InfiMM-WebMath-40B dataset to enhance the model's mathematical knowledge acquisition in a multimodal setting. The training is conducted for one epoch using DeepSpeed Zero2, with the AdamW optimizer, configured with a cosine learning rate scheduler, a maximum learning rate of 5e -5 , betas of (0.9, 0.95), and a weight decay of 0.1. The context length for training examples is set to 4096, with a maximum of 32 images per example. During this stage, the visual encoder remains frozen, and training focuses on learning the Perceiver Resampler module (the visual-language connector) and the LLM.Instruction Fine-tuning Stage In this stage of training, we fine-tune our models using instruction datasets, including PGPS9K [63] , Geo170k [15] , TABMWP [37] , ScienceQA [36] , Vflan [8] , Visual-WebInstruct, AI2D [25] , ChartQA [38] , DocVQA [39] , DVQA [23] , GeoQA [9] , and MAVIS [66] . We find that incorporating uni-modal text instruction datasets is crucial for enhancing the models' instruction-following capabilities. Therefore, we also include pure text instruction datasets such as Math [28] , MetaMathQA [59] , DART-Math [52] , and NuminaMath [5] . The objective of this stage is to acclimate the models to the common chat templates used in math VQA settings, thereby enabling them to better utilize the mathematical knowledge acquired in the previous stage.We freeze the vision encoder and update the parameters of the Perceiver Resampler and LLMs. As in the previous stages, training is conducted using DeepSpeed Zero2 for one epoch, with the AdamW optimizer, configured with 2000 warmup steps, a maximum learning rate of 5e -6 , betas of (0.9, 0.95), a weight decay of 0.1, and cosine decay to 5e -7 . The batch size is set to one per GPU, and the context length of the training examples is set to 4096. We utilize 32 A100-80G GPUs for the 1.3b models and 64 A100-80G GPUs for the 7b models.In this section, we conduct ablation studies on models (1) trained with and without continue pretraining (CPT), and (2) models fine-tuned on the MAVIS dataset versus a more extensive instruction fine-tuning (IFT) dataset. Specifically, we compare models trained with and without our own mathematical multi-modal pre-training dataset, InfiMM-WebMath-40B. Additionally, we evaluate two IFT dataset configurations: (a) a combination of MAVIS-Caption-to-QA, MAVIS-Existing- As shown in Table 6 , in the 1.3B model, CPT improves the MathVerse scores by 4.9 and 4.6 points when IFT is performed with MAVIS and Extended datasets, respectively. Similarly, Table 7 shows that in the 7B model, CPT improves the MathVerse scores by 4.8 and 5.3 points with MAVIS and Extended datasets, respectively. In contrast, using broader IFT datasets typically enhances model performance by approximately 2 points. These results highlight the significant mathematical capabilities imparted to the models through our InfiMM-WebMath-40B for CPT.We also employ an LLM-based classifier for high-precision filtering, Appendix C shows the comparison. 



Reviews: review: title: title: A Step Forward in Multimodal Mathematical Datasets
main_review: main_review: This paper introduces InfiMM-WebMath-40B, a comprehensive multimodal dataset designed for pre-training mathematical reasoning models. The authors outline the dataset construction process, evaluate its effectiveness through benchmark tasks, and demonstrate improvements over current methods. While this work represents a meaningful contribution to the field, some aspects of the presentation and methodology could be refined.

Presentation Wise :
1. The paper follows a clear and logical structure, typical of academic research.
2. The abstract concisely highlights the main contributions and findings.
3. Figures and tables are well-integrated and enhance the narrative.

Significance:
1. The introduction of a large-scale, open-source multimodal dataset specifically for mathematical reasoning is a notable advancement.
2. It addresses a critical resource gap for training multimodal language models (MLLMs) in mathematics.
3. The improved performance on benchmark tasks indicates the dataset's potential for advancing future research and practical applications.

Soundness:
1. The dataset construction methodology is meticulously described and appears rigorous.
2. The experimental setup and benchmark evaluations are appropriate and well-executed.
3. Ablation studies provide valuable insights into the impact of various components in the approach.
4. However, more detailed statistical analysis and error examination would strengthen the findings.

Strengths of the paper:
1. A comprehensive data curation pipeline that includes text extraction, language filtering, and domain-specific filtering.
2. Development of specialized tools for extracting mathematical content and equations from web sources.
3. The creation of a large-scale dataset (40 billion tokens, 85 million image URLs) combining text and visual data.
4. Demonstrated performance improvements across multiple benchmark tasks compared to existing models.
5. Thorough ablation studies that show the benefits of continued pre-training and instruction fine-tuning.

Weaknesses of the paper:
1. A lack of detailed error analysis or qualitative examples of model outputs.

Additional Comments and Questions:
1. How does the dataset address potential copyright issues related to web-scraped content?
2. What steps were taken to ensure data quality and remove harmful or biased content?
3. How does the performance of models trained on this dataset compare to human performance on the same tasks?
4. Are there plans to update or expand the dataset to keep it current?
5. How could this approach be adapted or applied to domains beyond mathematics?
6. How does the dataset handle varying mathematical notations or conventions across different sources?

This paper makes a contribution to the field of MLLMs in mathematical reasoning. The introduction of a large-scale, open-source dataset addresses a key resource gap and delivers promising results on benchmark tasks.
title: title: InfiMM-WebMath-40B: Advancing Multimodal Pre-Training for Enhanced Mathematical Reasoning
main_review: main_review: The paper introduces InfiMM-WebMath-40B, a large-scale, high-quality multimodal dataset designed to enhance mathematical reasoning in Large Language Models (LLMs). ​The authors detail their data collection and processing pipeline, emphasizing the integration of both text and visual data. ​

Pros
- Comprehensive Dataset: InfiMM-WebMath-40B fills a critical gap in the open-source community by providing a large-scale, high-quality multimodal dataset specifically for mathematical reasoning. ​
- Detailed Methodology: The paper offers a thorough explanation of the data collection and processing pipeline, ensuring reproducibility and transparency. ​
- Strong Experimental Results: Models trained on InfiMM-WebMath-40B achieve state-of-the-art performance on benchmarks like MathVerse and We-Math, validating the dataset's effectiveness. ​
- Innovative Filtering Techniques: The use of both high-recall and high-precision filtering methods, including the development of specialized tools for mathematical content extraction, enhances the dataset's quality. ​
- Significant Research Impact: The dataset has the potential to significantly advance research in multimodal mathematical reasoning, providing a valuable resource for the open-source community. ​

Cons
- Limited Vision-Only Performance: The models underperform in the Vision-Only category, indicating potential limitations in the vision encoder used. ​
- Resource Intensive: The model training processes are resource-intensive, which may limit accessibility for smaller research groups.
title: title: Review of "InfiMM-WebMath-40B"
main_review: main_review: # Summary of paper

The paper, "InfiMM-WebMath-40B" (omitting subtitle) contributes a large scale dataset of contributed images and web document pairs. The authors describe the collection, data cleansing process, (section 3) and tunes several models with the data collected (section 4). 

The paper is generally well written in clear English. However, for a dataset which is poised to be reused writ large by the Math-AI and the deep learning community more generally, there are a few items to consider to vastly improve the clarity and reusability of the contributed dataset. 

My comments are mostly focused on improving clarity and increasing the future citation counts of this manuscript and the contributed data. 

# Major points: 

1. There should be summary statistics on the images and the size of the documents in the contributed dataset. Both by # of images and by pixels and image file formats.

2. Given that the authors use LLMs to assess quality-and subsequently threshold filter the quality score-they should also classify the type of math problem associated with the document and associated images. In "Smart Vision-Language Reasoners" presented at ICML 2024, the authors  found that the improvement on a model using images and text varied according to the math problem category (e.g. algebra vs geometry etc) therefore having problem type categories would be beneficial for ablations. 

3. Given the cleansing process employed there is a likelihood of false positives and false negatives in the cleansing and filtration process. While estimating confusion matrices for each binary classifier employed may be infeasible, the top level classifier steps presented in figure 1 should have estimates for confusion matrix entries based on a small subset of randomly selected document/image pairs.

4. The math extraction tool used for extraction should be open sourced as part of the submission, no indication of whether this will be done was indicated, nor was an anonymized link to code provided. Providing code used is vital for reproducibility in science and furthering research in the Math-AI space. 

# Minor points: 

1. Bibliography contains multiple duplicate entries that are also not formatted correctly. 

2. Authors need to clarify the process for Chinese language processsing and how the data differ for Chinese vs English, eg. summary breakdowns. 

3. In appendix D the authors indicate they use a custom `<NUM>` token, to represent numeric values. Unless I misunderstand there is the potential to generate duplicates if the `<NUM>` tokens obscure distinctly different values from pages generated by web browsers in a templated manner. Was this considered and the deduplication was done before the `<NUM>` token overwrite? 

3. Where will the data be made publicly available?