STEM-PoM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing

Authors: jiaru  zou
qing  wang
pratyush  thakur
nickvash  kani
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-19 04:14:35

Abstract: Advances in large language models (LLMs) have spurred research into enhancing their reasoning capabilities, particularly in math-rich STEM documents. While LLMs can generate equations or solve math-related queries, their ability to fully understand and interpret abstract mathematical symbols in long, math-rich documents remains limited. In this paper, we introduce STEM-PoM, a comprehensive benchmark dataset designed to evaluate LLMs' reasoning abilities on math symbols within contextual scientific text. The dataset, sourced from real-world ArXiv documents, contains over 2K math symbols classified as main attributes of variables, constants, operators, and unit descriptors, with additional sub-attributes including scalar/vector/matrix for variables and local/global/discipline-specific labels for both constants and operators. Our extensive experiments show that state-of-the-art LLMs achieve an average of 20-60\% accuracy under in-context learning and 50-60\% accuracy with fine-tuning, revealing a significant gap in their mathematical reasoning capabilities. STEM-PoM fuels future research of developing advanced Math-AI models that can robustly handle math symbols.

Main paper: 
 introduction: 
 Large language models (LLMs) have demonstrated exceptional reasoning abilities across numerous fields [17, 13, 12, 25, 15] . With the increasing shift towards applying LLMs to complex tasks [6, 23, 39] , the need for supplementary data beyond the general pre-trained datasets has become increasingly important. Among these, mathematical reasoning tasks [10, 18] have recently drawn the attention of several researchers [19, 2, 45, 28] (see Backgrounds in Appendix B). In particular, Partof-Math Tagging [43] , the mathematical analog to part-of-speech tagging [36] where mathematical tokens are classified according to a given taxonomy of attributes, continues to gain interest but lacks the foundational datasets that can support advanced NLP tasks [43, 38, 37] . In addition, integrating mathematical language into NLP models remains a substantial challenge [3, 29] , especially in the realm of document parsing [8, 24, 44] . Traditional semantic parsing methods like LateXML [31] or arXMLiv [22] often fall short when applied to math-rich documents, where precision and structured syntax are paramount [14, 32, 41] . These methods struggle to accurately perform pattern matching between abstract mathematical symbols and their corresponding XML tag notations. Similarly, recent advanced LLMs, such as ChatGPT [26] , also face difficulties in understanding and reasoning with abstract mathematical symbols due to their contextual polymorphism [35] (as Figure 3 shown).Figure 1 : The overall pipeline for constructing the STEM-POM dataset. We extract math symbols with corresponding text information to formulate the dataset. Each math symbol is initially classified into one of four primary categories based on its definition. Then, the symbol is further categorized into secondary categories by the context in which it appears or by the symbol's dimensionality. An LLM is evaluated via the first-level and second-level classification tasks.For example, in the linear equation: y = mx + p, y is categorized as a variable. Whereas in the cross-entropy loss function: L(x, y) = -N i=1 x i log(y i ), the symbol y represents the fixed target labels, which is considered a constant for a given dataset. Without the corresponding contextual information of a mathematical symbol, LLMs are unable to distinguish between different attributes of the symbol and cannot effectively process related mathematical reasoning tasks. Thus, tagging math symbols within domain-specific contexts is essential for language models.In this paper, we introduce a novel benchmark dataset, STEM-POM, designed to evaluate the reasoning capabilities of language models on mathematical symbols across different domains. The STEM-POM dataset consists of 2,109 instances extracted from a random sampling of over 10,000 arXiv manuscripts, which are math-rich documents spanning domains such as Mathematics, Physics, Chemistry, and more. We provide a mathematical symbol for each dataset instance, its order in the document, its main and sub-level attributes, and the corresponding text or expression from the original arXiv paper containing the symbol. Each mathematical symbol in the dataset is classified according to two levels of attributes [42] . The first-level attribute categorizes the symbol as variable, constant, operator, or unit descriptor. The second-level attribute further classifies the symbol into one of six types based on its first-level category: scalar, vector, matrix, local, global, or discipline-specific. Figure 1 illustrates the dataset's category distribution. To further enrich the STEM-POM dataset with additional arXiv manuscripts and other math-rich document resources, we also design the STEM-PoM Labeler, a feasible method for assisting dataset generation by automatically searching, extracting, and recording hand-labeled mathematical symbols and their corresponding context from long-text documents.We conduct thorough experiments on the STEM-POM dataset to assess the mathematical reasoning abilities of seven open-and closed-source language models, including LSTM [11] , Mixtral-8x7B [20] , Llama2-13B [40] , Llama3-80B [9] , Claude-3.5-sonnet [4] , GPT-3.5, and GPT-4o [1] with various prompting and fine-tuning strategies. The experimental results indicate that STEM-POM distinguishes the performance levels across different LLMs, offering insights into the mathematical symbol reasoning abilities of these models. In addition, we investigate and analyze the influence of context length on the ability of language models to understand mathematical symbols. , where we apply the STEM-PoM Labeler to identify and extract mathematical symbols from the ArXiv dataset. We include a detailed description of each source dataset in Appendix A.2. 
stem-pom dataset: 
 Dataset Construction. After obtaining the mathematical symbols, we categorize each symbol into different attributes and assign corresponding information to construct the STEM-POM dataset. Specifically, we first extract the file name and symbol order for each mathematical symbol. Then, for each symbol, we extract the contexts in which the symbol appears, using several predefined lengths. Following this, we manually classify each symbol into four primary categories: Variable, Constant, Operator, and Unit Descriptor. For Variable, Constant, and Operator, we further categorize them into sub-level categories. The variable is classified as Vector, Scalar, or Matrix, while Constant and Operator are categorized as Local, Global, or Discipline-Specific. Table 2 outlines the overall dataset structure. We manually examine each entry of the dataset thoroughly to ensure its robustness and correctness. We provide a detailed explanation of the dataset structure in Appendix A.3 and the definitions of each level's attributes in Appendix A.4. Additionally, the STEM-PoM Labeler is described in Appendix A.5.We summarize the key statistics of our dataset in this section. Table 1 presents the categorical statistics, including the math symbols along with their first-and second-level attributes. The distribution of Variables, Constants, Operators, and Unit Descriptors is 58.5%, 18.2%, 17.2%, and 6.1%, respectively. In addition, Figure 2 illustrates the discipline distribution of the source arXiv papers. Our dataset covers mathematical symbols from various fields, including Mathematics, Physics, Chemistry, Economics, Computer Science, etc. Table 4 : Second-level classification accuracy with full manuscript input (Ten-sentence input for LSTM). We abbreviate "Discipline Specific" as "DS".3 Experiments 
setup: 
 Models. To thoroughly evaluate our dataset across models with varying parameter sizes, we utilize the following models: LSTM framework [11] , Llama-2-13B [40] , Mixtral-8x7B-v0.1 [20] , and GPT-3.5-turbo-0125 [1] .Evaluation Metrics. We apply the Precision Accuracy as our metric for the mathematical symbol classification task, the metric can be formulated as: Precision Accuracy = Number of correct predictionsTraining & Inference Details. We evaluate several models under both pre-training and fine-tuning settings. Specifically, we train an LSTM model with varying layers and apply the LoRA method [16, 47] , a PEFT technique, to GPT-3.5. We evaluate other models in the in-context learning setting.Appendix C provides the training and model parameter details.Table 3 presents the accuracy results for different models across varying context lengths. The result shows that the small-parameter-size language model such as the LSTM struggles with lower accuracy, achieving between 18.7% and 22.6%. In contrast, larger models, such as Llama2-13B and Mistral-8x7B, show marked improvements as context length increases, with Mistral-8x7B reaching up to 53.6% on the full manuscript. In addition, Claude3.5-Sonnet achieves comparable performance with GPT-4o across all context lengths, with accuracy consistently above 63.7% and up to 66.7%. GPT-based models exhibit stronger performance overall, with GPT-3.5 achieving between 56.8% and 60.6%. GPT-4o further improves across all context lengths, outperforming other models with an overall accuracy of 68.5% with the full manuscript input. We observe that the performance gap between models remains consistent as context length increases. For instance, GPT-4o outperforms Llama3-80B by 16.0%, 14.4%, and 16.8% for context lengths of one sentence, ten sentences, and the full manuscript, respectively. This consistent performance gap suggests that larger models with more pre-trained knowledge, such as GPT-4o and Claude3.5-Sonnet, exhibit superior scalability with longer contexts. These models are able to more effectively leverage extended context lengths to distinguish between mathematical symbols and other nuanced elements in the input prompts. On the other hand, the overall performance gain from increasing context length is more pronounced in smaller models, such as Llama2-13B and Mistral-8x7B, which have less pre-trained knowledge. These models benefit more from extended context as they rely on additional information to compensate for their limited pre-training. Larger models like GPT-4o and Claude3.5-Sonnet, which come with extensive pre-trained knowledge, show relatively smaller performance gains as context length increases.. By horizontally comparing the same model performance on different sub-attribute classifications, we find that the attribute Constants are generally easier to classify compared to Variables and Operators across all sizes of models, as seen by the overall higher accuracy in Constant-related tasks. However, Matrix and DS classification continue to present challenges, even for the largest models, indicating that certain structures or content types within manuscripts remain difficult to categorize accurately at the sub-attribute level.Overall, performance across all models on both first-level and second-level classification tasks shows a clear trend of improvement with increasing context length, highlighting the importance of context for accurately classifying mathematical symbols. Additionally, both small and large-size language models show a relatively higher accuracy in identifying Unit Descriptors and Operators compared to Variables and Constants, indicating that symbols with more distinct contextual or syntactical patterns are easier for models to classify. Through the above results, we aim to gain insights into the extent to which different category attributes of mathematical symbols influence LLMs' understanding of math-rich documents by correctly classifying the symbols in real-world scenarios. We leave additional experiments in Appendix D. 
conclusion: 
 In this paper, we introduce STEM-POM, a comprehensive benchmark for evaluating language models' mathematical reasoning abilities to classify math symbols from scientific texts. The dataset includes over 2,000 math instances sourced from ArXiv papers. Extensive experiments show that the best-performing model, achieves only 73.8% and 60.5% for first and second-level Part-of-Math Tagging classification accuracy, highlighting the challenge of extracting and categorizing mathematical symbols from large text corpora. 
 
 Appendix: A.1 Frequency Analysis on math symbols A.2 Source Dataset MTDE [14] contains approximately 10,000 entries of mathematical symbol names along with their defined contexts. Each entry includes a 'short' definition and a 'long' definition. A short definition is a single-word definition, while a long definition consists of one or more words. The data was collected through random sampling from mathematical and scientific arXiv preprint manuscripts, covering a broad range of disciplines such as Physics, Computer Science, and Biology. For pre-processing, we ensured that the candidate data was generated via a corpus crawler and subsequently pruned and cleaned manually.ArXiv Paper Dataset [22] contains 1.7 million arXiv articles, spanning a wide range of disciplines, including Mathematics, Physics, Chemistry, Economics, and Computer Science. We randomly sample 10,000 articles from this raw dataset and manually ensure that each manuscript is math-rich, containing numerous mathematical expressions and symbols. For pre-processing, we utilize the STEM-PoM Labeler to extract these symbols along with their surrounding context, ensuring that the data is representative of real-world mathematical usage across various scientific fields. Additionally, File Reading: We initiate the data importing operation progress by importing files from the designated arXiv folder, ensuring a structured and accessible repository of mathematical documents for subsequent processing.Symbol Identification and Contextualization: For each file, we enumerate and display essential information: the current file being processed, the total number of symbols within, the sequence number of the current symbol, the graphical representation of the symbol, and the contextual content surrounding the symbol. This feature aids in providing a comprehensive overview and facilitates accurate symbol annotation.Annotation Interface: We then present a user-friendly interface offering a set of predefined tagging options for each symbol. Through the designed interface, we easily select the most appropriate tag from these options, standardizing the labeling process and enhancing the consistency of the dataset.Data Recording: Upon the selection of a tag for a symbol, We record this association by appending a new line to the dataset, capturing the symbol along with its assigned tag. This systematic data recording ensures the integrity and scalability of the MTCE dataset.Dataset Evaluation: After constructing the dataset, we manually evaluate the quality and applicability of the annotated data. Specifically, we process the evaluation process through the following steps: Consistency Check, Inter-annotator Agreement, Statistical Analysis, and Benchmark Testing.Part-of-Math (PoM) Tagging : The part-of-math tagging task draws inspiration from similar tagging tasks such as part-of-speech tagging [36] . In the PoM context, the goal is to label individual mathematical tokens or expressions in math formulas with their corresponding mathematical roles. Such a task is essential for enabling a deeper semantic understanding of mathematical content by machines. Several datasets or benchmarks have been developed for the part-of-tagging task, but there also remain several challenges. Abdou [43] collects mathematical content, such as formula representation and tagging for specific mathematical formula translations and verifications, including converting formulae into semantic LaTeX or testing with tools like CAS (Computer Algebra Systems). However, this focus on structured and narrow formula translations does not align with the broader, more diverse text-based tasks required to assess NLP models, due to the lack of scalability features in the collected math symbols. Ruocheng [37, 38] recently evaluated the potential of leveraging LLMs for automated annotation and Part-of-Math tagging of math symbols. However, their PoM tagging was conducted on the Digital Library of Mathematical Functions (DLMF) [27] . Since the source of math symbols is only one manuscript, the mathematical tokens collected only have a single classification type and are self-consistent. In contrast, our dataset incorporates the inherent messiness of published literature across several STEM subjects, where these domain-specific math symbols can have multiple classifications or meanings depending on the discipline and related context information.Large Language Models: Pre-trained large language models (LLMs) have become a cornerstone in modern NLP [34, 46] . These models, which assign probabilities to word sequences by decomposing the probability of a sequence into the product of conditional probabilities of subsequent tokens, have evolved significantly over time. Early approaches were based on N-gram models, but with the advent of distributed word embeddings [5, 30] , neural language models gained prominence. The scalability and performance improvements introduced by these models, along with the availability of vast textual data, have enabled the unsupervised pre-training of LLMs. These models, often referred to as foundation models [33, 23] , can then be fine-tuned on smaller, task-specific datasets to adapt them for various downstream applications. For STEM-POM, we apply one traditional sequence-based NLP model, LSTM [11] , and several recent LLMs for our dataset evaluation.Training Details In our experiments, we train an LSTM with varying numbers of layers for the mathematical symbol classification tasks. For LLMs, we choose GPT-3.5 and apply a common parameter-efficient fine-tuning (PEFT) method, LoRA [16] , to evaluate the model precision performance. We split our dataset into 80%/10%/10% for training/validation/testing sets.Model Parameters For the LSTM model, we use different layer sizes from {128, 256, 512, 1024}.The hidden state size is set to 256, the learning rate is set from {0.1, 0.01, 0.001}, the training epoch is 5, and the batch size is 16. We utilize the Adam optimizer [21] . For GPT-3.5 fine-tuning, we use the GPT-3. Table 5 : First-level classification with various context lengths on GPT-3.5 and fine-tuned GPT-3.5.Table 5 shows the comparison result on main attributes between fine-tuned and directly vanillareferenced GPT3.5. Notably, the fine-tuned GPT-3.5 model achieves an accuracy of 67.4% in the one-sentence context. However, its performance diminishes as the context length increases, with a noticeable drop to 66.9% for ten sentences and further down to 62.2% for full manuscript-length context. The decreasing trend suggests that while the fine-tuning process improves performance for shorter contexts, the model's ability to handle longer contexts is hindered.The vanilla inference results also show a similar pattern of improvement with context length, but the gap between fine-tuned and vanilla inference narrows as the context length grows. For instance, the difference in overall accuracy between fine-tuned and vanilla models is 10.6% for one-sentence contexts but only 1.6% for full manuscripts.Overall, the diminishing return for fine-tuned models with longer contexts indicates that fine-tuning amplifies sensitivity to the introduction of noisy or less relevant information when longer contexts are involved. The observation also could point to challenges in the fine-tuning process for long-context LLMs, which require more refined techniques to handle context length effectively.Model  



Reviews: review: title: title: a novel benchmark dataset
main_review: main_review: This paper presents a novel benchmark dataset named STEM-POM, which is designed to assess the reasoning capabilities of language models when interpreting mathematical symbols within intensive scientific texts. The dataset consists of 2,109 instances covering fields such as Mathematics, Physics, and Chemistry. Each instance of the dataset includes a mathematical symbol, its position within the document, its primary and secondary attributes, and the surrounding context from the original paper.

1. By concentrating on mathematical symbols and their contextual interpretations across multiple domains, the dataset addresses a crucial gap in existing resources for testing language models' comprehension of abstract mathematical concepts.

2. The evaluations in the paper highlight the challenges that these models encounter in accurately interpreting mathematical symbols.

My query is whether your datasets can distinguish between current state-of-the-art models. I hypothesize that Claude 3.5 and GPT-4o are quite proficient at this type of task. Have you ever evaluated them? (I have not even seen the results of GPT4). I am apprehensive that your dataset may become obsolete as time passes.
title: title: Well-described benchmark with potential limitations in model evaluation
main_review: main_review: Summary: 
The paper introduces a new benchmark designed to evaluate language models' mathematical reasoning abilities to classify math symbols from scientific texts.

Strengths:
- Very good description of the benchmark (dataset statistics, discipline distribution, frequency analysis, etc.) and how it was constructed (sources, STEM-PoM Labeler, etc.). 

Opportunities for improvements:
- It would be valuable to evaluate more recent and stronger models on STEM-PoM (GPT-4o, o1-preview, Llama 3, Claude, etc.) to better understand the challenge level for SOTA models. 

Final remark:
The paper introduces a new benchmark with a clear description. However, I have some concerns regarding its impact, as newer models might already perform very well, limiting its usefulness for pushing the boundaries of mathematical reasoning in AI. For this reason, I opt for a rating of 6.
title: title: A Review of STEM-PoM: Evaluating the Mathematical Reasoning of Language Models with STEM-POM
main_review: main_review: The paper "STEM-POM: Evaluating Language Models Math-Symbol Reasoning in Document Parsing" presents a novel benchmark dataset, STEM-POM, designed to evaluate the reasoning capabilities of language models (LLMs) on mathematical symbols within scientific texts. The authors aim to address the gap in LLMs' ability to understand and interpret abstract mathematical symbols in math-rich documents, a crucial aspect for advancing AI in STEM fields.

#### Quality:
The paper is well-structured and provides a comprehensive overview of the challenges faced by LLMs in mathematical reasoning. The introduction clearly outlines the motivation behind the study, and the methodology for dataset creation is detailed and robust. The experiments are thorough, covering a range of models and settings, and the results are presented with clarity.

#### Clarity:
The paper is generally clear, with a logical flow from the introduction to the conclusion. The authors provide sufficient background information and context for readers unfamiliar with the specific challenges of mathematical reasoning in NLP. However, some sections, particularly those involving technical details of the dataset construction and experimental setup, could benefit from additional clarification or simplification for broader accessibility.

#### Originality:
The introduction of the STEM-POM dataset is a significant contribution to the field, as it addresses a specific and underexplored area of LLM capabilities. The focus on mathematical symbol reasoning within the context of scientific documents is novel and timely, given the increasing application of LLMs in STEM fields.

#### Significance:
The findings of this paper are significant for the development of more advanced LLMs capable of handling complex mathematical reasoning tasks. The benchmark dataset provides a valuable resource for future research and development in this area. The results highlight the current limitations of state-of-the-art models, emphasizing the need for further advancements in this domain.

#### Pros:
- **Novel Dataset**: The creation of the STEM-POM dataset fills a critical gap in the evaluation of LLMs' mathematical reasoning abilities.
- **Comprehensive Evaluation**: The paper conducts extensive experiments across multiple models and settings, providing a clear picture of current capabilities and limitations.
- **Relevance**: The focus on mathematical reasoning in scientific texts is highly relevant to the ongoing integration of AI in STEM fields.

#### Cons:
- **Complexity**: Some sections of the paper, particularly those detailing the dataset construction and experimental setup, are complex and may be challenging for readers without a strong background in the field.
- **Limited Model Performance**: While the paper highlights the limitations of current models, it does not propose specific solutions or directions for improving LLMs' mathematical reasoning capabilities.