Primitive Vision: Improving Diagram Understanding in MLLMs
Authors: Shan Zhang
Aotian Chen
Yanpeng Sun
Jindong Gu
Yi-Yu Zheng
Piotr Koniusz
Kai Zou
Anton van den Hengel
Yuan Xue
Venue: ICML 2025 poster (2025)
Keywords: Multimodal Large Language Models
Visual Mathematical Understanding
Visual Perception
Geometric Primitive Grounding
Primary area: deep_learning->foundation_models
TL;DR: We introduce PRIMITIVE, a geometry-grounded model that boosts fine-grained visual understanding in Multimodal LLMs, significantly improving performance on mathematical diagram reasoning by addressing geometric perception challenges.

Abstract: Mathematical diagrams have a distinctive structure. Standard feature transforms designed for natural images (e.g., CLIP) fail to process them effectively, limiting their utility in multimodal large language models (MLLMs). Current efforts to improve MLLMs have primarily focused on scaling mathematical visual instruction datasets and strengthening LLM backbones, yet fine‐grained visual recognition errors remain unaddressed. Our systematic evaluation on the visual grounding capabilities of state‐of‐the‐art MLLMs highlights that fine‐grained visual understanding remains a crucial bottleneck in visual mathematical reasoning (GPT-4o exhibits a 70% grounding error rate, and correcting these errors improves reasoning accuracy by 12%). We thus propose a novel approach featuring a geometrically‐grounded vision encoder and a feature router that dynamically selects between hierarchical visual feature maps. Our model accurately recognizes visual primitives and generates precise visual prompts aligned with the language model’s reasoning needs. In experiments, PRIMITIVE-Qwen2.5-7B outperforms other 7B models by 12% on MathVerse and is on par with GPT-4V on MathVista. Our findings highlight the need for better fine‐grained visual integration in MLLMs. Code is available at github.com/AI4Math-ShanZhang/SVE-Math.

Primitive Vision:
Improving Diagram Understanding in MLLMs
Shan Zhang" 12
Aotian Chen
Yanpeng Sun Jindong Gu Yu-Yu Zheng Anton van den Hengel Yuan Xue
Piotr Koniusz" 27 Kai Zou
Abstract
both art and a science (see (Tufte, 1991) for particularly insightful exposition of the art). Despite their importance in human communication, current visual feature transforms frequently misinterpret diagrams, often hindering overall model performance in the greater task.
Mathematical diagrams have distinctive struc- ture. Standard feature transforms designed for natural images (e.g.. CLIP) fail to process them ef- fectively, limiting their utility in multimodal large language models (MLLMs). Current efforts to improve MI.LMs have primarily focused scal- ing mathematical visual instruction datasets and strengthening LLM backbones, yet fine-gra grained visual recognition errors remain unaddressed. Our systematic evaluation on the visual grounding ca- pabilities state-of-the-ar MLLMs highlights that fine- tine-grained visual understanding remains a crucial bottleneck in visual mathematical reason- ing (GPT-4o exhibits 70% grounding error rate, and correcting these errors improves reasoning accuracy by 12%). We thus propose a novel ap- proach featuring a geometrically- grounded vision encoder and a feature router that dynamically se- lects between hierarchical visual feature maps. Our model accurately recognizes visual primi- tives and generates precise visual prompts aligned with the language model's reasoning needs. In experiments, PRIMITIVE- 2-wen225-77 outper- forms other 7B models by 12% on Math MathVerse and is on par with GPT-4V on Math Our findings highlight the need for better fine-grained visual integration in MLLMs. Code is available at github. MAtaa//AAAMMMMaSSaaaaa///////
The visual information represented in diagrams crucial in mathematical problem-solving. When humans perform reasoning, visual thought facilitates detailed reasoning and textual thought supports logical reasoning (Lehmann et al., 2010). While Large Language Models (LLMs) have demon- strated impressive capabilities in textual mathematical rea- soning (Yu et al., 2023; Ying et al., 2024; Azerbayev et al., 2023). their proficiency often diminishes when tasks require integrating visual data (a.k.a., Multimodal Large Language Models (MLLMs)). Recent advances in MLLM (Zhang et al., 2024b; Shi et al., 2024; Kazemi et al., 2023) rely mainly on constructing large-scale mathematical visual in- struction datasets, which are costly and labor- intoo-ittesivve to create and often involve the use advanced models like GPT-4o (OpenAI, 2023b) to generate diverse prompts for synthetic datasets. Despite extensive effort, they still strug- gle to perceive and ground basic geometric primitives in mathematical diagrams.
We systematically analyzed MLLMs' ability to describe geometric entities using a meticulously collected set of 100 images from the Geo170K dataset (Gao et al., 2023a). We then manually reviewed its responses to categorize the cor- rect descriptions and error types. As demonstrated in Fig. la, observed that GPT-4o misperceived visual information in approximately 70% of cases involving geometric entities. Correcting these visual perception errors led to 12% over- all accuracy improvement on corresponding mathematical questions (refer to Fig. 5a in the Appendix). This finding highlights that misunderstanding of geometric primitives, such as lines, circles, angles, boundaries, and junctions, re- mains critical bottleneck in the mathematical reasoning capabilities of MLLMs.
1. Introduction
Diagrams are critical to many forms of communication, from corporate reports to news media. Their construction is
AAustralian Institute for Machine Learning 10Cta6
'The Ohio State University "National University of Singapore Uniiersityy of Oxford 'NetMind.ai' Australian National Univer- sity "The Commonwealth Bank of Australia. Correspondence to: Anton van den Hengel <anton, sandenhengel@adellae www.eeedeeddddduu...... Yuan Xue <yuan @du>..
To mitigate above challenges, we propose a novel PRIMI- TIVE (PRImitives in the MathematIcal in inTerpretatlon of Visually Encoded information). which shifts the focus from scaling mathematical visual instruction datasets enhanc- ing fine-grained visual perception. Specifically, we train
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Bbux
Geometric Shape Errors
Basclinc
Angle Size Errors
10.0%
Correct
Shape
0.4
0.6
50
Junction
Bbox
3,5
30.0%
Accuracy (%)
Boundary
16.0%
Junction
40
Bhox Shape
Errors 70%
All
44.2
Length Proportion 10.0%
Shox+Shape
30
Errors
8.0%
18.0%
Boundary
5.7
Parallel/Peppendicull
8.0%
Point Errors
Relationship Errors
LLaVA
LLaVA (Inaccurato)
20
Position Relationship Errors
T-40
GPT-4o (Inaccurate)
IntemVL2
InternLL2 (Inaccurate)
LLaVA
GPT-4o
Intern
(b)
(c)
Figure 1: Performance of MILMs mathematical visual reasoning tasks from GeoQA test set. GPT-4o misperceives geometric entities in 70% of cases (Fig. 1a). Providing optimal geometric information improves performance, while excessive visual cues reduce top-1 accuracy-even below the baseline for text-only questions (Fig. 1c). Model performance highly sensitive to visual cue accuracy, with a -13.6% drop in GPT-4o's top-1 accuracy when given incorrect bounding box locations and shape names SBaox+ (Fig. lb).
an auxiliary visual encoder, namely GeoGLIP (Geometric- Grounded Language-Iman Pre-training), tailored to rec- ognize geometric primitives. While existing mathematical datasets lack bounding box and pixel-level annotations, gen- erating such training data is simple yet highly efficient, e.g., through the Matplotlib Python library. Furthermore, training protocols for visual-centric tasks are more straightforward than those for MLLMs.
GPT-4o), emphasizing the importance of relevance (see A.5 for a case study). Performance is also sensitive to visual cue accuracy, with significant drops when given ran- domly generated incorrect coordinates. Given GeoGLIP's inherent uncertainty in detecting geometric primitives, in- stead of directly prompting LLMs with primitive locations (e.g., hard coordinates; see $A.4 for ablation), we leverage global pyramid feature maps that encode essential infor- mation for pixel-to-shape detection. These visual cues dynamically selected by the designed feature router, gener- ating the so-called visual soft prompts.
It is known that AB parallel to CD. ABB-200. 2 CDE=60'. Since parallel lines have equal corresponding interior angles, we have <BED = <CDE + ZABE 80", Therefore, the answer is B
B
20
E
We evaluate PRIMITIVE on several public mathematical benchmarks, and experimental results demonstrate its su- perior performance compared to models of the same or even larger sizes. Specifically, our model outperforms other 7B-parameter models and achieves comparable results to advanced 13B parameeter MLLMs, all while using smaller dataset for visual training (40K) and 60K 1 110K for align- ment and instruct learning, compared to the 588K + 834K dataset used in MAVIS (Zhang et al., 2024b). These results highlight the effectiveness of our approach and underscore the importance of precise visual perception in mathematical visual reasoning. Our contributions are as follows:
C
60
D
Given AB parallel to CD, we need find <BED. Since ZABE. <BED. and <CED form a straight line, their sum is 180". Substitutng 2ABB-20" and 2CED-60', we have 20*+2BED 10 Solving gives 2BED=100", Thus, the answer is
Question:
As shown in the figure, AB parallel to CD, then the degree of angle BED is?
Choices:
A: 40"
B: 80" C: 90" D: 100*
GPT-40 struggles correctly interpret mathematical el- ements, leading to errors in reasoning and relational de- scriptions in LLMs. By integrating GIOGLIP, PRIMITIVE effectively grounds geometric elements and their spatial relations (e.g., ZCDE), enabling more accurate reasoning. See the Appendix for additional examples.
We design GeoGLIP to directly tackle the root cause of geometrical visual recognition errors in mathematical reasoning tasks by enhancing the visual encoder to perceive geometric primitives. Compared with large- scale mathematical instruction datasets, the training data generation process is simple yet highly efficient.
Incorporating PRIMITIVE enables MLLMs to identify the critical visual components of mathematical problems before engaging in reasoning. Our hypothesis and specific designs are inspired by observations illustrated in Fig. l and Fig. Ic Instructing MLLMs with fine-grained visual information, such as junction points and object locations, improves top- 1 accuracy compared to providing only worded questions. However, excessive visual cues reduce accuracy (-4.2% in
We design a connector mechanism that effectively inte- grates the relevant geometric visual information into the language model, boosting performance without altering the reasoning components.
2

---

Primitive Vision: Improving Diagram Understanding in MLLMs
GeoGLIP integrates seamlessly with diverse LLM backbones without modifying their reasoning compo- nents. Extensive experiments demonstrate that PRIMI- TIVE outperforms existing models of comparable and larger sizes on math benchmarks.
images are often misinterpreted as similar, highlighting sys- tematic shortcomings in CLIP's visual perception. These findings underscore the need for more specialized visual en- coding methods tailored to mathematical contexts, as well as more rigorous evaluations of MILMM' visual capabilities.
Open-Set Object Detection. Open-set object detection identifies arbitrary classes using existing bounding box an- notations and language generalization. Methods like OV- DETR (Zareian et al., 2021). ViLD (Gu et al., 2022), Det- CLIP (Yao et al., 2022), and Grounding DINO (Liu et al., 2024d) integrate language models with detection frame- works to improve category-specific detection. However, these models often struggle with small-scale object detec- tion due to insufficient fine-grained visual understanding. GLIP (Li al., 2022b) addresses this limitation by inte- grating textual information with visual region features early in the pipeline via a language-aware deep fusion mecha- nism, enhancing region-level embeddings. GLIP improves detection of smaller objects and demonstrates strong zero- shot capabilities. While GLIP's potential has been explored in various fields (Surís et al., 2023; Peng et al., 2023; Li et al., 2023), its application to mathematical reasoning, par- ticularly in precise geometric entity description and fine- grained detail identification in mathematical diagrams, re- mains largely unexplored. Our work extends these con- cepts, developing a geometric-groundee language-image pre-training model (GeoGLIP) tailored for the unique de- mands of visual mathematical reasoning.
2. Related Work
Multimodal Large Language Models for Mathematics. Large Language Models (LLMs) have garnered significant attention, with much research focused on text-based mathe- matical gooblem-solving expanding mathematical datasets and utilizing data augmentation (Yu et al., 2023; Yue et al., 2023b; 2024; Luo et al., 2023). Meanwhile, advancements in languae alignment models, such as CLIP (Rad- ford et al., 2021) and BLIP (Li et al., 2022a), have signifi- cantly progressed multimodal tasks, leading to the develop- ment of Multimodal Large Language Models (MLLMs)) (Bai et al., 2023; Gemini Team, 2023; Ye et al., 2023a; Lin et al., 2023; Gao et al., 2024; Hu et al., 2024). With the rise of instruction projects visual tokens into LLMs using a linear layer, while MiniGPT-4 (Zhu et al., 2023) reduces computation by re- sampling visual tokens into fixed-length tokens.
LLMs, LLaVA (Liu et al., 2024b)
Building on these advancements, researchers have explore visual mathematical problem-solving using MLLMs. Uni- fied frameworks like UniGeo (Chen et al., 2022a), Uni- Math (Liang et al., 2023), and Geom Verse (Kazemi et al., 2023) expand multimodal mathematical datasets and im- prove MILM performance in geometry and diverse tasks. Leveraging current datasets, G-LLaVA (Gao al., 2023a) constructed the Geo170K dataset, enhancing geometric problem-solving and surpassing GPT-4V (OpenAI, 2023b) on the Math Vista benchmark (Lu et al., 2023) with only 7B parameters. GeoGPT4V (Cai et al., 2024a) further improved model performance on Math Vista and Math sion (Wang al., 2024) by creating a high-quality geometric problem dataset using GPT-4 and GPT-4V. MAVIS (Zhang et al., 2024b) specializes in mathematical tasks with three- stage training pipeline including a math-specific vision en- coder, while Math-LLaVA (Shi et al., 2024) introduced MathV360K, a large-scale dataset with high-quality images and diverse question- -asswrr pairs to improve multimodal mathematical reasoning. These math-specific MLLMs have shown promising performance across several benchmark datasets (Yue et al., 2023a; Zhang et al., 2024a).
Junction and Boundary Detection. Junction and bound- ary detection are crucial in object recognition (Dollar et al., 2006; Maire et al., 2008; Parida et al., 1998), and can play a pivotal role in mathematical reasoning with geometric dia- grams. Junctions represent points where lines intersect, and boundaries delineate object shapes. Traditional methods like Canny edge detection (Canny, 1986) and the Hough Trans- form (Duda & Hart, 1972) struggle with complex diagrams and fine- tne-grained details required for accurate mathemati- cal reasoning. Recent deep learning approaches, such as junction detection networks (Huang et al., 2018), 1), detect key points by considering surrounding regions. Boundary detection models like Field of Junctions (FoJ) (Verbin & Zickler, 2021) use a bottom-up approach with 'generalized M-junctions' to detect contours and junctions.
3. Methods
Despite these advancements, MLLMs still face challenges in multimodal mathematical tasks, particularly due to limi- tations in visual perception (Sun et al., 2025). While CLIP remains a common choice for many mathematical MILMS and is known to benefit multimodal tasks, its limitations have also been identified. For instance, (Tong et al., 2024) examines 'CLdd-blind pairs', revealing that visually distinct
3.1. Overview
PRIMITIVE integrates visual understanding of geometric primitives with textual analysis to enhance the model's capa- bility in solving mathematical problems involving visual el- ements. As illustrated in Fig. 2, our pipeline builds upon the LLaVA-1.5 (Liu et al., 2023b) architecture (refer $A. D).
3

---

Primitive Vision: Improving Diagram Understanding in MLLMs
<text> questions & choices
Tokenizer
Large Language Model
Connector
000000
Encoder
(CLIP)
Feature Router Ih
Answer
<image>
Encoder (GeoGLIP)
22 mm
Projectors
frozen
trainable
Figure 2: The diagram presents the architecture of PRIMITIVE, highlighting key innovations in the gcometric-grounde vision encoder (GeoGLIP) and the feature router. Fine-grained visual understanding is achieved through feature pyramid comprising ge F3 pixel-to-shape detection, respectively. The feature router dynamically adjusts the contribution of these features generate visual soft prompts. These prompts are then combined with CLIP visual tokens and textual inputs before being fed into the language model (LLM), enabling accurate visual perception and enhanced mathematical reasoning.
geo* g geo (attention maps displayed on the left), capturing hierarchical visual features essential for
introducing key innovations in the GeoGLIP and visual fea- ture connector (additional implementations with DeepSeek- Math-7B--nntruuc (Shao et al., 2024), and Qwen2.5-Mait 7B-Instruct (Yang et al., 2024) are also provided). Feature maps from different layers of the GeoGLIP encoder are processed through the connector, where a feature router optimally integrates the feature pyramid into visual soft prompts by leveraging geometric information. These vi- sual prompts are then fused with CLIP vision tokens, either along the sequence dimension or the channel dimension, and aligned with text embeddings via projection layers for visual understanding. Since channel-wise fusion offers bet- ter computational efficiency and comparable performance to sequence-based fusion in our experiments, set channel- wise fusion as the default approach.
perception, we use a cross-resolution mixture to integrate low-resolution features into high-resolution ones, aiding boundary and junction detection (as discussed below).
Boundary and junction detection. GLIP-T utilizes Swin- Tiny as its backbone, producing five-level feature pyramid (1 where each level's resolution is progres- sively downscaled by factor of 2. To enrich the high- resolution features with semantic information, we first pass the high-resolution tensor g00 (as the Query) and the low- resolution tensor F1. (as the Key and Value) to a Multi- Head Self Attention (MHSA) module. The resulting feature maps are upsampled by factor of 2 and element-wise added to gen- producing F The rationale behind this design is to fully integrate the hierarchical object concepts at vari- ous scales produced by the downsampling layers with the high-resolution spatial information encoded by the initial embedding layer. Taking Fl" as input, then adopt two decoders for boundary and junction detection (see Fig. 9).
3.2. Geometric-Groundee Language-Image Pre-training
Our proposed GeoGLIP extends GLIP (Li al., 2022b) to perform shape grounding, boundary and junction detec- tion tasks with no human annotations. The architecture of GeoGLIP is shown in Fig. 8 of the Appendix. For shape grounding. we follow the same pipeline structure as the original GL.P model for bounding box detection (refer to $A.2 for pipeline details) but train on the mathematical domain. The training datasets are discussed in Sec. 3.4.
The boundary decoder consists of two successive percep- tion blocks, each comprising an upsampling operation using nearest- neighbor interpolation, followed by a 3 x convo- lution (Conv2d), batch normalization (BN2d), and ReLU activation. The final output is resized to the original image resolution using bilinear upsampling. A junction represents the intersection of lines, determined by the intersection co- ordinates and the orientations of the lines. Accordingly, our junction decoder has two branches. The first branch estimates the confidence a junction falling within each grid cell of the original image (using a 60 60 grid) and its relative position to the cell's 's center coordinates. The second
The feature pyramids in the visual encoder capture different levels of diagram information: higher-resolution features are responsible for fine-grained pixel detection, while lower- resolution features capture semantic information necessary for shape detection. To further enhance -neignd visual
4

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Image: 1000 x 1000
Boundary
Junction
Shape
Coordinates
Circle
[185,142,593,593]
B
Rectangle
[116,56,311,441]
RAJ
PiHn
NA.
776,,99.111,,68,
Text
[142,382,148,77]
Figure 3: Process for generating synthetic data with box- and pixel-level annotations, used tranin our GeoGLIP visual encoder. "Text' is random string of alphanumeric characters with length between 1 and 10, placed alongside other geometric objects, i.e., circles and rectangles. Refer to Fig. 6 in the Appendix for the detailed flow chart.
branch predicts the orientations of the intersecting lines and their confidence in falling into one of 15 evenly spaced bins within each grid cell, where each bin covers 24 degrees, ensuring the full 360-degree range is divided evenly (15 bins x 24 degrees 360 degrees). In the junetion decoder, is first processed through perception block,
Next, we explore strategies for fusing the soft prompts Foso with Feip. either sequence-wise or channel-wise. In the sequence-wise method, additional tokens are added after the CLIP tokens, extending the sequence length. In contrast, channel- wise fusion combines all visual tokens along the channel dimension, maintaining the same sequence length. To
the input F geo where it is upsampled to resolution of 60 x 60. Then, two separate Conv2D units predict the cell confidence and location, with output sizes of 60 x 60 x 1 and 60 x 60 x 2, respectively, Additionally, two other Conv2D units predict the bin confidence and orientation, both producing outputs of x 60 15. For further details, refer to training step in 1. and the pipeline in Fig. 9 of the Appendix.
Seq--wise
Cha--wise
A O
Projector
Fati
enable the subsequent L.L. to understand these visual components, the fused vi- sual tokens then fed into projection layers, which project the visual modality into the LLM'' embedding space. Fol- lowing the LLaVa-1.5 approach, tive MLP projectors (linear layer GELU + linear layer, a.k.a., mlp2x-gelu for this task. In the sequence-wise ap- proach, two separate projectors are applied for CLIP and soft prompts, respectively. For example, the projection matrices for the two linear layers, per projector, Φ and $2, have sizes of 1, 024 x4, 096 and 4, 096 x4, 096, where 096 responds the text embedding dimension. In the channel- wise approach, a single projector (4,, 5, 120x and D E R 2,006xx40006 is used to process the combined visual tokens.
geo
3.3. Connector Design
employ highly effec-
Selecting key visual cues enhances mathematical visual problem-solving, while redundant information can hinder it. To manage feature contributions, we propose dynamic Multi-Layer Perceptron
feature router R, implemented (MLP). The router takes spatially averaged pooled feature maps from each level of GeoGLIP (Fis geo E IR1x256)aa the CLIP feature map (Falip R x1,04) as input, calculating routing weights for each feature (fw" "/w""eeeae (1.2,344) E R R1x4). This serves as soft router (Puigcerver et al., 2024), with alternative router types like sparse and constant routers dis- cussed in Sec. The soft router's process defined as:
3.4. Training Samples for Visual-centric GeoGLIP
F w' MLP G Fri
w'
R([
Feii)),
To enable GeoGLIP to perceive fine ane-grained mathematical elements, we supervise its training using datasets with box- and pixel-level annotations. The model trained with a clas- sical detection loss Caet (Eq. 2). a junction loss Ljune (Eq. 3), and boundary loss Loodr (the & loss between predicted heatmap values and ground truth values). The detection loss Caet is applied to the shape grounding task, using synthetic images and FigureQA (Kahou et al., 2018) training data annotated with bounding boxes and shape names (left panel of Fig. 3) These annotations are stored a COCO-style JSON file for seamless integration with standard GLIP. See $A.3 for details on the synthetic data engine and dataset statistics (Figures 5b and 5c).
(1)
where pi is resized (9) to match the spatial dimensions of Feip and processed by an MLP to align its channel dimen- sions. The symbol @ denotes the operation flow from right to left. The scalar routing weights w' are then applied to the respective features. Depending on the fusion strategy with Feipp the final F Feco is generated either by element-wing addition of the weighted features Fgco = Σi= Fi geo where the weights w' are normalized (σ) using the function Soft- Max (i.e., Σ/=1 w' = 1), or by channel-wist concatenation of the weighted features, where the weights are processed through a Sigmoid function.
5

---

Primitive Vision: Improving Diagram Understanding in MLLMs
For boundary and junction detection tasks, we leveraged off-the-shelf models (Huang et al., 2018; Verbin & Zickler, 2021) to extract junctions and boundaries as ground truth. In addition to our synthetic samples, we incorporated the public dataset GeK (Chen et al., 2021b) and generated the corresponding ground truth. Specifically, junction labels include intersection coordinates and line orientations. As noted, each grid cell and bin are responsible for predicting the coordinates and the orientations, and we have 60 x 60 cells& 15 bins per cell. The labels are formatted as J Pa where Tij denotes the junction center coordinates, Cij E {0, } indicates the presence of a junction, lick is the angle of the k-th bin, and c²js E {0,11) the indicator for that bin (right panel of Fig. 3).
base LLM, with a 5.5% improvement and achieves com- parable top-1 accuracy to the most powerful open-source LLaVA-NeXT (8B) (Liu et al., 2024a) despite being smaller in size (19.3% vs. 21.2%) When using DeepSeek-Mand -Bt Instruct (Shao et al., 2024) as the base I.IM, our model's performance further increases by an additional +3. 33.1% 1%. No- tably, even on the challenging Math Vista benchmark, our model outperforms the advanced SPHINX-Plus 13B (Gao et al., 2024), and is on par with close-sourced GPT-4V (Ope- nAI, 2023b), as shown in Table 2. These results highlight the critical role of fine-grained visual perception in advancing mathematical reasoning capabilities in MLLMs. Tables 3 and 4 report our model's performance on plane geometry and function analysis tasks, respectively. Compared to the second-best model, MAVIS (Zhang et al., 2024b), which is trained on an 8x larger mathematical visual instruction dataset. PRIMITIVE with LLaMA2-7B demonstrates supe- rior reasoning and generalization capabilities. As menon- strated by these comparisons, constructing large instruction datasets for training MLLMs is labor-intensive and costly, whereas synthetic datasets for training visual-centric tasks offer a more scalable and efficient alternative.
(mag; Caj, (O.ss, Lg J
4. Experiments
4.1. Experimental Setup
Implementation details. Our work follows a structured three-stage training pipeline, including multi-task visual perception training for GeoGLIP, visual-language alignment, and mathematical instruction tuning for MLLMs (refer to 1 for training& & implementation details).
Baseline Comparison. The effectiveness of geometric soft visual prompts is validated through a comparison be- tween PRIMITIVE-7B and G-LLaVA across Tables 1-3. Both models utilize the same LLM (LLaMA2-7B) and instruction training dataset, ensuring a controlled evalua- tion. PRIMITIVE-7B demonstrates clear improvements with +7.7% on Mathe + +12.3% on Math Vista, and +2.8 To GeoQA. To further evaluate generalizability and effec- tiveness across different LLMs, we extend the comparison to two additional base models, e.g., Deepseek-Math-71 7B and Qwen-Math- on the Math Vista benchmark, as shown in Table 5b, where '(-)" denotes variants without geometric soft visual prompts.
In summary, for multi-modal training, we freeze GeoGLIP encoder in Stage 2 and train only the projection layers to align diagram-language pairs. In Stage 3, we unfreeze both projection layer and LLM for instructionnfollowiii tuning. Our GeoGLIP together with a pre-trained CLIP ViT-L are integrated into the language models LLAMA-2. DeepSeck- Math-7B-Instruee and Qwen2.5- -at..-- In-Itructtt Images are resized 448 448 pixels for CLIP and 1000 x 1000 for GeoGLIP. We train PRIMITIVE for one epoch for cross- modal alignment and two epochs for instruction tuning Geo170K 70K (Gao et al., 2023a), evaluating on GeoQA. For fair comparison, we train on Math V360k (Shi et al., 2024) with a batch size of 16 for one epoch, evaluating on Math- Vista (Lu et al., 2023) and MathVerss (Zhang et al., 2024a).
The above variants are math-specific models trained solely mathematical text-diagram pairs (Math & Geol 70K). In contrast, generic models like Qwen2.5-VL- 7B are pre-trained on large-scale multimodal instruction data (2TB+), enabling broader reasoning capabilities. To assess the effectiveness of our method in a generalist setting, Specifi-
Evaluation benchmarks and metric. We evaluate PRIMI- TIVE on three public benchmarks: Math Verse (Zhang et al., 2024a), GeoQA (Gao et al., 2023a), and Math Vista (Lu et al., 2023). Math Verse focuses on multi-modal reasoning, combining text and diagrams. GeoQA emphasizes geomet- ric reasoning, and Vista includes diverse tasks, e,g., IQTest, PaperQA, and IconQA, covering various problem- solving domains. The predicted answers are compared to ground truths to determine top-1 accuracy, following the respective dataset protocols.
we perform an ablation study on Qwen2.5-VL- cally, we fine-tune both the projector and LLM using LoRA on Math V360K & Geol70K, Under this setup, integrating soft visual prompts yields performance gains, whereas direct fine-tuning slightly reduces performance (Table 5c).
4.3. Ablation Analysis
4.2. Main Results
Hierarchical feature maps & feature router types. The lowest-resolution feature maps &oo (semantic-rich) assist in shape grounding (Fig. 4a). while highest fea- ture maps Fi. (geometric-rich) aid pixel-level boundary
Table 1 presents the comparison results the testmini set of Vatseerse, where PRIMITIVE-7B achieves state-of-the-art performance among all models using LLaMA2-7B as the
6

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Table I: Results testmini set of Math Verse with the accuracy metric. The highest results for closed-source and
open-source MLLMs are highlighted in red and blue respectively.
All
Text Dominant
Text Lite
Vision Intensive
Vision Dominant
Vision Only
Model
Base L.M
Acc
Acc
Acc
Acc
Acc
Acc
Baselines
Random Chance
12.4
12.4
12.4
12.4
12.4 68.3
12.4 66.7
Human
67.7
71.2
70.9
61.4
L.Mss
CharGPT (Ouyang et al.. 2022)
26.1
18.9
GPT-4 (OpenAI, 2023a)
33.6
46.5
46.5
Closed- source MI..Ms
Owen-VL-Plus (Bai et al., 2023)
11.8
15.7
11.1
9.0
13.0
10.0
Gemini-Pro (Gemini Team, 2023)
23.5
26.3
23.5
23.0
22.3
Qwen-VL-Man VL-Max (Bai et al., 2023)
30.7
24.1
24.1
GPT-4V (OpenAI, 2023b)
39.4
54.7
41.4
34.9
34.4
31.6
Open-source MLLMs
1LaVA-1.5 (Liu al., 2023a)
V3
7.6
SPHINX-Plus Plus (Gao el al., 2024)
12.2
13.9
11.6
11.6
13.5
10.4
SPHINX-MoB (Gao et al., 2024)
15.0
22.2
16.4
14.8
12.6
9.1
G-LLaVA (Gan et al., 2023a)
LJaMA2-7R
16.6
20.9
20.7
17.2
14.6
9.4
IntemLM-XC2. (Dong et al., 2024)
EatemLM2-7B (Cai 2024b)
16.5
22.3
17.0
15.7
16.4
11.0
ShareGPT4V (Chen al., 2023b)
Vicums 138
13.1
16.2
Math-LLaVA (Shi al., 2024)
Vicums
20.2
LLaVA-NeXT
2024)
1LMMA3-SB (Met, 2024)
24.9
20,9
20.8
16.1
13.8
PRIMITIVE-7B
LLaMA2-7B
21.2
26.4
23.2
22.9
18.0
15,4
PRIMITIVE-Deessete
Deepseck moth 2084)
24.3
31.1
26.9
25.6
19.3
PRIMITYVE-Qwen2.5777
Quec mane-mmath-7B 7B (Acodeny. 2023)
28.5
36.9
34.1
31.4
25.1
15.2
Table 2: Results on testmini set of Math Vista with the accuracy metric. The highest results for closed-source and open-source MILMs are highlighted.
Model
Base LLM
All
FQA
GPS
MWP
TQA
VQA
Acc
Acc
Acc
Acc
Acc
Acc
Baselines
Random Chance
17.9
18.2
21.6
3.8
19.6
26.3
Human
60.3
59.7
48.4
73.0
63.2
Closed-souree MLLMs
Qwen- VL-Plus (Bai et al., 2023)
43.3
54.6
33.5
31.2
48.1
51.4
GPT-4V (OpenAl, 2023b)
49.9
43.1
50.5
57.5
65.2
38.0
Open-source MILMs
mPL.UG-OW12 (Ye et al., 2023b)
LLaMA-7B
22.2
22.7
23.6
10.2
27.2
27.9
MiniGPT-v2 (Chen ct al., 2023a)
LLaMA2-TB (Tanron 2023)
23.1
18.6
26.0
13.4
30.4
30.2
G-LLaVA (Gao et al., 2023a)
1LaMA2/7B
25.1
19.1
48.7
3.6
25.0
28.7
LLaVA-1.5 1.5 (Liu et al., 2023a)
Visum-13B
27.7
23.8
22.7
18.9
43.0
30.2
SPHINX-PIus (Gao et ul., 2024)
139
36.7
54.6
16.4
23.1
41.8
43.0
PRIMITIVE-7H
1LaMA2 7B
37.4
31.9
53.9
29.0
41.4
30.8
PRIMITIVE-Desee Deeesskk--De PRIMITIVE-Qwen22
merteeok-matt--11 711 (Shao 2019)
48.7
37.6
63.0
48.7
48.1
35.8
Qwan-math:7B
50.4
38.7
67.3
$1.2
31.8
detection but still fail to capture fine details (Fig. 4b). Our cross-resolution mixture method, producing F=" geo achieves superior fine-detail perception (Fig. 4c). See $A.4 a detailed analysis and more variants. We then examine three
types of routers: constant, sparse, and the default soft router (Tab. 6a). The constant router assigns equal weights : 0.25 to each Fi Fg.. while the sparse router selects one feature map of Ge with wf E (0, As expected, in

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Table 3: Comparison of geometric numerical answer accuracies (%) on GeoQA.
Table 4: Comparison of model performance on Func-
tionQA of MlathVista
Model
Accuracy (%)
Model
Accuracy (%)
Random Chance
25.0
Random Chance
22.5
Frequent Guesses
32.1
Closed-source MLLMs
CoT GPT-4 (OpenAl, 2023a)
35.0
Top-10 Accuracy
PoT GPT-4 (OpenAI, 2023a)
37.0
NGS (Chen et al., 2021a)
56.9
Multimodal Bard (Google, 2023)
45.5
DPE-GPP (Cao & Xiao, 2022)
62.7
SCA-GPP (Ning et al., 2023)
64.1
GPT-4V (OpenAI, 2023b)
69.5
Top-/ Accuracy
Open-source MLLMs
Geoformer (Chen al., 2022b)
46.8
LLaVA (Liu et al., 2023b)
20.5
AdaMAA-daater V2 (Gao al., 2023h)
32.0
UniMath (Liang et al., 2023)
50.0
LLaVA-NeXT (Liu et al., 2024a)
33.7
G-LLaVA (Gao et al.. 2023a)
64.2
SPHINX-MoE (Gao et al., 2024)
34.6
MAVIS-7B (Zhang et al., 2024b)
66.7
MAVIS- (Zhang al., 2024b)
40.3
PRIMITIVE-7B
67.0
PRIMITIVE-7B
40.5
DAMMIIIVE-Deessee-...
72.8
PRIMITTIVE-Deepsseek
45.1
PRIMITIVE--Qwen255501
79.6
PRIMITIVE-QQen22552
53.3
Table 5: Comparison of top-1 accuracies (%) on GeoQA w.r.t. different visual encoder variants (Table 5a). Table 5b shows top-1 accuracy on testmini set of Matha w.r.t. with (PRIMITIVE) or without our soft visual prompts (PRIMITIVE (-)) across different LLM base models. Table 5c shows the instantiation of our method on the Qwen2.5-VL-7B.
Type
Encoders
Accuracy (%)
Model
Acc (All)
MathVista Varhe
G-LLaVA
LLaMA2-7B
25.1
Model
Acctall)
Aociall)
encoders
GLIP+CLIP
65.3
PRIMITIVE
1aMMA2-7B
Dual encoders
GeoGLIIPPCLIPP
67.0
Desppeekk))
DeepSeck-math-722
Qwen2.5.VL-78
68.2
49.2
PRIMITIVE- Deeppookk
DeepScck-math-77
Single encoder
GeoGLIP
66.1
PRIMITIVE-QWa2255NU.
PRIMITIVES (wen2..5()))
Qwen2.5
Single encoder
CLIP
64.2
PRIMITIVE Qwen2.5
Qwen2.5 meen2.5-math-T 7B
50.4
PRIMITVEEQQee22...
69.7
(a)
(b)
(c)
Table 6: Ablation results w.r.t. top-1 accuracy on GeoQA. Tab. 6a shows results for feature router types; Results for connector designs are shown in Fig. 6b.
Shared param.
Seq-
Cha.-wise
66.58
63,93
64.726432
64.
64.32
Constant It
63.9
62.8
60
(a)
(b)
(c)
F-1*
64.2
64.9
Accuracy
pd
61.1
- 61.8
Figure 4: Hierarchical features capture different levels of geometric information, dynamically weighted by the fea- ture router. Semantic-rich feature F. effectively perceives geometric shapes but lacks sufficient resolution for bound- ary localization (Fig. 4a), while geometric-rich feature Fl enhances pixel-level boundary detection (Fig. 4b). The cross-resolution feature Fl+
Sparse
geu
peu
61.9
62.3
PS
61.9
+ 61.6
Soft R
66.6
67.0
Seg. -Wist
Cha-wrise
(b)
(a)
ants, with a summary of their impact shown in Table 5a. We designed a variant that excludes the CLIP visual encoder, relying solely on our soft prompts from the GeoGLIP vi- sual encoder. This resulted in accuracy drop from 67.0% to 66. 1, though it still outperformed the CLIP encoder alone (64.2%). These results demonstrate that while CLIP lacks fine-grained perception, its general visual features still benefit text-visual alignment in MLLM training.
geo captures fine-grained visual cues (Fig. 4c), enhancing boundary detection.
the sparse router, Ft., with rich visual information, achieves the highest accuracy (hierarchical attention maps in Fig. 2). The soft router outperforms the others, demonstrating its effectiveness for dynamic routing of multiple signals.
Necessity CLIP Impact of visual encoder variants. We present a comprehensive analysis of visual encoder vari-
Impact math-specific fine-tuning for GeoGLIP. We
8

---

Primitive Vision: Improving Diagram Understanding in MLLMs
leveraged the hierarchical pyramid features from the GLIP visual encoder (fine-trained on natural image datasets, such as MS COCO). The results are shown in Table Sa. To ensure a fair comparison, we utilized feature maps with the same resolution: the first layer with the largest resolution and the last three layers with smaller resolutions. This resulted in a performance drop from 67.0% to 65.3%, with only minimal +1.1% gain over G-LLaVA. The slight boost likely stems from integrating high-resolution vision features, which are not sensitive to geometric details, as GLIP fails to detect basic geometric shapes (Fig. 10). We also fine- tune DETR- based Grounding DINO (Liu et al., 2024c) as GeoDINO, reducing t accuracy on GeoQA from 67.0% to 66. % (vs. GeoGLIP)). See $A.4 for analysis.
Table 8: Effect of Gaussian noise on GeoQA top-1 accuracy.
Model
Acc. (w/o)
Acc. (w/ Gau, Noise)
PRIMITIVE
67.0
65.7
PRIMITIVE-D DDDepsscck
72.8
71.0
PRIMITIVE E-Quen2..5
79.6
76.9
PRIMITIVE (-)
64.2
61.1
PRIMITIVE (-)- (->Deepsock
66.1
61.9
PRIMITIVE (>>-5ee2..5
72.3
67.2
show modest drops of -1.3, -1.8, and -2.7, respectively. In contrast, their CLIP-based counterparts (PRIMITIVE (-)) exhibited larger declines, indicating that CLIP encoders are more sensitive to noise and propagate less reliable visual signals to the reasoning module.
The robust of GeoGLIP. To assess the robustness of PRIM- ITIVE in junction and boundary detection, evaluated it on custom set of 500 mathematical diagrams with vari- ous visual distortions. Standard metrics from natural image tasks are adopted: junction detection is measured by recall with a confidence threshold of 0.65, and boundary detection by Intersection over Union (loU), computed from binarized maps (threshold=200) via pixel-wise logical AND/OR be- tween predictions and ground truth. We applied the follow- ing distortions: 1) Gaussian Noise: Added with variance of 0.3 to simulate noisy conditions; 2) Resolution change: Reduced the shortest image dimension from 800 to 400 pix- els: 3) Line style modification: Replaced solid lines with dashed lines.
Connector designs. We examine the impact of the number of projection experts. The default channel concatenation setup utilizes a single expert with a m1p2x mlp2x.gelu. In the multi-expert ablation, where two sequential m1p2x-gell are applied, the top-1 accuracy drops from 66.98% to 64.32% (-2.66%), as shown in Fig.6b. For sequence-wise fusion, which uses two separate projectors by default, we ab- late shared parameters across these projectors, making them act as a Lunge-projecttoo expert. Fig. 6b shows that the multi-expert setup enhances sequence-wise performance compared shared parameters (a.k.a., a single expert), boosting accuracy from 64.32% to 66.58% (+2.26%). We hypothesize that the improvement in sequence-wise fusion may stem from the added flexibility in handling hetero- geneous inputs, whereas in channel-wise fusion, it could introduce unnecessary complexity and redundancy.
As shown in Table 7, GeoGLIP exhibits strong robustness to resolution and line width variations, attributed to training- time augmentations such as randomized line width, horizon- tal flipping, center cropping, and ratio- preserving resizing. The model is more sensitive to Gaussian noise, with a 4.6% drop in junction recall and 3.1% in boundary IoU. Dashed lines increase false positive junction detections, resulting in a 3.2% drop in performance. These results highlight both the strengths and limitations of the current system and moti- vate further improvements in data generation and training strategies to better handle diverse visual distortions.
More ablation studies. See $A.5 for case studies and $A.6.2 for efficiency analysis.
5. Conclusion
We mitigate the limitations of current mathematical MLLMs by directly tackling their deficiency in perceiving geometric primitives, which are essential for visual mathematical rea- soning. We proposed PRIMITIVE, a novel approach that enhances mathematical visual reasoning by integrating geometry-aware visual encoder trained through multi-task objectives. Our method avoids the building large-scale mathematical visual instruction datasets, offering a more efficient and scalable solution. By designing a feature router that dynamically adjusts the contribution of each visual cue, we generate soft prompts that guide the language model toward better mathematical reasoning without overwhelming it with redundant or irrelevant visual data. Extensive experiments across three public mathemati- cal benchmarks validate the effectiveness of PRIMITIVE. We believe our work offers a new paradigm for solving mathematical problems in a visual context, emphasizing the critical role of fine tne-grained primitive visual grounding.
Table 7: Robustness of GeoGLIP under visual distortions.
Distortion Type
Recall %
InU %
process of
None (w/o)
85.6
92.3
Gaussian Noise
81.0
89.2
+ Resolution Change
85.2
91.9
Line Width Variation
85,9
92.3
Dushed Lines
82.4
92.7
Impact of Detection Errors on Reasoning Accuracy. To assess how detection errors affect downstream reasoning, we introduced Gaussian noise (variance=0.3) to evaluation images on the GeoQA benchmark. Table 8 reports top- 1 accuracy with or without distortion. Our three variants

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Impact Statement
for Computational Linguistics. doi: 10. 10.18653/v1/2221. findings-acl.46. URL https: /ac http://aaaaatthhocc anthology. findinggss. acl. 46.
This paper presents a novel approach aimed at advancing the field of Multimodal Large Language Models (MLLMs)
for mathematical reasoning tasks. Our work focuses on enhancing fine-gres ine-grained visual perception through the pro- posed GeoGLIP framework, addressing a critical bottleneck in current MLLMs' performance. Our systematic analysis and "apples-to-apples" comparison provide critical insights for future research, highlighting the need for more effective integration of fine- aeeigra visual understanding in MI.LMs. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, J.. Tang, J., Qin, J., Liang, X., Liu, L., Xing, E. P., and Lin, L. Geoqa: A geometric question answering bench- mark towards multimodal numerical reasoning. arXiv preprint arXiv:2 14517, 2021b.
Chen, J., Li, T., Qin, J., Lu, P., Lin, I.., Chen, C., and Liang, X. Unigco: Unifying geometry logical reason- ing via reformulating mathematical expression. ArXiv, abs/2212.027466 2022a.
Chen, J., Li, T., Qin, J., Lu, P., Lin, L., Chen, C., and Liang, X. UniGeo: Unifying geometry logical reasoning via reformulating mathematical expression. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 3313-3323, 2022b.
References
Academy, A. D. Qwen: General-purpose large lan- guage models with mathematical reasoning abilities, 2023. URL https: http://hhggggggaf/ecee ct/cco///// co/Qwen/ Qwen 2. 5- -Btth-7B.
Chen, J., Li, D. Z. S. X., Zhang, Z. L. P., Xiong, R. K. V. C. Y., and Elhoseiny, M. Minigpt-v2: Large language model as a unified interface for vision-language multi- task learning. arXiv preprint arXiv:2310.09472 2023a.
Azerbayev, Z., Schoelkopf, H., Paster, K., Santos, M. D., McAleer, S., Jiang, A. Q., Deng, J., Biderman, S., and Welleck, S. Llemma: An open language model for math- ematics. arXiv preprint arXiv:2310. 10631, 2023.
Chen, L., Li, J., wen Dong, X., Zhang, P., He, C., Wang, J., Zhao, F, and Lin, D. Sharegpt4v: Improv- ing large multi-modal models with better captions. ArXiv, abs/2311. 2023b. URL https: seppiissmanticseee wrw.coooor..rg/ooousts..... 265308687.
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: versatile vision- language model for understanding, localization, text read- ing, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
Cai, S., Bao, K., Guo, H., Zhang, J., Song, J., and Zheng, B. Geogpt4v: Towards geometric multi-modal large language models with geometric image generation. arXiv preprint ar Arxiv:2466. 1503, 2024a.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang. H., Zheng, L., Zhuang. S., Zhuang, Y.. Gonzalez, J. E., Stoica, I., and Xing, E. P. Vi- cuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https: / 1msys.mss.o org/blog/ 2023 03 30- 100-viicna//// March 2023.
Cai, 7., Cao, M., Chen, H., Chen, K., Chen, K., Chen, X., Chen, X., Chen, Z., Chen, Z., Chu, P., et al. Internlm2 technical report. arXiv preprint 172.7:7403.17297, 2024b.
Dollar, P., Tu, Z., and Belongie. S. Supervised learning of edges and object boundaries. In 2006 IEEE computer society conference on computer vision and pattern recog- nition (CVPR'06), volume 2, pp. 1964-1971. IEEF, 2006.
Canny, J. A computational approach to edge detection. IEEE Transactions on pattern analysis and machine in- telligence, 8(6):679-699) 698, 1986.
Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., et al. Internim- xcomposer2: Mastering free-form text-image composi- tion and comprehension in vision-language large model. arXiv preprint arXiv:2401. /6420, 2024.
Cao, J. and Xiao, J. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In Proceedings of the 29th International Con- ference on Computational Linguistics, pp. 1511-1520, 2022.
Duda, R. O. and Hart, P. E. Use of the hough transformation to detect lines and curves in pictures. Communications of the ACM, 15(1): 1-15, 1972.
Chen, J., Tang, J.. Qin, J., Liang, X., Liu, L.,, Xing, E., and Lin, L. GeoQA: A geometric question answering bench- mark towards multimodal numerical reasoning. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 313-523, Online, August 2021 12. Association
Gao, J., Pi, R., Zhang, J., Ye, J., Zhong, W., Wang, Y., Hong, L., Han, J., Xu, H., Li, Z., et al. G-llava: Solving geo- metric problem with multi-modal large language model. arXiv preprint arXiv:2312.1 1370, 2023a.
10

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Gao, P., Han, J., Zhang, R., Lin, Z., Geng, S., Zhou, A., Zhang, W., Lu, P., He, C., Yue, X., Li, H., and Qiao, Y. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15011 2023b.
Li, B., Zhang, K., Zhang, H., Guo, D., Zhang, R., Li, F., Zhang, Y., Liu, Z., and Li, C. Llava- next: Stronger Ilms supercharge multimodal capabilities in the wild, May 2024. URL
io/blog///////// 2024-00 000 10- 1lava seeaaanexxtttee strongerroeerr 11ms/.
Gao, P., Zhang, R., Liu, C., Qiu, L., Huang, S., Lin, W., Zhao, S., Geng, S., Lin, Z., Jin, P., et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-ing immge pre-training for unified vision- vision-language understanding and generation. In International confer- ence on machine learning, pp. 12888-12900. PMLR, 2022a.
Gemini Team, G. Gemini: a family of highly capable multi- modal models. arXiv preprint arXiv:3312.11805, 2023.
Li, L. H., Zhang, P., Zhang, H., Yang, J., Li, C., Zhong, Y., Wang, L., Yuan, L.,, Zhang, L., Hwang, J.-N., et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965-10975, 2022b.
Google. Bard, 2023. URL https:: https::/barr..gog... com/.
Gu, X., Lin, T.-Y., Kuo, W., and Cui, Y. Open-vocabulary object detection via vision and language knowledge dis- tillation, 2022. URL https: /att://axxii /aaaaaivvvo..o org/ abs/ 2104. 13921.
Li, Y., Liu, H., Wu, Q. Mu, E,, Yang, J., Gao, J., Li, C., and Lee, Y. Gligen: Open set grounded toxaeto-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 22521, 2023.
Hu, Y., Streteu, O., Lu, C.-T., Viswanathan, K., Hata, K., Luo. E., Krishna, R., and Fuxman, A. Visual program distillation: Distilling tools and programmatic reason- ing into vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9590-9601, 2024.
Liang, Z.. Yang. T., Zhang, J., and Zhang, X. Unimath: foundational and multimodal mathematical reasoner. In EMNLP, 2023.
Huang, K., Wang, Y., Zhou, Z., Ding, T., Gao, S., and Ma, Y. Learning to parse wireframes in images of man-made environments. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 626-635, 2018.
Lin, Z., Liu, C., Zhang. R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.0755 2023.
Jiang, A. Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D. S., de Las Casas, D., Hanna, E. B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L. R., Saulnier, L., Lachaux, M., Stock, P., Sub- ramanian, S., Yang, S., Antoniak, S., Scao, T. L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mixtral experts. Arxiv 04088, 2024.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning, 2023a.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In NeurIPS, 2023b.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., and Lee, Y. J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-vll htt...........b..ooooooo/o 2024 2024-01-30-11ava-n0
Kahou, S. E., Michalski, V., Atkinson, A., Kadar, A., Trischler, A., and Bengio, Y. Figureqa: An anno- tated figure dataset for visual reasoning, 2018. URL ht https:/ /aaxiviii..../////a///a///aaaava www.iv..r///...... wwt//bgg//li/ooeeeeeeeeeee/see//
07300.
Kazemi, M., Alvari, H., Anand, A., Wu, J., Chen, X., and Soricut, R. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv preprint arXiv:2312.12241, 2023.
Liu, H., Li, C., Wu, Q., and Lee, Y. Visual instruction tun- ing. Advances in neural information processing systems, 36, 2024b.
Liu, S., Zeng, Z.,, Ren, T., Li, F,, Zhang, H., Yang. J., Jiang, Q. Li, C., Yang, J., Su, H., Zhu, J., and Zhang, L. Grounding dino: Marrying dino with grounded pre- training for open-set object detection, 2024c. URL https:/ http://aww...orra////0//00/
Lehmann, D., Pascual-Marqui, R. D., Strik, W. K., and Koenig, Core networks for visual-concrete and ab- stract thought content: a brain electric microstate analysis. Neuroimage, 19(1):1073-107,, 2010.

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang. J., Jiang, Q. Li, C., Yang, J., Su, H., Zhu, J., and Zhang, L. Grounding dino: Marrying dino with grounded pre- training for open-set object detection, 2024d. URL https: / arxiv. www.ivvooooroovoooooooooo org/ abs 2303. 05499.
Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts, 2024. URL https: / /arxiviiivvvv/////i///a////aaaaa org/ http//bb///...../
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning trans- ferable visual models from natural language supervi- sion. In International Conference on Machine Learning 2021. URL https: //psi www.st.eani.... www.onc...cc.................... org Co:pooID:
Lu, P., Bansal, H., Xia, T., Liu, J., yue Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. 2023.
ArXiv, abs/2310.0222
Luo, H., Sun, Q. Xu, C., Zhao, P., Lou, J., Tao, C.,, Geng, X., Lin, Q. Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
Shao, Z., Wang. P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprini urXiv:240..03300, 2024.
Maire, M., Arbelaez, P., Fowlkes, C., and Malik, J. Using contours detect and localize junctions in natural im- ages. In 2008 IEEE Conference Computer Vision and Pattern Recognition, pp. 1-8. IEEE, 2008.
Shi, W., Hu, Z., Bin, Y., Liu, J., Yang. Y., Ng. S.-K., Bing, I.., and Lee, R. K.-W. Math-llava: Bootstrapping mathe- matical reasoning for multimodal large language models. arXiv preprint arXiv:2406. 17294, 2024.
Meta, A. Introducing meta Ilama 3: The most capable openly available Iim to date. Meta AI, 2024.
Sun, Y., Zhang, S., Tang, W., Chen, A., Koniusz, P., Zou, K., Xue, Y., and van den Hengel, A. Mathglance: Multimodal large language models do not know where to look in mathematical diagrams, 2025. URL https://arxiis org/ / abs 2500 20745.
Ning, M., Wang, Q- Huang, K., and Huang, X. A symbolic characters aware model for solving geome- try problems. In Proceedings of the 31st ACM In- ternational Conference on Multimedia, MM '23, pp- 7767-7775, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400701085. doi: 10. 1145/35581883336125557 URL 3612577
Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual inference via python execution for reasoning. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 11888-16 11898, 2023.
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08874,
Tong, S., Liu, Z., Zhai, Y., Ma. Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1069955655555 9578, 2024.
2023a.
OpenAI.
GPT-4V(ision) system card, 2023b. URL https: /opp:aa/./paa..... resarreeercc////
gpt- 4v- EBl-Av-system- card.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
Ouyang, L., Wu, Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, I.., Simens,
M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. Advances Neural Information Processing Systems, 2022.
Tufte, E. R. Envisioning information. Optometry and Vision Science, 68(4):322-324 1991.
Parida, L., Geiger, D., and Hummel, R. Junctions: Detec- tion, classification, and reconstruction. IEEE Transac- tions Pattern Analysis and Machine Intelligence, 20 77x:6876998, 1998.
Verbin, D. and Zickler, T. Field of junctions: Extracting boundary structure at low snr. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 6878, 2021.
Peng. Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Wang, K., Pan, J., Shi, W., Lu, Z., Zhan, M., and Li, H. Mea- suring multimodal mathematical reasoning with math- vision dataset. arXiv preprint arXiv:2402.1 /4804, 2024.
12

---

Primitive Vision:
Improving Diagram Understanding in MLLMS
Yang, A., Zhang, B., Hui, B., Gao, B., Yu, B., Li, C., Liu, D., Tu, J., Zhou, J., Lin, J., al. Qwen2. 5-math techni- cal report: Toward mathematical expert model via self- improvement. arXiv preprint arXiv:2409. 12122, 2024.
Zhang, R., Wei, X., Jiang, D., Zhang, Y., Guo, Z., Tong, C., Liu, J., Zhou, A., Wei, B., Zhang, S., et al. Mavis: Mathematical visual instruction tuning, arXiv preprint urXi::2407.88739 2024b.
Yao, L., Han, J., Wen, Y., Liang, X., Xu, D., Zhang, W., Li, Z., Xu, C., and Xu, H. Detelip: Dictionary-en ennicched visual-concept paralleled pre-training for open-world detection, 2022. URL https:://// ptttps://axiiy /awxiv.iivv.////ii//...///aaaavm org/ /bs/ 2209. 09407.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304 10592, 2023.
Ye, Q. Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang. J., Hu, A., Shi, P., Shi. Y., Jiang, C., Li, C., Xu, Y., Chen, H., Tian, J., Qian, Q.. Zhang, J., and Huang, F. mplug-owl: Modularization empowers large language models with multimodality, 2023a.
Ye, Q. Xu, H., Ye, J., Yan, M., Hu, A., Liu, H., Qian, Q.. Zhang. J., Huang, F., and Zhou, J. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023b.
Ying, H., Zhang, S., Li, L.., Zhou, Z., Shao, Y., Fei, Z., Ma, Y., Hong, J., Liu, K., Wang, Z., et al. InternIm- math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402 06332, 2024.
Yu, L., Jiang, W. Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint arXiv:2309.122:
2023.
Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S.. Jiang. D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang. Z., Liu, Y., Huang, W., Sun, H., Su, Y. and Chen, W. Mmmu: A massive multi multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311. 2023a.
Yue, X., Qu, X., Zhang, G., Fu, Y., Huang, W., Sun, H., Su, Y., and Chen, W. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0566 05653363.,.. 2023b.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mam- moth2: Scaling instructions from the web. arXiv preprint arXiv:2405.03548. 2024.
Zareian, A., Rosa, K. D., Hu, D. H., and Chang, S.FF. Open- vocabulary object detection using captions, 2021. URL https:: //arxiv org/ abs/ 2011 1067.
Zhang, R., Jiang, D., Zhang. Y., Lin, H., Guo, Z., Qiu, P., Zhou, A., Lu, P., Chang, K.-W., Gao, P., et al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint arxiv:2403.14624 2024a.
13

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Primitive Vision: Improving Diagram Understanding in MLLMs
A. Appendix
In this supplementary material, we illustrate the related background for our method ($ A.1), provide a detailed description
for GeoGLIP Snoometrie-rrundder Language-Imase Image Pre-training) pipeline ($ A.2), explain the process of synthetic data generation, and outline the datasets used for training GeoGLIP ($ A.3), present more ablation studies of PRIMITIVE ($ A.4), offer case studies that illustrate the practical application of our feature router mechanism and thain-of-thought (CoT) reasoning results ($ A.5), demonstrate the training details/effiiiency of PRIMITIVE ($ A.6) and examine our model's limitations while outlining potential directions for future work ($ A.7).
A.1. Background
Grounded Language-Image Pre-training (GLIP). GLIP (Li et al., 2022b) unifies detection and grounding by reformulating object detection as phrase grounding. It accepts paired image-text inputs, where the text consists of candidate detection categories, such as the 80 COCO object class names joined by '., Le., person. bicycle. car. ... toothbrush. In GLIP, object classification logits in the box classifier (traditional object detection) are replaced with word-region alignment scores, computed as the dot product between region visual features and phrase language features. GLIP operates as a two-stage detector, composed of: 1) A Swin Transformer to Region Proposal Network (RPN) to generate region coordinates, and then corresponding region features O₃ cropped from Fr; 2) A pre-trained BERT model as the language encoder, to embed the input text Xt. into token embeddings Pr. 3) A language-aware deep fusion module F that fuses O, and Pt in the last few encoding layers. The final alignment scores Sground calculated as:
visual encoder, which extracts features Fr of images X; and passes Fj
0 = RRPN(SSin(X))) P1 = BERT(XE), O₁,P% FO,, Sground O',
Large Language and Vision Assistant (LLa VA). We adopt (Large Language and Vision Assistant) LLaVA's architecture (Liu et al., 2023b) as the basis. LLaVA leverages the complementary strengths of pre-trained large language models and visual encoders to perform multi-modal tasks, consisting of a large language model fo (Vicuna (Chiang et al., 2023)), vision encoder (CLIP, ViT-L/14) (Radford et al., 2021), and a projection layer. The projection layer projects the visual embedding from the vision encoder into the text embedding space. LLaVA begins by processing an input image X2 through the CLIP visual encoder, which extracts visual features F, = CL)P(X,). To bridge the gap between the image features and the language model's word embedding space, LLaVA applies simple linear projection matrix &, converting visual features Fr into visual tokens H4,, which are compatible with the language embedding space:
H + F1, with F1 = ())
The visual tokens H, and language instruction tokens Pr, are passed into the language model for joint reasoning and language generation as foHH,,
A.2. GeoGLIP
The GeoGLIP pipeline is shown in Fig. 8, where the RPN and language-awars deep fusion details are omitted for clarity. The GeoGLIP takes image-text paired as input: an image containing geometric shapes and text listing the shape classes (i.e., 'circle. trapezoid. triangle.
line.". )) These inputs processed by the GeoGLIP encoder, which generates feature pyramids at multiple scales (F1 F2 geo geor F1 goo (. Each feature pyramid contains different levels of detail, capturing geo varying levels of geometric information. These features are then routed to three separate detectors: 1) Shape Detector: identifies and localizes basic geometric shapes by generating bounding boxes for objects within the image; 2) Junction Detector: detects junctions or intersections of geometric entities in the image; 3) Boundary Detector: identifies boundaries of geometric shapes, refining their outlines for more accurate representation. The combination of the feature pyramids with task-specific detectors allows GeoGLIP perform fine- -ne-ggained visual tasks in a mathematical context.
In Fig. 9, we illustrate detailed designs about junction and boundary detectors:
Junction Detector: The detector processes the feature Fl". through a decoder, identifying the confidence of junction points within each grid cell and their relative positions. It also predicts the orientations and confidence levels of intersecting lines within the grid, split into multiple angular bins to cover the 360-degree range.
14

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Circle
Geometric Shape
Emon
Bllipse
7.8%
Text
1000
980
Angle Size Errors
6.0
5.7%
846
14.0%
Rectangle
800
744
Correct 42.0%
11.1%
628
Length Proportion
Errors
10.0%
Errors 58%
52.5%
600
5.56
522
Triangle
5.4%
ParallelPerpendiollar Relationship Errors
8.0%
400
318
7.4%
Parallelogram
200
Position Relatiosshpp
10.2%
Erros
Point
Errors
0
Trapezoid
5
(a)
bb)
(c)
Figure 5: Fig. 5a presents the statistics of top-1 accuracy after manually correcting the visual perception errors shown in Fig. la of the main paper, which initially caused incorrect answers to mathematical questions. Specifically, we restated the output of GPT-4o w.r.t. each type of visual recognition error and calculated the accuracy of its answers. Overall, correcting these visual perception errors led an approximate 12% increase in accuracy the corresponding mathematical questions. Fig. 5b and Fig. 5c present the data statistics for synthetic math-specific datasets, including the distribution of geometric shapes/classes and the number of objects per image. Each geometric object has 70% probability of being assigned an alphanumeric text, leading to higher proportion of the 'Text' class.
Start
Synthetic Data
Create White Board
Size: 1000 1000
Pittn
Initialize
Properties
C914
Random
2W
Font Color
Line Width Range:1-5
Font Size
Text Length
2m
70%: Black
chars: 40-70
70%::14 chars
OY
RGB
chars: 10-30
chars
egg
Random
Shape Type
cirele, ellipse, rectangle, trinngle, parallelogmam tapezoid, text
No
yve1
Text Prob. 0.7
Yes
Synthetic Data
End
whot DbKZ
Add Text to Shape
drXi
OV
O7WV
Text Position
Figure 6: The flow diagram depicts the process for generating synthetic math-specific datasets, along with visualizations of the generated data samples.
Boundary Detector: It employs two successive perception blocks and upsampling operations restore the feature map to the original image resolution for boundary decoding.
Both detectors use multi-ress resolution feature maps from the GeoGLIP encoder, and specific design for each task is optimized to capture relevant geometric properties, contributing to enhanced mathematical visual reasoning. Refer to $ 3.2 of main paper for more details.
15

---

Primitive Vision: Improving Diagram Understanding in MLLMs
A.3. Training Dataset for GeoGLIP
Notably, our synthetic math-specific datasets differs from the traditional mathematical instruction datasets, and we do not create or use any additional self-generated instruction datasets beyond the publicly available Geol70K (Gao et al., 2023a) and MathV360K (Shi et al., 2024) datasets for MLLM training. Instead, our synthetic samples, annotated with box/pixel-tevel details, are exclusively utilized to train the GeoGLIP. Compared to constructing mathematical instruction datasets, our synthetic data generation process is significantyy more efficient and resource-friendly. It does not require manual labeling, as all data can be programmatically generated, e.g., through the Matplotlib Python library. In contrast, constructing instruction datasets often relies on GPT-4o to create diverse prompts and necessitates human intervention, making the process labor-intensive and costly.
Shape grounding. To generate synthetic datasety for object grounding tasks, we employ an automated Sython-based approach that efficiently creates images containing geometric shapes and text with associated bounding boxes, class labels, and annotations. The geometric categories include shapes like circles, ellipses, rectangles, triangles, parallelograms, trapezoids, and text. A variable number of basic geometric shapes and alphanumeric text elements are generated, with font sizes dynamically adjusted according to text length. These shapes are randomly distributed within a 1000 1000 pixel canvas, while text is positioned either inside or adjacent to the shapes with a 70% probability. Bounding boxes are then calculated for each shape and text element, ensuring they remain within image bounds. Finally, shapes and text are assigned class labels and coordinates, saved in COCO-style JSON file for seamless integration with standard GLIP. Fig. 6 shows the detailed flow diagram. Fig. 5b and Fig. Fig, 5c present the data statistics for synthetic math-specific datasets, including the distribution of geometric shapes and the number of objects per image. In addition to 10,000 synthetic images, we incorporated 20,672 images from the FigureQA training dataset with bounding box annotations for the shape grounding task.
Junction and boundary detection. We utilized off-the-shelf models (Huang et al., 2018; Verbin & Zickler, 2021) to extract junctions and boundary as ground truth on both our synthetic dataset and public Geo training images (9,426 diagrams). We then designed junction and boundary heads, parallel to the object detection head, with all tasks sharing the same visual encoder. Through this multi-task learning approach, our GeoGLIP can perceive rich visual information in the mathematical domain.
A.4. Quantitative analysis
GeoGLIP detection visualizations. Fig. 10 illustrates shape detection results Geo170K. FigureQA and our synthetic test dataset, while Fig. 11 presents the results for boundary and junction detection. Our detector successfully localizes basic geometric shapes and junction points while providing pixel-level boundary results in most cases. However, in complex scenarios such as overcrowded or occluded settings, the detector may struggle. Moreover, in junction detection, some failure cases involve numerous detections but with low accuracy. This issue arises due to noisy ground truth during the training phase, as manually labeling junctions is tedious and time-consuning. To address this, we use an off-the-shelf model (Huang et al., 2018) to generate ground-truie truth labels for junction detection. However, since this model was trained on images of man-made environments, it faces an out-of-domain challenge when applied to geometric objects, resulting in labels that are not fully accurate. Improving the accuracy of these labels would significantly enhance junction detection performance.
Effect of cross-resolution mixture. We designed four additional variants to demonstrate the effectiveness of our cross- resolution mixture approach. Recall that we have five feature levels {T (11.233333) with different resolutions, each with different resolutions, ranging from geometric-rich to semantic-rich information. The cross-resolution mixture aims to generate the input F1", for the boundary and junetion decoders, with the expectation that F! captures more informative visual information to benefit boundary and junetion detection tasks.
goo
goo
Using boundary detection as example, we first used the semantic-rich F5 as input to the boundary decoder. As shown in Fig. 7a, the decoder fails to generate clear boundaries, resulting in a blurred output. Next, we used the geometric-rich Fl.. which performs better (Fig. 7b), showing some visible boundaries. To further enhance the results, we applied cross-resolution attention mechanism (classic Multi-Head Sclf-Attention, MHSA) between F2 and Fd.. improving boundary detection as seen in Fig. 7d. Since boundary detection benefits from geometric-rich information, we upsampled the cross-correlated features by a factor of 2 and added them element-wise with goo- producing the best visualization results, especially for finer details (Fig. 7c). Finally, to assess the importance of cross-resolution attention, we replaced it with element-wise addition. As expected, the boundaries became blurred (Fig. 7c) due to the reduced receptive field. Replacing addition with the attention mechanism yields similar boundary results but decreases shape grounding performance from 95.3% to 92.4% mAP our synthetic test set. Therefore, our mixture process integrates both cross-resolution attention and
16

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Figure 7: Qualitative boundary visualization results. Semantic-rich features with the lowest resolution lead to blurred boundaries (Fig. 7a), while geometric features with the highest resolution improve clarity (Fig. 7b). The cross-resolution mixture yields the best results (Fig. 7e), compared with using either element-wise addition (Fig. 7c) or MHSA alone (Fig. 7d). Zoom in for best view.
27. M
E
Q
0
(a)
(b)
(c)
(d)
(c)
addition operations.
Necessity of GLIP architecture. GLIP is an open-set object detector capable of identifying arbitrary classes by matching visual features with corresponding language embeddings. Unlike traditional object detectors with learnable classification weights, GLIP's multi-modal architecture offers greater generality to novel objects and surpasses previous traditional object detectors. To evaluate alternatives, we replaced GLIP with another open-set object detector, Grounding DINO (Liu al., 2024c), and fine-tuned it our math-specific dataset. Experimentally, we found that Grounding DINO struggles to effectively detect small-scale geometric primitives. We hypothesize that this limitation stems from architectural differences. Grounding DINO, as DETR-based detector, relies exclusively the last-layer features of its visual encoder for cross- attention with query embeddings during final detection. In contrast, GLIP, as a Bester-RCNN -asee detector, leverages multí-scale features for both bounding box regression and classification, enabling superior small-object detection capabilities. When integrating the fine-tuned Grounding DINO encoder into our pipeline, the top-1 accuracy on the GeoQA benchmark dropped from 67. 77.0% to 66. 1%, further supporting GLIP's advantages for our tasks.
Instructing LLMs with coordinates of primitives. We have conducted experiments for directly providing geometric- relevant information the model. Since existing mathematical instruction datasets include detailed location information for geometric objects (e.g., bounding box coordinates or junction points), we generated this data by inferring Geo 70K training images using GeoGLIP to extract the relevant location information. This information was appended to the special token (image) as supplementary descriptions for each image, using instructions such as: "there is a bounding box at (r, y. wo, h) or there is junction at (I. y) with lines directions (f)". When tested on the Geo170K test set the GeoQA benchmark, the top-1 accuracy dropped from 67. 37.0% to 63.2%. This result is close to the variant of our constant router 62.8% (assigning equal weights all features in Table 6a). This performance drop is consistent with our systematic analysis in Fig. lb and Fig. Ic: Inaccurate instructions would harm the performance, and relevance is key- excessive visual cues interfere with problem-solving.
A.5. Case studies
Selective visual information helps reasoning. Fig. 12 showcases GPT-4o's responses based on additional visual information from geometric primitives, alongside the question, choices, and diagram (image) inputs. We provide hard-coded coordinates for bounding boxes and junctions using instructions such as: "there is a bounding box at (I, y, w. h) (the normalized center point and width/height)" with shape names (geometric shape) (if shape information is provided). or "candidate junction point (z. y). For boundary information, we use "(boundary image) is the boundary sketch related to the main diagram" as instructions. The right side visualizes the provided visual cues the original geometric diagram for clarity, though these images are not input into GPT-4o. Fig. 12 highlights the importance of providing relevant visual prompts for each case; otherwise, redundant information may interfere with the solving process. For example, in case 1, bounding box coordinates per object can be distracting when solving perimeter question compared to junction locations. In contrast, pixel-level visual information (boundary) aids the model in perceiving complex geometric shapes, such as polygons and circles, and is beneficial for calculating overlap regions, while relying on junctions may lead to biased answers. In practice, selecting supporting information for each case is labor-intensive and requires the involvement of math experts. We address this challenge by using the feature router, which automatically learns which grained visual information is important during the training stage.
17

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Response comparison. Fig. 13 presents case studies comparing our PRIMITIVE-DDepsek-77B with GPT-4o on the Math Verse testmini set. These examples highlight the strengths of PRIMITIVE- in providing precise geometric visual information, enabling clear and logically grounded mathematical reasoning in its responses. For instance, our model demonstrates sensitivity to the positions of individual points/junctions, effectively capturing the relationships between different lines. As shown in Fig. 13a, it successfully identifies angle 1 and its relationship with angle BEF, enabling correct reasoning and answers. In contrast, GPT-4o fails to recognize these relationships, leading to flawed reasoning and incorrect answers.
Fig. 14 and Fig. 15 present Chain-of-Thought (CoT) comparison among HELMITIVE-Deepsecke Intern The results clearly demonstrate that providing geometry-awe -awaee visual cues significantly aids LLMs in under- standing the relationships between geometric elements, thereby enhancing the entire reasoning process. In contrast, the other two MLLMs fail to achieve this level of understanding, leading to incorrect reasoning and outcomes. This demonstrates that without accurately recognizing visual elements, even strong LLMs struggle with reasoning tasks. As shown in GPT-4V's output, its initial misidentification of mathematical elements results in an incorrect Chain-of- Chaon-of-Thought (CoT) response.
GPT-4V, and
A.6. Mathematical Visual Training and Efficiency
1. TRAINING DETAILS
Our work follows a structured three-stage training pipeline, including multi-task visual perception training for GeoGLIP, visual-language alignment, and mathematical instruction tuning for MLLMs.
Stage To enable the visual encoder in GeoGLIP to ground geometric entities in mathematical diagrams, we utilize synthetic and FigureQA training images annotated with bounding boxes for the grounded pre-training. Specifically, we fine-tune pre-trained GLIP-T model (with Twin-Tiny as the backbone), adhering to the GLIP detection loss defined as:
Latt = firpn + Leets + frreg
where Crpn refines the region proposals generated by the RPN, Lets applies binary sigmoid loss to alignment scores, and Creg uses smooth & loss for bounding box regression.
Following the process in (Huang et al., 2018), for the junction detection task, the input image is divided into mesh grids, with each grid cell responsible for detecting junction if its center falls within the cell. Each ij-th cell predicts confidence score Czj, indicating the likelihood of junction in that cell. Since junction represents the intersection of lines, the number of predictions per cell varies depending on the number of lines intersecting. To capture orientations, each cell is further divided into K' equal bins (default K = 15), with each bin spanning 24 degrees to cover the full 360-degree range. Each junction is represented as JP = (E.j, Cijz (0isk: & K 1) where Tij denotes the junction center coordinates, Gj E [0, 1] Lyk is the confidence score for the presence of junction, 0igk is the angle of the k-th bin, and Ly is the confidence score for that bin.
The loss function for junction detection consists of four terms. Given a set of ground truth junctions JP =P= jPI,...... jPN in an image, the loss function is formulated as:
Ljunn Aiac. (Cioe + Lloc) + demn (Ceons + Cecng)).
(3)
The default values for the weights in Eq. 3 are Atoc = 0.1 and Xemn 1, where the superseripts and refer to cell and bin, respectively. Specificallyy, we apply the binary cross-entropy loss for both Ceonf and C.cnn. and use &g loss to measure the relative position of the predictions against the ground truth for Li and C) Refer to (Huang et al., 2018) for more details. In the boundary detection task, Lbodr minimizes the Eg loss between the estimated heatmap values and the ground truth values.
Our final loss function for multi-task visual perception training is defined as:
Lyss Ldet + Ljunn + 5 Lbodrr
(4)
where the weight for Grodr is set to 5, while the weights for Ldet and Liune are kept at 1.
Stage 2 & 3: During both phases, we freeze the GeoGLIP encoder. In Stage 2, we train only the projection layers to align llngram- pairs. In Stage 3, we unfreeze both the projection layer and the LLM to perform comprehensive
18

---

Primitive Vision: Improving Diagram Understanding in MLLMs
instruction-follovving tuning, culminating in PRIMITIVE-7B. For these two stages, we employ the conventional LLaVA loss, formulated as:
Cim Σ logp (farlfo(ssar
(or) Sin: 11}.
where fo denotes the model parameterized by ø, I corresponds the figure, Stor and Sin represent the target and input sentences, respectively; Slar refers the i-th token of the target output, and I. denotes the sequence length.
Training is conducted on 8 A100 GPUs with batch size of 32. The base learning rate is set to 1 x 10-5 for the language backbone and 1 x 10-4 for all other parameters, and it is decreased by a factor of 0.1 at 67% and 89% of the total training steps. We employ the same data augmentation strategies as GL.P, including random horizontal flipping and aspect ratio-preserving resizing with a minimum size of 800 pixels.
For multi-modal training, we freeze the GeoGLIP encoder. In Stage 2, we train only the projection layers to align pairs. In Stage 3, we unfreeze both the projection layer and the LLM perform comprehensive instruction-followwe tuning. Our GeoGLIP together with a pre-trained vision transformer (CLIP ViT-L)(Radford et al., 2021) are integrated into the language models LLAMA-2(Touvron et al., 2023), MeapSeee-Math-7BBBle
(Shao et al., 2024), and Qwen2.5-Math-775 In-ntruet (Yang et al., 2024), respectively. Images are padded to squares and resized to 448 x 448 pixels with a white background for processing by CLIP, and to 1000 x 1000 pixels for processing by GeoGLIP. We train PRIMITIVE for one epoch for cross-modal alignment and two epochs for instruction tuning on the Geo170K(Gar et al., 2023a) dataset, evaluating the model on GeoQA (Gao et al., 2023a). For fair comparison, we train our model on MathV360k (Shi al., 2024) using a batch size of 16 for one epoch with an initial learning rate of 10 evaluating on Math (Lu et al., 2023) and the minitest set of Math (Zhang al., 2024a).
A.6.2. EFFICIENCY
PRIMITIVE-7B introduces minimal computational overhcad, as detailed in the below comparison Table 9. The GeoGLIP and Connector contribute an additional parameter size of 32.65MB and 8. 8.73MB, and the Projectors accounting for 16. 13M1. The inference time per sample increases slightly, from 19.80s to 20.04s (+0.24s). Training is conducted on 8 A100 GPUs with batch size of 128 using the V360K dataset, which includes 40K images and 360K question-ann -nestion-answer answer pairs. The total training time shows only marginal increase, from 10.35h to 10.54h (+0.19h), demonstrating scalability for larger models and datasets.
Table 9: Comparison of computational overhead and parameter size for G-LLaVA and PRIMITIVE.
#Parameter size GeoGLIP
Connector
Projectors
Time (inference/sample)
Time (training/Mathh V360K)
G-LLaVA
16.52MB
19.80s
10.35h
PRIMITIVE
32.65MB
8.73MB
31.20MB
20.04s
10.54h
A.7. Limitations and Further research
Our research aims to offer a new perspective on solving mathematical visual reasoning problems by first training a vision- centric model to provide visual prompts for LLMs, rather than focusing on creating large visual instruction fine-tuning datasets for MLLMs. Despite the effectiveness of our approach, there several limitations consider. First, the reliance on synthetic data for visual tasks may not fully capture the complexity of real-world geometric problems, potentially limiting generalization to more diverse datasets. Additionally, while the feature router provides automatic selection of relevant visual cues, it may not always perfectly align with human intuition or domain-specific knowledge, particularly in cases requiring more intricate reasoning.
For future research, one promising direction is to extend our method by incorporating real-world mathematical datasets to improve generalization and robustness. However, this will require some human labeling processes, as existing mathematical datasets lack detailed box or pixel-level annotations. Incorporating such annotations would provide a more accurate and fine- fine-grained understanding of visual elements in mathematical problems, allowing models to better generalize to real-world scenarios. Developing efficient semi-automated labeling techniques or combining expert annotations with synthetic data could also help reduce the manual effort required. With improved detection performance, we may explore more advanced methods for designing soft prompts, such as object-level prompts/visual tokens. Further refinement of the feature router, such as combining with interpretable methods to better understand its decision-maken process, could also enhance
19

---

Primitive Vision: Improving Diagram Understanding in MLLMs
<image>
Encoder (GeoGLIP)
F1
'eo
F2
<txt
F3
circle. trapezoid. triangle. ... line.
F
F5 geo
BERT
Deep Fusion
Shape Detector
Junction Detector
Boundary Detector
text:0.84
triang 92 rectangle:0.93
X
text:0.83
42
text:0.91
38
42
42
38
Figure 8: GeoGLIP pipeline. A geometric multi-task detector. GeoGLIP simultancously detects multiple tasks, including basic geometric shapes, junctions, and boundaries, utilizing multi-scale features to capture fine-graie fine-grained geometric entities.
the model's performance. By making the feature router transparent, could gain insights into how it selects and prioritizes visual cues, allowing for fine-tuning that aligns better with human intuition and task-specific requirements. This, in turn, would allow for more informed adjustments, leading to better generalization and accuracy in complex mathematical problem-solving scenarios.
20

---

Primitive Vision: Improving Diagram Understanding in MLLMs
F1 geo
F3 geo
F4 geo
geo
ggo
Perception Block
Q
K,
Upsampling
MHSA
C Conv2d
BN2d
ReLU
DxWxIl F.
Perception Block
C E R 1x60x600
x2
Dx4
Perception Block
E R2x60x60
38
42
C R15X60X60
R15600X60
jik
38
Dx60x60
1 E (1, ...,, 60}
k (1,.,..11 15)
Figure 9: Designs for the junction and boundary detectors: We first use an attention mechanism (MHSA) to fuse two-scale features, followed by upsampling and addition with the highest resolution features, resulting in F1"
geo Separate perception
blocks are then applied for junction and boundary detection. For junction detection, the detector provides cell confidence (C), cell location (X), bin confidence (C*), and bin orientation (0). Green check-marked features indicate candidate features for soft prompts, with D, W, Hl representing channel dim., and spatial resolution (width&heighht.
21

---

Primitive Vision: Improving Diagram Understanding in MLLMS
Th000.57
No labels
textsO.53
trlamgle:0.79
W Wrrlangle:0885
trlangle:0.75
textc
taxb:077
X
Pline:0.73
t.0000.80
X
text=OOt
fertsdaeo
24
b8as -a5
24
Saut
61
20
30
Laxda0.80
UHOY
triangle:0
Sg
Nerbd.88
Mine:0.77
P
v2ai
5x"
2x"
behoas
12
Q
(b)
(c)
The visualization of shape detection on FigrueQA, Geo170K and our synthetic test dataset. The left panel (a) displays accurate shape detection results generated by GeoGLIP where even small-scale x-ticks are correctly recognized (zoom in 280% for details). GeoGLIP successfully classifies bars in histograms and rectangular shapes in geometric diagrams. The middle panel (b) represents failure cases, with all errors highlighted using magnifying glass. For instance, in the first row figure, the cyan line is misrecognized, and three crowded lines are incorrectly grouped within a single bounding box. The results in the last panel (c) are generated by the original GLIP, trained on natural images. is evident that most geometric shapes are misclassified as lines or text, and GLIP struggles to recognize small-scale objects, where GeoGLIP excels.
Figure 10:
K
K
M
10 ft
P
R
15x*
12 ft
R
N
M
W
X
1011
fer-
8)
Ty++ 28
22 mm
30
10y+ 10)*
(ax+
14)
(b)
Figure 1I: The visualization of junction and boundary detection results. The left panel (a) illustrates accurate detections, while the right panel (b) represents failure cases. Junction detection failures frequently exhibit redundant detections.
22

---

Primitive Vision: Improving Diagram Understanding in MLLMs
Case 1: Junction Point Scenario
Required Information
Distracting Information
Question: Find the perimeter of the figure. Round to the nearest tenth if necessary.
2
Choices: (A) 20 (B) 21 (C) 24 (D) 25
B
GPT-40 with Junction Point: (C) /
GPT-4o: (D) X
GPT-4o with Bounding Box: (D) X
Case 2: Bounding Box Scenario
Required Information
Distracting Information
Question: The segment is tangent to the circle. Find Round to nearest tenth.
V
Choices: (A) 7.5 (B) 8.5 (C) 9.2 (D) 13.0
GPTT-4o with Bounding Box: (B) V
GPT-4o: (A) X
GPT-4o with Bounding Box & Shape: (D) X
Case 3: Bounding Box & Shape Scenario
Required Information
Distracting Information
Question:
Circle has radius of 10 units, Circle K has radius
of units, and BC = 5.4 units. Find CK. Choices: (A) (B) 2.6 (C) 2.8 (D)
GPT-4o with Bounding Box & Shape: (B) GPT-44 with Boundary: X X
GPT-40: (C)
Case 4: Boundary Scenario
Required Information
Distracting Information
Find the area of the shaded region formed by the circle and regular polygon. Round to the nearest tenth.
Question:
Choices: (A) 76.4 (B) 237.8 (7.8(6 (C) 266.6 (D) 314.2
GPT-4o with Boundary: (A) /
10
GPT-4o: (C)x
GPT-4o with Junction Point: (B) X
Figure 12: A case study the 70K dataset (Gao et al., 2023a) highlights the importance providing relevant visual information for each math visual question answer. Zoom in for best view.
g
Since AB parallel to CD, we have the relacionships 41+2 BEF 180". 130",
and
tangent cirele O, we have 4OAP
EG bisects ZBEF, we have 2BBG z BEG 65", Thus, the answer is
OBP 90". Alse, LP 50".
Therefore, ZAOB 360" 90" 90"
D
50" 130" Thus, 2ACB ZAOB Therefore,
Question:
Since AB is panallel CD and EF mnneversal, the altemate interior angles are equal. Thus, 2BEE-ZEEG-- EG bisects BEEF which 2GEF-25',
Using the tangent-seeant theorem, the angle between tangent and chord (cAPB) equals the angle subtended by the chord in the alternate segmeat (ZACB). Given ZAPB-50'. that Z.CBB-S... Thus, the size of ZACB is 50%, and the correct option is
As shown the figure, AB parillel CD. straight line
Questlon:
point intersecis CD al point EG bisects ongle BFE, intersects CD poinr angle 50.0, Ihen angle squal
-BEF-
the figure, given that angle P = 50.0, then the size of angle ACB is?
4BGP
are supplementary (they form lincar along EG), JBGPP500-22 501-25--255 Finally, 22-2GEF-2EGGP Thus.
follows
Choices:
Choices:
the answer
50
(a)
(b)
Figure 13: Response comparison of PRIMITIVE-DDeppsek and GPT-4o. Refer the main text for detailed analysis. Zoom in for best view.
23

---

half Primitive Vision: Improving Diagram Understanding in MLLMS
PRIMITIVE
GPT-4V
Intern
Since AB:EB-5:1, AB-5 EB
of length EB is given 5:1. This 5 5 x EB
Since AB:EB-5:1, have ABm5 x EB
congruent to and
X E point AB, and ED parallel to AC, When line is drawn parallel to one side of triangle and two sides, it forms similar triangle. Therefore, A ABC is similar to ABED
is congruent LD,
The 4ABC and ΔEBC share the
conclude that AABC similar same height from point C
ABED
the other
Question:
Thus, the ratio of their same the ratio of their buses (AB to EB), which 5:1
the
Therefore, the ratio of the areas of AABC ARED equal the square ratio to EB, which (AB/EBy-25
If the ratio of the length of AB to the length of EB is S:1, and the area of the triangle BED is Sa + 10, what is the area of triangle ABC?
X The of AABC 5 (5a" 10) 25 x
Since AB:EB-5:1, the ratio of areas will be 57:1-22 57:1-5:11
The
AABC 25 (5a" 10) 125 of 4-20 Therefore, the answer is
Therefore, the inswer
The area of AABC is (5a 10) x 25 125 x x 250 Therefore, the answer
Choices:
A; 5a + 10
B: 25a + 50
C: 25a + 100
1 250
E: Cannot determined
Figure 14: Chain-of- Thought (CoT) response comparison of our PRIMITIVE- DeeDMIIIVVE-Deppscck-BBB GPT-4V and Intern VL2. Refer the main text for detailed analysis. Zoom in for best view.
PRIMITIVE
GPT-4V
InternVL2
As shown in the figure, OB- OC, 2 OBC
X
The image shows circle with The figure shows a circle with center and points A, and the circumference, Angle OBC is given as 40 degrees.
40%, therefore
we central angle ZOBC, of know that the central angle is equal to the measure of the are intereepts.
ZOCB
40
COB = 180 -40 400 100"
In circle, the angle subtended by an are at the conter twice the angle subtended at any point on same side of the arc. This known the Inscribed Anglo Theorem
Therefore ZA 1/2 ZCOB 50"
So, the are BC would also degrees.
Question:
So the answer is A
As shown in the figure, when angle OBC 40.0, the degree of anaglc A is?
So, the BC is 40 degrees, then angle A. which inseribed angle intereepling that are, would be of that
the circumference
X <OBC 40 degrees. LA is the angle subtended by the are BC at point A. Based on the Inseribed Angle Theorem, ZOBC= 40% 2 20". The given choices do not include 20"
Choices:
X Therefore
20"
A: 50" B: 55" C: 65
However, this result does match any of the given options
Figure 15; Ch Thoughh (CoT) response comparison of our PRIMITIVE-Deepseedk -B GPT-4V and Intern Refer the main text for detailed analysis. Zoom in for best view.
24


Reviews:
[{"review_id": "WzgdUxYbrx", "reviewer": "Reviewer_PhN5", "summary": "This paper proposes PRIMITIVE, a multi-modal large language model for mathematical problem-solving. The contribution of PRIMITIVE is two-folds: a mathematical vision encoder, GeoGLIP, and an MLP-based feature router. The GeoGLIP is pre-trained using synthetic data with box-level and pixel-level loss within the GLIP architecture. The feature router adopts a soft weight strategy to combine different levels of visual feature adaptively. The performance of PRIMITIVE is good on different benchmarks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for insightful questions that help refine our work further.\n## 1. The formula (1) in Section 3.3 should contain more comprehensive illustration of newly defined labels.\n\nThank you. Duty noted. Eq. (1) describes the soft router process. The scalar routing weight for each level is denoted as $w^i$, where $i \\in \\{1, 2, 3, 4\\}$; the symbol $\\circledcirc$ denotes the operation flow from right to left. For instance, $\\mathcal{G} \\circledcirc F_{\\text{geo}}^i $ indicates that the different levels of GeoGLIP features $\\(F_{\\text{geo}}^i\\)$ are resized $(\\mathcal{G})$, which is essential for aligning spatial size of CLIP features.  $\\sigma$ denotes the normalization function, which could be either a SoftMax or a Sigmoid function, depending on the feature integration process (channel-wise or sequence-wise concatenation). \n\nWe have  added a more detailed description of each component and its role in the revised manuscript to ensure that the formulation and its application are transparent to the readers.\n\n## 2. In Section 4.1, authors use two checkpoints for evaluation on GeoQA and MathVerse/MathVista, which is not considered fair for performance comparison since the standard practice in this field is to test all benchmarks using a single trained model.\n\n\nThank you for highlighting this important point. GeoQA primarily evaluates geometric problem reasoning, and our baseline G-LLaVA-7B is trained exclusively on the Geo170K dataset for this benchmark. To ensure a fair comparison, we maintain consistency in the training dataset and test on GeoQA.\n\nRegarding the MathVista and MathVerse benchmarks, which cover a diversity of subjects including scientific topics, PaperQA, and IconQA, we applied the additional MathV360K dataset for fine-tuning our model (Geo170K+MathV360K), a combination commonly used by other mathematical MLLMs like Math-LLaVA and MAVIS. In response to this concern, we test our model trained on Geo170K+MathV360K on the GeoQA benchmark. The performance improved, with PRIMITIVE-7B's top-1 accuracy increasing to 71.3 (67.0 in Tab. 3). We adopt this version as the default choice and will release the checkpoint to the public.\n\n## 3.  The main motivation and training approaches of this paper are similar to MAVIS, e.g., improving vision encoder and three-stage training, although with different techniques. I suggest discuss more about the relation and difference of this paper and MAVIS in the Intro and Related Work parts.\n\nThank you for your suggestion. We have expanded the discussion regarding our model in comparison to MAVIS in the Related Works section.\n\nAlthough our high-level motivation and MLLM training strategies share similarities with MAVIS, our different training techniques for the visual encoder and MLLM design enable our model to achieve comparable reasoning performance using an 8$\\times$ smaller visual instruction training dataset.  For visual encoder training, MAVIS utilizes 588K caption-diagram pairs with alignment loss similar to CLIP, which provides image-level supervision. In contrast, our model employs only 40K samples with box-level and pixel-level supervision, offering more detailed local feature perception.  Researchers have shown that CLIP-style encoders  fail to capture fine details (e.g., Yiwu Zhong et al., \"RegionCLIP: Region-based Language-Image Pretraining\"; Zhe Gan et al., \"Vision-language pre-training: Basics, recent advances, and future trends\").\n\nA more intuitive demonstration is shown through the visualization of attention maps. MAVIS's visual encoder, depicted in Fig. 1(a) of MAVIS's paper, primarily highlights coarse-level boundary information and often overlooks detailed features. In contrast, our model, as shown in Fig. 2, produces attention maps that are more fine-grained, clearly highlighting not only the boundaries but also detailed elements such as dotted lines and right-angle symbols. This stark contrast underscores the superior local feature detection capability of our model.  Furthermore, our model design incorporates a soft router that adaptively selects visual cues ranging from semantic-rich to geometric-rich. This adaptability enhances the model's ability to accurately solve problems by leveraging relevant visual information effectively.\n\n## 4. The title of the paper is a bit overclaimed. PRIMITIVE utilizes localization information to enhance the vision perception and performance, but does not enable LLMs to know where to focus within math figures.\n\nThank you. Duty noted.  We will revise the title to more accurately reflect the model's enhanced visual perception and avoid overclaiming.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The formula (1) in Section 3.3 should contain more comprehensive illustration of newly defined labels.", "experimental_designs_or_analyses": "It is not very fair for performance comparison. In section 4.1, the authors mention that they evaluate PRIMITIVE on GeoQA and MathVerse/MathVista separately using different checkpoints, that is, Geo170K and MathV360K, respectively. This is my main of concern, since previous works in this field or the default choice of this community, is to test all different benchmarks using one trained model, either by combining training data in one stage or setting different training stages. If PRIMITIVE cannot perform well on different benchmarks within one checkpoint, the generalization capability of this approach will be questioned. The author is encouraged to provide experiments or illustrations.", "relation_to_broader_scientific_literature": "No concern", "essential_references_not_discussed": "The main motivation and training approaches of this paper are similar to MAVIS, e.g., improving vision encoder and three-stage training, although with different techniques. I suggest discuss more about the relation and difference of this paper and MAVIS in the Intro and Related Work parts.", "other_strengths_and_weaknesses": "Strengths:\n\n1. Exploring the role of vision encoder in math problem-solving is interesting and reasonable, which is mostly ignored by existing research. Although a few works in recent months also focus on this point, this paper provides unique insights with localization loss.\n\n2. The curated datasets for pre-training GeoGLIP is useful to the community.\n\n3. The figure is clearly presented.\n\nWeaknesses:\n\n1. Please refer to the experimental fairness above.\n\n2. The title of the paper is a bit overclaimed. PRIMITIVE utilizes localization information to enhance the vision perception and performance, but does not enable LLMs to know where to focus within math figures.", "other_comments_or_suggestions": "Overall it's a good paper, while remains some issues to be solved.", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I17U8AkmK2", "reviewer": "Reviewer_Ay8j", "summary": "This work proposes a novel approach named PRIMITIVE, aiming to address the deficiencies of current mathematical multimodal large language models (MLLMs) in geometric perception, thereby enhancing their capabilities in visual mathematical reasoning. Experiments conducted on three benchmarks demonstrate the effectiveness of the PRIMITIVE method.\n\n## Update review\n\nIn the previous round, I thought the feature pyramid used by the authors lacked novelty. The authors replied that they did not consider the feature pyramid technique as a major contribution. I hope the authors can clarify this in the revision. \n\nAdditionally, I still have concerns about the authors using existing models to extract junctions, which may introduce unpredictable noise, as the performance on MathVerse and MathVista is not very promising. I hope the authors will address this point in the revised version.\n\nThe study of how fine-grained visual perception influences downstream reasoning in MLLMs is meaningful. Therefore, I change my opinion to 'Weak Accept'.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for insightful questions that help refine our work further.\n\n## 1.  The authors use existing models to extract junctions and boundaries as ground truth—could this introduce noise given the variation in geometric shapes and domains?\n\nFor junction detection, we discuss noisy cases in lines 802–807 of Appendix A.4 and show failure cases in Fig. 11. We use a CNN model (Huang et al., 2018) trained on Man-Made Environments to generate ground truth. Some label noise arises from out-of-domain (OOD) diagrams, which we mitigated using a 0.85 confidence threshold and manual correction. While 100\\% accuracy isn’t guaranteed across 20K+ samples, the noise level is acceptable. The test set recall is 85.6%, and GeoGLIP remains robust to distortions (see Q2 for AR sjVj). Improving junction labeling and data synthesis, as discussed in Sec. A.7, could further boost performance.\n\nFor boundary detection, we use the FoJ model (Verbin & Zickler, 2021), which applies a machine learning algorithm for generalized M-junctions. Unlike CNNs, FoJ generalizes better to OOD domains and is less dataset-biased. Its noise resilience is supported by prior work. GeoGLIP achieves an IoU of 92.3% and remains robust under various distortions (see Q2 for AR sjVj).\n\n## 2. PRIMITIVE-Qwen2.5-7B shows lower  MathVerse/MathVista performance compared to Qwen2.5-VL-7B (68.2%), and even earlier MLLMs.\n\nMLLMs fall into two categories: generic (e.g., InternVL2.5, Qwen2.5-VL, GPT4o) and math-specific (e.g., G-LLaVA, Math-LLaVA, MAVIS, ours), differing significantly in training recipes and datasets when integrated with LLMs and visual encoders. \n\nGeneric models are trained on large-scale visual instruction datasets covering a wide variety of multimodal data, including OCR, academic questions, localization data, documents, and video descriptions ( e.g., Qwen2.5-VL uses 2TB+ data), while math-specific models utilize only mathematical text-diagram pairs, which require fewer training resources and provide an efficient test base for proposed math module designs. Our model is fairly compared with math-specific MLLMs, as it is trained solely on MathV360K+Geo170K. In controlled experiments, adding GeoGLIP and the soft router boosts baseline G-LLaVA by 4.6% on MathVerse and 12.3% on MathVista. We've also extended our approach to DeepSeek-Math-7B and Qwen2.5-Math-7B, achieving consistent +6% gains on MathVista (Tab. 7). Note: baselines use only CLIP visual encoders and the same training setup—not generic DeepSeek-VL or Qwen2.5-VL.\n\nTo compare with state-of-the-art 7B models, we implemented our designs on Qwen2.5-VL-7B and fine-tuned it using its official checkpoint. As the training code is not publicly available, significant effort was required to rewrite the functions, data loader, and training processes within the transformers and Hugging Face frameworks. For alignment, we trained only the projectors;  Due to time limits, we trained both projectors and LLMs using LoRA to accelerate SFT stage on the MathV360K+Geo170K dataset. \n\nUnder the same training setup, we developed two versions: (1) Qwen2.5-VL-7B+, fine-tuned with math-specific visual data; and (2) PRIMITIVE-Qwen2.5-VL-7B, integrated with our GeoGLIP and soft router. Our integrated models are lightweight (<50MB, see Sec. A.6.2) and already show performance gains. We plan to further improve results by exploring full fine-tuning (beyond LoRA) and partial unfreezing of the visual encoder with lower learning rates.\n\n\n|Model|MathVista/All (acc)|MathVerse/All (acc)|\n|:-:|:-:|:-:|\n|Qwen2.5-VL-7B|68.2|49.2|\n|Qwen2.5-VL-7B+|68.5|49.8|\n|PRIMITIVE-Qwen2.5-VL-7B|69.7|51.0|\n\n## 3. PRIMITIVE uses a feature pyramid to extract fine-grained features—a technique common in natural images but not specifically tailored for geometric tasks.\n\nThank you for your observations regarding the PRIMITIVE architecture. Although the feature pyramid structure effectively extracts fine-grained visual cues in natural images, it is not central to enhancing mathematical perception in our MLLM.  Instead, the key factor lies in the mathematically sensitive GeoGLIP, which incorporates visual-centric mathematical training datasets and fine-grained box- and pixel-level supervision (refer to Q3 for AR PhN5). \n\nOur empirical evidence, detailed in lines 840-848 and Tab. 6 of the appendix, supports this. Specifically, coupling the original GLIP (trained on natural images) with the feature pyramid and soft router techniques decrease performance from 67.0 to 65.3 on GeoQA. Additionally, Fig. 10 (right panel) demonstrates GLIP's inability to accurately perceive basic geometric primitives. \n\nThese results confirm that fine-grained primitive perception ability depends on a geometrically sensitive visual encoder, and then this ability could be further enhanced by employing pyramid techniques. Our specific adaptations and innovations for GeoGLIP are deliberately designed to tackle the unique challenges of mathematical image tasks.", "claims_and_evidence": "No, please refer to Part of [Methods And Evaluation  Criteria]", "methods_and_evaluation_criteria": "- 1) The authors mentioned that existing models would be used to extract junctions and boundaries as ground truth. For geometric shapes with diverse variations in shapes and domains, could this process introduce noise?\n\n- 2) The performance on MathVerse and MathVista is not very promising. For example, the proposed PRIMITIVE-Qwen2.5-7B achieves lower performance on the MathVista evaluation set compared to InternVL2.5-8B (64.4%), Qwen2.5-VL-7B (68.2%), and even previous-generation MLLMs such as InternVL2-8B (61.6%), InternVL2-4B (57.0%), and Qwen2-VL-7B (58.2%).\n\n- 3) The relatively low experimental results on the MathVista and MathVerse evaluation sets raise doubts about the effectiveness of the proposed approach. Although the work claims that the approach can be integrated into multiple model baselines, the experimental results for its combination with baselines such as Qwen2.5-7B and DeepSeek-7B remain unsatisfactory.", "theoretical_claims": "This research work does not propose theoretical analyses.", "experimental_designs_or_analyses": "- 1) The PRIMITIVE architecture used in this work is built upon LLaVA-1.5, which is somewhat outdated. Moreover, PRIMITIVE employs a feature pyramid structure to extract fine-grained geometric visual features, a method that is generic for natural images and not specifically tailored for geometric image tasks.", "relation_to_broader_scientific_literature": "This work proposes MLLMs in mathematical scenarios, which could potentially aid in analyzing the reasoning capabilities of MLLMs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "[Strengths]\n\n- 1) This work analyzes the proportion of perception errors in geometric question answering, which is insightful.\n\n- 2) The authors achieved high performance on the GeoQA leaderboard.\n\n[Weaknesses]\n\n- 1) The authors mentioned that existing models would be used to extract junctions and boundaries as ground truth. For geometric shapes with diverse variations in shapes and domains, could this process introduce noise?\n\n- 2) The performance on MathVerse and MathVista is not very promising. For example, the proposed PRIMITIVE-Qwen2.5-7B achieves lower performance on the MathVista evaluation set compared to InternVL2.5-8B (64.4%), Qwen2.5-VL-7B (68.2%), and even previous-generation MLLMs such as InternVL2-8B (61.6%), InternVL2-4B (57.0%), and Qwen2-VL-7B (58.2%).\n\n- 3) The relatively low experimental results on the MathVista and MathVerse evaluation sets raise doubts about the effectiveness of the proposed approach. Although the work claims that the approach can be integrated into multiple model baselines, the experimental results for its combination with baselines such as Qwen2.5-7B and DeepSeek-7B remain unsatisfactory.\n\n- 4) The PRIMITIVE architecture used in this work is built upon LLaVA-1.5, which is somewhat outdated. Moreover, PRIMITIVE employs a feature pyramid structure to extract fine-grained geometric visual features, a method that is generic for natural images and not specifically tailored for geometric image tasks.", "other_comments_or_suggestions": "The figure captions should be placed below the figures. Please make this correction in future versions.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SbqN5Vlqa", "reviewer": "Reviewer_nRX6", "summary": "This paper addresses a significant challenge in multi-modal large language models (MLLMs), specifically focusing on their limited ability to accurately interpret geometric primitives (e.g., points, lines, boundaries, and junctions) in mathematical diagrams. The authors conduct a comprehensive analysis revealing that existing models such as GPT-4o frequently misinterpret these visual elements, negatively impacting the accuracy of subsequent reasoning tasks. To address this, the authors propose PRIMITIVE, a novel approach featuring a dedicated geometric-aware visual encoder (GeoGLIP) and a dynamic feature router. GeoGLIP utilizes a multi-task learning framework based on Mask R-CNN architectures to effectively detect boundaries and junctions. The feature router dynamically selects and integrates multi-scale features from GeoGLIP and the CLIP encoder, resulting in improved reasoning performance on several benchmarks, including MathVerse, GeoQA, and MathVista. Experimental results demonstrate substantial improvements in accuracy compared to existing methods, emphasizing the importance of precise geometric visual understanding in mathematics-related tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for insightful questions that help refine our work further.\n\n## 1. The impact of visual encoders—e.g., comparing Qwen2.5-VL and PRIMITIVE-Qwen2.5-7B with fixed LLM weights.\n\nTo address the concern about visual encoders' impact on reasoning, we compare variants in controlled settings, as detailed in lines 840–847 and Tab. 6 of the appendix.\n\nAlthough Qwen2.5-VL and PRIMITIVE-Qwen2.5-7B share the same LLM backbone, they differ significantly in training recipes and dataset scale (~2T vs. ~600K). For details, see Q2 (Ay8j). In Tab. 6, we control for training settings and datasets, isolating the visual encoder for direct comparison.  See Q1 (sjVj) for analysis.  For qualitative response differences, we analyzed model reasoning steps and found that the GLIP+CLIP dual encoder struggles with basic shape perception and interrelationship descriptions, often leading to hallucinations and incorrect answers. We will include their rationale demos in the revised paper.\n\nTo ensure a fair comparison with Qwen2.5-VL, we conducted new experiments resulting in Qwen2.5-VL-7B+ and PRIMITIVE-Qwen2.5-VL-7B (see Q2 for ARAy8j), with more detailed comparisons to be added in the revision. \n\n## 2. The PRIMITIVE pipeline seems limited in contribution, as it mainly integrates existing LLMs with minor modifications.\n\nWe respectfully disagree that our contributions are merely incremental. Our core contribution is the design of a geometrically sensitive visual encoder that enhances MLLMs' fine-grained primitive perception—addressing a key bottleneck in reasoning, as shown by the analysis in Figs. 1 and 5a.\n\nUnlike single-modality LLMs, MLLMs rely on accurate visual cue interpretation to support abstract reasoning. Misinterpretations at this stage can misguide reasoning processes. To address this, we designed a geometrically sensitive visual encoder with box- and pixel-level supervision, going beyond the image-level supervision used in CLIP (see Q3 for AR PhN5). This was enabled by a custom data engine generating diagrams with shape, junction, and boundary annotations.\n\nFurthermore, we integrated the trained visual encoder into LLMs using two methods: hard coordinates (Sec. A.4) and soft visual prompts. For the latter, we introduced a feature router that dynamically selects features from semantic- to geometric-rich levels. Our designs were implemented in LLaMA2-7B, Deepseek-math-7B, and Qwen-math-7B, with ablations showing notable gains over baselines across three math reasoning benchmarks.\n\n## 3. The paper lacks the parameter size and citation for InternVL2, limiting clarity and reproducibility; adding comparisons with recent state-of-the-art visual models would strengthen the experimental validity.\n\nWe apologize for the oversight—InternVL2 has 8B parameters, which we will clarify as InternVL2-8B. For reproducibility, we will release model weights for PRIMITIVE-7B, PRIMITIVE-Deepseek-7B, and PRIMITIVE-Qwen2.5-7B, along with training and inference code. To address comparisons with state-of-the-art models, we have integrated our design into Qwen2.5-VL (see Q2 for ARAy8j).\n\n## 4.  It lacks analysis of remaining reasoning errors, which may stem from LLM hallucinations rather than perception.\n\n Our primary goal is to enhance the visual grounding of MLLMs to complement their reasoning, not to solve the full spectrum of mathematical reasoning tasks. We acknowledge that hallucinations in reasoning remain an important challenge, influenced by many factors such as data distribution, vision encoders, modality alignment, and the LLM's inherent knowledge [1–5].  \n\nPerceptual enhancement is one feasible path to reducing such hallucinations [4,5], and our work focuses on improving object- and pixel-level perception in mathematical contexts. We agree that other factors also impact reasoning accuracy and plan to explore them in future work. As suggested, analyzing caption-based descriptions could provide intuitive insights into hallucinations tied to diagram elements. However, building such datasets and benchmarks is non-trivial and beyond the scope of this rebuttal, but it remains a key direction for future research.\n\n[1] Jaemin Cho et al., Fine-grained image captioning with clip reward.\n\n[2]  Ziwei Ji et al., Survey of hallucination in natural language generation.\n\n[3]  Lei Huang et al., A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.\n\n[4]  Shunqi Mao et al., Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding.\n\n[5]  Hanchao Liu et al., A Survey on Hallucination in Large Vision-Language Models.\n\n## 5.  Additional visualization examples are needed, and Fig. 3 and Tab. 5 exceed page margins.\n\nThank you. We will add more qualitative comparisons between PRIMITIVE and baselines for clarity and fix the formatting issues in Fig. 3 and Tab. 5.", "claims_and_evidence": "- The paper clearly identifies a crucial limitation in current MLLMs regarding precise geometric visual perception, presenting a thorough and systematic analysis of existing model failures. \n- Although the authors claim to perform an 'apples-to-apples' comparison, the evaluation lacks a detailed analysis of how different visual encoders affect the reasoning performance when keeping the LLM weights fixed. For example, a comparison between Qwen2.5-vl and PRIMITIVE Qwen2.5-7B, analyzing both the performance differences and qualitative differences in responses, would provide deeper insights.", "methods_and_evaluation_criteria": "- The introduction of GeoGLIP as a specialized visual encoder effectively addresses the challenge of accurately detecting small-scale geometric primitives, significantly advancing the state-of-the-art in math-oriented visual understanding. \n- The proposed feature router mechanism is innovative, enabling adaptive fusion of multi-scale visual information from GeoGLIP and CLIP, thus demonstrating significant empirical improvements. \n- The technical contributions of the proposed PRIMITIVE pipeline appear limited, as the approach primarily integrates existing LLM architectures with incremental modifications.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "- Comprehensive experimental validation across multiple challenging mathematical visual reasoning benchmarks showcases clear and convincing performance advantages over established baseline methods. \n- Throughout the paper, the authors do not specify the parameter size of InternVL2 nor provide any relevant citation, which limits the reproducibility and clarity of their comparisons. Additionally, InternVL2 is currently not the most state-of-the-art open-source model; therefore, including a comparison with the latest state-of-the-art visual models would significantly strengthen the experimental validity.\n- Although PRIMITIVE demonstrates promising performance and improved interpretability in Figures 14 and 15, the authors have not adequately addressed the reasoning errors that still persist in problem-solving scenarios. These errors may not necessarily stem from perceptual inaccuracies but could potentially result from hallucinations within the MLLMs' reasoning processes. It would be beneficial to include additional experiments or analyses in caption-based tasks to better understand and address these reasoning limitations.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "- While the authors extensively evaluate their method on mathematical visual reasoning tasks, additional qualitative analyses or visualization examples illustrating the feature routing decisions and specific cases where PRIMITIVE significantly outperforms baselines could further clarify the interpretability and practical effectiveness of the dynamic feature routing mechanism. \n- Figure 3 and Table 5 extend beyond the page margins, which compromises the visual presentation and overall readability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ce0l45ngbG", "reviewer": "Reviewer_sjVj", "summary": "The paper focus on the problem of math diagram understanding --- multimodal math problems with visual inputs. The paper has two main contributions, (1) GeoGLIP, a vision encoder that is more geometrically-grounded; (2) PRIMITIVE, a VLM trained with GeoGLIP + a feature selection module, which demonstrates strong performance on math diagram understanding tasks.\n\n## Update after rebuttal\nThe rebuttal addresses my major concerns, I raised score accordingly; I believe adding these results are important to make the claims more convincing;", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# We thank the reviewer for insightful questions that help refine our work further.\n## 1. Claim 1—that GeoGLIP outperforms CLIP—is unconvincing due to missing baselines using the same VLM and data with original vision encoders, leaving it unclear whether gains come from the backbone or training data.\n\nWe have conducted an ablation study in Sec. A.4  (lines 840-848 and Tab. 6), which utilizes different visual encoders trained on the same  backbone and datasets.\n\n* We compare a **single encoder** setup using either CLIP or GeoGLIP, with the same backbone VLM (LLaVA) and dataset (Geo170K). GeoQA accuracy is 64.2 for CLIP and 66.1 for GeoGLIP.\n\n* We evaluate **dual encoders** combining GLIP or GeoGLIP with CLIP, showing that without math-specific fine-tuning, GLIP's performance drops (67.0→65.3) due to limited geometric sensitivity, as illustrated in Fig. 10.\n\nIf the reviewer considers this ablation crucial, we are open to moving it to the main paper.\n\n|Type|Model|Top1 Acc (GeoQA)|\n|:-:|:-:|:-:|\n|Dual encoders|GLIP+CLIP|65.3|\n|Dual encoders|GeoGLIP+CLIP|67.0|\n|single encoder|GeoGLIP|66.1|\n|single encoder|CLIP |64.2|\n\n## 2. Additional analysis is needed to assess GeoGLIP’s robustness to visual variations (e.g., resolution, line style/width, junction size) in junction and boundary detection, and how detection errors impact final performance.\n\nIn response to this concern, we evaluated the robustness of our model using a custom set of 500 images with various distortions, as no public benchmark exists for junction and boundary detection in mathematical diagrams. We adopted standard metrics from natural image tasks:\n\n* Junction Detection: Recall, using a confidence threshold of 0.65.\n\n* Boundary Detection:  Intersection over Union (IoU), with boundary maps binarized using a threshold of 200 (1 for boundary and 0 for non-boundary); IoU is computed via pixel-wise logical AND/OR between prediction and ground truth.\n\nDistortions Applied: \n* Gaussian Noise: Variance of 0.3.\n\n* Resolution Change: Shortest side reduced from 800 to 400 pixels.\n\n* Line Width Variation: Increased from 1–4 to 1–8 pixels,  correspondingly impacting junction size.\n\n* Line Style Modification: Changed from solid to dashed lines.\n\nOur results (see table) show GeoGLIP is robust to resolution and line width changes, aided by diverse training data and augmentations (e.g., varying line widths, flipping, cropping, keep-ratio resize). It is more sensitive to Gaussian noise (−4.6 junction, −3.1 boundary) and dashed lines (−3.2 junction, due to higher false positives). These results highlight the need for improved data generation and training strategies to better handle visual distortions.\n\n| Distortion Type      | Junction Detection (recall \\%) | Boundary Detection (IoU \\%) |\n|:-:|:-:|:-:|\n| w/o                   | 85.6                          | 92.3 |\n| +Gaussian Noise       | 81.0                           | 89.2  |\n| +Resolution Change    | 85.2                           | 91.9 |\n| +Line Width Variation | 85.9                          | 92.3|\n| +Dashed Lines         | 82.4                          | 92.7|\n\nAs Gaussian noise and dashed lines are the most impactful factors, we apply Gaussian noise for further reasoning evaluation due to limitations in modifying line styles without access to the original drawing codes of evaluation images.  On GeoQA, top-1 accuracy dropped by −1.3 (PRIMITIVE-7B), −1.8 (PRIMITIVE-Qwen2.5-7B), and −2.7 (PRIMITIVE-Deepseek-7B). In contrast, the baseline model using only the CLIP encoder (PRIMITIVE(-)) showed larger drops of −3.1, −4.2, and −5.1, indicating that CLIP is more sensitive to distortions and provides less reliable visual input for reasoning.\n\n| Model                 | Top1 Acc (wo/w Gau. noise) |\n|:-:|:-:|\n| PRIMITIVE-7B          | 67.0/65.7|\n| PRIMITIVE-Deepseek-7B | 72.8/71.0 |\n| PRIMITIVE-Qwen2.5-7B  | 79.6/76.9 |\n| PRIMITIVE(-)-7B      | 64.2/ 61.1  |\n| PRIMITIVE(-)-Deepseek-7B  | 66.1/  61.9   |\n| PRIMITIVE(-)-Qwen2.5-7B | 72.3/ 67.2   |\n\n# 3. Preservation of general ability.\n\nWhile our model doesn't lead on FigureQA and VQA in Table 2, it significantly outperforms the G-LLaVA baseline with gains of +19.6 and +2.1, respectively. To address concerns about general ability, we further evaluated PRIMITIVE-7B on SEED-I and MM-Bench. On SEED-I, it maintained comparable performance to LLaVA-1.5-7B (66.2 vs. 66.9), and on MM-Bench, it achieved a +0.6 improvement. Most gains are observed in categories like instance interaction, counting, and spatial localization.\n\n# 4. Inconsistent notations and formatting issue.\n\nWe apologize for the confusion. The four features transferred from GeoGLIP to the Connector include one mixed feature ($F^{1*}$) and three unmixed features. We will clarify this in Sec. 3.3, including revising the index notation in Eq. (1). We will adjust the title to meet the formatting requirements.", "claims_and_evidence": "The main claims/contributions of the paper are stated as: \n1. GeoGLIP feature is better than CLIP\n2. The Feature Router is needed and effective\n\n---\nAlthough Claim 2 is well supported by ablation Table 5, Claim 1 is not very convincing.\n- For Claim 1. the paper lacking important baselines that finetune the same backbone VLM, with the same data, but using their original vision encoder; this makes it unclear whether the main benefit is just mainly from the a better backbone model, or maybe better training data", "methods_and_evaluation_criteria": "The method and evaluation setting are mostly clear for the claim, but some remaining questions need to be answered:\n\n1. Additional analysis is needed for demonstrating: how robust is the GeoGLIP (especially junction and boundary detection) to the variance/noise in the visual input, for example, change of resolution, different line width, different line type (solid vs dashed), different junction size, etc. And how will the these detection error be propagated to the final performance?\n\n2. Another concerning result is that the performance on FigureQA and VQA is not good, showing that the GeoGLIP feature might not be able to easily generalize to more diverse domains other than in-domain data as the synthesized dataset. And it is unclear that whether the performance on general multimodal benchmarks such as MME, MMBench can be retrained.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "As mentioned in the Claims And Evidence, a set of important baseline is missing", "relation_to_broader_scientific_literature": "The key contributions are related to addressing the long-lasting problem in VLMs --- fine-grained visual perception and grounding.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Some notations of the features are a little bit confusing; For example, in Figure 2's caption, it seems that the four features from GeoGLIP to Connector contains a mixed feature F1* and 3 un-mixed feature; but in Section 3.3 text, there is no notation referring to mixed feature (*); this make is unclear what exactly is the input to the Connector", "other_comments_or_suggestions": "Minor: The title is exceeding the textlength, this might violate the formatting requirement, please fix that", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper identified problems in fine-grained visual understanding for math VQA, and proposed a new encoder and a routing method to address it. The paper received unanimous positive ratings (3, 3, 3, 4). The problem is well-motivated, and the empirical evidence are strong for the proposed improvement, according to the reviewers. The AC recommends acceptance.
Decision: Accept (poster)