G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model

## 1 Introduction

Large language models (LLMs) exhibit human-like proficiency in reasoning (Wei et al., ; Wang et al., ; Zhou et al., ) and generation (Ouyang et al., ; Touvron et al., ), which encourages extensive research on their application in mathematical problem solving (Fu et al., ; Gou et al., ; Yue et al., ; Luo et al., ; Zhao et al., , ; Jiang et al., ). These problems often require highly sophisticated and symbolic reasoning capabilities, often considered impossible to solve before the era of LLMs.

It is an intuitive approach to use LLMs for mathematical reasoning problems presented in a textual form. Nevertheless, a substantial proportion of mathematical reasoning problems necessitate the comprehension of geometric information. Moreover, even when certain problems do not overtly pertain to geometric information on the surface, the integration of geometrical-based methods often holds significant practical implications (e.g., analytic number theory). With the advent of GPT-4V (OpenAI, ), Gemini(^1^11Gemini, a concurrent work, was released one week before our submission. Consequently, our work is primarily benchmarked against GPT4-V and other MLLMs.) (Google, ), and numerous multi-modal large language models (MLLMs)  (Zhu et al., ; Liu et al., ; Dai et al., ; Li et al., ; Bai et al., ; Lai et al., ; Gao et al., ; Pi et al., ), recent work has progressively looking into employing MLLMs to tackle geometric reasoning problems in mathematics (Yang et al., ; Lu et al., ; Yue et al., ).

Figure: Figure 1: State-of-the-art MLLMs suffer severe hallucination on geometric figures, which greatly hinders their abilities on solving geometric problems. On the other hand, our G-LLaVA’s ability to interpret geometric figure is boosted after the alignment phase with our curated dataset.
Refer to caption: x1.png

However, we have observed that even with the most advanced MLLMs, current systems still exhibit limitations in addressing geometric problems due to challenges in accurately comprehending geometric figures.
For instance, as demonstrated in Figure [1](#S1.F1), GPT4-V often produces inaccurate descriptions for geometric figures. Specifically, the model struggles with understanding the relationships between fundamental elements like points and lines, and in accurately interpreting elements such as the degree of an angle.
We presume that the underlying reason for this may be the fact that these MLLMs are typically trained with images and descriptions from the general domain, and the ability to understand such semantics differs significantly from that required for geometric reasoning.

To address this issue, one of the most direct and effective approaches is to enhance current MLLMs by augmenting them with data containing high-quality descriptions of geometric information (Ye et al., ; Meng et al., ).
However, a significant challenge arises from the limited size of the largest publicly available geometric problem dataset, which contains only a few thousand question-answer pairs.
Additionally, the current datasets lack descriptions of geometric images and exhibit a limited range of problem-solving methods, which constrains the model’s ability to understand basic geometric elements and affect its problem-solving capabilities.

In this paper, we propose to synthesize geometric visual-text data leveraging existing datasets via text-only LLMs (e.g., ChatGPT).
More specifically, we utilize the geometry characteristic to construct a multi-modal geometry dataset, building upon existing datasets. The data generation process involves incorporating utilizing uniqueness of geometric logic form, geomertic representation uniqueness, geometric scalability, etc (as shown in Figure [2](#S2.F2)).
We term our generated dataset Geo170K, which contains around 60,000 geometric image-caption pairs and more than 110,000 question-answer pairs. This dataset is 28 times larger than GeoQA+, greatly expanding the coverage of geometric problems. With our collected Geo170K, we derive G-LLaVA, a MLLM capable of solving geometric problems,
surpassing SOTA MLLMs by a large margin.
Specifically, G-LLaVA-13B outperforms LLaVA-13B by 27.4 on GPS minitest split of MathVista (Lu et al., ). In addition, with only G-LLaVA-7B, it is able to surpass the powerful GPT4-V on the geometry problem solving questions. Code and data will be available at [https://github.com/pipilurj/G-LLaVA](https://github.com/pipilurj/G-LLaVA).

## 2 Related Work

##### Multi-Modal Large Language Model.

Recent years have witnessed transformative advancements in the development of large language models (LLMs), characterized by a series of pioneering studies (Brown et al., ; Scao et al., ; Chowdhery et al., ; Smith et al., ; Hoffmann et al., ; Ouyang et al., ; Touvron et al., ; Bai et al., ). These breakthroughs have significantly elevated the capabilities of language understanding and generation, showcasing near-human proficiency across diverse tasks. Concurrently, the success of LLMs has inspired explorations into vision-language interaction, leading to the emergence of multi-modal large language models (MLLMs) (Liu et al., ; Li et al., ; Dai et al., ; Zhu et al., ; Dai et al., ; OpenAI, ; Bai et al., ; Su et al., ; Gao et al., ). These models have exhibited remarkable capabilities in synthesizing detailed descriptions and engaging in dialogue based on visual inputs. However, we observe that even the state-of-the-art MLLMs face challenges in resolving geometric problems using diagrams and figures.

##### Geometry Problem Solving.

The Geometry problem reasoning is an challenging visual mathematical reasoning problem. Early efforts by Seo et al. (); Sachan et al. (); Alvin et al. (); Sachan and Xing () focused on creating datasets through manual efforts. More recent approaches have introduced enhanced methods and datasets, including Geometry3K (Lu et al., ), GeoQA (Chen et al., ), GeoQA+ (Cao and Xiao, ), UniGeo (Chen et al., ), UniMath (Liang et al., ), and SCA-GPS (Ning et al., ), aiming to improve both performance and explainability. However, the scale of current datasets remains limited, and the performance of traditional models in this domain has not achieved the level observed in other areas of mathematical problem solving, particularly when compared to methods that utilize large language models for solving math word problems (Cobbe et al., ; Wei et al., ; Gou et al., ).

Figure: Figure 2: Framework of our multi-modal geometric data generation using the characteristics of geometry problems.
Refer to caption: x2.png

##### Data Generation via LLM.

Bootstrapping data from pretrained models has long been an active area of research. Ye et al. (); Meng et al. () generates training data using pretrained language models such as GPT-2 for classification tasks. Gao et al. () improves the quality of generated dataset via bi-level approach. Ye et al. () utilizes influence function to select in-context examples to aid data generation. Recently, automatic data generation becomes more ubiquitous with the advent of powerful LLMs such as ChatGPT, a line of recent works utilize ChatGPT-generated data to perform instruction tuning (Wang et al., ; Peng et al., ; Taori et al., ; Liu et al., ; Zhu et al., ; Bai et al., ; Pi et al., ; Su et al., ; Yu et al., ; Chen et al., ; Zhang et al., ).

## 3 Observation

We observe that most state-of-the-art (SOTA) MLLMs, although being adept at understanding daily visual scenes, have difficulty in comprehending geometric figures, even if they are simple and straightforward for humans. In Figure [1](#S1.F1), we demonstrate the descriptions generated by SOTA MLLMs for geometric figure. We observe that severe hallucination exists in all the generated descriptions.

More specifically, we find GPT4-V has difficulty understanding relationships between basic elements like points and lines, and also struggles with precisely interpreting these elements themselves (such as the angle B in Figure [1](#S1.F1)). Furthermore, smaller MLLMs like LLaVA1.5 and MiniGPT4 demonstrate even greater difficulty in accurately identifying the types of geometric shapes present in a figure.

This inadequacy in interpreting geometric diagrams may be one of the major causes for the failure in solving geometric problems. In contrast, actual geometric diagrams typically exhibit clear and well-defined relationships among their elements. This geometry characteristic can be utilized to develop datasets that help mitigate the above issues and mitigate hallucination.

**Table 1: Full geometric diagram description generation via inverse information recovery. The description is generated based on the textual QA pair. The upper section shows the QA pair employed to instruct text-only ChatGPT, while the lower section (in blue) shows the responses produced by ChatGPT.**
| QA Pair: |  |
| --- | --- |
| Question: As shown in the figure, circle O is the circumscribed circle of triangle ABC, and it is |  |
| known that angle ABO = 30.0, then the size of angle ACB is () |  |
| Answer: In triangle AOB, OA=OB, angle ABO=30°; Therefore, angle AOB=180°- 2 angle ABO |  |
| =120°; Therefore, angle ACB=1/2angle AOB=60° |  |
|  |  |
| Diagram Description: |  |
| The diagram consists of a triangle ABC inscribed within a circle, where the circle is denoted as circle O. Points A, B, and C are the vertices of the triangle, and they all lie on the circumference of the circle. The center of the circle is marked as point O. |  |

## 4 Geometric Data Generation

While previous efforts have been made to address multi-modal geometry problems (Chen et al., , ; Cao and Xiao, ), the availability of geometry datasets remains limited.
The key limitations of existing datasets are threefold: (1) limited data volume (a few thousands for the largest dataset), (2) absence of detailed descriptions for geometric images, and (3) a lack of diversity in problem-solving methodologies and answer pathways.
This limitation presents challenges for MLLMs in accurately understanding geometric elements and providing precise geometric solutions.

To address this issue, we utilize the geometry characteristic to construct a multi-modal geometry dataset based upon existing dataset. This dataset includes two parts: an alignment dataset to provide MLLMs with fundamental geometric knowledge and an instruction-tuning dataset to improve the assistant’s ability to understand user instructions and generate accurate geometry solutions.

**Table 2: Geometric image description and contrastive QA pairs for understanding basic elements. The generation process consists of two phases: 1) Translating the human-labelled logic form into detailed information items and a summary of the diagram description. 2) Generating contrastive QA pairs based on the provided information and summary. The sections in blue display the responses generated by ChatGPT. The detailed prompt will be included in the appendix.**
| Logic Form: |  |
| --- | --- |
| Similar(Triangle(A,B,C),Triangle(C,B,D)) |  |
| Triangle(A,B,C) |  |
| Triangle(A,C,B) |  |
| Equals(LengthOf(Line(C, B)), 5) |  |
| Equals(LengthOf(Line(C, D)), 3) |  |
| Equals(LengthOf(Line(B, D)), 4) |  |
| PointLiesOnLine(D, Line(A, B)) |  |
| Perpendicular(Line(A, C), Line(C, B)) |  |
| Perpendicular(Line(C, D), Line(A, D)) |  |
|  |  |
| Phase-1 Generation |  |
| Info Items: |  |
| The geometric figure have the following information: |  |
| Info 1: The figure represents a triangle with vertices A, B, and C. |  |
| Info 2: Triangle ABC is similar to triangle CBD. |  |
| Info 3: Point D lies on the line segment AB. |  |
| Info 4: Line segment AC is perpendicular to line segment CB. |  |
| … |  |
| Summary: |  |
| The figure depicts a triangle ABC and triangle CBD, which is similar to triangle ABC. Additionally, triangle ACB is also present in the figure. The lengths of the segments CB, CD, and BD are 5, 3, and 4 units respectively. Point D lies on the line segment AB, and perpendicularity is indicated between the line segments AC and CB, as well as between the line segments CD and AD. |  |
|  |  |
| Phase-2 Generation |  |
| QA pairs: |  |
| 1. Question: Does the diagram include a triangle formed by vertices A, B, and D? |  |
| Answer: No, the diagram does not include a triangle formed by vertices A, B, and D. The triangles present are ABC and CBD, and the point D lies on the line segment AB. |  |
| 2. Question: Does the diagram contain a triangle with vertices A, C, and B? |  |
| Answer: Yes, the diagram contains triangle ACB, which is the same as triangle ABC. |  |
| 3. Question: Is point D the lies on line BC? |  |
| Answer: No, point D is not on lines BC. The point D lies on the line segment AB. |  |
| 4. Question: Is line segment AC perpendicular to line segment CB? |  |
| Answer: Yes, line segment AC is perpendicular to line segment CB. |  |

### 4.1 Geometric Cross-Modal Alignment Data

#### 4.1.1 Geometric Image Caption Generation

Image-caption datasets play a significant role in training MLLMs for understanding the context of images, which is essential for aligning image and text modalities. In the field of geometry, there is a lack of such datasets that offer detailed descriptions of geometric diagrams. To address this issue, we propose the generation of image descriptions from labeled question-answer (QA) pairs, as illustrated in Table [1](#S3.T1). In particular, we use text-only ChatGPT 3.5 to create image captions based on these human-labeled QA pairs, which can be considered as a type of inverse information recovery. This approach leverages the strong understanding ability of ChatGPT to produce descriptions for geometric diagrams.

#### 4.1.2 Contrastive QA Pairs for Basic Elements

Our approach also involves generating QA pairs to facilitate the comprehension of geometric diagrams, focusing primarily on their basic elements. The process begins with the interpretation of human-labeled logical forms on Geometry3k (Lu et al., ). We employ text-only ChatGPT to convert these logical forms into clear descriptions that cover various geometric elements such as shapes, lines, and points, and their relationships.

After creating these diagram descriptions, the model begins to produce contrastive QA pairs. These pairs are designed to examine different aspects of the diagrams. Questions may explore the presence of certain geometric elements (e.g., "Are there triangular shapes in the diagram?") or check the accuracy of the relationships described (e.g., "Is point D the lies on line BC?"). This method enables the model to comprehend geometric concepts and to analyze and interpret the details in geometric diagrams accurately.
The generation example is shown on Table [2](#S4.T2).

**Table 3: The original example.**
| Question: |  |
| --- | --- |
| As shown in the figure, the length of the short arm of the railing at the railway crossing is |  |
| 1.0 and the length of the long arm is 16.0. When the end of the short arm drops by 0.5, the |  |
| end of the long arm rises () |  |
| Answer: |  |
| By the property of similar triangles,<br>the lengths of corresponding sides are proportional.<br>Since |  |
| the long arm is 16 times longer than the short arm, the height it rises is 16 times the height |  |
| the short arm drops, which is 0.5 multiplied by 16, resulting in a rise of 8 meters. |  |

### 4.2 Geometric Instruction Data

After performing alignment leveraging the constructed alignment data, the model is able to better interpret the geometric diagram (Figure [1](#S1.F1)). However, they are still limited at solving geometric problems. Therefore, we construct an instruction tuning dataset based on existing datasets with the help of powerful LLMs. Specifically, we design a series of strategies to expand the question-answer pairs in existing datasets. The resulting dataset contains more than 110k QA pairs, which is the largest public geometric QA dataset available. We will introduce the proposed strategies in detail below.

#### 4.2.1 Equation Solving (ES)

As shown in Table [5](#S5.T5), we replace the specific values in the original QA pairs with unknown variables and prompt the LLM to construct the solution by solving equation. Such data is helpful for the MLLM to generalize its understanding of the problem, which enables it to apply the similar reasoning and solution steps to different scenarios. The abstraction of the problem by using variables and solving equation helps the LLM focus on the underlying mathematical concepts and relationships, rather than getting caught up in specific numerical values.

#### 4.2.2 Value Scaling (VS)

As shown in Table [4](#S4.T4), we augment the data by scaling the length values in the QA pairs. Note that for the same diagram, the QA pair is still correct if all the lengths in a geometric problem are scaled simultaneously. However, note that it is not the case for quantities such as angles. When different scaling of values are applied, the LLM becomes more flexible in handling different numerical inputs. Involving a range of values that extends beyond the initial training dataset aids in refining the model’s computational and reasoning capabilities, thereby contributing to its generalizability.

#### 4.2.3 Re-Formulating Condition as Unknown (RCU)

Motivated by(Weng et al., ; Yu et al., ), we design new multi-modal QA pairs that ask questions backwards, as shown in Table [6](#S5.T6). Specifically, we reformulate questions to ask for the values originally present in the condition, and retain the generated data with correct answer only. In this way, the LLM is repeatedly exposed to the relationships between variables, equations, and their solutions.
This reinforcement helps the model learn the dependencies and connections between different elements in a mathematical problem.

#### 4.2.4 Sentence Paraphrase (SP)

We also conduct paraphrasing for both the question and answer pairs, as shown in Table [7](#S5.T7). This exposes the LLM to a broader range of phrasing and language variations. This helps the model become more robust in understanding and generating diverse sentence structures. Consequently, it can handle similar questions with different phrasings and provide accurate responses.

## 5 Model Architecture and Training

### 5.1 Model Architecture

We utilize the LLAVA (Liu et al., ) architecture for our model. The model mainly consists of a large language model (LLM) such as LLAMA-2 Touvron et al. (), a pretrained vision transformer Radford et al. () (ViT) as image encoder. In addition, a projection layer is required to map the visual features from the image encoder to the same dimension as the LLM.

**Table 5: Synthetic example via replacing values to unknown variables and solving the equation.**
| Question: |  |
| --- | --- |
| Given a railway crossing barrier with two arms, where the shorter arm has a length of ‘a’ |  |
| units (a = 1.0) and the longer arm has a length of ‘b’ units (b = 16.0), determine the elevation |  |
| ‘y’ units of the longer arm’s end when the shorter arm descends by ‘d’ units (d = 0.5). |  |
| Express your answer as a function of ‘a’, ‘b’, and ‘d’. |  |
| Answer: |  |
| To find the elevation ‘y’ of the long arm’s end, we apply the property of similar triangles and establish the equation $\frac{d}{y}=\frac{a}{b}$. Substituting the given values, we get $\frac{0.5}{y}=\frac{1.0}{16.0}$. Solving for ’y’ leads to $y=\frac{b\times d}{a}$, which simplifies to $y=\frac{16.0\times 0.5}{1.0}$. Therefore, ’y’ equals 8.0 units, indicating that the long arm rises by 8.0 units when the short arm descends by 0.5 units. |  |
| Question: |  |
| In a railway crossing lever system, the lengths of the shorter and longer arms are ‘s’ units (s = 1.0) and ‘l’ units (l = 16.0) respectively. If the shorter arm lowers by ‘deltaS’ units (deltaS = 0.5), what is the increase ‘deltaL’ in the length of the longer arm? |  |
| Answer: |  |
| The increase ‘deltaL’ in the longer arm can be found using the proportionality relationship of similar triangles.<br>The equation can be expressed as: $\frac{deltaS}{deltaL}=\frac{s}{l}$. Inserting the given values, the equation becomes $\frac{0.5}{deltaL}=\frac{1.0}{16.0}$. Solving for ‘deltaL’ yields $deltaL=\frac{l\times deltaS}{s}$, which simplifies to $deltaL=\frac{16.0\times 0.5}{1.0}$. Thus, deltaL is 8.0 units. |  |
| Question: |  |
| At a railroad crossing, the short arm of the barrier is 1.0 unit long and the long arm is 16.0 units long. If the short arm drops 0.5 units, by how many units will the long arm rise? |  |
| Represent the rise of the long arm as ‘h’. |  |
| Answer: |  |
| The triangles formed by the shorter and longer arms as they descend are similar.<br>The rise of the long arm ‘h’ can be calculated using<br>the property of similar triangles,<br>the equation is 0.5/h = 1/16. Solving for ‘h’ gives h = 16 * 0.5 / 1, which equals 8.0 units. |  |

**Table 6: Synthetic example via re-formulating condition as unknown.**
| Question: |  |
| --- | --- |
| At a railroad crossing, the short arm of the barrier is u unit long and the long arm is 16.0 |  |
| units long. When the end of the short arm drops by 0.5, the end of the long arm rises 8 units. |  |
| What is the value of unknown variable u? |  |
| Answer: |  |
| Denote the short arm of the barrier as variable u. By the property of similar triangles, |  |
| we can determine u by solving the equation 0.5/8 = u/16. Therefore, u equals 1. |  |

**Table 7: Synthetic example via sentence paraphrase.**
| Question: |  |
| --- | --- |
| In the illustration, the railing at the railway crossing has a short arm measuring 1.0 unit |  |
| in length and a long arm measuring 16.0 units. When the short arm drops by 0.5 units, |  |
| what is the corresponding rise in the long arm? |  |
| Answer: |  |
| The triangles are similar, and their corresponding sides are proportional. The long arm |  |
| is 16 times longer than the short arm, resulting in an 8-meter rise when the short arm |  |
| drops by 0.5 meters. |  |

During inference, given an image and a textual instruction, the image encoder first extracts the visual tokens from the image, which are then mapped to the dimension of LLM’s embedding space via the projection layer. Then, the mapped image features are concatenated with text embeddings to serve as the input to the LLM. Subsequently, the LLM begins to perform next-token-generation.

### 5.2 Model Training

We train our G-LLaVA in two phases, namely 1) geometric visual-language alignment, and 2) geometric instruction tuning. In both phases, we leverage the conventional language modeling loss, which can be formulated as follows:

$$ $\displaystyle\mathcal{L}(S_{tar},S_{in},I)=-\sum^{L}_{t=1}\log p\left[S^{t}_{ tar}|\mathcal{F}(s^{(<t)}_{tar},S_{in},I)\right]$ (1) $$

where $\mathcal{F}$ represents the model. $I$ represents the geometric figure; $S_{tar}$ and $S_{in}$ represent the target and input sentences, respectively; $S^{t}_{tar}$ denotes the $t^{th}$ token of target output, and $L$ stands for length.

## 6 Experiments

### 6.1 Setup

**Table 8: Comparison of model performance on the testmini set of MathVista benchmarks (Lu et al., ) on geometry problem solving (GPS) . For input, $Q$ represents for question, $I$ represents for image, $I_{c}$ represents for image caption generated by Bard, and $I_{t}$ represents fo OCR text detected in the image. Baseline results are obtained from Lu et al. (). Human performance and the results surpassing human performance are highlighted in grey. Our results are highlighted in blue.**
| Model | Input | Accuracy (%) |
| --- | --- | --- |
| Heuristics Baseline |  |  |
| Random Chance | - | 21.6 |
| Frequent Guess | - | 34.1 |
| Human | $Q,I$ | 48.4 |
| Close Source Model |  |  |
| Text-Only LLMs |  |  |
| 2-shot CoT Claude-2 | $Q$ | 29.8 |
| 2-shot CoT ChatGPT | $Q$ | 36.5 |
| 2-shot CoT GPT-4 | $Q$ | 44.7 |
| 2-shot PoT ChatGPT | $Q$ | 30.8 |
| 2-shot PoT GPT-4 | $Q$ | 33.2 |
| Visual-Augmented LLMs |  |  |
| 2-shot CoT Claude-2 | $Q,I_{c},I_{t}$ | 31.7 |
| 2-shot CoT ChatGPT | $Q,I_{c},I_{t}$ | 29.3 |
| 2-shot CoT GPT-4 | $Q,I_{c},I_{t}$ | 31.7 |
| 2-shot PoT ChatGPT | $Q,I_{c},I_{t}$ | 26.4 |
| 2-shot PoT GPT-4 | $Q,I_{c},I_{t}$ | 39.4 |
| Multimodal LLMs |  |  |
| Multimodal Bard | $Q,I$ | 47.1 |
| Gemini Nano 1 | $Q,I$ | 21.6 |
| Gemini Nano 2 | $Q,I$ | 23.6 |
| Gemini Pro | $Q,I$ | 40.4 |
| Gemini Ultra | $Q,I$ | 56.3 |
| GPT4-V | $Q,I$ | 50.5 |
| Open Source Model |  |  |
| IDEFICS (9B-Instruct) | $Q,I$ | 21.1 |
| mPLUG-Owl (LLaMA-7B) | $Q,I$ | 23.6 |
| miniGPT4 (LLaMA-2-7B) | $Q,I$ | 26.0 |
| LLaMA-Adapter-V2 (7B) | $Q,I$ | 25.5 |
| LLaVAR | $Q,I$ | 25.0 |
| InstructBLIP (Vicuna-7B) | $Q,I$ | 20.7 |
| LLaVA (LLaMA-2-13B) | $Q,I$ | 29.3 |
| G-LLaVA-7B | $Q,I$ | 53.4 |
| G-LLaVA-13B | $Q,I$ | 56.7 |

##### Dataset.

We generate the alignment data and instruction data utilizing training set of GeoQA+ (Cao and Xiao, ) and Geometry3K (Lu et al., ). More specifically, the contrastive question-answer (QA) pairs in the alignment data are generated using Geometry3K, which features human-labeled logical forms.
Note that GeoQA+ covers the training set of GeoQA (Chen et al., ), and share the same val/test set as GeoQA (Chen et al., ).
More details of data split on GeoQA and GeoQA+ is listed in Table [9](#S6.T9).
Our approach results in 60K alignment data samples, and more than 110K instruction data samples.

We compare our model with other MLLMs on the geometry problems on the minitest split MathVista (Lu et al., ), and compare our model with traditional in-domain model on the test split of GeoQA following (Chen et al., ; Liang et al., ). The geometry problems in MathVista minitest set is collected from four source datasets Geometry3K (Lu et al., ), GeoQA+ (Cao and Xiao, ), GEOS (Seo et al., ) and UniGeo (Chen et al., ).

**Table 9: Data Split of GeoQA and GeoQA+.**
| Dataset | Train | Validation | Test |
| --- | --- | --- | --- |
| GeoQA+ (Cao and Xiao, ) | 6027 | 745 | 754 |
| GeoQA (Chen et al., ) | 3499 | 745 | 754 |

**Table 10: Comparison of model performance with traditional methods on GeoQA.**
| Model | Input | Accuracy (%) |
| --- | --- | --- |
| Random Chance | - | 25.0 |
| Frequent Guess | - | 32.1 |
| Top-10 Accuracy |  |  |
| NGS (Chen et al., ) | $Q,I$ | 56.9 |
| DPE-GPS (Cao and Xiao, ) | $Q,I$ | 62.7 |
| SCA-GPS (Ning et al., ) | $Q,I$ | 64.1 |
| Top-1 Accuracy |  |  |
| Geoformer (Chen et al., ) | $Q,I$ | 46.8 |
| UniMath Liang et al. () | $Q,I$ | 50.0 |
| G-LLaVA-7B | $Q,I$ | 64.2 |
| G-LLaVA-13B | $Q,I$ | 67.0 |

##### Implementation Details.

We employ ChatGPT (gpt-3.5-turbo-0613) for data generation. A detailed description of our prompts will be provided in the appendix.
We use LLaVA (Liu et al., ) as our backbone.
More specifically, we utilize LLAMA-2 (Touvron et al., ) as the language model and employ the visual encoder of a pretrained vision transformer Radford et al. () (ViT). The resolution of the input image is 336 by 336. We conduct experiments with both 7B and 13B LLMs.
In the cross-modal alignment process, only the projection linear layer is trainable. During the instruction tuning phase, both the projection linear layer and the language model are trainable.

For training data, as we found the minitest split of MathVista contains some examples of Mix-train.pk of GeoQA+, we remove those samples that also appears in minitest split of MathVista.
The learning rate is set to $3e^{-5}$. We expand the images into squares during training, where the extended background color is set to white. For image augmentation, we set the maximum translation distance to 0.25 of the length of longer side.
If not otherwise specified, the models are trained for 1 epoch for cross-modal alignment and 2 epochs for instruction tuning, respectively. And the batch sizes are set to 6 per GPUs and 32 per GPUs, respectively.

##### Evaluation Metric.

We use accuracy as the metric for evaluation.
Note that several prior studies (Chen et al., , ; Cao and Xiao, ) report results using Top-10 accuracy (generating 10 sequences and selecting the first sequence that successfully addresses the problem as the prediction).
Our experimental results directly report Top-1 accuracy.
During instruction tuning, we enable the model to output the choice in a fixed format. For evaluation, we directly use regular expression to extract the predicted choices from the generated answers. The answer is considered false if the regular expression fails to extract a valid answer.

### 6.2 Main Experiment

We compared G-LLaVA with other MLLMs on minitest split of MathVista (Lu et al., ) benchmark on Table [8](#S6.T8). The results shows that, geometric cross-modal alignment and instructing tuning on our dataset is effective in improve MLLMs’ geometric problem solving ability. Our specific in-domain model G-LLaVA-7B can even surpass the strong GPT4-V on geometric problems.

### 6.3 Comparison with Conventional Methods

We additionally compare our method with conventional SOTA methods in geometry problem solving domain. As illustrated in Table [10](#S6.T10), our method demonstrates a notable improvement in Top-1 accuracy over the existing SOTA techniques. Moreover, our model’s top-1 accuracy outperforms the baselines’ top-10 accuracy, demonstrating a significant improvement in predictive precision.

### 6.4 Performance Across Problem Difficulties

We compare G-LLaVA with the baselines models on problems with different difficulty levels, as shown in Table [11](#S6.T11). Specifically, OP represents the number of “operations", or reasoning steps that needs to be taken for solving the problem. The results verify that our G-LLaVA consistently outperforms baseline models by a large margin across various difficulty levels.

**Table 11: Different difficulty problems on GeoQA.**
| Model | OP=1(%) | OP=2(%) | OP=3(%) | OP>=4(%) | Total(%) |
| --- | --- | --- | --- | --- | --- |
| LLaVA-7B | 16.8 | 20.9 | 15.5 | 22.9 | 18.7 |
| LLaVA-13B | 19.1 | 21.3 | 18.5 | 24.6 | 20.3 |
| G-LLaVA-7B | 77.5 | 60.8 | 54.8 | 40.9 | 64.2 |
| G-LLaVA-13B | 79.0 | 64.9 | 55.5 | 49.1 | 67.0 |

### 6.5 Performance Across Different Types of Questions

We compare G-LLaVA with the baselines models on problems with different type of questions, as shown in Table [12](#S6.T12). The results suggest that G-LLaVA performs better than the baseline models in various geometric problems such as angle, length, and area problems.

**Table 12: Performance of different types of questions on GeoQA.**
| Model | Angel | Length | Area | Other | Total |
| --- | --- | --- | --- | --- | --- |
| LLaVA-7B | 16.1 | 22.2 | 17.0 | 14.3 | 18.7 |
| LLaVA-13B | 17.5 | 23.0 | 25.5 | 28.6 | 20.3 |
| G-LLaVA-7B | 70.7 | 56.5 | 55.3 | 42.9 | 64.2 |
| G-LLaVA-13B | 71.5 | 61.1 | 63.8 | 57.1 | 67.0 |

**Table 13: Effectiveness of alignment in the pre-training phase. Top-1 accuracy is reported.**
| Model | Input | Accuracy (%) |
| --- | --- | --- |
| Random Chance | - | 25.0 |
| Frequent Guess | - | 32.1 |
| Geoformer (Chen et al., ) | $Q,I$ | 46.8 |
| UniMath Liang et al. () | $Q,I$ | 50.0 |
| G-LLaVA-7B | $Q,I$ | 64.2 |
| w/o alignment phase | $Q,I$ | 62.8 |

Figure: Figure 3: Demonstration of geometric problem solving by GPT-4-V and G-LLaVA.
Refer to caption: x3.png

### 6.6 Effectiveness of Cross-Modal Geometric Alignment

To evaluate the alignment phase’s effectiveness, we conducted the analysis of the model’s performance with and without alignment phase in Table [13](#S6.T13). The results suggest that the alignment phase enhances the model’s ability to interpret images, which is also illustrated
by the qualitative result
in Figure [1](#S1.F1).

## 7 Conclusion

In this paper, we make the attempt to address the limitations of current MLLMs in solving geometric problems. We propose strategies to enrich the data by leveraging LLMs, resulting in our augmented dataset, Geo170K. With this dataset, our G-LLaVA outperforms GPT-4-V on the geometric split of MathVista benchmark, with as few as 7B parameters. We hope our work provides new insights on improving multimodal LLMs’ ability of solving geometric problems.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Will the RCU process generate a question with multiple answers, since sometime the cause and effect are not necessarily sufficient and necessary conditions? And unfortunately, the data correctness verification cannot recognize such case.
2. In Table 10, the performance gain led by cross-modal alignment seems limited, and why the fixed LLM performs better than tunable LLM?
soundness: 3
strengths: - This paper curates a large-scale geometry dataset (28 times larger than the biggest existing geometry dataset) containing both detailed description of geometric images and reasoning paths, which may help MLLMs improve their ability in geometric comprehension.
- Data augmentation techniques like value scaling, equation solving, Re-Formulating conditions as unknown, and sentence paraphrase make it convenient to enlarge the size of dataset and enhance robustness.
- Extensive experiments validate G-LLaVA’s superiority in geometry problem solving. Experiments on MathVerse demonstrate its generalization ability.
summary: Current Multi-Modal Language Models (MLLMs) struggle to solve geometry problems for lack of comprehension of geometry information. This paper generates a large-scale geometric problem dataset named Geo170K by GPT3.5, which contains more than 170K geometric image-caption and QA pairs. The data is made up of two components: the alignment data and instruction data. The former is generated through logical forms in Geometry3K while the latter augments QA pairs in GeoQA+. Besides, a model based on LLaVa is proposed to be trained on Geo170K. Numerical results show that G-LLaVA outperforms existing traditional methods and MLLMs like GPT4-V on the geometry problem solving (GPS) tasks.
weaknesses: - The experiments are all about choice questions. Some works also involve proving[1] and completion[2] questions, which can be more difficult. Experiments on such questions are neglected.
- In many experiments, the strongest existing model GPT4-V is not included, making the result less convincing.
- The proposed G-LLaVA selects LLAMA-2 as LLM and a pretrained ViT as the vision encoder. A comparative analysis or experiment on other architecture or pretrained model may help justify this choice.

[1] Chen J, Li T, Qin J, et al. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression[J]. arXiv preprint arXiv:2212.02746, 2022.

[2] Li Z Z, Zhang M L, Yin F, et al. LANS: A layout-aware neural solver for plane geometry problem[J]. arXiv preprint arXiv:2311.16476, 2023.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: In Table 11, why does just including SP increase accuracy by 8%? This type of data augmentation doesn't enhance the model's understanding of geometry. Does this imply that the performance gains are due more to an improved understanding of the problem description rather than geometry?
soundness: 3
strengths: 1. Understanding geometric problems is important for MLLMs.
2. Presents a dataset with geometric alignment and instruction data.
summary: This paper presents G-LLaVA, an MLLM focused on geometric problems. The authors propose that the difficulty in geometric problems lies in logical structure compactness. Subsequently, they introduce a new dataset, Geo170K, with geometric cross-modal alignment data and geometric instruction data to enhance the geometric perception ability of MLLM.
weaknesses: 1. The novelty is limited, as it primarily fine-tunes an existing MLLM.
2. The models compared in Table 14 need to be updated; stronger baselines, such as QwenVL2, InternVL2 should be considered.
3. Since the authors claim that G-LLaVA enhances geometric understanding from image input, it would be interesting to see the G-LLaVA's  performance improvement when only input $Q$ is provided, in Table 8, 9 and 14.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please check the previous sections.
soundness: 3
strengths: 1. This work explores an interesting topic, geometric problem solving, and currently, most MLLMs perform poorly on this task.
2. The expression and organization of this manuscript make it easy for readers to understand.
3. The dataset proposed in this manuscript is relatively large and includes image-caption and question-answer pairs data.
summary: This paper investigates the capabilities of existing MLLMs in geometric problem-solving tasks and introduces the Geo170K dataset, which contains over 170K image-caption and QA pairs. It also proposes training the G-Llava using the proposed dataset and validates its performance on the GeoQA dataset, demonstrating its effectiveness.
weaknesses: 1. Novelty and Technical Contribution: The proposed geometric dataset construction approach relies on the existing logical annotation data and uses ChatGPT for automatic labeling. Besides, the overall model structure and training strategy mainly apply the existing techniques, lacking sufficient innovation. Please explain the differences in model structure or training strategy during the rebuttal phase and whether the authors made customized designs on a geometric domain. No specific considerations for the geometric domain were found.
2. Data Quality Issues: Since annotations are generated automatically by ChatGPT, is there any hallucination in the dataset? Is there any quality inspection conducted to support the claim of a "high-quality dataset"? 
3. Insufficient Evaluation: The paper lacks quantitative comparisons on geometry-task-related benchmarks like UniGeo, Geometry3K, and PGPS9K, commonly used in prior studies [1-4].

[1] Chen J, Li T, Qin J, et al. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression[J]. arXiv preprint arXiv:2212.02746, 2022.
[2] Liang Z, Yang T, Zhang J, et al. Unimath: A foundational and multimodal mathematical reasoner[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023: 7126-7133.
[3] Zhang M L, Yin F, Liu C L. A multi-modal neural geometric solver with textual clauses parsed from diagram[J]. arXiv preprint arXiv:2302.11097, 2023.
[4] Li Z Z, Zhang M L, Yin F, et al. LANS: A layout-aware neural solver for plane geometry problem[J]. arXiv preprint arXiv:2311.16476, 2023.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to the Weaknesses for more details. All in all, the authors are encouraged to append the discussion or experiments about recent progress in both dataset and methods.
soundness: 2
strengths: It's a pioneering work on MLLM's mathematical reasoning especially focusing on geometry problem. Compared with its earlier version, the authors added more experiments on recent datasets like MathVerse.
summary: The paper is a pioneering work about Geometry Problem Solving using MLLM. The contribution is to provide a dataset and also a baseline using LLaVA.
weaknesses: The work was a milestone in developing mathematical skills of MLLM and the authors tried to append results on a latest dataset. However, the research in the area moves fast in both dataset construction and methods. 
Despite the updates, the paper is still flawed in
1. Its targeted problem, i.e., geometry problem solving (GPS), is limited nowadays. The mathematical reasoning of MLLM evolves fast with the corresponding dataset. Now, it also covers the function plots in MathVerse and chart in MultiMath.
2. The baseline needs a fair comparison. Even though the problem is confined to GPS and methods in comparison is confined to open-source methods, the methods like MathLLaVA, MultiMath and Math-PUMA are encouraged to be compared with.

Meta-review: This paper introduces G-LLaVA, a Multimodal Large Language Model fine-tuned with Geo170K, a dataset of 170,000 geometric image-caption and QA pairs. It is recognized as a pioneering effort in addressing geometric problem-solving in MLLMs, with an innovative data generation approach that effectively utilizes geometric logic, demonstrating scalability and substantial performance improvements.

While concerns were raised about methodological novelty and data quality assurance, the authors provided thorough explanations and evidence to address these issues. They emphasized that the primary contribution lies in the customized data generation tailored to the geometric domain, enabling effective training without altering the model architecture. The adoption of their dataset by leading institutions further highlights its value.  

This paper makes a significant contribution to advancing MLLMs' geometric reasoning capabilities through an innovative, data-centric approach. It addresses a critical gap in the field and offers valuable resources for future research.

Decision: Accept (Poster)