LANS: A Layout-Aware Neural Solver for Plane Geometry Problem (2024)

Abstract: Geometry problem solving (GPS) is a challenging mathematical reasoning task requiring multi-modal understanding, fusion, and reasoning. Existing neural solvers take GPS as a vision-language task but are short in the representation of geometry diagrams that carry rich and complex layout information. In this paper, we propose a layout-aware neural solver named LANS, integrated with two new modules: multimodal layout-aware pre-trained language module (MLA-PLM) and layout-aware fusion attention (LA-FA). MLA-PLM adopts structural-semantic pre-training (SSP) to implement global relationship modeling, and pointmatch pre-training (PMP) to achieve alignment between visual points and textual points. LAFA employs a layout-aware attention mask to realize point-guided cross-modal fusion for further boosting layout awareness of LANS. Extensive experiments on datasets Geometry3K and PGPS9K validate the effectiveness of th layout-aware modules and superior problemsolving performance of our LANS solver, over existing symbolic and neural solvers. We have made our code and data publicly available.1

heading: 1 Introduction
text: Automatic geometry problem solving (GPS) is a long-standing and challenging research topic in both computer vision and natural language processing communities (Bobrow, 1968; Chou et al., 1996; Seo et al., 2015). Each geometry problem consists of a geometry diagram and a textual problem in different modal forms, complementing each other. GPS necessitates comprehensive mathematical reasoning and multi-modal understanding, making it a pivotal testbed for evaluating the high-level multimodal reasoning ability of artificial intelligence. Past research works of GPS were mainly focused
* Equal Contribution † Corresponding Author 1https://github.com/zzli2022/LANS
on symbolic solvers (Seo et al., 2015; Sachan and Xing, 2017; Lu et al., 2021), which are criticized in respect of complex rules and poor adaptability. With the development of deep learning, neural solvers (Chen et al., 2021, 2022; Zhang et al., 2023, 2024), treating GPS as a special vision-language reasoning task, have attracted dominant attention recently.
Layout information is typically defined as positional coordinates of elements such as text, paragraphs, tables, and figures within images (Xu et al., 2020; Gupta et al., 2021). Supplying layout details for elements in document images facilitates parsing reading sequences, executing information extraction, and enhancing document comprehension (Appalaraju et al., 2021; Wang et al., 2021; Hong et al., 2015). In the layout of geometric diagram, the coordinate positions of geometric points and symbols play a crucial role in understanding the elements within geometric diagrams. For example, the coordinate positions of geometric symbols "A" as shown in Figure 1, determine which geo-
1 2596
metric points are named A, while the coordinate position of the non-geometric symbol "120◦" determines the numerical assignment of ̸ ABD instead of other angle.
Despite considerable efforts devoted to constructing proficiently crafted representations for geometric diagrams, the explicit fusion of positional information into geometric diagrams remains unexplored. Existing neural solvers have adopted different diagram representation schemes, such as feature maps (Chen et al., 2021; Cao and Xiao, 2022; Ning et al., 2023), image patches (Chen et al., 2022; Ning et al., 2023) and textual clauses (Lu et al., 2021; Zhang et al., 2023). For methods based on image patches and feature maps, several representative geometric problem solvers have employed extensive pre-training strategies, such as jigsaw location prediction (Chen et al., 2021), geometry elements prediction (Chen et al., 2021), masked image modeling (Ning et al., 2023), and character alignment (Ning et al., 2023), to bridge the gap between geometric and natural scene images (Anderson et al., 2018; Yu et al., 2019; Ding et al., 2022). Although rough image pre-training methods have achieved some effectiveness, they often fail to capture finer-grained details. Conversely, methods based on text clauses extract the crucial structural and semantic information of geometric problems in the form of clauses. Currently, clausebased approaches yield superior inference results through clause-based deductive reasoning (Lu et al., 2021) or clause pre-training (Zhang et al., 2023). We attribute this to the structured nature of clauses, which makes them more adept at capturing structural information in geometric problems. For example, The structural clause “line B C D" describes a structural relationship that points “B", “C" and “D" lie on one line in order. The semantic clause “m B̂C = 120" illustrates a semantic relationship for the degree of arc “B̂C" and text “120◦".
Although the textual clauses are capable of capturing the primary layout relationships within the images, they lose significant spatial information during the conversion process of diagram parsing (Lu et al., 2023; Trinh et al., 2024). They cannot distinguish the geometry diagram (a) and (b) displayed in Figure 1 because of loss of position information. For example, “ ̸ CBA" in Figure 1(a) and (b) need the spatial relationship to determine whether it is acute or obtuse. The lack of positional indicators for geometry elements (such as "A," "B," etc.) makes it challenging for neural solvers based

heading: Self-limited GRU Decoder

heading: Sum N0 N1 V0 Gougu N1 V1 V0 Multiple V1 C2 V2 Get V2

heading: Layout-Aware Fusion Attention
text: on text clauses to distinguish between these ambiguous scenarios.
Considering the under-representation of geometry diagrams, we propose a layout-aware neural solver called LANS. LANS inputs the diagram image, the textual clauses parsed from a diagram, and the textual problem, and outputs the explainable solution program to solve the geometry problem. As shown in Figure 2, two new modules, multimodal layout-aware pre-trained language model (MLA-PLM) and layout-aware fusion attention (LA-FA), are proposed to endow LANS with layout awareness. We introduce a point-match pretraining (PMP) method within MLA-PLM. This method, based on contrastive learning, aims to model the relationship between text clauses and diagrams using layout information in a data-efficient manner. When integrated with structural-semantic pre-training (SSP) in PGPS solver (Zhang et al., 2023), it shows promising outcomes. Then, to better utilize pre-trained multimodal representations, LA-FA module with the layout-aware attention mask is employed in LANS to fuse the diagram and text clauses representation via point positions. LA-FA further enhances the layout awareness in cross-modal fusion.
The contributions of this work are summarized in four folds: (1) We propose a layout-aware neu-
2
ral solver LANS for GPS, which can represent and fuse geometry diagrams effectively. (2) We introduce the MLA-PLM module with two pretraining strategies SSP and PMP, realizing global relationship modeling and cross-modal alignment of point primitives. (3) We design the LA-FA module, equipped with a layout-aware attention mask directed by point positions, to further strengthen the layout awareness of LANS. (4) Our LANS outperforms existing symbolic solvers, neural solvers, and current multimodal large models significantly on Geometry3K and PGPS9K datasets.

heading: 2 Related Work

heading: 2.1 Geometry Problem Solving
text: GPS is a special type of multimodal reasoning that examines geometric spatial structure cognition and mathematical logical reasoning, and also requires the application of geometric theorem knowledge, which make it highly challenging. Existing works of GPS can be classified into two categories: symbolic solvers and neural solvers. The symbolic solvers (Seo et al., 2015; Sachan and Xing, 2017; Lu et al., 2021; Peng et al., 2023) parse the diagram and textual problem into a unified formal language first, and then perform symbolic reasoning by path search and condition matching based on the geometric theorem knowledge. However, symbolic solvers are carefully designed with complex rules and are hard to extend. The neural solvers treat GPS as a visual question answering task and design a special interpretable program to represent the problem-solving process. NGS (Chen et al., 2021) and Geoformer (Chen et al., 2022) use auxiliary self-supervised tasks such as location prediction, elements prediction, and knowledge classification to boost cross-modal semantic representation. PGPSNet (Zhang et al., 2023) expresses the geometry diagram with textual clauses and fuses multi-modal information through structural and semantic pretraining, data augmentation, and self-limited decoding. SCA-GPS (Ning et al., 2023) tries to align characters in text and diagram and enhance the diagram understanding through multi-label classification and masked image modeling pre-training. Although existing neural solvers have achieved impressive performance, they are still coarse-grained at the modal understanding and fusion, especially for geometry diagrams with complex layouts. In this paper, we propose a layout-aware neural solver to improve the understanding and fusion of geome-
try diagrams and therefore promote GPS.

heading: 2.2 Multimodal Pre-training & Layout-Aware Learning
text: Multimodal pre-training realizes alignment and understanding between different modalities by a series of designed auxiliary tasks and then applies to the specific downstream tasks. Common strategies involve image-text contrastive learning (Radford et al., 2021), image-text matching (Kim et al., 2021), image-grounded text generation (Cho et al., 2021), and masked object classification (Li et al., 2020). With a large amount of pre-training data, these strategies exhibit good performance in multimodal tasks for natural images. However, their alignment methods are coarse-grained and straightforward and do not fit for complex multi-level and fine-grained tasks. Most relevant to our work is the research on document analysis (Liu et al., 2023a). Existing advanced document pre-training methods (Xu et al., 2020, 2021) incorporate textual and visual blocks with fine-grained position embeddings, and adopt masked visual-language modeling and text-image alignment to pretrain document layout, whereas they still do not apply to GPS due to the specificity of geometry objects and small-scale of GPS datasets. DocFormer (Appalaraju et al., 2021) and LayoutReader (Wang et al., 2021) employ meticulously designed attention mechanisms targeting information within text boxes to enhance their perception abilities regarding document content. Our LANS proposes targeted and data-efficient pretraining methods and a geometry layout-aware attention to implement geometry layout awareness.

heading: 3 Method
text: Before presenting the neural solver model, we first describe the formal definition of GPS task here. Given a geometry problem P including a geometry diagram D and a textual problem Tprob, the goal is to solve the problem by applying geometric knowledge and obtaining the solution steps S, formulated as P = {D,Tprob} ⇒ S. Then solution steps are verified in the form of fill-in-the-blank, multiple-choice, or logical reasons.

heading: 3.1 Overall Framework
text: To fully understand and represent the geometry diagram, we propose a layout-aware neural solver called LANS as displayed in Figure 2. First, the diagram is parsed into the textual clauses using the geometry diagram parser PGDPNet (Zhang
3
et al., 2022), where the structural clauses Tstru describe the connection relations among geometric primitives and the semantic clauses Tsem depict the semantic relations between non-geometric primitives and geometric primitives (Zhang et al., 2023). Besides, the visual information of diagram image is represented as patches. Therefore, the input of LANS could be further expressed as{ D={di}NDi=1 , T={Tstru, Tsem, Tprob}={tj}NTj=1 } after token concatenation, where ND is the diagram patch number and NT is the text token number. Then, these modal tokens are fed into the multimodal layout-aware pre-trained language model (MLA-PLM) and input into the bidirectional GRU encoder to perform fusion encoding. Next, the mixed encoding context H = {hi}ND+NTi=1 leverages the layout-aware fusion attention (LA-FA) to further boost diagram layout awareness. Finally, the enhanced context is decoded by the self-limited GRU decoder and generates the sequential solution program S in the manner of autoregressive.

heading: 3.2 Multimodal Layout-Aware Pre-training
text: Geometry problems are often solved by humans by depicting the geometric structure in visual form no matter whether it has the geometry diagram or not. Previous neural geometric solvers, such as the NGS (Chen et al., 2021), PGPSNet (Zhang et al., 2023) and SCA-GPS (Ning et al., 2023), do not utilize the diagram layout adequately, thus resulting in unsatis-
factory performance of GPS. In this paper, we propose the multimodal layout-aware pre-trained language model (MLA-PLM), with two pre-training strategies: structural-semantic pre-training (SSP) and point matching pre-training (PMP) illustrated in Figure 3, to boost the diagram layout-aware ability during the pre-training stage.

heading: Revisit Structural-Semantic Pre-training To
text: enable the multimodal pre-training module to comprehend text clauses and gain a preliminary understanding of the content and layout of geometry diagrams, we adopted the structural-semantic pretraining (SSP) (Zhang et al., 2023) method used in PGPS. MLA-PLM is trained to recover the masked text in a unified text generation manner, and the training loss denotes as LSSP . Concretely, inputs of MLA-PLM include the diagram patch embeddings eDi and textual token embeddings e T j , where eDi is obtained via patch projection and patch-level positional encoding, and eTj fuses not only positional encoding but also embedding of class tag and section tag following (Zhang et al., 2023) as:
eDi =PatchProj(di)+PosEmb(i), 1≤ i≤ ND eTj =TokenEmb(tj)+PosEmb(j)+ClassEmb(tj)
+ SectEmb(tj), 1 ≤ j ≤ NT ,
(1) where PosEmb(∗) is the sequential position encoding of sequences instead of the spatial position of the diagram layout. The concatenated eDi and e T j are modeled by MLA-PLM and then output e′Di
4
and e′Tj . For SSP in MLA-PLM, we mask 30% of text tokens tj with mask token [M ] following (Cho et al., 2021) but keep tags unchanged.
Point-Match Pre-training We propose the PMP based on contrastive, learning modeling to achieve cross-modal alignment between visual points (one type of geometric primitives in the diagram) and textual tokens of the points. For PMP, we match image patches and points inside image patches with the cosine contrastive loss (He et al., 2020; Grill et al., 2020) as follows:
LPMP = −1 |P|
∑ j∈P log exp(cos⟨e′Tj , e′D+⟩/τ)∑ND i=1 exp(cos⟨e′Tj , e′Di ⟩/τ) ,
(2) where P = {j | Class(tj) = [P], 1 ≤ j ≤ NT } is the index list of text tokens corresponding to points, e′D+ is the embedding of the diagram patch that the point tj is located in, and τ is the temperature coefficient that empirically set as 0.1. Combining SSP and PMP, our pre-training loss is a multi-task learning loss with the mixed training loss Lall = LSSP + LPMP .
By combining two pre-training strategies SSP and PMP, the solver strengthens the cognition of complex geometry layout. In SSP, the modeling of local relationships leads to the global relationship understanding, for example, we can infer that the mask token in the semantic clause “BD ⊥ EA on [M]” is “C” according to structural clauses “line B C D” and “line E C A”. Via PMP, the textual points become aware of layout position from positional encoded image patches by alignment. We do not adopt the simple and direct way of fine-grained 2D position embedding such as in LayoutLM (Xu et al., 2020, 2021). This is because existing GPS datasets do not support large-scale layout understanding pre-training. It is also akin to human geometric cognition in that accurate positioning is not required to understand geometry layout.

heading: 3.3 Layout-Aware Fusion Attention
text: Although LANS has already acquired a certain level of layout understanding through the pretraining strategies above, this ability can fade to some extent during downstream training because of the different training targets of GPS. To compensate for the loss of layout awareness in the GPS training phase, we propose layout-aware fusion attention (LA-FA) to enhance the intra-modal and cross-modal token fusion. LA-FA is located be-
tween the bidirectional GRU encoder and the selflimited GRU decoder.
As shown in Figure 4, the LA-FA module is similar to the transformer encoder block (Vaswani et al., 2017) which also contains layer normalization, feed-forward layer, and residual connection except the layout-aware self-attention. Our layout-aware self-attention uses the carefully designed layoutaware attention mask which allows visibility to all intra-modality tokens but restricts cross-modality visibility if the textual point is not inside the image patch in the visual space. Specifically, we construct the mask matrix Mi,j (1≤ i, j≤ND+NT ), which consists of value 0 as invisible and value 1 as visible:
Mi,j=    1, if (i, j) ∈ VV 1, if (i, j) ∈ TT 1, if (i, j) ∈ VT&Pos(tj) ∈ Reg(di) 0, otherwise (3) where VV = {(i, j) | 1≤ i, j ≤ND} is the mask region of visual intra-modality, TT = {(i, j) |ND+ 1 ≤ i, j ≤ ND+NT } is the mask region of textual intra-modality, VT = {(i, j) | 1 ≤ i, j ≤ ND+NT }−VV −TT is the mask region of crossmodality, Pos(tj) denotes the visual position of point token tj and Reg(di) refers to the visual region of image patch di. Moreover, layout-aware fusion-attention (LA-FA) could be computed by:
LA-FA(Q,K, V,M)=softmax ( QKT√ mk ·M ) V
(4) where Q,K, V are query matrix, key matrix, and value matrix all transformed from encoding context H , and mk is the dimension of the key vector.
In summary, in the process of cross-modal fusion, LA-FA leverages the point position to guide
5
the attention between diagram and text, strengthening the understanding of diagram layout. For mitigating the optimization burden, we only use one LA-FA block, as adding more blocks does not bring extra improvement according to our experiments.

heading: 4 Experiments

heading: 4.1 Setup
text: Model Architecture The patch projection module for diagram chooses the CNN architecture, selecting a light-weight ResNet10 (He et al., 2016) to extract feature map before meshing. Feeding with diagram images resized as 256×256, the patch projection module maps diagram into 8×8=64 image patches. In default, we employ a 6-layer, 8-head, 256-input, and 1024-hidden dimensional transformer (Vaswani et al., 2017) as the architecture of MLA-PLA, and a multi-head attention with the same head number and feature dimension for LA-FA. The bidirectional GRU encoder and selflimited GRU decoder in LANS are adopted following the same architecture as PGPSNet (Zhang et al., 2023). Besides, a dropout layer with the value 0.2 is added behind the patch projection module to prevent overfitting during the training stage.
Training Hyperparameters Details We choose the AdamW optimizer (Loshchilov and Hutter, 2017) with the weight decay 1× 10−2 and the step decline schedule with the decay rate of 0.5, and the training batch size is set as 128. We provide a more detailed description of the remaining parameters we use during the pre-training and fine-tuning stages in the appendix B.1.
Datasets and Metrics We evaluate the performance of proposed LANS on two plane geometry problem datasets: Geometry3K (Lu et al., 2021) and PGPS9K (Zhang et al., 2023). They all have fine-grained diagram annotation and interpretable solution programs. The textual clauses and point positions used in this paper are converted from the diagram annotation. The solution program consists of several solving steps, each step consists of an operator and associated operands, where the operator corresponds to a geometric theorem and operands are arranged according to the theorem formula. The paired program executor based on Python calculates the numerical results of solution programs. The MLA-PLA module of LANS is pre-trained from scratch on PGPS9K dataset that
masks solution programs, because of the shortage of geometric corpus and the great distribution gap in contrast with natural corpus.
Similar to PGPSNet (Zhang et al., 2023), we use three evaluation metrics to assess the numerical performance of our LANS, namely Completion, Choice, and Top-3. In the Completion, the neural solver selects the first executable solution program as the Completion result. The Choice is defined as choosing the correct option from four candidates but selecting one randomly if the outputted answer is not in. In the Top-3, the solution is considered correct if it is among the top three confidence solutions. We set the Completion as evaluation metric for ablation study in section 4.3 by default. Given the outstanding capabilities of multimodal large models in addressing multimodal reasoning problems, we compared popular existing open-source multimodal large models in Table 1 with the currently most powerful multimodal model, GPT-4V. Evaluation was conducted in both Completion and Choice modes, where in Completion mode, the large model was required to directly provide answers, and in Choice mode, reference options were added to the prompt for the large model.

heading: 4.2 Comparison with State-of-the-art Solvers
text: We compare LANS with state-of-the-art models, including neural solvers, symbolic solvers, and multimodal large models in Table 1, in terms of both performance and parameter quantity. The results indicate that our LANS achieves excellent model performance by incorporating efficient parameters.
As to symbolic solvers, InterGPS (Lu et al., 2021) solved geometry problems by searching and matching with unified formal language. According to the input source of formal language, InterGPS presents three types of results, e.g., “Predict" means that all formal language is predicted by its parsers, “Digram GT" denotes that formal clauses of diagram use ground truth, and “All GT" indicates that formal clauses of diagram and textual problem are all ground truth. GeoDRL (Peng et al., 2023) improved the search strategy of Inter-GPS with logical graph deduction and deep reinforcement learning. Experimental results show that our LANS outperforms symbolic solvers on all datasets and in all evaluation metrics. Even compared with InterGPS (All GT) which uses annotated formal clauses designed carefully, LANS gains a 3.1% improvement in Completion and a 6.4% improvement in Choice mode on Geometry3K Dataset.
6
As to neural solvers, NGS (Chen et al., 2021) and Geoformer (Chen et al., 2022) relied primarily on textual problems to solve problems. Even though re-implementing them with the textual clauses parsed from the diagram and the same augmentation strategies, performance gaps between these two solvers and our LANS are still significant, 32.6% and 31.1% lower in Completion on PGPS9K, respectively. SCA-GPS (Ning et al., 2023) shows similar performance as InterGPS (All GT) because diagram understanding methods, character alignments, and masked image modeling, are coarse-grained and ineffective. PGPSNet (Zhang et al., 2023) employed textual clauses to model diagram layout but lost lots of visual information. Our LANS is enhanced at modal alignment and fusion for better layout awareness and surpasses PGPSNet by 7.1% and 4.0% in Completion on Geometry3K and PGPS9K. The improvements in Top-3 are less than in Completion because most of the correct solutions are concentrated among highly confident candidates.
Our approach far surpasses the performance of current multimodal large models. This may be attributed to the presence of complex symblic OCR information, layout details, and abstract elements in geometric images, where the perception capabilities of the Gemini (Team et al., 2023) and GPT4V (Achiam et al., 2023b) models are insufficient. Similar phenomena have also been observed in the MathVista (Lu et al., 2024) benchmark. For evaluation results and detailed information on the Com-
pletion and Choice multimodal large-scale models, please refer to section C.

heading: 4.3 Ablation Study
text: Effect of Modules To examine the effect of our proposed modules in LANS, we conducted ablation experiments on the Geometry3K dataset, taking PGPSNet solver (Zhang et al., 2023) who owns the SS-PLM module but without the LA-FA module as the baseline. Experimental results presented in Table 2 illustrate that MLA-PLM module with multimodal pre-training is superior to SS-PLM module with only text-modal pre-training and obtains a 5.4% improvement. LA-FA module further boosts GPS via multi-modal feature fusion in the training phase and achieves a 72.1% accuracy, over baseline 7.1%.
Role of Pre-training Strategies To validate the role of pre-training strategies within MLA-PLM, we did ablation experiments on both SSP and PMP pre-training strategies. Ablation experiments involved two processes: first pre-training with various strategies and then fine-tuning on Geometry3K. Table 3 verifies that SSP and PMP pre-training strategies all improve GPS, where SSP promotes
7

heading: Problem:

heading: Problem:
text: global relationship recognition and PMP aligns visual points and textual points. The comparison between row 2, row 3, and row 4 demonstrates that the combination of SSP and PMP realizes complex layout understanding synthetically, thus promoting problem-solving together.
Role of Attention Mask To validate the role of attention mask within the LA-FA module, we compare three types of attention masks: w/o LA-FA, vanilla attention mask (Vaswani et al., 2017), and layout-aware attention mask. Compared with the vanilla attention mask with global visibility, layoutaware attention mask guided by point positions promotes modal fusion and strengthens diagram understanding. The results in Table 4 also indicate the significance of layout-aware attention.

heading: 4.4 Case Analysis and Fail cases
text: We also conducted a case analysis to discuss the strengths and weaknesses of solvers. Figure 5 displays four plane geometry problems (a)-(d) involving various geometric layouts, and they rely on good layout awareness to solve them. In case (a), the position of C relative to F determines if ̸ JIK and ̸ DEC are corresponding or alternate angles. Results show LANS identifies corresponding angles accurately, unlike other solvers. In case (b), the perception of polygon edge number is the key to solving this problem. Contrary to LANS, other solvers cannot count edge numbers correctly through the diagram or textual clauses, resulting in a wrong solution. Case (c) is the same problem as shown in Figure 1 in which textual clauses cannot identify diagram uniquely. In contrast with PGPSNet, LANS can judge the orientation and type of ̸ ABC and get the right solution.

heading: 5 Conclusion
text: We propose a layout-aware neural solver LANS to understand complex layouts of plane geometry diagrams. Benefiting from the multimodal layoutaware pre-training, LANS is endowed with abilities of global relationship cognition and cross-modal point alignment. Thanks to layout-aware fusion attention, LANS further improves cross-modal fusion directed by point positions. The experimental results demonstrate the superiority of LANS enhanced with layout awareness.
8

heading: Limitations
text: LANS is still limited to point primitives to carry out layout understanding. In the future, we will try to align higher-level geometric primitives to obtain better layout understanding and modal fusion. Besides, LANS may generate redundant solution sequences. Case (d) in Figure 5 is a complex layout scenario that none of the solvers can solve correctly. In conclusion, the case analyses above fully indicate that LANS promotes GPS with enhanced layout awareness. Integrating richer layout information and symbolic cues of elements through multimodal pretraining is a direction worthy of further exploration.
Ethical Impact
As a neural solver addressing multimodal mathematical problems, LANS has the potential for application in educational settings, specifically for the automatic resolution of mathematical problems. This utilization can contribute to promoting educational equity.

heading: Acknowledgments
text: This work has been supported by the National Key Research and Development Program Grant 2020AAA0109700, and the National Natural Science Foundation of China (NSFC) Grant U23B2029.

heading: A Dataset Details
text: We evaluated our method on two datasets, Geometry3K and PGPS9K, each containing high-quality diagram images. The Geometry3K dataset exists in two versions, provided by PGPSNet and InterGPS, respectively. These two versions have different annotation formats tailored for training symbolic solvers and neural solvers. As a neural solver, we adopted the Geometry3K provided by PGPSNet. It is worth noting that the Geometry3K and PGPS9K datasets provided by PGPSNet are different splits of the same dataset.
10

heading: B Training Details
text: To ensure the reproducibility of the paper, we provide here the key hyperparameters used during training, as well as the data augmentation methods employed. Additionally, within our method framework, how Patch Projection is relied upon and the granularity of Patch division are crucial for achieving the effectiveness of our approach as described in the paper. We discuss here the impact of these parameters on the replicability of the model.
LANS, like PGPSNet, follows a two-stage training process. The model is first pre-trained using our proposed pre-training task. It is then fine-tuned meticulously. We used the clauses and key point location information provided in PGPS9K for pretraining.

heading: B.1 Optimzation Parameters Details
text: During the pre-training phase, the learning rate is initialized to 5× 10−4 and the learning rate decay is applied at 1,000, 1,800, 2,400, and 3,000 epochs with a total of 3,500 epochs. During the training stage, all modules of LANS train together with an initial learning rate as 1e−4 for language model MLA-PLM and 1e−3 for other modules, decaying at 160, 280, 360, 440 and 500 epochs uniformly with a total 520 epochs.
All experiments were conducted on an 8-GPU Titan XP server. Training of the MLA-PLM module took approximately 20 hours on a 4-GPU machine, while fine-tuning of LANS on 4 GPUs took 8 hours.

heading: B.2 Data Augmentation Details
text: We scale the image to 256 on the longest side and place it in the center of 256×256 blank screen. The diagram is flipped randomly and changes the point positions accordingly. For text, following the work (Zhang et al., 2023), we apply four augmentation strategies: token replacement, connection rotation, representation transposition, and clauses shuffle. These augmentation strategies not only improve the diversity of geometry problems but also provide geometric solvers with basic geometric representation knowledge.
B.3 Impact of hyperparameters
Discussion on the Granularity of Patch Division. To assess the influence of image patches, we adopted four configurations of patch numbers: 1×1, 4×4, 8×8, and 16×16. In Table 5, we
observe that LANS benefits from fine-grained partitions of the diagram, based on the comparison of row 1 with rows 2, 3, and 4. However, according to the comparison of row 3 with row 4, problemsolving performance declines if the diagram is over-segmented. The possible explanation is that redundant and blank image grids, which are generated from patch partition, interfere with model attention while increasing the burden of model computation. Therefore, considering overall performance and speed, we choose the 8×8 configuration as our model setup.
Discussion on the Projection Method of Image Patches. To validate the impact of patch projection schemes, in Table 6, we tested three types of patch projection modules: None, linear layer, and CNN model. None refers to not using the patch projection module and also not inputting image patches. In our experiments, we find that a redundant placeholder in None does harm to GPS due to additional meaningless optimizations. The linear-based patch projection maps image grids linearly and produces corresponding image patches, which is also commonly adopted in recent transformer architectures (Kim et al., 2021; Li et al., 2022). However, this module does not fit to geometry diagram because it may damage the geometric structure. CNN-based patch projection first extracts global features and then mesh feature maps. That module could better understand the overall layout, bringing with higher solving performance and more stable training, and it is also set as the default patch projection module.
11

heading: C Mulitmodal LLM Eval Details

heading: C.1 Eval Prompt Details
text: We illustrate in Table 7 with examples of how the prompts used for evaluating the multimodal large model vary across different Eval Mode. Our Prompt consists of several components, including Role Prompt, Diagram, Question, Choices, and Format Prompt. The Role Prompt specifies the type of problem the large model is tasked with solving and the actions it needs to perform. Diagram depicting the form’s content, textual description of the Question, and Choices. To ensure the large model generates standardized output for easy scoring, we have set a Format Prompt. The main difference in evaluation between Choice and Completion modes lies in that, in completion mode, the prompt does
not provide reference options. We conducted repeated tests using multiple sets of different System Prompts and obtained the final average value.

heading: C.2 Generation Parameter Details
text: We list the relevant important parameters used for evaluation across different multimodal large models in Table 8.
C.3 Output detail and discussion of MLLM Table 1 demonstrates that multimodal large models perform poorly in solving geometric problems. Some open-source models exhibit performance lower than random guessing for select questions. We attribute this to the fact that the visual comprehension component of current multimodal large models is primarily tailored to natural scene images and struggles with abstract forms. For instance, in Figure 6, both GPT4V and LLaVA-v1.5 fail to grasp the relational elements within geometric diagrams, such as understanding the reference to Angle 1. Additionally, models like LLaVA-v1.5 may also generate severe model hallucinations, even output in an incorrect format like "So, the answer is [28.0, 56.0, 62.0, 90.0]," which prevent the extraction of correct results and result in low accuracy.

heading: D Discussion about the impact of clause source on LANS model performance
text: Like InterGPS, we also evaluated the performance of LANS under different clause sources. The results show that the results obtained by using PGDP
12
extraction can still achieve excellent solution performance.
D.1 Discussion about GPT4V’s ability to understand clauses
We removed the clauses generated by PGDP in GPT4V’s Prompt and found that the performance of GPT4V dropped significantly, which shows the importance of using clauses or structured language to understand geometric problems and that GPT4V may have a large number of perceptual illusions.
13

heading: Future Work
text: In the future, we will try to align higher-level geometric primitives to obtain better layout understanding and modal fusion. Besides, LANS may generate redundant solution sequences. Case (d) in Figure 5 is a complex layout scenario that none of the solvers can solve correctly