GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning

Authors: mehran  kazemi
hamidreza  alvari
ankit  anand
jialin  wu
xi  chen
radu  soricut
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-05-22 15:38:10

Abstract: Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual. Many mathematical reasoning problems, however, contain both text and image. With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial. 
In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems. 
We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes, thus enabling a systematic evaluation.
The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry (and, by generalization, other topics requiring similar reasoning) as suggested by previous benchmarks. This is made especially clear by the construction of our benchmark at various depth levels, since solving higher-depth problems requires long chains of reasoning rather than additional memorized knowledge.

Main paper: 
 introduction: 
 Multi-hop reasoning is a fundamental element in intelligence: it allows us to combine multiple pieces of information to answer questions or solve problems. While formal reasoning such as automated theorem proving (Robinson, 1965; Kovács & Voronkov, 2013; Schulz, 2002) has been a key focus in the AI literature, recent years have witnessed a great amount of progress in multi-hop reasoning with natural language thanks to the advances in pre-trained large language models (LLMs) (Wei et al., 2022; Nye et al., 2022; Kazemi et al., 2023b; Saparov et al., 2023; Yao et al., 2023;  Figure 1 : Sample GeomVerse problem. Question: If the ABEF shape is a rectangle where a semi-circle has been removed from one side of it, the perimeter of the ABEF shape is 34 [...] compute the degree of the DAB angle. Assume π = 3.14. Round computations to 2 decimal places. Solution: The diameter of the semi-circle in the ABEF shape is equal to the side of the rectangle with length 7 so the shape has two sides with equal but unknown lengths, one side with length 7, and one semi-circle arc with diameter 7. So the perimeter is 2 * U nknownSide + 7 + 7π 2 [...] the length of the AB side is 16.01 2 = 8. [...] the final answer is 28.69. Pan et al., 2023) . Among various types of multi-hop reasoning, mathematical reasoning has turned into a key focus domain for AI researchers (Lu et al., 2022; Lewkowycz et al., 2022) with many recent works targeting to solve open problems in mathematics (Fawzi et al., 2022; Davies et al., 2021) . It is an appealing domain for AI research due to various reasons: it is a primitive skill that is essential for many tasks, it has an open-ended nature, and due to various challenges such as limited data it still remains a challenge for LLMs and modern AI systems. Recently, the International Math Olympiad (IMO) grand challenge (Selsam et al., 2020) was announced where the goal is to build an AI system that can win a gold medal in IMO, one of the most prestigious competitions. Not only research, with advancements in LLMs, many new applications and products are leveraging AI research for education to build personalized tutors (Abdelghani et al., 2023; Khan, 2023) . One of the key challenges so far has been to improve the performance of these systems in STEM subjects.Due to the vast popularity of mathematical problem solving both from research and product perspectives, several datasets have been developed for measuring and improving the mathematical reasoning of LLMs (Cobbe et al., 2021; Ling et al., 2017; Hendrycks et al., 2021) and are widely adopted by the research community. While existing datasets mostly focus on textual problems, there are several bodies of mathematical problems that require both textual and visual understanding of the problem. Being one of the main school curriculum and having a high presence in many math competitions including IMO, geometry is a key domain in this space. With the fast pace in adoption of the vision-language models (VLMs) (Chen et al., 2022b; OpenAI, 2023) in various aforementioned applications, it is crucial to measure and improve their performance on such problems. Previous work has created a number of datasets with geometry questions based on high-school, college, or SAT exams, and developed specific models for this task. While evaluating VLMs on such datasets may provide a holistic understanding of the general capability of the models, such evaluation may provide little information about the specific areas of strengths and weaknesses of VLMs and hence provide little guidance on where research should focus. Recent years have witnessed a surge of interest in synthetic datasets that allow for a systematic evaluation of the boundaries of capabilities and the limitations of the state-of-the-art models (see, e.g., (Lindström & Abraham, 2022; Borisov et al., 2022; Kazemi et al., 2023a; Gekhman et al., 2023; Vaska & Helus, 2023; Fatemi et al., 2024) ).In this paper we create GeomVerse, a dataset of synthetically generated geometry questions that require multi-hop mathematical reasoning over text and image. We bridge reasoning about geometry problems and logical reasoning, allowing us to measure model performances on reasoning factors that may go beyond geometry and may be present in many (mathematical) reasoning problems on text and image. In other words, GeomVerse allows for unveiling the reasoning ability of VLMs across several axes, by using geometry as a lens. We also measure model performances on geometry-specific axes of difficulty. This enables a systematic evaluation of VLMs on this task. A sample generated problem and solution can be viewed in Figure 1 . Some of the main findings from our systematic evaluation on GeomVerse are summarized below. Firstly, through the unique property of GeomVerse that allows for constructing benchmarks at various depths, we find that current VLMs are not as capable in subjects like geometry as suggested by previous benchmarks, showing that they may still be immature for product applications such as AI tutoring. Importantly, since several of the difficulty axes we study are not specific to geometry, our results reveal a number of important failure modes as well as a significant gap in the reasoning capacity of state-of-the-art VLM that may go beyond geometry. Secondly, finetuning VLMs to produce the entire solution substantially improves their performance for in-distribution problems but that does not generalize to out-of-distribution problems. Thirdly, VLMs struggle more with increasing in depth rather than width of reasoning. And fourthly, VLMs are rather robust to the question and image representation. 
related work: 
 Our work is related to several research directions in the literature as summarized below.Vision-Language Models (VLMs): Recent VLMs (Chen et al., 2022b; Allaway et al., 2022; Alayrac et al., 2022; Li et al., 2023; Wang et al., 2022; Chen et al., 2023) have demonstrated promising performance on a wide range of image and video tasks including captioning, question answering and visual reasoning. However, the capabilities of performing multi-modal multi-hop (mathematical) reasoning are less investigated. Because these VLMs are generative black-boxes, understanding how well they can comprehend and answer the multi-hop questions is a critical topic.There are a number of datasets available in the literature that require multi-hop logical (Tafjord et al., 2021; Kazemi et al., 2023a; Zhong et al., 2021) and mathematical (Cobbe et al., 2021; Ling et al., 2017; Hendrycks et al., 2021) reasoning over text. Previous work has also developed a number of geometric reasoning datasets (Seo et al., 2015; Lu et al., 2021; Chen et al., 2021; 2022a; Zhang et al., 2023) that require reasoning over both text and image. Table 1 provides an overview of the existing datasets and compares them along four axes: 1-requiring textual understanding, 2-requiring visual understanding, 3involving mathematical reasoning, and 4-automatic control of the difficulty level (thus allowing for a systematic evaluation).Multi-Hop Reasoning Approaches: Some of the approaches for improving the multi-hop reasoning of LLMs and VLMs range from pre-training on relevant data (Hendrycks et al., 2021; Lewkowycz et al., 2022) , finetuning with (Nye et al., 2022; Dalvi et al., 2021; Zelikman et al., 2022; Kazemi et al., 2023a) and without (Clark et al., 2021; Betz et al., 2021) explicitly generating the solution, in-context learning with solutions (Wei et al., 2022) , decomposing the problem into smaller pieces and solving them separately (Zhou et al., 2023; Khot et al., 2023) (Masry et al., 2022 ) GeoS (Seo et al., 2015) GeoQA (Chen et al., 2021) Geometry3k (Lu et al., 2021) UniGeo (Chen et al., 2022a ) PGPS9K (Zhang et al., 2023) GeomVerseTextual Understanding ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Visual Understanding ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Mathematical Reasoning ✗ ∼ ✗ ✓ ✓ ∼ ∼ ✓ ✓ ✓ ✓ ✓ ✓ Automatic Difficulty Control ✓ ✓ ✗ ✗ ✗ ∼ ∼ ✗ ✗ ✗ ✗ ✗ ✓Table 1 : A comparison of GeomVerse with some of the recent and/or widely-used multi-hop (logical or mathematical) reasoning datasets. We use ∼ when a dataset contains a property to a limited extent.LLMs/VLMs as tools within classical algorithms (Kazemi et al., 2023b; Creswell et al., 2023) . In the realm of reasoning about geometry problems, existing work typically develops specialized models or tools (e.g, (Trinh et al., 2024) ) or resorts to distillation strategies (e.g., (Gao et al., 2023) ); measuring the reasoning ability of general-purpose VLMs is less studied. 
the geomverse dataset: 
 We start with some preliminaries and terminologies. Then, we explain how GeomVerse is created. The dataset will be publicly available upon the acceptance of the paper.A logical theory consists of facts and rules. Consider the following theory as a running example:F acts : {a, b} Rules : {a ⇒ c, a ∧ b ⇒ d, d ⇒ e}The theory contains two facts specifying a and b are true, and three rules specifying a implies c, a and b imply d and d implies e. Starting from the facts, one can apply deduction on the set of facts and the rules to derive new facts and answer queries (e.g., we can query whether e holds). We define the depth of a query as the number of hops of reasoning required to prove it, and the width of a query as the maximum number of branches in the proof of the query. For a given query, any fact or rule not necessarily in the proof of the query is referred to as a distractor. For example, if we query a both the depth and width are 0, if we query c the depth is 1 and the width is also 1, if we query d the depth is 1 and the width is 2, and if we query e both the depth and the width are 2. When we query e, the rule a ⇒ c is a distractor. Note that queries with width 1 correspond to a chain of reasoning, whereas higher width queries correspond to a tree of reasoning.Geometry problems often provide values for certain elements (e.g., sides, angles, areas). Using geometric rules and formulas, we can deduce the values of the remaining elements one by one. The elements whose values are given to us can be thought of as facts in logical theories, the geometry rules and formulas can be considered as the rules in logical theories, and the process of deriving the hidden values can be thought of as the deduction.As an example, the solution to the problem in Figure 1 can be formulated in logical form as:F acts : {A AHID , A ABCD , P ABEF , L BE } Rules : {A AHID = ⇒ AD, P ABEF , L BE = ⇒ L AB , A ABCD , L AD , L AB = ⇒ D DAB }where A x , P x , L x and D x represent the area of a shape, perimeter of a shape, length of a side, and degree of an angle respectively. We note two key differences with logical reasoning: 1-unlike in deductive logical reasoning, the rules are not given to the model and the model has to use its own geometry knowledge (learned from pre-training or finetuning) to apply the right geometry formulae and derive new values, 2-in the case of geometry, applying rules involves computations.To create GeomVerse, we fix a set of 12 standard and nonstandard shapes S as demonstrated in Figure 2 and gather a number of rules/formulas F s for each shape s ∈ S (e.g., the Pythagorean theorem) with a total of 68 formulas across all shapes. We further use supplementary and complementary angles as two special shapes with only a single formula each. For a formula f , we let f in represent the input elements and f out represent the element whose value can be computed based on the formula and the inputs (e.g., for the Pythagorean theorem, the two sides can be the input and the hypotenuse can be the output). Then, similar to several existing works on constructing multi-hop, textual logical reasoning datasets or textual stories (Kazemi et al., 2023a; Ye et al., 2022) , we adopt a backward generation strategy where we start by generating a question, and then adding rules to increase the depth and width of reasoning to the desired amount.To generate an example with depth 1, we can simply sample a shape s ∈ S and formula f ∈ F s . Then, we let f acts = f (in) , query = f (out) , and with the only required rule in the solution being rules = {f }.Increasing the Depth: Let f 1 be the formula we sampled for the depth 1 example and f (in) 1 and f (out) 1 be the inputs and output of f 1 . To increase the depth to 2, we select one of the elements e in f (in) 1and do not provide it in the facts. Instead, we sample a new shape s 2 and formula f 2 such that f (out) 2 has the same type as e and we tie the values of e and f (out) 21 . For example, if e is one of the sides of a triangle, then s 2 can be a square and f 2 can be the formula of deriving the side of a square from its area, where the square and the triangle share the same side. Thenf acts = (f (in) 1 -e) ∪ f (in) 2 , query = f (out) 1, and the required rules are rules = {f 1 , f 2 } with f 2 providing the value for e and then f 1 using this value to answer the query. The depth can be further increased in a similar way by appending a new shape and formula to one of the elements in f (in) 2.Increasing the Width: Let s and f be the shape and formula we sampled at some depth for the construction of an example and e 1 and e 2 be two connectable elements (side or angle) in f (in) . We can include only f (in) -{e 1 , e 2 } in the facts, and append new shapes and formulas as explained above so that the values for e 1 and e 2 can be derived.Distractors: Distractors can be added in a post processing step. Consider a Depth 2 (Width 1) example and suppose e is the element that has to be computed in the first hop and be used in the second hop. If we provide the value of e as input, then the model turns into a Depth 1 problem with a distracting shape and corresponding values. In Figure 1 , for example, if we provide the value of the AD side as out) .type != τ Append s to other shapes on e. Randomly assign values to f (in) . Provide f (in) values as facts.Algorithm 1 BackwardGenerate Input: Shared element e, Shared element type τ Depth d [t] if d == 0 then do s = RandomSelect(S) f = RandomSelect(F s ) while f (else do s = RandomSelect(S) f = RandomSelect(F s ) E = ConnectableElements(f (in) ) while f (out) .type != τ OR |E| = 0 Append s to other shapes on e. Randomly assign values to f (in) -E. Provide f (in) -E values as facts. e 1 , . . . , e m = SampleElems(E, p branch ) for e ∈ {e 1 , . . . , e m } doBackwardGenerate(e, e.type, d-1) input, then the square and its corresponding values can be considered as distractors.The Generation Algorithm: Algorithm 1 adopts the highlevel idea of the GenerateTheory algorithm from Kazemi et al. (2023a) for recursively generating geometry problems (as opposed to logical theory problems) in a backward fashion. Initially, we select one element type τ to be asked for in the question (e.g., side, angle, area, perimeter, etc.), and a desired depth d. Then we call the BackwardGenerate function. If d = 0, we sample a shape s ∈ S and formula f ∈ F s such that the type of the element in f (out) is τ , append the shape s to the previous shapes on the shared element, assign random values to the elements in f (in) and provide them as facts2 . Otherwise, we sample a shape s ∈ S and formula f ∈ F s such that 1) the type of the element in f (out) is τ and 2) there is at least one connectable (side or angle) element in f (in) . Then, we select a subset E of the elements in f (in) for expanding the number of hops. If p branch = 0, we only select one of the elements from f (in)Figure 2 : The standard shapes (top row) and non-standard shapes (bottom row) used in our dataset.which introduces no branching and so no increase in width.Otherwise, with probability p branch we select a second element as well, which leads to a branching and so increases the width. We append the shape s to the previous shapes on the shared element, assign random values to the elements in f (in) -E and provide them as facts. Then, for each element in E, we recursively call the BackwardGenerate function to append new shapes such that these values can be derived. A visual example of the procedure is provided in Appendix C.Automatic Question and Solution Generation: We automatically produce a question that provides the facts as input and asks for f (out) where f is the first formula used.We also keep track of the required rules (including shapes, formulas, and the shared elements) during the generation process (excluded from Algorithm 1 for brevity) and automatically produce a solution by applying deduction and computations on the rules and facts.Text-Only vs. Text-Image: We create two versions of our problems. In one version, all the required information is given in the question and the image is not needed for answering the question (although the presence of the image can make it easier to understand the problem), and in the other version some information is given in the image and some in the question text so both the image and text of the question are required. We use the former to experiment with text-based LLMs and the latter to experiment with VLMs.Coverage: The connection between Algorithm 1 and logical reasoning helps specify what classes of geometry problems are covered by Algorithm 1. Specifically, Algorithm 1 can generate any geometry problem P containing a tree of shapes where each shape is connected to its parent shape via a single side or a single (vertical) angle, and where the solution can be found by finding the values of the shared elements bottom-up on the tree.Further Considerations: While Algorithm 1 can generate problems with overlapping shapes, to ensure the quality of the generated examples remains high without any human involvement in the generation process, we only accept the generated examples where the shapes are non-overlapping.To ensure high quality for the questions, the solutions, the images, and the labels, we did two quality checks. Firstly, we generated all possible Depth 1 problems and manually verified their quality and correctness. Secondly, we asked 10 well-educated people to verify a total of 100 problems (from various depths and with various properties) and identify as many issues as possible with the questions, solutions, labels, or images. A list of the issues identified in this round are provided in Appendix E. All the raised issues were then fixed, and the process was repeated with 100 new examples to ensure no issues remained. Additionally, to get human performance on these problems, a separate set of four people solved 60 sampled problems (20 from each depth) and raised no issues, indicating another level of quality check for the generated dataset. 
experiments: 
 We experiment with two state-of-the-art VLMs: PaLI (Chen et al., 2022b) and GPT4V (OpenAI, 2023), and a state-ofthe-art LLM: PaLM 2 Large (Anil et al., 2023) , in four settings: 1-zero-shot, 2-few-shot with chain-of-thought (CoT) prompting (Wei et al., 2022) (hereafter referred to as FS-CoT), where the CoT corresponds to the solution, 3finetuning to directly predict the label (hereafter referred to as FT), and 4-finetuning to predict the solution/CoT (hereafter referred to as FT-CoT). We do the first experiment with GPT4V3 , the second with PaLM 2 Large and PaLI 55B (the largest PaLI model), and the last two experiments with PaLI 5B to keep the required computations manageable.Following Methani et al. (2020) and Masry et al. (2022) , we measure performance in terms of relaxed accuracy, where a prediction is considered correct if it is within δ percent of the golden label. We do this to accommodate for the slight variation in computations introduced due to the rounding strategy (e.g., due to the order of the computations). We empirically found δ = 3 to be appropriate so we consider a prediction p correct if 0.97 * label ≤ p ≤ 1.03 * label. We remove from our dataset any example where the difference between the label computed with and without rounding intermediate steps is more than 3%.We provide results on subsets of our dataset with different properties. In each case, we generate 1000 examples randomly given the described parameters and report the results on those examples. We also generate a separate pool of train, validation, and fewshot examples for our experiments. The implementation details are presented in Appendix B. In contrast, the text-only model can solve a portion of the Depth 2 and 3 problems as well. While the presence of the image should make the problem easier to understand and solve, this results hints that LLMs may be stronger in mathematical and multi-hop reasoning compared to their counterpart VLMs. Moreover, while finetuning helps VLMs learn to do some reasoning, as the depth of reasoning increases the performance drops monotonically and quite significantly.Notice that FT-CoT outperforms FT substantially for all depths. While such improvements have been previously observed for reasoning with textual inputs (Suzgun et al., 2022) , this result shows the importance of showing CoT to VLMs as well. This result also hints at the quality of the automatic solutions in GeomVerse.We also measured human performance on our dataset by having 4 well-educated (but not necessarily expert in geometry) people solve a total of 20 problems per depth. The results show a stark gap between the best model performances and the human performance; we also observe that our problems can be challenging to solve even for humans. The mistakes made by humans where due to various issues including wrong/forgotten degree to radians conversion, using wrong formulas, and making wrong assumptions. Due to the low performance of the zero-shot and FS-CoT PaLI models, hereafter we only experiment with the FS-CoT Text-Only and finetuned PaLI models.We generate Depth 2 examples (medium difficulty in terms of depth) with p branch =0, 0.5, and 1.0, and report the performances in Figure 5 . We observe that while increasing the width negatively affects the performance in several cases (especially for the FS-CoT model) the amount of decrease The results hint at the ability of the models at learning to deal with higher width examples. This could be because the main added difficulty from higher width problems is that the model needs to solve more independent Depth 1 problems, on which they showed good performance according to Figure 3 .We next measure how well models can deal with distracting information, a phenomenon which is common in real problems. We create a version of the Depth 2 problems where we provide the hidden value as input. This effectively turns the Depth 2 problem into a Depth 1 problem with some extra (distracting) shapes and values. The model performance is reported in Figure 6 . Comparing Depth 1 results with and without distractors, the performance drops significantly for all models in presence of a distractor. Comparing Depth 1 with distractor and Depth 2 without distractor, while the text-only model has taken advantage of the value for the hidden element in some cases, for the finetuned VLM models the performance degrades to as low as that of the Depth 2 dataset.4 Part of the reason for this observation could be because we have only 30/68 formulas that have more than 1 connectable elements in their inputs and so even in the case where p branch = 1.0, we still generate a number of examples that correspond to chains as opposed to trees. To measure how sensitive the VLM models are to the lowlevel visual features, we create separate test sets each varying in one low-level feature and measure the performance of the trained models on these new sets. Specifically, we experiment with changing the opacity of the shape colors, the line width of shape boundaries, and the font size of the texts on the images. In figure 7 , we observe that the models are robust against opacity and line width, but not against font size changes.Our experiments so far focus on general factors that may be present in many problems requiring reasoning on text and image. In Appendix A, we experiment with various other axes of difficulty that are more specific to geometry problems (including shapes, source of information, image annotation, adding variablized inputs, and decomposing performance based on question type). 
limitations & risks: 
 While GeomVerse covers a wide range of geometry questions, there are problems that cannot be produced using Algorithm 1 with our current set of shapes and formulas.The connection between Algorithm 1 and logical reasoning makes evident the class of problems that cannot be represented by the algorithm. In particular, let P be the class of geometry problems containing a tree of shapes where each shape is connected to its parent shape via a single side or a single (vertical) angle, and where the solution can be found by finding the values of the shared elements bottom-up on the tree. Algorithm 1 cannot generate any geometry problem that is not in P. For example, let ABC be a triangle, D be a point on the AC side dividing ABC into two triangles ABD and ACD, where some property of ABC should be computed based on the properties of ABD and ACD. This problem cannot be produced by Algorithm 1 as it does not correspond to a tree of connected shapes as described above.However, note that one can add such cases to our set of non-standard shapes in a similar way we added the other non-standard shapes.The problems in GeomVerse can be solved with a logical deduction procedure and may not require much creativity. For this reason, our evaluation should not be considered as measuring the creativity of the models in solving problems, but rather their ability in following a deduction procedure.For our finetuning experiments, to make computations manageable, we used the small PaLI 5B model. Finetuning larger and more capable models such as Gemini (Team et al., 2023) or GPT4V (OpenAI, 2023) can provide more insight into the performance of the finetuned VLMs. 
conclusion: 
 In this work, we procedurally generated a synthetic dataset of geometry reasoning questions that require multi-hop reasoning over both text and image. Through the lens of the geometry problems, we conducted a systematic analysis of various general and geometry-specific reasoning abilities of VLMs and found the gaps and strengths in their reasoning capabilities. Future work can verify the merit of finetuning models on synthetic geometry problems for improving their performance on real datasets. In an initial experiment, we measured the performance of the PaLI 5B model on Ge-ometry3k with and without finetuning on GeomVerse and observed modest improvements (from almost 0 to almost 2 percent accuracy). We believe this is due to the difference in the visual and textual features of the Geomety3k and GeomVerse, as well as the poor generalization of PaLI to geometry problems beyond its training distribution. Better aligning the textual and visual features and using more powerful models can yield more gains.  
 
 Appendix: Besides the experiments in the main text, we also consider a number of other axes of difficulty for a systematic evaluation.In what follows, we describe these axes and present the experimental results. In Section D, we provide samples corresponding to each of the axes of difficulty.In Figure 2 , we outlined the standard and non-standard shapes used in GeomVerse. Conceptually, it should be more difficult to solve problems involving non-standard shapes as they require more computations. We compare the performance of various models on problems that contain all shapes vs those that involve only standard shapes. To fix other axis of difficulty, we only consider depth 2 examples for this experiment where the problems are at a medium level of difficulty. The finetuned models are finetuned on all images in both cases. The results are in Figure 8 . According to the results, we observe that while the FS-CoT model performs better on the standard shapes, this is not the case after finetuning. This shows that finetuning can teach the models to effectively deal with non-standard (but in-distribution) shapes. Some of the information can be provided either in the text of the question or on the image. For example, the degree of an angle can be provided in the image, or can be provided in the text.We generate examples where the information is given mostly in text and examples where it is given mostly on the image, and report model performances in Figure 9 . For the FT model, we see that the former case results in lower accuracy which could be because in this case the model needs to first map those information to the elements in the image and then reason with them. FT-CoT almost closes the gap; this could be because the provided CoTs teach the model how to map information from text to image.We consider two types of image annotation: 1-individual annotation: we refer to each side with a single lower-case letter, each angle with a Greek letter, and each shape with its (distinct) color, and 2-coordinate annotation: we assign upper-case letters to the coordinates on the image and refer to sides with the letters on the two coordinates, to angles with the three coordinates, and to shapes with all their coordinates. We generate a test set with coordinate annotation and another with individual annotation and report model performances on these two sets in Figure 10 . The two models show different behaviour with the FT model performing slightly better on the individual annotation case, but the FT-CoT model slightly performing better on the coordinate annotation case.Instead of providing the exact values of the input elements (e.g., the α angle is 30 degrees), it is common in geometry questions to provide a variablized version of them (e.g., the α angle is 2x + 1) in which case one needs to first infer the value of the variable based on the given information and then use that to infer the value of an element. As an example, we can either directly provide two of the angles of a triangle as input and ask for the third one, or we can provide variablized values for the three angles and ask for one of them. To generate variablized questions, when we use a formula f , instead of directly providing the values for f (in) as input and expecting the model to apply the formula to derive the value of f (out) , we provide variablized values for (some of) the elements in f (in) and f (out) and expect the model to use the formula for deriving the value of the variable x and use that to derive the numerical value of f (out) . We selected 17/68 of our formulas for which a variablized version of the problem only requires solving an extra 1-d linear equation.We then conducted an experiment where, whenever one of the 17 rules was selected during generation, we provide a variablized version of it with probability ρ. Figure 11 demonstrates the results for ρ = 0, ρ = 0.5 (corresponding to level = medium) and ρ = 1.0 (corresponding to level = high). We observe that as we include variablized inputs, the performance of the models degrade, especially for the FT-CoT model. This shows VLMs (and also LLMs) struggle to work with variables when solving geometry problems.Our questions involve asking about the length of a side, the degree of an angle, or the area/perimeter of a shape. In Figure 12 , we report model performances for each of these question types. We observe that the FS-CoT Text-Only model performs almost equally across all three types, with slight preference for angle and area/perimeter questions.For the FT model, questions about angles are substantially easier, followed by questions about side. We conjecture that part of the reason for the high performance of the FT model on angle questions might be because the degree of an angle can be estimated from the figure without actually solving the problem. This could be in part validated by the results of the FT-CoT model, where the jump in accuracy is substantially higher for side and area/perimeter questions. In the case of FT-CoT, we see that side questions are easier than the other two; this may in part be because these questions involve easier arithmetic operations (e.g., some of the angle questions require computing arcsin which might be difficult for a pre-trained model). We finetuned our model for 10k steps with a learning rate of 0.0005 and a batch size of 128, measured the model performance on the validation set every 2000 steps, and reported the results on the test sets for the checkpoint achieving the best performance on the validation set.For our fewshot experiments, we manually selected 4 examples from the training set and used those examples as fewshot demonstrations across all experiments. These examples were selected to ensure many aspects of the test set are covered (e.g., to ensure there are examples at various depths, widths, with and without variables, with different question types, etc.).Rounding Errors: Note that depending on how we round intermediate computations, the final answer can be slightly different. For example, consider the expression 2.26 * 3.14 4 . If we first multiply the numerator, round it and then divide by 4 and round again, we will get 2.26 * 3.14 4 = 7.1 4 = 1.78. However, if we first do both computations and then round at the end, we will get 2.26 * 3.14 4 = 1.77. For this reason, we reported relaxed accuracy in our experiments to account for the differences in the way we computed the final results and the way the model may compute it.In Figure 13 , we provide a visual demonstration of the process in Algorithm 1 for generating an example with Depth 3. In Step 1, we select a shape from our set of shapes and then select one of the formulas. The shape selected in this example is a rectangle and let the selected formula be to compute the area of a rectangle given its height and width; so f = {A ABCD } where L AC and L CD represent the length of AC and CD and A ABCD represents the area of ABCD (note that we could also select L BC and L AB instead). We then select which element(s) from f (in) 1 we will provide explicitly and which element(s) should be derived. Assume we decide to provide L AC explicitly and append other shapes to derive the value of L CD . In this case, we assign a random value to L AC and provide it in the set of facts.In Step 2, we need to select a shape where one of its sides is CD, and select a formula from which the length of this side can be derived. In the provided example, the selected shape is a right triangle and let us assume the selected formula is to compute a side of a right triangle given the hypotenuse and the opposite angle. So f we will provide explicitly and which element(s) should be derived. Assume both elements should be derived (corresponding to increasing the width of reasoning). So none of the elements will be added to the facts.In Step 3, we need to select a shape where one of its sides is CE, and select a formula from which the length of this side can be derived. In the provided example, the selected shape is a semi-circle. Assume the formula is to compute the diameter of the semi-circle L CE given its perimeter P SemiCircle . Since we want to generate Depth 3 examples, we add P SemiCircle to the set of facts.In Step 4, we need to select a shape that can be connected to the CED angle, such that D CED can be derived from that new shape. In the provided example, the selected shape is a supplementary angle, and the formula is that the sum of two supplementary angles is 180. We provide D DEF in the facts so D CED can be derived based on that.Putting it all together, we get the rightmost shape in Figure 13 . The facts include {L AC , P SemiCircle , D DEF } and the query is A ABCD . Based on the rules we used, we can apply deduction to produce a solution as follows:D DEF = ⇒ D DEC P SemiCircle = ⇒ L CE D DEC , L CE = ⇒ L CD A B C D A B C D E A B C E D A B C E D F A B C E D FStep 1Step 2Step 3Step 4 Putting it all togetherFigure 13 : A visual demonstration of the process in Algorithm 1 for generating a example in Depth 3.L CD , L AC = ⇒ A ABCDTo generate the question, we turn the facts (and the extra information needed to know such as the shapes, whether some angles are vertical or complementary, etc.) into a question using a template. We also provide the shape names when necessary. For Figure 13 , for example, the elements whose values have to be provided as input are recorded during the generation process; this includes the length of AC, the measure of the DEF angle, and the perimeter of the semi-circle. We also take note of the other information that must be provided; this includes the fact that CDE is a right triangle and that DEF and DEC are complementary. We then use templates to turn each of these pieces of information into a textual format and concatenate them; we also textify the question using templates and append at the end. The final question will look like: If the length of the AC side of the ABDC rectangle is 10, CDE is a right triangle, the DEF angle is 120 degrees, the DEF and the DEC angles are complementary, and the perimeter of the semi-circle is 20, compute the area of the ABDC rectangle.In this work, we experimented with several variations of GeomVerse. Here, we provide samples from these different variations to better illustrate how each test set looks like. The questions and solutions are provided in Tables 3 and 4 and the corresponding images are provided in Figure 14 .As mentioned in the main text, the dataset went through multiple rounds of quality check. In what follows, we provide some of the examples of the issues found during the quality check by non-authors.• Text repetition: In two cases, the quality checkers found the text of the question to be repetitive. This happened in the cases where, e.g., the question was "the length of the AB side of the ABC triangle is 10, the length of the BC side of the ABC triangle is 6, the length of the AC side of the ABC triangle is 8". We updated our templates to remove repetitions.• Unnecessary information in the question: An issue raised by multiple quality checkers was that we provided the value of π = 3.14 even when it was not used in the solution. We made sure we only provide it when needed. Misprinting a formula in the solution:In one case, a formula was misprinted in the solution where a squaring operation was missing (this did not affect the final result though because it was a misprint). This was fixed.• Unsolvable Variablized Inputs: The quality checkers identified that when we provided variablized inputs, sometimes the problem became unsolvable. This happened, e.g., in the case where we provided an input such as "the length of the three angles of a triangle are x + 30, -2x + 60 and x + 45" where after summing the three values, x disappeared.• Missing Coordinates: In one case, one of the characters corresponding to an image in the coordinate was missing. We identified the root cause and fixed this.• Spacing issues: Since the solutions were generated automatically, there were a number of cases where a space was either missing between two words or there were multiple spaces. Table 3 : Questions and solutions corresponding to the images in Figure 14 . = 102 -14.13 = 87.87. Therefore, the length of the BC side is 87.87 6 = 14.65. The length of the hypotenuse of the ABC triangle is 14.65 and the length of the side opposite to the BCA angle is 7, so the BCA angle equals asin( 7 14.65 ) = asin(0.48) = 28.69. Therefore the final answer is 28.69. (d) If the length of the height of the ABCD trapezoid is 8, the area of the blue semi-circle is 189.97, the BCFGH shape is a combination of a rectangle and an equilateral triangle and the perimeter of the BCFGH shape is 42, compute the area of the ABCD trapezoid. Assume π = 3.14. Round computations to 2 decimal places.The area of the blue semi-circle is 189.97 so the length of the AD diameter can be computed as 8 * 189.97 π ) =1519.76 π = √ 484.0 = 22. The side of the equilateral triangle in the BCFGH shape is equal to the side of the rectangle with length 8 so the shape has two sides with equal but unknown lengths, one with length 8, and two triangle sides with length 8. The perimeter of the BCFGH shape is 42 so 2 * (unknown side) + 3 * 8 = 42. So 2 * (unknown side) = 42 -24 = 18, and the length of the BC side is 18 2 = 9. The lengths of the AD and the BC bases of the ABCD trapezoid are 22 and 9 and the height of the trapezoid is 8, so the area of the trapezoid is 1 2 * (22 + 9) * 8 = 1 2 * 31 * 8 = 124. Therefore the final answer is 124. (e)If the perimeter of the ABE triangle is 42, the BEFG shape is a rectangle where a semi-circle has been removed from one side of it and the perimeter of the BEFG shape is 62, compute the perimeter of the ABCD rectangle. Assume π = 3.14. Round computations to 2 decimal places.The diameter of the semi-circle in the BEFG shape is equal to the side of the rectangle with length 10 so the shape has two sides with equal but unknown lengths, one side with length 10, and one semi-circle arc with diameter 10. So the perimeter is 2 * U nknownSide + 10 + 10 * π 2 . So 2 * U nknownSide + 10 + 10 * 3.14 2 = 62. So 2 * U nknownSide = 62 -10 -10 * 3.14 2 = 62 -10 -31.4 2 = 62 -10 -15.7 = 36.3. Therefore, the length of the BE side is 36.3 2 = 18.15. The lengths of the AE and BE sides of the ABE triangle are 10 and 18.15 and the perimeter is 42, so the lengths of the AB side equals 42 -10 -18.15 = 13.85. The lengths of the AD and the AB sides of the ABCD rectangle are 15 and 13.85, so the perimeter of the ABCD rectangle is 2 * (15 + 13.85) = 2 * 28.85 = 57.7. Therefore the final answer is 57.7.Google DeepMind. Correspondence to: Mehran Kazemi <mehrankazemi@google.com>.The first AI for MATH Workshop at the 41st International Conference on Machine Learning, Vienna, Austria.Copyright 2024 by the author(s).Note: e should have a type that allows it to be connected to another shape (e.g., side or angle).During random value assignment, we test multiple factors to ensure the assigned values are sensible (e.g., the sides of a right triangle are smaller than its hypotenuse) and re-assign values until these criteria are met. Sometimes, this becomes impossible due to some values that are derived from other hops; in these cases, we simply discard the example and generate another example from scratch.Based on the GPT4 responses, we notice that it uses zero-shot CoT(Kojima et al., 2022) under the hood. 


Decision: Nice paper. It proposes a new dataset of geometry problems to benchmark the performance of visual language models (VLMs) with clear presentation and comprehensive empirical study. It could serve to the AI for Math community in the future as a common benchmark in this direction.

Reviews: review: paper_summary: paper_summary: This paper presents a detailed assessment of VLMs using geometry problems. It introduces the GeomVerse dataset, a procedurally generated set of geometry questions designed to test multi-hop mathematical reasoning involving both text and images. The authors demonstrate that while VLMs can handle simple reasoning tasks, they struggle with more complex problems requiring deeper reasoning and generalization. Key findings include the models' limitations in dealing with higher-depth problems and distractors, highlighting the need for further advancements in VLMs for them to be effective in educational applications and beyond.
strength_weakness: strength_weakness: N/A
questions: questions: N/A
paper_summary: paper_summary: This paper has a very clear structure with details. It also has relatively good literature review. Moreover, the proposed method to generate school geometric problems is systematic and innovate. In general it is a good paper and is worthy a contributed talk or good poster at least.
strength_weakness: strength_weakness: I would recommend this paper in general, at least for a contributed talk or a very good poster. The only little concern here is that the topic discussed in this paper  seems not one of the central themes of the conference, though related.
questions: questions: 1. In Section 3.2, it is mentioned that the majority of the geometric reasonings in this study are based on some values and the logics between them. The sample questions all have a clear numerical answer. Have the authors considered creating and testing geometric questions whose answers are not a determined number? The question descriptions may not include specific values.

2. In Section 4.1, when discussing "Generalization", it is discussed that a subgroup of shapes are used to finetune the model; does the choice of this subgroup have impact on the experiment?