Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models

Authors: hyunsik  chae
seungwoo  yoon
chloe  yewon
gyehun  go
yongin  cho
gyeongmin  lee
ernest k ryu
Venue: NeurIPS_cc_2024_Workshop_MATH-AI
Published: 2024-09-25 19:28:43

Abstract: Recent Vision Language Models (VLMs) have demonstrated impressive multimodal comprehension and reasoning capabilities, but they often struggle with trivially simple visual tasks. In this work, we introduce the Atomic Visual Skills Benchmark (AVSBench) to evaluate whether VLMs possess capabilities to understand basic geometric features, which we refer to as atomic visual skills. Specifically, we systematically categorize the atomic visual skills and handcraft a set of 5,073 diverse questions designed to assess each individual atomic visual skill. Using AVSBench, we evaluate the current leading VLMs and find that they struggle with most of these atomic visual skills that are obvious to humans.

Main paper: 
 introduction: 
 Recent Vision Language Models (VLMs), also referred to more generally as Multimodal Large Language Models (MLLM), integrate vision components into language models and demonstrate an impressive breadth of multimodal comprehension and reasoning capabilities [7] . At the same time, however, VLMs often struggle with trivially easy visual tasks as shown in Figure 1 , a puzzling phenomenon that seems almost contradictory to their remarkable performance [11, 43] . We propose two hypotheses to explain the observed shortcomings of current vision-language models: Hypothesis 1: The comprehension of complex visual diagrams requires the composition of smaller atomic visual skills.Hypothesis 2: Current vision language models are incapable of such atomic visual skills.In this work, we introduce the Atomic Visual Skills Benchmark (AVSBench) to test Hypothesis 2.AVSBench is designed to rigorously evaluate VLMs' ability to comprehend fundamental geometric features, which we refer to as atomic visual skills. We systematically categorize 36 atomic visual skills that encompass diagrams arising in high school-level geometry and handcraft a set of 5,073 diverse questions designed to assess the understanding of the individual atomic visual skills.We then evaluate the state-of-the-art VLMs on AVSBench, and the results clearly support Hypothesis 2.While our problems are designed to be trivial to humans, VLMs struggle; state-of-the-art models like Gemini-1.5-pro and GPT-4o score around 70%-75% on problems with the "easy" categorization, score around 60% on the "medium" problems, and 30% on "hard" problems. The confirmation of Hypothesis 2 also lends support to Hypothesis 1, and suggests a promising direction of future work of training vision language models specifically on atomic visual skills to improve their performance in comprehending complex visual diagrams. 2 Atomic Visual Skills Benchmark (AVSBench)Many visual reasoning tasks in existing benchmarks, such as the ones listed in Section A, are composite tasks that can be broken down into more elementary components. This observation leads us to define a set of atomic visual skills based on the following criteria: (i) each skill is intuitive and trivial for adult humans, (ii) each skill cannot be decomposed further, or doing so would be unnatural, and (iii) the list of atomic visual skills should comprehensively cover the abilities required for comprehending geometric diagrams arising in high-school level mathematics. While this definition is not a fully rigorous one, we found it to be sufficiently clear and substantive for our work.Using these criteria, we identified 36 atomic visual skills, including the ability to understand concepts such as angle, boundary, orthogonality, curvature, and direction. The complete list and further illustrations are provided in D.For adult humans, these skills are trivially simple and require little to no reasoning to perform. Therefore, we use the term comprehension instead of reasoning to emphasize our belief that these skills do not require much reasoning or thinking to perform, for both humans and VLMs. This belief is partially supported by Findings 3 of Section 3.1.We then constructed the Atomic Visual Skills Benchmark (AVSBench) to evaluate VLMs' ability to perform the 36 atomic visual skills. AVSBench, as summarized in Figure 2 , comprises 5,073 new handcrafted image-question-answer triplets with the following characteristics: Figure 3 : Evaluation results on AVSBench. +CoT implies the performance of the model on the right with chain-of-thought (CoT) prompting [22] . The area ratios of each colored section are aligned with the actual ratio of problem counts. Details about the models including their full name are in E. Full quantitative results are illustrated in Table 5 .• Originality. All images and questions are newly generated, ensuring that they are free from data contamination concerns.• Diversity. Although we focus on the set of only 36 skills, the problems feature diverse expressions and formats, as illustrated by the sample problems in C.• Skill isolation. Each question targets a specific atomic skill, minimizing the overlap with other skills. Recognizing the impossibility of achieving complete isolation, our method incorporates diverse tasks to mitigate the influence of any task or their relevant overlapping skills. For instance, to minimize the influence of other skills while evaluating the cardinal skill, we asked about cardinals of various concepts and objects, from colors to points, lines, and other figures.• Focus on high school geometry. We focus on the visual skills required to solve high schoollevel geometry problems for the following reasons: (i) the scope of the high school mathematics curriculum is more or less clearly defined, (ii) (as our results of Section 3 show) these atomic visual skills are sufficiently challenging for current VLMs, (iii) the range of skills is broad enough to be applicable to other visual comprehension tasks, such as interpreting charts, tables, and scientific or mathematical figures.3 Current vision language models struggle with atomic visual skillsWe evaluate three types of VLMs on AVSBench: (i) state-of-the-art proprietary models: GPT-4o [40, 39] and Gemini-1.5-pro [49] , (ii) popular mid-sized open-weight models: LLaVA-Next (7B, 13B) [30] , LLaVA-OneVision (7B) [26] , Phi-3.5-Vision (4B) [1] , InternVL2 (8B) [10] , Deepseek-VL (7B) [31] , and (iii) VLMs specifically trained for geometry or other visual data: Math-LLaVA (13B) [47] , Table-LLaVA (7B) [60] . Further details of model versions are provided in Section E.The evaluation protocol consists of three steps. First, we provide the VLM with the image-question pair and solicit a response. As we further discuss later, we also explore the effect of the chain-ofthought (CoT) prompting [54, 22] . Second, we extract the answer from the VLM's response using GPT-4o mini [40] . Third, we ask GPT-4o mini to score the answer by comparing the extracted answer with the answer key. We award 1 point for a correct answer and 0 points otherwise without any partial credit. More details on our evaluation protocol are provided in F. 
experimental results and findings: 
 Figure 3 presents the results comparing the selected VLMs and the baseline corresponding to random guessing. Details including exact values are provided in G. On "easy" problems, models with about 10B parameters score between 32.5% and 51.0% while closed-source models, including GPT-4o and Gemini-1.5-pro, achieve over 70%, far above random chance (22.4%). On "medium" problems, models with about 10B parameters score between 23.8% and 37.8%, slightly above random chance (19.1%). Closed-source models achieve between 58.6% and 64.6%. For "hard" problems, most open models score close to random chance (11.7%). The closed-source models GPT-4o (32.3%) and Gemini-1.5-pro (26.9%) scored significantly better than random chance but clearly struggled.Findings 1: Models share strengths and weaknesses. Figure 4 presents the accuracies of selected models on each skill. The performances across skills varied significantly. For example, most VLMs performed well on OCR, Absolute Position, and Shapes, but performed poorly on tangency, parallel, and angle. Interestingly, the different models largely shared the same set of skills they did well on and the same set they found challenging.Findings 2: Domain-specific models are not better. Surprisingly, Math-LLaVA [47] and Table-LLaVA [60] , which are VLMs specifically trained for geometry or visual data, did not perform better than general VLMs of similar size, on almost any skills within AVSBench as the results of Table 5 show.Findings 3: Chain-of-thought is not helpful in enhancing atomic visual skills. We also evaluated the best-performing models-GPT-4o and Gemini-1.5-pro-with chain-of-thought (CoT) prompting [22] on AVSBench. Surprisingly, CoT did not help for most skills, and for some skills, it even worsened the performance, as shown in Table 5 . This contrasts with prior work, which found CoT to be beneficial for certain visual reasoning tasks [32, 53] . We attribute this difference to our hypothesis that the atomic visual skills of AVSBench require simple "comprehension" and, therefore, do not benefit from the additional "reasoning" steps afforded by CoT prompting. More concrete comparison, see G. 
conclusion: 
 We present the Atomic Visual Skills Benchmark (AVSBench), a benchmark designed to rigorously evaluate VLMs' ability to perform atomic visual skills in a decomposed manner. We then show that current state-of-the-art VLMs struggle with such atomic visual skills.The failure of VLMs to carry out such simple atomic visual tasks raises the question: How is it that VLMs are sometimes successful at performing complex visual tasks? For this, we hypothesize that the existing impressive performance on complex tasks is due to overfitting or unimodal shortcuts. Indeed, recent studies such as [11, 18, 57, 50] report that VLMs tend to depend on language shortcuts, as we further reference and discuss in Section A of the appendix.Recall that our Hypothesis 1 posits that the atomic visual skills are necessary subcomponents for comprehending complex visual diagrams. In future work, we plan to train and fine-tune VLMs directly on the atomic visual skills and ascertain Hypothesis 1. 
 
 Appendix: VLM benchmarks and language shortcuts. Existing VLM benchmarks evaluate models on their ability to solve diverse vision-language problems from general real-world tasks [3, 13, 15] , tasks that require specific skills such as high-school geometry [32, 20, 9, 8] , analyzing charts and tables [34, 60, 35] , and other scientific visual data [19, 21] . However, most VLM benchmarks do not contain a mechanism for verifying whether a correct solution is based on correctly comprehending the visual information, allowing the models to sometimes rely on linguistic biases to find a solution [7] . Lin et al. [27] revealed that by simply avoiding implausible or less fluent sentences, blind language models can distinguish the correct description of an image from wrong ones on CREPE [33] , VL-Checklist [59] , and ARO [56] . Mathverse [57] observed that, when solving geometry problems, VLMs rely mostly on textual inputs without correctly interpreting diagrams.Some recent work has started to seek unbiased ways to measure visual capabilities. Winoground [50] prevents choosing image captions based on the plausibility of the sentence structure, by providing two images with same objects or concepts but with different relationships. Blink [11] and CV-Bench [51] present novel vision-oriented tasks with minimized effects of linguistic biases.Compositional reasoning. There has been intensive recent research on the compositional capabilities of Language Models [4, 55, 17, 48, 44, 58, 23, 38, 42] . VLMs have additionally shown compositional capabilities in visual tasks [25, 37, 33, 59, 56] . However, such studies left the visual portion with less attention, thus vulnerable to linguistic shortcuts such as removing grammatically wrong setnences or choosing more realistic sentences as answers. To mitigate this issue, Sugar-Crepe [18] generated sentences with ChatGPT to provide incorrect captions of given images, with different compositional structures while as realistic as the ground truths.Research on atomic skills of LLMs. To understand the capabilities of LLMs, there has been prior work on studying LLMs in simple idealized experiments. This includes research on in-context learning [28, 36] , arithmetic (addition and multiplication) [38, 16, 24] , fact search and reverse fact search [2, 6, 12] , and programming [5, 45, 14] .However, there have been far fewer studies of this kind for vision language models. Paiss et al. [41] focused on counting objects in image and suggested CountBench. Shen et al. [46] suggest a skillbased approach to evaluating VLMs, but their list of skills is not atomic. CV-Bench [51] evaluates 4 vision-centric skills: spatial relationship, object count, depth order, and relative distance. MMVP [52] challenges VLMs to understand 9 visual patterns.BlindTest [43] observed failures of VLMs with 7 simple tasks focusing on fundamental geometric features, some of which share similar approaches with AVSBench. While these tasks are novel and effective, they lack diversity in color, shape, or word choice. Their Task 1 for instance, uses only one red and one blue line, each with exactly one sharp turn. For Task 3, they adopted only three types of strings and a red circle to generate visual context. Moreover, these 7 tasks are insufficient to collectively evaluate the full spectrum of visual capabilities, leading to limited scope in evaluation objectives. In contrast, our AVSBench offers a systematic, comprehensive framework for evaluating a comprehensive set of visual skills of VLMs with an extensive dataset that is rich in color, shape, and other variations.The problems are categorized into three difficulty levels: easy, medium, and hard. Problems categorized as easy or medium should be quickly solvable by humans, whereas hard questions, although more time-consuming, are designed to be clear and easily verifiable. We clarify that the difficulty levels were determined by the authors, so there is a degree of subjectivity to the categorization.We present 99 sample problems to provide the readers with the general characteristics of our AVS-Bench dataset, along with the responses of Gemini-1.5-pro [49] , GPT-4o [40] , and LLaVA-Next-13B [29] . For readability, we report a summary rather than the full text of the model response. See Figure 5 for examples of full model responses.1Response : The red line intersects with the black lines 9 times in the image. Extracted Answer : 9 Score : 0How many times does the red line segment intersect with the black line segments in the image?Response : The red line intersects the black lines **9** times. \n Extracted Answer : 9 Score : 0Response : The red line intersects with the black lines at three points.Extracted Answer : 3 Score : 0 BlueYellow(2/3), Pink(1/3)In the image, there are nodes labeled A through E and 1 through 3, connected to each other by straight lines. Among A, B, C, D, and E, which node is not directly connected to node 2?A GreenThere is a gradually changing colors in the shapes on the first row. What will be the color best fits in a shape labeled "?" ? Choose one from "A", "B", "C", "D" and "E". BIn the image, there are black, red, orange, green, blue, and purple regions, with some regions sharing sides. Choose all the regions that share at least one side with the red region and list their colors.GreenGreen, Blue, Orange In this problem, we say two areas are adjacent to each other if one is directly above, below, to the left, to the right, or diagonal to the other. Answer with all the colors of the areas that are not adjacent to box A: red, orange, yellow, light green, green, light blue, blue, purple, gray, and white.Yellow, Purple In the image, there are shapes of different colors.Identify the color of the only convex shape. Choose from: red, purple, orange, green, or blue.Orange Gemini Orange GPT Orange LLaVA Blue QuestionIn the image, there is a 9 by 9 grid with some blocks colored. Among blue, purple, red, green, and yellow, whose area's shape is convex?Purple In the image, there are two circles and a curve. Which color curve is most likely to be part of a circle? Choose from: red, orange, blue, purple, green, yellow, black.Blue Gemini Blue GPT Blue LLaVA Blue QuestionThe image shows a graph of a function with labeled points: A, B, C, and D. Choose the point where the function is most sharply bent. State the label of the point. AIn the image, there are a cloud, a sun, a moon, a lightning, and a three-headed arrow. Identify the shape that is NOT pointed at by the arrow. Choose from cloud, sun, moon, and lightning.CloudCloudLightningIn the image, there are dots in various colors and an arrow. Guess the color of the dot the arrow is pointing at. Choose from: red, orange, blue, green, purple, yellow, brown, gray, black.YellowNo dot Read this rotated image.EverydayRead the six-letter word displayed in the picture. Helium Gemini STREAM GPT HELLO! LLaVA Not RecognizableWhat are the two characters marked with a blue circle in the phrase in the image? Answer with the two characters.EWGemini E W GPT E, W LLaVA N YWhich of the alphabets is not present in the image: "A", "U", "E", "F", "W", "N", "Y"?Ground TruthN Gemini N GPT N LLaVA URead the text in the given image.Sejong GreenGreenThere are several letters in the image. What is the letter that is fourth from the above? You should also consider its case. For example, if 'z' is fourth from the above, your answer should be "z". Red Gemini Brown GPT Brown LLaVA PurpleIn the image, there are several line segments with distinct colors. Given that there is a unique line segment that is parallel to the yellow line, what is the color of the line? Choose from brown, green, blue, and black.GreenBrownGreen Question Which color's text is written parallel to the black line in the image? Choose from: red, blue, green, orange, or purple.GreenNoneBlue Question Among the six red lines labeled 1, 2, 3, 4, and 5 respectively, choose all the red lines that are parallel to the blue line in the given image.1, 3, 4 In the image, there are curves labeled as: a, b, c, x, y, and z. Guess the only curve that is smooth everywhere.x Gemini z GPT c LLaVA aIn the image, there are multiple squares with a number or a letter labeled inside. Name the only label that is written at the bottom of its corresponding square. Lower Right Gemini Lower Right GPT Lower Left LLaVA Upper RightIn the image, there is a slice of yellow cheese with holes. Which hole forms a perfect circle in the cheese? Choose from: v, w, x, y, z.x Gemini w GPT y LLaVA V Question Which color's area is totally contained inside the blue area? Choose just one from: red, green, pink, gray, purple, yellow, brown. Yellow Gemini Gray GPT Purple LLaVA Green QuestionOrange Gemini Orange GPT Orange LLaVA Orange QuestionIn the image, point P is the midpoint of the line segment BD. Similarly, choose the point that is most likely to be the midpoint of the line segment AC. The answer should be one of E, F, G, or H.E Gemini F GPT F LLaVA EIn the image, there are squares of the same size, colored red, yellow, black, green, blue, orange, pink, and purple. Identify the color of the square closest to the top. Blue Gemini Blue GPT Black LLaVA Blue QuestionUpperUpper LeftUpper LeftUpper LeftAmong black, blue, red, green, and yellow, choose the color of the pair of points that is not line-symmetric about the black line.YellowIn the image, there are a black triangle on the left of the line and five triangles on the right of the line. Choose the triangle on the right that is symmetric to the black triangle with respect to the line. Choose your answer from "red", "blue", "green", "yellow", "purple". For example, if black and red triangle are symmetric with respect to the line, your answer should be "red". Blue Gemini Green GPT Purple LLaVA BlueIn the image, there are four arrows and three line segments of different colors. The black arrow is the original arrow, and the other arrows are the results of reflections with respect to the line segment that matches the color of the resulting arrow. Among the three colors, one is incorrectly reflected. Which color is not correctly reflected? Choose from blue, gray, and purple.Gray Gemini Purple GPT Blue LLaVA Purple QuestionIn the image, there is a black line and four pairs of curves (each curve has the same color as its partner), each representing a reflection of a figure. Which pair failed to correctly represent a reflection with respect to the black line? Answer with the color of the pair: blue, green, black, or red.Red Gemini Red GPT Blue LLaVA Blue QuestionIn the image, there is a 9 by 9 grid with some blocks colored. Among blue, red, orange, green, and yellow, which color blocks are line-symmetric about the black line? CircleGuess which color's line is the only line orthogonal to the bold black line in the image. Choose from: blue, green, red, orange, purple, brown. In the given picture, there are line segments which become the sides of some shape. What is the type of the shape in the picture? Choose one the following and answer in a single word: triangle, rectangle, parallelogram, pentagon, hexagon, circle, star, or heart.Hexagon In the image, which line is tangent to the orange curve at point O among the green, blue, black, and yellow line segments? State the color of the line segment in lowercase.In the image, there are 9 alphabet letters, labeled A to I, along with one red curve and one blue curve. List all the alphabet letters that are located between the red curve and the blue curve. In this section, we provide detailed definitions of the 36 atomic visual skills, together with a corresponding problem sample from AVSBench.1. Angle is a skill to understand how an angle is visually represented. Angle is the primary factor in how a polygon looks, how two or more objects are related, and many other situations.Cardinal is a field about counting distinct objects or specified concepts. Mastery of cardinals implies measuring quantities or dealing with multiple objects. Especially, it should take into account everything that satisfies given conditions, giving a difference from the skill of understanding Ordinals.Congruence is a skill of detecting objects with the exact same scale and shape, and understanding their correspondence. Congruence is a primary component of visualizing various symmetries including translation, rotation or flipping. Congruence distinguished from other equivalence because it requires the objects to be equal at all levels of measurement.6. Convexity is a skill of understanding convexity of given shapes. The skill is also closely related to detecting bumps or indentations and understanding convex and concave functions.Intersection is a mastery of detecting intersections of lines and curves. The skill is necessary for interpreting relationships among 1-dimensional objects, and also among higher dimensional objects from 1-dimensional representations of their boundaries.8. Line is a skill to detect line segments and understand their roles in the image. This skill is a fundamental unit in understanding various objects as polygons, graphs and diagrams. 9. OCR is a skill to detect and read characters from visual inputs.10. Ordinal is a skill to count certain objects or concepts in a given order. Mastery of this skill requires not just counting but also focusing on specific portions and order of targets, giving a difference from Cardinal Understanding.11. Overlap skill is about correctly recognizing two or more objects sharing a common area.The skill is crucial in understanding overlapping shapes or complex shapes such as diagrams.12. Interior is a skill of distinguishing between interior and exterior of the target area. This skill is essential in perceiving different areas.13. Relative Position is an ability to identify positional relationships between objects that cannot be simply described such as inside, outside, or moved in a certain direction. This skill requires comprehension of complex relationships such as "positioned in between," or "at the same side of."14. Reflection is a field of recognizing linear symmetries. It requires detecting the axis of reflection and induced correspondence of objects.15. Length is a skill to handle lengths of different objects. It involves comparing different lengths and measuring distances of objects.16. Rotation is an ability to identify changes in positions and angles induced by rotation, and detecting the axis of rotation. 17. Rotational Symmetry is a field of symmetric representations with respect to rotations. The skill involves understanding invariant geometric features under specified rotations.18. Symbol is a skill to detect symbols, understand their roles in the image, and combine them with other visual information to attain the correct interpretation of the image.19. Texture is a skill to understand textures of objects in the image. The skill is essential as texture is another main component of visual representation of objects, and is used to distinguish different objects with same shapes, such as line styles.Width is a skill to understand thickness and width of objects or areas. The skill is essential in measuring area or proportion of images together with length understanding.Adjacency is a skill to recognize when two or more objects are next to each other. The skill is crucial in understanding features induced by close positions such as forming clusters.22. Absolute Position is a skill to correctly understand where the objects are represented as a part of the visual input, independently of other objects. This involves recognizing objects posited at corners of an image, or comparing heights of objects represented in the image.Area is a skill to handle 2-dimensional volumes, including comparing areas.Cardinal Direction is a skill to understand primary directions including up, down, left, right, or diagonals. This involves recognizing North, South, West, and East directions.29 25 . Orthogonality is a skill to identify orthogonal relations of objects in the image, including a right angle formed by two lines. Understanding orthogonality is fundamental in geometry, design, and engineering.Tangency is a skill to detect tangent objects. This skill focuses on geometric representation of tangent curves or boundaries, and is different from understanding adjacency that rather focuses on positional information.Connectedness is a skill to identify connected components and detect links between objects. This is crucial in understanding interactions and distinguishing distinct components.28. Parallel is a skill to recognize parallel lines or curves. This is essential in identifying fundamental objects like squares.29. Similarity is a skill to understand equivalence of geometric representations independent of scale. It also involves understanding of rescaling or comparing aspect ratios.Color is an ability to perceive, distinguish different colors, and understand the change in saturation and brightness.Coordinate is a skill to recognize and acquire correct information upon coordinate systems. We provide and acquire information about different systems such as polar coordinates.Point is a fundamental capability to detect points and understand their roles in the image. It also involves understanding nodes in different graphs.33. Shape is a skill to understand details of shapes and compare different shapes independently of positions or tilts. It also involves identifying popular shapes such as triangles, rectangles, circles, and stars.Curvature is an ability to measure and compare curvatures of different curves. This involves distinguishing between straight lines and wavy curves, and detecting bends in shape.35. Sharpness is a skill to detect pointy parts of a shape. This is essential in understanding the representations of non-smooth objects such as points of a function that are not differentiable.Orientation is a skill to correctly distinguish clockwise and counterclockwise tendencies induced by not only rotations but also other movements that result in clockwise and counterclockwise directional change. The name originated from the mathematical definition of orientation in differential geometry.We evaluated closed-source models ChatGPT [40] , Gemini [49] and open-weight models LLaVA-NeXT [29] , LLaVA-OneVision [26] , Math-LLaVA [47] , Table -LLaVA [60] , Phi-3.5-Vision [1] , InternVL2 [10] , DeepSeek-VL [31] . Tables 1 and 2 describe further details about the model sizes and versions. For closed-source models, we used the commercial APIs. All models' temperatures were set to 0. We used GPT-4o mini to extract answers from model responses and to judge correctness. Few-shot in-context learning prompts we provided to GPT-4o mini as described in Tables 3 and 4 . To verify the reliability of this pipeline, we randomly selected 128 problems from our dataset and compared the scores from GPT-4o mini with human annotations. Reassuringly, GPT-4o mini and the human annotators agreed on the scoring of the 128 problems. We attribute this high level of reliability, in part, to the straightforward and clear design of our questions and answers.In this section, we provide full details of Section 3 and further analysis of Findings 3.Table 5 presents our full evaluation results on AVSBench. "Random Chance" represents the expected accuracy of randomly choosing multiple-choice questions and scoring 0 from all non-multiplechoice questions. The subcolumn named "overall" indicates the accuracy across all problems of its corresponding skill. The column named "TOTAL" describes the overall accuracy across all problems in AVSBench, involving all skills.As mentioned in Findings 3, CoT did not provide meaningful performance gains when used with GPT-4o and Gemini-1.5-pro. Gemini-1.5-pro had only a 0.1% gain from applying CoT, and GPT-4o had a 1% gain. However, CoT worsened performances of skills including OCR, Length, and Symbol.As in the case of Figure 17 , by inspecting the responses of GPT-4o with and without CoT prompting, we observe that the additional reasoning steps afforded by CoT are not helpful in comprehending visual inputs.bThe shape labeled \"z\" is different from the others. While all the other shapes (a, b, c, x, y) are arranged horizontally, the shape labeled \"z\" is arranged vertically. In the image, there are different shapes labeled as a, b, c, x, y, and z. Guess which shape is different from the others. Rotational Symmetry Shape Sharpness Similarity Symbol Tangency Texture Width easy medium hard total easy medium hard total easy medium hard total easy medium hard total easy medium hard total easy medium hard total easy medium hard total easy medium hard total Table 5 : Full details of evaluation results on AVSBench. Each value represents the accuracy of the model of its row, on problems with the difficulty and for the skill of its column.Direction is an ability to recognize linear direction in an image. It is a fundamental skill in human vision, supporting representation of linearity and multi-dimensional relations.Boundary is a skill to understand the ends of objects or areas, and to detect visual representation of edges. The skill is used in distinguishing between distinct objects, or detecting boundaries between spaces. 



Reviews: review: title: title: The paper is overall well-written and the main ideas are clearly explained.
main_review: main_review: Paper Summary:
This paper introduces AVSBench, a novel benchmark designed to evaluate 36 atomic visual skills in geometry and conduct experiments on influential models to show that they struggle with fundamental geometric visual tasks.

Weakness and Strength:

(+) The paper is well-structured and presents the problem and findings. 

(+) This paper studies an interesting and well-motivated research topic: Benchmark to evaluate whether VLMs possess capabilities to understand basic geometric features.

(-) The introduction to the novel benchmark is somewhat limited.
title: title: Review
main_review: main_review: **Summary:**
The paper presents the *Atomic Visual Skills Benchmark (AVSBench)*, a dataset designed to evaluate VLMs on their ability to comprehend basic geometric features, termed *atomic visual skills*. The benchmark includes 5,073 handcrafted image-question-answer pairs covering 36 skills, such as understanding angles, boundaries, and congruence. It aims to highlight the limitations of current VLMs, which struggle with these seemingly trivial tasks, despite their success in more complex scenarios. Models like GPT-4o and Gemini-1.5-pro perform relatively well on easier tasks but struggle with harder ones.

**Strengths:**
1. **Useful Dataset with Detailed Categories:** AVSBench provides a comprehensive and systematic breakdown of 36 atomic visual skills, covering a wide range of essential visual features relevant to high-school level geometry and other visual comprehension tasks. This detailed categorization enables more granular evaluation of VLM capabilities.
   
2. **Well-Motivated Problem:** The paper addresses an important issue—VLMs often fail at tasks involving simple visual comprehension. The problem is clearly defined, and the benchmark helps shed light on the underlying limitations of these models, thereby contributing to the understanding of VLM behavior.

3. **Comprehensive Models for Evaluation:** The paper evaluates several state-of-the-art VLMs, including GPT-4o and Gemini-1.5-pro, along with other open models. The results are compared across different levels of task difficulty, providing insights into how various models perform across the spectrum of atomic visual skills.

**Weaknesses:**
1. **Not Enough Baselines:** The study only explores the performance of the models with chain-of-thought (CoT) prompting and does not investigate other advanced text or visual prompting methods. Incorporating more diverse baselines would strengthen the evaluation and provide a broader perspective on how different techniques could improve model performance.
title: title: Review of AVS for VLMs
main_review: main_review: This work focuses on evaluating VLMs' perception as understanding details in abstract shapes. The authors introduce a novel benchmark, that decomposes 36 different atomic skills for perception. The work evaluates existing VLMs and shows that these models struggle with such tasks. The work is interesting and significant for the field as it shows VLMs' shortcomings in perceiving abstract shapes in a more fine-grain way as opposed to previous work which had focused on the general performance of these models on such tasks.

I find these areas for improvement:
1) I am wondering if these skills are truly atomic. For instance, the "intersection" in Fig 1. is not an atomic task. I think there are at least two skills involved here: 1) the ability to find intersections, and 2) the ability to count found intersections. As we know about these models, they are not good at counting, so I assume a model might be able to find the interactions, but unable to count them (however, in the example provided in Fig 1., it is not the case because of the great distance between the correct answer and the one provided by GPT-4o). Overall, this issue concerns me about those questions that involve counting. I think authors can simply improve the reliability of their results by removing the questions that involve counting and more generally, looking for skills that are not truly atomic and removing them from AVSBench.

2) The models evaluated are very interesting as they show that specialized models such as Math-Llava also struggle with these tasks and CoT does not help that much. These results resonate with some other work that the authors might want to refer to as well ([1], [2], [3]). I am wondering if fine-tuning Llava might help with its performance. To further increase the significance of this work, it would be important if the authors evaluated a fine-tuned model that is specifically fine-tuned on these AVS skills (as mentioned in the conclusion section).

3) I find that maybe the most of atomic tasks are not high school level. For instance, all of the examples provided in Fig 1. are elementary level. It would be important if authors could provide fine-grained detail on the age appropriateness of these skills. Also, are the authors claiming that these skills are exclusive? If so, how did they make sure that they included all the atomic skills? 

4) I find the evaluation setup a bit overly complicated. Why did the authors use another LLM (GPT mini) to extract the response generated by the VLM? Perhaps they could have had the models generate their response in a specific format so they could have parsed the output with no further API calls.

Overall, I find this work a great contribution to the MATH-AI workshop as it shows important shortcomings in VLMs that need to be overcome in the future. I like the writing quality of the paper and I believe that it can benefit from the above points to be in great shape for future conference submissions.


[1] Fu, D., Khalighinejad, G., Liu, O., Dhingra, B., Yogatama, D., Jia, R., & Neiswanger, W. (2024). IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations. arXiv preprint arXiv:2404.01266.
[2] Rismanchian, S., Doroudi, S., & Razeghi, Y. (2024, May). Turtle-like Geometry Learning: How Humans and Machines Differ in Learning Turtle Geometry. In Proceedings of the AAAI Symposium Series (Vol. 3, No. 1, pp. 586-587).
[3] Melanie Mitchell, Alessandro B Palmarini, and Arseny Moskvichev. Comparing humans, gpt-4, and gpt-4v on abstraction and reasoning tasks. arXiv preprint arXiv:2311.09247, 2023.
title: title: Review of paper: Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models
main_review: main_review: # Review of paper: Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Models

# Reviewer Summary of paper:

The paper proposes a new benchmark dataset named, AVSBench which consists of 

# Major points: 

1. Hypothesis 1 is not directly answered in the work as far as I can tell and only indrectly answered via the results. While I appreciate the simplicity of using problems with a single skill, the authors should provide some descriptions of why multi-skill tasks were not included. 

2. MathVerse found that models use shortcuts in language only portion of models but also found 
mathematical training benefits performance, cf. Section 3.2 of the paper.
Moreover, the majority of the models did not design the neural network architecture for domain specific multimodal language models like math problems. 
In "Smart Vision-Language Reasoners", also presented at ICML 2024 in tandem with Mathverse, the authors found that designing and fine tuning for domain specific multimodal language models improved performance on the Smart-101 dataset, an image + text dataset which is similar in structure to the AVSBench data. 
Specifically, on the architecture aspect of the multimodal language models in "Smart Vision-Language Reasoners" they introduced via a cross attention layer to between the textual backbone and the vision backbone model to promote alignment of visual and textual features. 
This is in Contrast with the Llava architecture which uses a learned projection layer and an MLP to mix the weights of the vision backbone (CLIP) and the text backbone (Vicuna), cf. section 4.1 of the original Llava paper.

Suggest to reference these two papers: 

Are Deep Neural Networks SMARTer than Second Graders? CVPR 2023 

https://openaccess.thecvf.com/content/CVPR2023/papers/Cherian_Are_Deep_Neural_Networks_SMARTer_Than_Second_Graders_CVPR_2023_paper.pdf

Smart Vision-Language Reasoners, ICML Math AI workshop 2024

https://arxiv.org/abs/2407.04212

Note: The Smart-101 dataset focuses on problems for children ages 6-8 so differ from those of high school geometry.

3. The authors mention that the skill difficulties for each problem are subjectively determined by the authors themselves. However, no mechanism, e.g. other annotators were used. Recommend the authors use independent annotators or use the multimodal languages evaluated in the manuscript to confirm (or not) that these difficulties match. 

4. The skill categories are claimed to be fundamental/atomic for geometric problem solving but there are no references to indicate the source of the taxonomy nor support claims of being fundamental skills for geometry problems. 

# Minor points:

1. The author(s) claim without reference that the curricula of high school geometry are more or less agreed upon. The claim is made without a reference, after searching the PISA site and using a major search engine, I was unable to find something that showed a generally agreed upon high school geometry syllabus. 

The referee suggests to rephrase as "minimal common curricular elements" or something similar, or to provide a reference to the commonly agreed sylabus. 

2. The referee did a quick search engine check and AVSBench is a previously used name for an audio visual benchmark dataset, cf. https://paperswithcode.com/dataset/avsbench, perhaps a different name should be chosen to avoid namespace collisions?

# Additional remarks:

1. In addition to the, COT prompting, the authors might try visual-COT which has been proposed in the literature to compare with pure textual COT.