Smart Vision-Language Reasoners

Authors: denisa  roberts
lucas  roberts
Venue: ICML_cc_2024_Workshop_AI4MATH
Published: 2024-06-01 04:38:29

Abstract: In this article, we investigate vision-language models (VLM) as reasoners. The ability to form abstractions underlies mathematical reasoning, problem-solving, and other Math AI tasks. Several formalisms have been given to these underlying abstractions and skills utilized by humans and intelligent systems for reasoning. Furthermore, human reasoning is inherently multimodal, and as such, we focus our investigations on multimodal AI. In this article, we employ the abstractions given in the SMART task (Simple Multimodal Algorithmic Reasoning Task) introduced in \cite{cherian2022deep} as meta-reasoning and problem-solving skills along eight axes: math, counting, path, measure, logic, spatial, and pattern. We investigate the ability of vision-language models to reason along these axes and seek avenues of improvement. Including composite representations with vision-language cross-attention enabled learning multimodal representations adaptively from fused frozen pretrained backbones for better visual grounding. Furthermore, proper hyperparameter and other training choices led to strong improvements (up to $48\%$ gain in accuracy) on the SMART task, further underscoring the power of deep multimodal learning. The smartest VLM, which includes a novel QF multimodal layer, improves upon the best previous baselines in every one of the eight fundamental reasoning skills. End-to-end code is available at \href{https://github.com/D-Roberts/smarter}{github.com/D-Roberts/smarter}.

Main paper: 
 introduction: 
 Human intelligence is oftentimes associated with the ability to operate on mathematical abstractions. In (Chollet, 2019) the author conducts an in depth discussion and formulatesThe first AI for MATH Workshop at the 41st International Conference on Machine Learning, Vienna, Austria. Copyright 2024 by the author(s). a formal definition of intelligence based on algorithmic information theory. Several meta characteristics of intelligent systems are listed as scope, generalization difficulty, priors and experience. On a different but related axis, (Didolkar et al., 2024) speaks of metacognitive capabilities of large language models, abilities that underlie all problem solving, including math problems. In a related work in the multimodal domain (Cherian et al., 2022) , a Simple Multimodal Algorithmic Reasoning Task (SMART) is introduced with visual-linguistic puzzles designed for children in the 6-8 age group (the US Kangaroo Olympiad style). In this work, an explicit categorization of underlying skills utilized by humans in problem solving are labeled and tallied as they get employed in solving puzzles as measure, path, pattern, logic, math, algebra, and spatial skills. Furthermore, reasoning must be multimodal because humans have multiple senses whose inputs are amalgamated to reason at higher abstractions. Better abstractions are akin to better mental representations. Deep neural networks excel at learning You have to block some locations in the maze so that the feline cannot reach the bird. Which of the following options to block will fail? Answer Options: A: 1, 2, and 3; B: 4; C: 5, 6, and 7; D: 8 and 9 ; E: 10, 11, and 12. Counting Question: The entire pie is divided among several children. Each child receives a piece of pie, and each piece of pie looks identical. The maximum possible number of children there is: Answer Options: A: 7; B: 2; C: 1; D: 4; E: 3. Algebra Question: The entire pie is divided among several children. Each child receives a piece of pie, and each piece of pie looks identical. The maximum possible number of children there is: Answer Options: A: 5; B: 4; C: 2; D: 0; E: 6. Measure Question:A student had a few canes with a height of 1 cm and a length of 5 cm. Using the canes, she built the arrangement illustrated. What is the width of the arrangement? Answer Options: A: 20; B: 30; C: 15; D: 5; E: 35. Spatial Question: Cristina made a setup using some green blocks and 94 white blocks. How many of these white blocks are not visible in the figure? Answer Options: A: 28; B: 61; C: 64; D: 90; E: 79. Logic Question: Emily has 7 toy items: a remote, a hair brush, a truck, an eraser, a rubber duck, carrots, and a toe ring. She keeps each toy at a different row of the shelf. The carrots lower to toe ring. Remote lower to truck and toe ring higher to truck. Toe ring higher to rubber duck. She keeps carrots as shown. On which row can the rubber duck not be placed? Answer Options: A: 4; B: 3; C: 7; D: 5; E: 6. Pattern Question: Which picture on the right matches with the left, if we invert the colors? Answer Options: A; B; C; D; E.(artificial) representations (Bengio et al., 2013) .We base our investigations in this article on a few conjectures with respect to mathematical reasoning and general problem-solving:• Intelligence is related to multimodal reasoning. If a person is deaf and cannot hear more than 50% of what is being said, the speech modality input is supplemented with reading faces and other visual aids (vision modalities), captions (text), as well as other modalities (all the other senses, as well as enhanced reasoning and computation abilities). Therefore, it is worth striving to improve multimodal deep learning architectures and their reasoning abilities.• Training leads to learning and improved reasoning.Math and physics Olympiad expert competitors practice more than non-experts in environments that value the pursuit. Taking a classic example, the Polgar sisters-three chess grandmasters-were trained to excel at chess. Similarly in the musical domain, Mozart and Beethoven were trained from a young age to excel in music. In fact, one interpretation of evolution is the act of training and learning. We learned to grow better brains out of necessity. Therefore, training neural networks specifically to enhance reasoning makes evolutionary sense.• Intelligence is related to better abstractions and those are related to better representations. Expert chess players, thespians, martial artists, mathematicians, and coders all develop fine-grained relevant mental representations so they can better reason and imagine new ideas rapidly. Therefore, improving the representations derived with deep learning architectures is worth pursuing (better image, text etc. representations as well as their cross-play).• If we make neural networks better at reasoning (we include here a few types of reasoning such as creative problem-solving in math, physics, logic and coding algorithms, puzzles and IQ tests, learning, planning and decision making), these skills may be transferable to science, strategy, medicine, law, and commonsense, with far reaching real world impact. 
related work: 
 Reasoning Surveys of deep learning for mathematical reasoning such as (Lu et al., 2022; Sun et al., 2023) mentioned the relatively smaller subset of works on multimodal / visionlanguage models in this space, with datasets and models which are smaller, niche, and mostly using visual question answering frameworks. These approaches are lacking since they are trained on natural images and not on models trained on vision and language datasets. Subsequent works such as (Zhang et al., 2024b; Wu et al., 2024; Lu et al., 2023) and this article, aim to enrich the multimodal mathematical reasoning domain.Vision-Language Models Opportunities for improvement of vision language models still exist along problem-solving and algorithmic reasoning ability, visual grounding, as well as architectures for encoding, decoding and aligning (Karamcheti et al., 2024; Tong et al., 2024; Liu et al., 2023; Wu & Xie, 2023) . In this article we focus on the reasoning ability along eight dimensions of reasoning. In the reasoning realm, much recent work focuses on evaluating vision-language models on general multimodal tasks (Yue et al., 2023; Lu et al., 2023) or applying a chain of thought approach for in context learning (Zhang et al., 2023; 2024a) . In (Azerbayev et al., 2023) large language models are pretrained to solve text only questions bringing on the full power of heavy-weight LLM, but without taking the visual signal into account. Then a separate line of work builds very large general Vision-Language Models (VLM) akin to an LLM. In (Li et al., 2023b) a vision-language architecture, the Query Transformer, adds transformer layers to frozen image and text encoders and learns in a contrastive pretraining paradigm on massive datasets. Llava (Liu et al., 2024a; 2023) versions (Liu et al., 2023) emerge as a multimodal instruction tuned large model finetuned on a science dataset. In (Gao et al., 2023) authors use an LLM and parameter-efficient visual instruction tuning, focusing on learning efficiently only the adapters, with early fusion of visual tokens in LLM layers. In (Tong et al., 2024 ) the deficiencies in visual grounding of large multimodal models is investigated and mixtures of visual features are proposed to improve vision modality. In (Li et al., 2023a) , another pretrained and visual instruction tuned framework is proposed, employing CLIP (Radford et al., 2021 ), Llama (Touvron et al., 2023) and Perceiver adapters (from Flamingo) (Alayrac et al., 2022) as well as a dataset, MIMIC-IT. The pretrained vision encoder in DinoV2 (Oquab et al., 2023) aims to leverage different techniques and diversity of images to pretrain backbones for the purpose of using them as general-purpose features. However, it is not necessarily true that even this general purpose encoder will encode all the visual signals, so fusing with representations from a backbone pretrained in a different fashion, for instance the SigLIP (Zhai et al., 2023) representation, may provide additional visual signal boost. Specifically, SigLIP improves on CLIP (Radford et al., 2021) for language-image pretraining by employing a sigmoid loss instead of the constrastive learning with softmax normalization and performs better across tasks.Multimodality and representation learning in Math AI.MATH-AI research has focused primarily on language models, but problem solving is inherently multimodal (text, image, diagrams, tables, numbers, symbols). In the current article we utilize the vision and text modalities to encode all concepts included in each puzzle, without employing separate encodings and/or representations for symbols or numbers. Multiple works focus on representation learning and reasoning with more than the specialized mathematical visual modality, such as symbols and numbers on symbolic reasoning (Li et al., 2023c) , specialized representation learning works for numbers (Golkar et al., 2023) , as well as more complex hierarchical math concepts with graphs in (Rute et al., 2024) , which would be interesting to include as further modalities in future works. In a related vein, (Wu & Xie, 2023) and (Tong et al., 2024) , proffer the reasoning ca-pabilities of multimodal large language models and explores their visual representation learning abilities.So how can we help (deep) artificial neural networks reason better? In (Cherian et al., 2022) experiments show that the visual signal is very important in solving complex multi-reasoning skill puzzles and, despite being very large, language-only models lag behind visual language models in terms of performance. Conversely, in (Zhang et al., 2024b ) the conclusion appears to be that large multimodal models cannot truly understand the visual diagrams for mathematical reasoning, along the line of weak visual grounding and poor attention to visual detail in (Tong et al., 2024) and (Wu & Xie, 2023) for large multimodal models for math, question answering, and other reasoning tasks. The Simple Multimodal Algorithmic Reasoning Task (SMART) introduced in (Cherian et al., 2022) contains puzzles that measure intelligence across eight different reasoning skill classes: counting, math, logic, path, measure, logic, and pattern.Problems include an image and a text question and are formulated as multiple choice. We can see a few examples of problems in Figure 2 . Baseline models trained in (Cherian et al., 2022) struggle to solve this task, especially when employing transformers. In the past, specialized neural networks such as (Mikuła et al., 2023) have been developed to solve specific reasoning tasks, specifically premise selection in automated theorem proving. In this article, we investigate how we can craft and train deep neural networks which employ several types of deep learning blocks and multimodal inputs from deep frozen transformers to reason better across the eight meta reasoning axes in the SMART task.The SMART reasoning task and baselines. A set of visionlanguage models are trained as benchmarks in (Cherian et al., 2022) Contributions. In (Awad et al., 2023) the authors demonstrated how a deep learning module which encode a sequence of image-and-text items using diverse representations composed on several modalities, across time steps, and across pooling methods, obtained impressive results in sponsored search and recommendations. Inspired by the ADPM in (Awad et al., 2023) and using tricks to train vision transformers in (Dolev et al., 2023) and (Karamcheti et al., 2024) , a smarter VLM is built. In this article, we make the following contributions on the VLM reasoning axis:• Introduce a novel multimodal QF-layer to learn a hidden representation from the vision and language modalities.• Improve the MLP decoders in (Cherian et al., 2022) through GELU activations, residual connections, and layer normalization.• Improve the sequence decoder by replacing the LSTM with a GRU.• Strengthen the vision modality by learning an adaptive visual representation on top of two fused vision backbones: SigLIP (Zhai et al., 2023) and DinoV2 (Oquab et al., 2023) similarly to (Karamcheti et al., 2024) . In this way, the model makes better use of the puzzle's image.• Strengthen the text-vision alignment by using a frozen SigLIP language encoder together with the vision modality which includes the SigLIP vision backbone.The pretrained text encoder does not overpower the visual signal as much as an LLM as seen in (Tong et al., 2024; Zhang et al., 2024b) .• Furthermore, the smarter VLM reasoner includes a composite hidden representation through the concatenation of language-only representations, an adaptive image-only representation learned on top of the fused frozen foundation backbones, and the QF multimodal layer representation which includes a language-vision cross-attention sublayer. Ablation studies in Section 4 show that the QF layer is essential to the smarter VLM reasoner. The use of cross-attention improves the ability of the reasoner to make use of the puzzle's visual cues.• These model improvements lead to up to 48% accuracy gain across several of the meta reasoning skills measured by the challenging SMART task. 
methodology: 
 We formalize the problem as supervised learning with classification loss. For each image-question instance, we predict the probability of one of five answer options. When the options are in the form of a sequence, a decoder module decodes the answer sequence first, and then the answer is translated to one of the {A, B, C, D, E} multiple choice options. In this article, the decoder is a recurrent neural network (Cho et al., 2014) . Furthermore, we focus on training deep learning architectures from scratch for the SMART task with inputs from diverse pretrained frozen backbones. We focus the investigation on the eight skill classes counting, (counting, math, logic, algebra, path, pattern, measure, spatial) rather than individual puzzle groups, since these reasoning skills are of more general interest across domains and trademarks of intelligence. In (Chen et al., 2019) authors demonstrate that strong pretrained backbones can perform without meta-learning, so we do not employ metalearning as in (Cherian et al., 2022) . The accuracy metric calculated on the validation set is used to tune the models and evaluate method success, and the accuracy for the five-class classification is evaluated on the test set. The accuracy is calculated overall, and more interestingly, broken down by reasoning skill class (counting, math, etc.).We derive a multimodal representation through a novel layer, the QF layer, inspired by the ADPM in adsFormers (Awad et al., 2023) , the QFormer in (Li et al., 2023b; Zhu et al., 2024) and VilBERT in (Lu et al., 2019) . More recently, (Karamcheti et al., 2024) and (Tong et al., 2024) combine multiple image representations to leverage diversity of signal in vision-language models, in a similar vein to the ADPM. Additionally, inspired from (Cherian et al., 2022) , we learn an adaptive image representation on top of the fused frozen backbone. Moreover, we directly include the frozen SigLIP text encoder to encoder the question tokens.The choice of the SigLIP text encoder is two-fold: firstly, the alignment with the SigLIP vision encoder; secondly, to tame the language power by not employing a very large language model (LLM) such as (Gao et al., 2023) , which standard VLM such as BLIP (Li et al., 2023b) or Llava (Liu et al., 2023) employ. In (Tong et al., 2024) we see that visual grounding is lacking in large multimodal models. Furthermore, in (Cherian et al., 2022) the visual signal is quite important, the accuracy loss is more when removing the image rather than the text question, so the visual signal needs to be protected as it is critical for the reasoning skill necessary to solve the puzzles.The QF layer representation learned from the imagequestion input is concatenated to the average pooled text representation and the puzzle-specific adaptive image representation from the fused image encoders. As depicted in Figure 1 , the resulting composite representation, denoted compositeR, takes as input three component representations, r 1 , r 2 , and r 3 , defined as follows.• Text representation r 3 is an average pooled encoding of the question sequence of max length 110 tokens. Each token is first encoded using the frozen SigLIP text model into a representation of size 768. ThenEQUATION• An image representation r 1 from the puzzle-specific image encoder block of dimension 128 seen in 1. The dimension is a hyperparameter selected via optimization. The image encoder consists of two feed forward layers with a GELU unit, with separate weights for each puzzle head, for the 101 separate puzzle groups (e.g. one loss calculated per puzzle-group). Each encoder takes as input the image representation from the two fused pretrained vision backbones, DinoV2 (Oquab et al., 2023) and SigLIP (Zhai et al., 2023) , each of dimension 768. Specifically, for an image X, r 1 isEQUATIONfor i ∈ {1, . . . , 101}, a distinct puzzle group.• A QF representation, r 2 , is produced by the QF layer which takes as input the encoded image representation r 1 and the SigLIP-encoded sequence of text tokens (before average pooling). First, the SigLIP frozen language backbone encodes the 110-long question text sequence. Then, the QF layer passes the text sequence through a multi-head self-attention block (Vaswani et al., 2017) . The resulting hidden representation is fed to a cross-attention layer as query, with keys and values coming from the adaptive image encoder representation, marginally inspired from the QFormer in (Li et al., 2023b) and VilBERT (Lu et al., 2019) but with multiple differences. Distinctly from these works, in our case the image encoder in the cross-attention sublayer is a per-puzzle group adaptive representation learned on top of the frozen fused concatenation of DinoV2 and SigLIP vision backbones. Finally, an intermediate stack of fully connected layers, with residual connections (He et al., 2016) , dropout (Srivastava et al., 2014) , and layer normalisation (Ba et al., 2016) produces the QF text-and-vision multimodal representation. Specifically,EQUATIONFinally, the composite QVFusion layer aggregates these distinct representations (text-only, vision-only, and textand-vision QF multimodal) via concatenation producing the composite representation CompositeR ∈ R 2 * 768+128 , and then passing it through a two-layer feed forward module with Gaussian error linear units (Hendrycks & Gimpel, 2016) in between , before being read by the puzzle specific decoder.The composite representation isCompositeR = CLayer([r 1 , r 2 , r 3 ]) (6) = LayerN orm(Concat([r 1 , r 2 , r 3 ]). (7)The QVFusion layer in 1 isEQUATIONFinally, the decoder, which is either a stack of three fully connected layers separated by GELU activations, or a gated recurrent neural network (GRU) (Cho et al., 2014) for sequence-type answer puzzles, produces predictions fed to a cross-entropy loss. The introduction of GELU units with layer normalization boosts performance, as they do in many recent attention-based multimodal neural networks (Liu et al., 2023; Alayrac et al., 2022; Li et al., 2023b) , by allowing for a smoother loss landscape than rectified linear units with batch normalization layers. 
experiments and results: 
 We train several baselines from (Cherian et al., 2022) and give results in Table 1 . We chose to move forward with the frozen BERT+ResNet50 as baseline for two reasons: 1. Note that the numbers are extremely close between the frozen and unfrozen variants but the frozen variant does better on Math, a top skill of interest for this investigation; 2. The backbones are frozen which we favor in this article for a few reasons. The first reason is efficiency of training. Frozen backbones result in fewer parameters to update. Secondly, as noted in (Karamcheti et al., 2024) Training and Evaluation. Tables 2 and 6 Final results. As can be seen in Table 5 , the best trained models display massive gains in accuracy (eg. +48% gain over the baseline in the counting skill) across reasoning skills. Recall that the baseline was chosen as the strongest in the math skill. Truly so, the baseline was hardest to beat in math vs. other skills. Tables 2 and 6 (in the Appendix) list the hyperparameter setting and other training choices experimented with to arrive at the best results in Table 5 . In summary, the smartest VLM reasoner was trained using the following deep learning choices, inspired as a starting point from results in (Awad et al., 2023; Dolev et al., 2023; Roberts, 2019; Olteanu Roberts, 2021 ):• Adam optimizer with decoupled weight decay from (Loshchilov & Hutter, 2018) with weight decay of 0.2, eps = 1e -8, and beta2 = 0.98.• Cosine learning rate scheduler (Loshchilov & Hutter, 2016) with ten warmup steps, using the implementation in the HuggingFace repository (Wolf et al., 2019) .• Clipping the gradient norm to no more than one to avoid explosions.• Layer normalization throughout the architecture modules with eps = 1e -6 for better learning and generalization.• A SigLIP frozen language backbone and fused DinoV2 and SigLIP vision backbone.• A composite hidden representation with a QF layer with two attention heads, concatenated with text-only and adaptive vision-only representations.• Dropout probability of 0.2 anywhere dropout is used.• Employing Gaussian error linear units instead of rectified linear units in the decoder leads to improvements across the eight skills. The MLP decoder is akin to SigLIP's MLP block (a stack of feed-forward layers with GELU activation).• Hidden representation sizes of 128 (for instance for the adaptive image representation), except the hidden size within the GRU which is 256. Note that we learn an adaptive per-puzzle group visual representation by using a fully connected layer on top of the fused vision backbone.• GRU decoder for problems with sequence answer, as they are easier to train than LSTMs. QF layer ablations. The QF layer employs a multihead self-attention (MHA) sublayer using the SigLip language encoder representations and a cross-attention sublayer which uses the text hidden MHA representations as queries and the adaptive image representations as keys and values. There-fore the hidden sizes in the MHA QF subnetworks are fixed to the representation sizes of the language and vision encoder representations. There is flexibility on the intermediate subnetwork composition as well as the number of heads, addition of normalization, dropout, and residual layers. We have performed extensive ablation experiments to understand the impact of these different choices for the QF layer, and how the learned QF representation performs as part of the QVFusion composite representation where it is concatenated with vision and language-only representations.Constituency of composite representation ablation results. From Tables 3 and 4 we can see that the inclusion of the QF layer representation together with the vision-only, language-only, both vision and language or QF representation-only improves accuracy on all skill sets. Intuitively, the model can better use the word-language crosssignals to make sense of the puzzles. Interestingly some skill sets benefit from visual cues more, where only QF and vision representations are included in the composite representation and not language (pattern and measure) while other skills (math, spatial) benefit from using QF and language representations (no image representation concatenated in the composite QVFusion), which perhaps is an indication that in some math puzzles the model cannot make good use of the diagram/image as mentioned in (Zhang et al., 2024b) .The insight here is that the model can still make sense from the cross-signal from text-image encoded through the QF layer which includes a learned cross-attention sublayer with language as queries and the adaptive vision representation (learned on top of the fused frozen vision encoders) as keys and values. Intuitively this makes sense because humans will rely on the visual cues more for some type of problems and more on the verbal cues for others. Furthermore, one theory of learning postulates that some humans are more "visual learners" while others are more "auditory" learners, relying more on speech and language (Pennsylvania Higher Education Assistance Agency).Activation function used inside the QF layer. We ablated the activation function to use the self-gated activation (Ramachandran et al., 2017) instead of GELU, as well as ReLU.Several of the large language models, such as (Touvron et al., 2023; Jiang et al., 2024) , use the SiLU activation, which motivated our choice. We found that GELU works best on most skills, except on math-type puzzles, where both ReLU and SiLU work better. Based on these results only we do not have an intuition of why this might happen. We will perform additional future experiments on a larger dataset for deeper understanding.QF intermediate sublayer ablations. The QF layer includes a multihead attention (MHA) sublayer which takes the frozen text representations as input, a cross-attention layer which uses the MHA hidden representations as queries and the adaptive image representations as keys and values, and an intermediate final stack consisting of two fully connected layers and a residual connection with dropout and layer normalization. Exclusion of the residual connection (together with dropout and layer normalization) had a detrimental impact across skills, confirming the importance of residuals and regularization techniques for learning in deep transformers and for generalization. Furthermore, ablation on the dropout level, confirms better generalization with more regularization from dropout. The ablation of the intermediate layer sizing had mixed results across reasoning skills and we proceeded with the symmetry of using the hidden size used elsewhere throughout the smartest reasoner architecture (256). The QF layer includes two types of multihead attention (language only self-attention and language-vision cross-attention) which share the number of heads. An ablation on number of heads (1, 2, 4, 8) showed mixed results across skills with very similar results for average accuracy (except one head, which underperforms all the other choices) and is worth experimenting with further on larger datasets in future work.Development process and scaling side note. For a deep understanding of the behavior of the models with various architectural and hyperparameter choices, in the development phase, we started with training and evaluation on a very small subset of data for quick insight and iteration: only 20 questions per puzzle, with a batch size of 16, on a split ratio of train : val : test = 40 : 20 : 40. The experimental results are available in Tables 7 and 8 in the Appendix and were tracked with CometML (Comet.com, 2021) and publicly accessible at vlm-reasoners. 
discussion and future work: 
 In this article, we show how deep learning architectural innovations as well as hyperparameter and training choices led to improvement in model performance on the SMART reasoning task. Multimodal transformer-like architectures, deep learning representations, and stronger visual grounding led to improvements in eight fundamental reasoning skills of vision language models.Future work. Considering the different learning dynamics for the eight skill classes, a multitask learning ap-proach (Caruana, 1997; Lu et al., 2022; Dolev et al., 2023) with eight tasks may afford modulating the impact of eight weighted losses to account for the different dynamics. A mixture-of-experts approach (Zhao et al., 2019) Efficient training techniques employing compression (Dettmers et al., 2024) , autodiff variations (Roberts & Roberts, 2020) , or variations of multimodal transformers' efficient training (Liu et al., 2024b) may facilitate access to larger model sizes. In this article, visual and text representations are combined through concatenation. Experiments with an approach similar to (Ramrakhya et al., 2024) , where learned representations on the decoder path take frozen visual features as inputs and are conditioned on text embeddings via an element-wise product, show faster learning and better performance in the context of Math AI, according to initial experiments. Decoder-only architectures (Radford et al., 2019) took generative modeling by storm, and exploring other decoding architectures for the VLM reasoner instead of recurrent neural networks, such as Perceiverinspired layers (Alayrac et al., 2022) , or convolutional neural networks (He et al., 2016; Roberts, 2019) , may show interesting results. Furthermore, utilizing a frozen text encoder pretrained to outperform in mathematical reasoning and for efficiency, such as Mistral in (Jiang et al., 2023) , or Mixtral in (Jiang et al., 2024) , in conjunction with the strengthened vision encoders and cross-attention layers introduced in this article, is a further avenue worth exploring for next generation VLM reasoners.Additionally, in (Hu et al., 2024) LLMs are instructed to follow rules for general problem solving, instead of relying on the cases already seen in training. We expect that in the multimodal smart puzzle solving case, where we have multiple generated instances of each unique root puzzle, a similar investigation may improve generalization to unseen root puzzles. In fact, based on the progress in the quality of visual representations generated for multimodal mathematical knowledge (Wu et al., 2024) , more problems could be automatically generated using machine generated representations based on models, for further data augmentation to improve learning and generalization. While we deep dive into the SMART benchmark, several other multimodal evaluation benchmarks were recently published in the AI for Math space, as mentioned in the literature review section, and our architecture can be applied to any of them, which we leave for future work. Finally, other Math AI tasks, such as theorem proving and other tasks mentioned in the excellent survey article (Lu et al., 2022) , can benefit from our methodology.This paper presents work whose goal is to advance the field of Machine Learning and Math AI. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Results in Table 6 correspond to the algebra, measure, spatial, and pattern reasoning skills for the ablation experiments presented in the main text and a continuation to the results in Table 2 which included accuracy results for the counting, math, logic, and path skills.Furthermore, experimental results in Tables 7 and 8 from vlm-reasoners were obtained on a small subset of the SMART-101 dataset and can be reproduced using the end-to-end code github.com/smarter-vlm/smarter for the same hyperparameter and architectural choices. Table 7 gives results for the first four skills, counting, math, logic, and path, and Table 8 gives results for the remaining four skills, algebra, measure, spatial, and pattern.  
 
 Appendix:  


Decision: The paper evaluates the effectiveness of large vision-language models (VLMs) in solving complex multimodal reasoning tasks. The paper is novel in combining the frozen vision backbones with the new QF layer and in evaluation with the SMART task framework. The author could further improve the paper by providing more details and ablation on the QF layer and giving a more in-depth analysis.

Reviews: review: paper_summary: paper_summary: The paper explores the enhancement of vision-language models (VLMs) for complex multimodal reasoning tasks, particularly in mathematics and problem-solving. Using the SMART task framework, the authors assess VLMs across eight reasoning skills. They employ frozen vision backbones (DinoV2, SigLIP) and introduce a novel QF layer to improve model performance. The paper compares various training strategies and hyperparameters, showing accuracy gains (up to 48%) in skills like counting and math. The research addresses challenges related to computational resources and proposes efficient solutions for data handling. Reproducibility is ensured through detailed experiment tracking and standardized settings.
strength_weakness: strength_weakness: 1. The writing could be improved for clarity, as some sections are hard to follow and could benefit from more straightforward language. The reviewer can see that a lot of work went into the paper, but the key messages could be highlighted better to make the main contributions stand out.

2. The effectiveness of the proposed QF layer isn't clearly demonstrated; the paper lacks an ablation study to isolate the impact of this specific component. This makes it difficult to assess how much the QF layer contributes to the overall performance improvements. Including such an analysis would greatly strengthen the claims made about the QF layer's effectiveness.
questions: questions: 1. Could you provide more detailed results or an ablation study to isolate the impact of the QF layer on the overall performance of your model?
2. Are there specific examples or case studies that demonstrate the practical benefits of the proposed QF layer in real-world applications?
paper_summary: paper_summary: The paper explores the concept of "Smart Vision-Language Reasoners" that leverage multimodal learning to improve mathematical reasoning and problem-solving abilities. The authors investigate the ability of vision-language models to reason along eight axes: math, counting, path, measure, logic, spatial, and pattern. The proposed smarterVLM model achieves significant improvements (up to 48% gain in accuracy) over baselines on the SMART task, a multimodal algorithmic reasoning task.
strength_weakness: strength_weakness: -- While the paper presents a novel application of vision-language models to mathematical reasoning, the methodology itself is not particularly innovative. The use of transformers-like layers and frozen pretrained backbones is well-established in the literature .

-- The paper's evaluation is limited to a single benchmark (SMART task) and does not provide a comprehensive comparison with existing state-of-the-art methods. The authors should consider evaluating their approach on a broader range of tasks and comparing their results to existing methods.

-- The paper's results are presented in a concise manner, but lack a more in-depth analysis of the strengths and limitations of the approach. The authors could improve this by providing a more detailed breakdown of the results, including error analysis, ablation studies, and sensitivity analysis. This would provide a more comprehensive understanding of the approach's performance and limitations.
questions: questions: * Can the authors provide a more detailed explanation of the QF layer and how it differs from existing multimodal fusion techniques?
* How did the authors select the specific vision and language models used in their approach, and what were the criteria for choosing these models?
* Can the authors provide a more detailed description of the training process, including the hyperparameter tuning and optimization methods used?
* How do the authors plan to address the limitations of the SMART task, such as its limited scope and potential biases?
* How do the authors plan to address the challenges of deploying their approach in real-world settings, such as scalability and robustness?