Title: MuSLR: Multimodal Symbolic Logical Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Symbolic Reasoning

Multimodal Learning
Abstract: Multimodal symbolic logical reasoning, which aims to deduce new facts from multimodal input via formal logic, is critical in high-stakes applications such as autonomous driving and medical diagnosis, as its rigorous, deterministic reasoning helps prevent serious consequences. To evaluate such capabilities of current state-of-the-art vision language models (VLMs), we introduce the first benchmark MuSLR for multimodal symbolic logical reasoning grounded in formal logical rules. MuSLR comprises 1,093 instances across 7 domains, including 35 atomic symbolic logic and 976 logical combinations, with reasoning depths ranging from 2 to 9. We evaluate 7 state-of-the-art VLMs on MuSLR and find that they all struggle with multimodal symbolic reasoning, with the best model, GPT-4.1, achieving only 46.8%.
Thus, we propose LogiCAM, a modular framework that applies formal logical rules to multimodal inputs, boosting GPT-4.1’s Chain-of-Thought performance by 14.13%, and delivering even larger gains on complex logics such as first-order logic. We also conduct a comprehensive error analysis, showing that around 70% of failures stem from logical misalignment between modalities, offering key insights to guide future improvements.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MuSLR, the first multimodal, multi-domain benchmark for VLMs to perform symbolic reasoning using both textual and visual information. MuSLR requires VLMs to either 1) decide the truth value of an argument based on the given information or 2) select the argument that best corresponds to the input.\n\nIn addition, this paper introduces the LogiCAM framework, which is a multi-round CoT procedure that guides in-depth reasoning iteratively. In each step, LogiCAM first attends to the most valuable information at the current step, then decides whether to apply formal symbolic reasoning or commonsense heuristics. Next, it performs reasoning and finally checks if the derived results are sufficient to answer the question.\n\nEmpirical evaluations show that MuSLR is a difficult benchmark even for current SOTA VLMs (e.g., GPT-4.1), which achieve an average accuracy of less than 50%. The proposed LogiCAM method significantly boosts model accuracy by around 14.13% on average, demonstrating its effectiveness in iteratively guiding VLMs to solve complex multimodal reasoning problems.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1.  The proposed benchmark is highly relevant to VLM research and could serve as a valuable resource for VLM researchers seeking to improve the general reasoning abilities of VLMs, or for neural-symbolic researchers seeking benchmark tasks for multimodal models.\n2.  The proposed LogiCAM approach is intuitive and effective.\n3.  The writing is clear, and the paper contains a rich analysis of different aspects of the dataset and the CoT framework, making the paper informative for readers in related research areas.\n\n**Weaknesses**\n\n1.  The design of the proposed LogiCAM framework is largely heuristic. There is no clear ablation study to pinpoint the effectiveness of each module or to determine how it might be improved.\n2.  LogiCAM uses VLMs directly to perform reasoning. While this is a natural choice for heuristic commonsense reasoning, it is less reliable for formal symbolic reasoning. This approach leads to the \"Incorrect Application of Logical Rules\" errors reported in the paper. Given that a symbolic representation of facts arises naturally in the CoT process, the authors might consider incorporating a formal symbolic reasoner to aid this process and provide verifiable results.\n\nQUESTIONS:\n1.  It is interesting that model performance varies significantly with different families of logic, since they are all used to describe the same scenarios and, therefore, should not produce significantly different results. IMHO this point is not thoroughly discussed in the current manuscript.\n    -   **Is the entire dataset within the expressive reach of PL?** Specifically, when constructing the dataset, were there any instances that could not be represented with PL alone, thus requiring the introduction of FOL to capture all necessary information?\n        -   If the entire dataset can be expressed within PL, it would be understandable for FOL to compromise model performance, as it introduces many redundant concepts and axioms.\n    -   From my understanding, Figure 8 (Error Distribution) is summarized regardless of logic type. **Could the authors provide some quantitative, or at least qualitative, comments on the error distribution for different logic types? From your perspective, why might these differences occur?**\n    -   It is worth noting that after applying LogiCAM, the performance divergence between different logic types is greatly reduced. In some cases, FOL even achieves the best results. **Could the authors briefly explain why this happens and what kinds of errors LogiCAM helps to avoid?**\n2.  Following up on weakness 1, **is there any way to formally or informally analyze the effectiveness of each module in LogiCAM?** This would make it easier for other researchers to make future improvements and reference the work. I understand that a complete ablation study may not be feasible, so any comments on this would be greatly appreciated.\n3.  **What is the relationship between LogiCAM and other CoT frameworks in prior art, for example, for visual or textual reasoning?** Is it a straightforward application of existing methods, or is it tailored for multimodal tasks?\n\nLIMITATIONS:\nThe proposed MuSLR dataset is valuable; however, the choice of different logic types (PL, FOL, NM) seems somewhat arbitrary and may not be fully informative for downstream VLM evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 2, "DATE": "2025-06-09"}

comments: SUMMARY:
This paper introduces Multimodal Symbolic Logical Reasoning (MuSLR), a novel task requiring Vision Language Models (VLMs) to perform formal logical reasoning by integrating visual and textual inputs. The authors create MuSLR-Bench, a benchmark dataset with 1,093 instances across 7 domains featuring reasoning depths from 2-9 steps, and propose LogiCAM, a modular framework that improves GPT-4.1's performance by 14.13%.

STRENGTHS AND WEAKNESSES:
Strenghts: I think this paper is really great and deserves a publication. The proposed benchmark addresses a key challenge that most previous works just focused on symbolic reasoning in a single text modality. The proposed method is also reasonable to me. Furthermore, the code is published for reproducibility. 

Weaknesses: Although authors have done a lot of experiments, a deeper analysis of the results will better help the community.

QUESTIONS:
1. Is it a typo in line 24 for the word "text"? Should it be "visual" or anything else?
2. Seems that all the rules are first order logic rules right? Did you consider a higher-order case to represent a more natural real-world case?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

comments: SUMMARY:
This paper studies VLMs on multimodal symbolic logical reasoning tasks. First, the authors propose a new benchmark MuSLR for this task and evaluate multiple state-of-the-art VLMs on MuSLR, finding that they all struggle with multimodal symbolic reasoning. To this end, the authors then propose LogiCAM to improve the performance of existing VLMs on this task.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The contribution of this work is solid. After introducing of the background and motivation, the paper 1) raises a new task MuSLR with a clear definition, 2) introduces MuSLR-Bench as a benchmark for MuSLR task including detailed data statistics and construction process, and 3) proposes LogiCAM as a framework to adress challenging issues in MuSLR task with a clear flowchart and process description. Exhaustive experiments and in-depth analyses offer insights of this task.
2. The writing of this work is clear and easy to follow.
3. Part of the code and dataset is provided.

Weaknesses:
1. The MuSLR seems not entirely new. Some previous work like [1] also uses multimodal input and includes a symbolic reasoning process. The authors should better discuss about their differences with such prior studies.

[1] Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning

QUESTIONS:
What is the differences between the proposed MuSLR task and a task that receives multimodal input and includes a symbolic reasoning process?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a task requiring models to perform formal logical reasoning over combined visual and textual inputs. The authors create (1) MuSLR-Bench, a dataset of ~1k instances spanning 7 domains with ground-truth reasoning chains using propositional logic, first-order logic, and non-monotonic logic. They (2) evaluate 7 VLMs, finding that GPT-4.1 achieves 46.8% accuracy. They (3) propose LogiCAM, a prompt-driven framework that contains premise selection, reasoning type identification, and formal inference components, which improves GPT-4.1's performance by 14%.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. (Originality and Significance) The task genuinely important, and the dataset contribution is timely. As mentioned by the authors, prior work like LogicVista and VisuLogic don't test multimodal formal logical rules, but just do logical reasoning in visual contexts. The novelty here comes from the explicitness of symbolic rule annotation in the dataset (compared to implicit logical reasoning in many VQA datasets).  The benchmark will likely be valuable for the community.\n2. (Quality) The experimental design is sound, with multiple evaluation metrics. There are some actionable insights, particularly the finding that 70% of failures stem from cross-modal alignment issues. \n3. (Clarity) The paper contains helpful visualizations, and workflow diagrams to communicate the approach. \n\n**Weaknesses**\n1. (Quality) I think the paper has a glaring gap in comparing LogiCAM against techniques presented in the literature. LLMs+Solver paradigms already exist. For example : Logic-LM (cited) and many more papers (not cited) ; already combines LLMs with symbolic solvers, decomposing reasoning into symbolic steps. Multimodal symbolic reasoning has also been shown to work in papers like Proof-of-Thought (not cited). If you have a new framework to replace these, you should compare it against adapted versions of other frameworks previously proposed, and not just the base language models.\n2. The distinction between \"symbolic\" and \"heuristic\" reasoning in LogiCAM seems arbitrary and inadequately defined.\n3. Why certain logic types (FOL vs PL vs NM) have different difficulty levels needs better understanding. Lines 233-240 do not line up with standard understanding of complexity. In non-monotonic reasoning, you may invalidate previous conclusions, and the solver would need to handle belief revision and consistency maintenance, which would make it more complex than first order logic.\n4. There are a number of typos strewn across the paper, which could use thorough proof-reading. Here are some examples:\n\t1. Fig 1 : Step 4 \"pposing\" instead of \"opposing\"; Step 3: Minor Premise 1: \"woad\" instead of \"road\"\n\t2. Some references use \"CoRR, abs/XXXX\" while others use full venue names\n\t3. Table 1 Formatting InstructBlip row: \"33,33\" should be \"33.33\" (comma instead of period)\n\t4. Figure 9 : Context text says \"arounod\", should be \"around\".\n\nQUESTIONS:\n1. Dataset composition says 35+976 ≠ 1,093. May you please elaborate on the composition?\n2. Are you using a solver?\n3. Figure 7 shows accuracy performance degradation at higher depths. Why?\n4. Could you please elaborate on Weakness 3.\n5. I am willing to revise my scores upwards if you can address weakness 1.\n6. Could you please elaborate on your dataset construction methodology. I find this very interesting. More specifically, I want to understand how you deal with the following\n\t1. How do you do semantic disambiguation in the autoformalization process.\n\t2. Is this quantifier free? How do you deal with quantifiers, which may add significant complications to autoformalization.\n\t3. How do you formalize inherently imprecise natural language statements? How do you deal with implicit meaning in language queries? Did you face any of these issues\n\t4. How do you go about determining the necessary entities, properties, and relations for the problem's domain, defining the universe of discourse. How do you go about choosing the appropriate level of abstraction? How do you frame invariants?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}