MuSLR: Multimodal Symbolic Logical Reasoning
Authors: Jundong Xu
Hao Fei
Yuhui Zhang
Liangming Pan
Qijun Huang
Qian Liu
Preslav Nakov
Min-Yen Kan
William Yang Wang
Mong-Li Lee
Wynne Hsu
Venue: NeurIPS 2025 poster ()
Keywords: Symbolic Reasoning
Multimodal Learning
Primary area: applications

Abstract: Multimodal symbolic logical reasoning, which aims to deduce new facts from multimodal input via formal logic, is critical in high-stakes applications such as autonomous driving and medical diagnosis, as its rigorous, deterministic reasoning helps prevent serious consequences. To evaluate such capabilities of current state-of-the-art vision language models (VLMs), we introduce the first benchmark MuSLR for multimodal symbolic logical reasoning grounded in formal logical rules. MuSLR comprises 1,093 instances across 7 domains, including 35 atomic symbolic logic and 976 logical combinations, with reasoning depths ranging from 2 to 9. We evaluate 7 state-of-the-art VLMs on MuSLR and find that they all struggle with multimodal symbolic reasoning, with the best model, GPT-4.1, achieving only 46.8%.
Thus, we propose LogiCAM, a modular framework that applies formal logical rules to multimodal inputs, boosting GPT-4.1’s Chain-of-Thought performance by 14.13%, and delivering even larger gains on complex logics such as first-order logic. We also conduct a comprehensive error analysis, showing that around 70% of failures stem from logical misalignment between modalities, offering key insights to guide future improvements.

MuSLR: Multimodal Symbolic Logical Reasoning
Jundong Xu', Hao Fei'; Yuhui Zhang,, Liangming Pan', Qijun Huang', Qian Liu", Preslav Nakov", Min-Yen Yen Kan', William Yang Wang', Mong-Li Lee', Wynne Hsu
National University of Singapore, Stanford University, Peking University. UniMelb, University of Auckland, MBZUAL," University of California, Santa Barbara
jundong xxwu. nus edu, haofei37 edu. ag, Tuhuiz@cs.stan liangmingpan@pku. eduu edu. cn, qijunhuanggstuddnt unimelb edu au, 1iu. gian@auckland. ac. nz, preslav nakovvmbzuai ac. ae, knmnyn@nus edu. sg, william@cs. ucsb. edu, dcsleeml @nus. edu. sg, dcshsuwdnus edu. sg
edu,
Abstract
Multimodal symbolic logical reasoning, which aims to deduce new facts from multimodal input via formal logic, is critical in high-stakes applications such as autonomous driving and medical diagnosis, its rigorous, deterministic reasoning helps prevent serious consequences. To eval- uate such capabilities of current state-of-the-un vision language models (VLMs), we introduce MuSLR, the first multimodal symbolic logical reasoning grounded in formal logical rules. We curate benchmark dataset for MuSLR comprising 1,093 instances across 7 domains, including 35 atomic symbolic logic and 976 logical combinations, with reasoning depths ranging from 2 to 9. We evaluate 7 statc-of-the-art VLMs on our benchmark and find that they all struggle with multimodal symbolic reasoning, with the best model, GPT-4.1, 1, achieving only 46.8%. Thus, we propose LogiCAM, a modular framework that applies formal logical rules to multimodal inputs, boosting GPT-4.1's Chain-of Stain-of--Thuglhe Thought performance by 14. (4.13% and delivering even larger gains on complex logies such as first-order logic. We also conduct a comprehensive error analysis, showing that around 70% of failures stem from logical misalignment between modalities, offering key insights to guide future improvements. All data and code are publicly available at ht htt:://mmmmmmmmmm/ gi github.io/MuSLL..
Introduction
Recent progress has extensively highlighted the pivotal role of reasoning capabilities in enhancing the generality and robustness of large language models (LLMs) 9 12 21) Yet, achieving human- level intelligence demands more than commonsense heuristic thinking. In particular, symbolic logical reasoning, grounded formal logic such as first-order logic, offers a rigorous, precise, and verifiable paradigm essential for high-stakes scenarios where reasoning errors have critical consequences. Although previous works have shown that LLMs can handle symbolic reasoning in purely textual contexts [22 36 37), these capabilities remain limited to unimodal inputs, i.e., text. However, many woal-world domains, such as autonomous driving, healthcare, law, and finance, demand reasoning that integrates multiple modalities, particularly combining visual and textual information, to support accurate and reliable conclusions. Consider an autonomous driving system that observes traffic sign (from camera image) indicating "Road Closed Ahead", given the traffic rule "Only if the road ahead is open ( (B), the vehicles may proceed straight ( (A)." From the image, the system detects that the road is in fact closed (-B), and must infer that continuing straight is not permitted (-A), forming a formal logical reasoning (Modus Tollens; (A + B) A -B + -A) to avoid traffic accidents. Despite the significance of such multimodal symbolic reasoning, no standard definition or benchmark currently exists for this capability.
To fill this gap. we introduce Multimodal Symbolic Logical Reasoning (MuSLR), a novel task that challenges VLMs to perform symbolic reasoning over combined visual and textual inputs. Figure]] illustrates the MuSLR task with the above example. We define MuSLR under two task formats: Truth
Corresponding author: Hao Fei
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

INPUT
Image
Text
Cunnett (Bale))
If drviing smsight in allwed thes no canicnnttutte
De vinal wittis parking dgy the cinsa dhank nilee es ensight lie
Uo trowii rad with anclane drunton th devvr damg: arive wasl vchatia changing lates
Queriios
Soving
GOLD SYMBOLIC REASONING
STEP
STEP
Mbjjer thy ebt ticlec aubraa
Vijer Princiee divetest to tri kiny "Jood lis deeccio
allned LAURSE LAUREE thes sorea commrniie
partiag en otter [2 -20
nokillin lir an ming aer lekke
Mieer Poveate
Siiter Proniied li priciis inib
Eassin las pas befles chat (TE
Supprting CSB
perting angs
Vinue Prommteds The deviet
Thrriter dhere should
chaugr lare sow
Craalasio: Dokkay
Coedisires The
tor mnigg ane chenggg
Systhoie rukes Modas Tale -8A0BB-0
byydooie mado deyaaitee Gfagleng (N
kyraballe rules Makk Frain
Figure 1: An example of a depth-4 propositional logic task, requiring the VLMs to apply formal symbolic logic rules and integrate multimodalities to reach the conclusion.
Evaluation and Multiple Choice, where given image 1, context T, the model must apply symbolic logical reasoning to identify the correct answer. To enable systematic evaluation, we then propose ResLR, high-quality benchmark dataset specifically designed to assess the symbolic reasoning abilities of statc-of-the-arr VLMs. Drawing from authentic web-sourced scenarios where visual and textual content naturally co-oceur, we annotate each instance with formal logical rules (e.g., modus ponens) and conduct rigorous quality checks to ensure correctness and logical validity. HuSLR-BBend' comprises 1,093 instances spanning 7 domains, including 35 atomic symbolic logic and 976 complex logical compositions, with reasoning depths ranging from 2 to 9 to reflect diverse difficulty levels. In pilot study, we evaluate seven leading open- and closed-weight VLMs of varying sizes on MuSLR-Benche revealing that even top models struggle substantially with multimodal symbolic logic inference.
To establish strong baseline for MuSLR, we further propose LogiCAM (Logical reasoning with Commonsense Augmentation in Multioddalities) which decomposes multimodal symbolic reasoning into modular steps through Chain-of of-Thought (CoT)) mechanism (cf. Figure First, the Premise Selector is designed address the difficulty of multimodal fusion. We next devise Reasoner module to integrate multimodal evidence and apply symbolic reasoning by approximating formal logical rules, enabling rigorous and systematic deduction to meet the core challenge of MuSLR. Then, the Reasoning Type Identifies er is designed to address the issue of incomplete information in MuSLR, where heuristics act as supplementary resources to complement symbolic rules when they are insufficient to reach the conclusion. Extensive experiments show that LogiCAM improves achieving even greater gains on complex
GPT-4.1's CoT performance by 14.13% on first-order logic tasks. Further analysis reveals that reasoning performance deteriorates sharply as logical complexity and chain depth increase, highlighting key limitations of current popular VLMs.
In summary, contributions are fourfold:
We introduce MuSLR, a pioneering task targeting multimodal symbolic logical reasoning, ad- dressing critical gap in real-world AI reasoning.
We curate MuSLR-Bench a high-quality dataset comprising 1,093 instances with diverse logical structures and depths, serving critical foundation for this topic.
We develop LogiCAM, a strong CoT-based baseline method that decomposes complex reasoning into more manageable and trackable modules.
Through extensive experiments and analyses, we pinpoint where and why current VLMs struggle with MuSLR, offering insights for future investigation of this area.
2 Related Work
Textual Symbolic Logic Reasoning and Benchmarks. Existing benchmarks for symbolic logical reasoning have primarily focused on purely textual settings under formal logic rules. For instance, FOLIO 7 is a human-annotated dataset for complex natural language reasoning equipped with first- order logic annotations to ensure the logical consistency of premises and conclusions. ProofWriter provides small English rulebases of facts and rules with associated questions, requiring models
28

---

Image Processing
Symbolic Rules Processing
STEP Collect dots
STEP
Your dra
STEP S: Select symbaic riler
STEP Connntme queetion
STEP 7:0uest control
Deterdaa whettc [ax)
Before
Jever sieel Tere chinge
3I nc: The ofer thodd change dine
After
Cam Lakereu
Real world Grounding
Quality Control
Figure 2: Pipeline of MuSLR data construction. We begin by collecting multimodal data and symbolic rules. These rules are then combined to form reasoning chains, which are grounded in real-world contexts to generate questions and answers, followed by a strict quality check.
to prove or refute statements (or answer "unknown" when proof is impossible) via multi-step natural language proofs. Likewise, Multi-LLgiiFval [24] evaluates multi-step logical reasoning across propositional. first-order, and even non-monotonic logic types, encompassing over 30 inference rules and various depths to test LLMs' deductive abilities. We further acknowledge numerous additional related works, such as ProntoQA (27), LogicBench [23]. and RuleArena [41]. However, these benchmarks assume fully specified, idealized inputs in single modality (text) and do not incorporate visual information, limiting their direct applicability to real-world scenarios.
Multimodal Reasoning and Benchmarks. In parallel, several benchmarks have introduced access- ing reasoning in vision and language 33). Logie Vista [35 evaluates VLMs' logical reasoning in visual contexts, with 448 annotated multiple-choice questions spanning a spectrum of logical reason- ing tasks and capabilities. Similarly, VisuLogic [38 targets vision-centric reasoning by constructing tasks that require robust visual logic without relying on textual descriptions or shortcuts. Meanwhile, broader vision-language benchmarks emphasize contextual reasoning rather than formal logic: for example, MMMU (39] offers college-level multimodal questions across six disciplines (e.g., charts, maps, chemical structures), testing demain-expert reasoning. Math Vista [8] targets compositional mathematical inference in visual scenarios. However, none of these multimodal benchmarks explicitly test the application of formal logical rules (c.g. Modus Ponens or De Morgan's 's Law) grounded in both visual and textual input. MuSLR addresses this gap by requiring explicit symbolic logical deduction from joint visual-textual inputs, integrating formal logic rules into multimodal understanding.
Neuro-Symbolie Reasoning Method. Many prior works adopt symbolic prover in the reasoning pipeline to achieve rigorous and reliable reasoning. Typically, an LLM is used to formalize natural language into symbolic form, after which theorem prover is employed to solve it [1] 19 22 26
34]. However, theorem provers only accept text input. In multimodal scenarios, this requires first converting visual or multimodal information into text, process that inevitably leads to information loss and thus limits adaptability. In contrast, our LogiCAM framework is designed to approximate symbolic reasoning using lision-language model (VLM), which has direct access multimodal information without relying on lossy translation.
3
Task Definition
The proposed tasks require models to integrate information from both an image T and text passage to perform reasoning, ensuring that neither modality alone is sufficient for correct inference. The tasks explicitly emphasize multimodal reasoning, where the fusion of visual and textual context is essential for deriving accurate and consistent conclusions.
Question. Given an image I, a text passage T. and an argument A, the model must determine the truth value of the argument based on the combined information from I and T. Specifically, the model outputs the truth value Truth(A) e (True, False, Unknown] } and generates sequence of reasoning steps I = [R, R2..... Rn where each RG represents an individual step that contributes to the final decision. Formally, the input is a triplet (1, T, A), and the output consists of Truth A) and R.
Task-I: Truth Evaluation (True/False/Unkna)
3

---

Statisties
Numbers
Each Death
Total instances
1093
NM
159
Total sources
3M
Domain (N)
34
Symbolic logic (f)
Atomic symbotic zules (#)
35
Grevrill
Heabkare
Symbolic rule combination (#)
976
11
44%
Instances Earh Source
200
Min rcasoning depth
2
7%
Max reasoning depth
Min context length
Max context length
Avg. longth
554.9
me m70
Teats opont
Figure 3: Dataset Statistics. The left table presents general dataset statistics. The middle pie chart illustrates the distribution across domains and symbolic logic. The right bar charts display the number of instances by reasoning depth and data source.
TIsk-II: Multiple Choice Question. Given an image 7, text passage T, and candidate arguments {A1, A2, A3, A4 J. the model must select the argument that best matches the image and text, denoted as BestArgument(1, ,') E {Ai, A2, A3, A4 Additionally, the model must provide detailed reason- ing steps R {Ri, .2...., .,, R. where each R; details a step in the reasoning process. Formally, the input a triplet (I,T, {Ai, A2, A3, A4)), ), and the output consists of BestArgument(1,22 and R.
4 MuSLR-Bench: A Benchmark for Multimodal Symbolic Logical Reasoning
Dataset Construction. We collect images from various sources such as COCO [14), Flickr30k 25]. nocaps (. Mimic [O), RVL_CDIP R ScienceQA (17), and manually collected Traffic Reports. Visual details for each image are extracted using GPT-40, ensuring diverse and fine-grained descriptions. We carefully select non-trivial logical inference rules, such as Modus Ponens and Hypothetical Syllogism, drawn from propositional logic (PL), first-order logic (FOL), and non- monotonic logic (NM). These rules then form meaningful but abstract reasoning chains through logical combinations. The abstract chains are grounded in real-world contexts by leveraging extracted visual features and relevant retrieved text from sources like healthcare, traffic reports, and Wikipedia. Questions and answers are then generated based on these instantiated reasoning chains, using rule- based substitution.
To ensure the quality and relevance of the dataset, both automatic and manual quality control procedures are employed. Automatic checks include assessing lexical similarity and commonsense plausibility,, while human annotators verify the accuracy of visual details and the real-world relevance of the generated context. Instances that fail these checks are filtered out, ensuring high-quality, logically sound, and contextually relevant dataset. Further details on the data construction and quality control processes are provided in the Appendix and C respectively.
4.1 Dataset Highlights
MuSLR consists of 1093 instances, where each instance includes multimodal context (image and associated text), a ground-truit logical reasoning chain, and corresponding question-answer pairs. The dataset is constructed to support both detailed symbolic logical reasoning analysis and challenging multimodal reasoning tasks. Below, we summarize the key features of the dataset:
Ground-Truth Reasoning Steps. Each instance is equipped with explicit, step-by-step ground- truth reasoning chain, enabling detailed analysis and training of models for symbolic logical reasoning.
Multi-Scenario Coverage. The dataset spans a wide range of domains, including science, enter- tainment, sports, social issues, general knowledge, traffic, healthcare, and finance. The distribution across these scenarios is illustrated in the pie chart in Figure
Diverse Symbolic Reasoning Types. MuSLR contains diverse symbolic logic: propositional logic (PL), first-order logic (FOL), and non-monotonie logic (NM), ensuring broad logical coverage.
Multimodality, To the best of our knowledge, this is the first dataset that combines both image and text modalities for symbolic logical reasoning tasks grounded in formal logical rules.
Diverse Difficulty Levels. The reasoning chains vary in depth from to 9 steps, offering a broad spectrum of difficulty levels and supporting evaluation across simple and complex reasoning scenarios.
Multiple Question Types. The dataset supports multiple question formats, including Truth Evalua- tion and Multiple-Chie Chhice questions, allowing for diverse model evaluation protocols.

---

STEP 1: Select critical premise
STEP 2: Identify reasoring hype
STEP 3: Perform recsoning
STEP 4:
Check for completion
Contrie
Syndere
Racsoving Pype
Cotp
dee paatin
Fros Thot for seciile ONL
Herrate
Add
Final answer
Figure 4: LogiCAM Workflow. The figure illustrates single iteration; the complete multi cilli-iteration reasoning process is detailed in Section
4.2 Challenge
MuSLR presents five key challenges for developing robust multimodal symbolic reasoning models: Integrate Multimodality. Can the model extract and integrate critical visual and textual context to construct valid reasoning chains? (See Section 7.5)
Step-by-Step Symbolic Reasoning Tracability. Can the model produce interpretable, verifiable, step-by-step reasoning processes in valid logic? (See Section (.2)
Blend Heuristies for Symbolic Reasoning. Can the model apply heuristic reasoning when symbolic logic is insufficient?
Diverse Symbolic Logic. Can the model handle various forms of symbolic logic (PL. FOL, and NM)? (See Section 7.1
Reasoning Depth Handling, Can the model reason over different depths, maintaining consistency in longer chains? (See Section 7.3)
Addressing these challenges requires models to integrate multimodal perception and systematic logical reasoning, thereby providing a solid foundation for advancing multimodal reasoning systems.
5 LogiCAM: A Modular MUSLR Framework
We propose a modular framework, LogiCAM (Logical reasoning with Commonsense Augmentation with Multimodality), which consists of three modules based GPT-4. as illustrated in Figure 4 Each module is designed to address a specific challenge posed by MuSLR. The modules work logether solve different problem components, which include: (I) the Premise Selector, (2) the Reasoning Type Identifier, and (3) the Reasoner module. Below, we explain how each module addresses its challenge and contributes to the reasoning chain.
Select Critical Multimodal Premises. The Premise Selector is designed to address the multimodal- ities integration challenge, which involves the need to process both visual and textual data to extract critical premises. Given an image and textual information T containing context T and question Q. this module directs the VL.M first select the most relevant symbolic rules R2 E T. The VL.M will then analyze the symbolic logic R, to determine which part is relevant to the image and extract the corresponding visual information V, In this way, the system ensures that only the most critical visual and textual details are extracted, avoiding unnecessary complexity and noise from abundant data. The symbolic rule R, and visual details V, will be combined and denoted leritical-
Identify Reasoning Type. The Reasoning Type Identifier addresses the blend of heuristics and symbolic, which involves determining whether symbolic reasoning or heuristics should be applied during each reasoning iteration. The core challenge deciding when symbolic logic sufficient and when heuristics should be used complement symbolic reasoning. To solve this, the Reasoning Type Identifier analyzes the selected premises Lenitiall and determines whether formal logical rules can be applied. If so, prioritize it. Otherwise, heuristics and commonsense reasoning are employed to com- pensate for the limitations of purely symbolic reasoning. In this way, the model maximizes the rigor and soundness of the reasoning by prioritizing symbolic reasoning while maintaining flexibility to supplement additional knowledge through commonsense-drinn heuristics when symbolic reasoning alone is insufficient.
Perform Reasoning. The Reasoner is central to addressing symbolic reasoning tracability, which uses VIM to approximate formal logical rules when symbolic reasoning is required. Depending on the outcome of the Reasoning Type Identifier, the Reasoner either applies symbolic reasoning or uses heuristic commonsense to complete the reasoning process. If symbolic reasoning is selected,
5

---

Model
Symbol Healtheare Traffic
Sports
Ent.
Social
Science
Finance
General
Three-shots CoT Open-Weight Weight VLMs
PL
50.00
33.33
33.33
42.67
36.49
48.54
54.17
46.31
Qwen
FOL
0.00
42.86
50.00
40.00
66.67
16.67
22.22
25.00
NM
43.18
25.93
43.75
35.42
23.26
43.75
54.24
37.50
PL
20.45
30.95
37.18
22.97
34.95
27.08
43.02
Llava
FOL
0.00
57.14
50.00
20.00
44.44
66.67
55.56
50.00
NM
31.82
37.04
45.31
43.75
39.53
47.06
25.42
45.31
PL 57.95
42.86
37.97
44,00
37.84
46.60
51.04
50.00
IntervVL
FOL
50,00
42.86
50.00
20.00
66.67
50,00
22.22
50.00
NM
38.64
29.63
46.88
35.42
46.51
49.02
45.76
43.08
PL 42,05
33,33
39.2
26.67
36.49
29.13
40.62
25.58
InstructBlip
FOL
50.00
28.57
25.00
40.00
55.56
16.67
22.22
25.00
NM
52.27
40.74
53.12
31.25
44.19
35.29
2.34
30.77
Three-shots CoT Closed- Weight VLMs
PL
44.32
26.19
24.36
26.67
28.38
35.92
36.46
34,88
Claude
FOL
50.00
14.29
50.00
20.00
55.56
0.00
44.44
75.00
NM
29.55
37.04
32.81
29.17
30.23
43.14
38.98
31.25
PL
45.45
40.48
33.33
37.50
34.72
37.00
28.99
43,90
GPT-4o
FOL
25.00
37.50
33.33
33,33
NM
52.27
48.15
35.48
50.00
52.38
45.10
41.46
32.81
PL
54.55
50.00
44.30
41.33
33.78
43.69
45.83
51.16
GPT-4.1
FOL
0.00
14.29
50.00
20.00
44.44
16.67
33.33
0.00
NM
47.73
59.26
46.88
50.00
53.49
56.86
61.02
40,62
PL
63.64
61.90
58.23
64,00
56.76
57.28
53,68
67.44
(+9.09)
(+11.90)
(+13.93)
(+22.67)
(+22.98)
(+13.59)
(+7.85)
(+16.28)
LogiCAM
FOL
50.00
60,42
50.00
60.00
44.44
40.00
75.00
75.00
(+50.00)
(+46.13)
(+0.00)
(+40.00)
(+0.00)
(+23.33)
(+41.67)
(+75.00)
58.23
74,42
64.71
55.38
(+15.91)
(+7.41)
(+11.35)
(+10.42)
(+20.93)
(+7.85)
(+13.12)
(+14.76)
Table 1: Main Results. Blue indicates the best open-weight VLM, and Red indicates the best cloged-weight VLM. The (red brackets) indicate improvement over the base model.
the module applies formal logical rules to the premises Teritical and derives conclusion C based on syllogism, which draws result from the major and minor premises. This reasoning process ensures that conclusions are drawn according sound logical principles. heuristics are selected, the module uses commonsense reasoning to bridge gaps left by symbolic logic. This design makes sure that the model can perform symbolic reasoning grounded in logical principle, while relax this restriction when heuristics are required. A full example can be found in the Figure 9
Check for Completion. Finally, the system checks whether the conclusion C' is sufficient answer the question Q. If so, it concludes the final answer. Otherwise, the system appends the conclusion C to the context T, resulting in T" T' U C, and starts over the whole reasoning iteration.
Experiments
6.1 Settings
Evaluation. We evaluate models based on two dimensions: direct answer match and reasoning accuracy. Direct answer match measures the correctness of the final answer, while reasoning accuracy evaluates the quality of the step-by-step reasoning. Reasoning accuracy is computed by comparing model-generatee steps with ground- truth steps using ROUGE-L [3] and BertScore-Fi [40]. We also assess ROSCOE (6), which measures logical coherence, factual grounding. and informativeness step
by step. More details are Section 7.2
Baseline. For benchmarking, we consider multiple state-of-the-art models. For open-source models, we benchmark Qwen2. 5- Instruct [3, Llava-1- 1.5va-1.5--7 5B [15] InternVL3-B3 [2], and
Instructblip-Vicun - (4). For closed-source models, we evaluate GPT-4o [20]. GPT-4.
2I and Claude-3. 7 7-Soneet These models are chosen to represent the current SoTA in multimodal reasoning.
Settings. To ensure reproducibility. all models are evaluated under standardized settings. We adopt three-shot Chain-of-Thoon (CoT) (32 prompting setup. For language model sampling, the temperature is set to 0.0 minimize randomness and encourage deterministic outputs.

---

Rouge-L
BertScore-F1
ROSCOE
Average
Qwen
645
Llava
0.8
Claude
0.59
0.6
GPT-4.1
0.4
LogiCAM
Avg
0.12
FOL
NM
Qwen
Llava
Claude
GPT-4o
GPT-4.1
LogiCAM
Figure 5: Accuracy of symbolic logic
Figure 6: Comparison of models' reasoning tracability
6.2 Main Results and Observations
The main results are presented in Table We have the following observations:
Ceised-weight models generally outperform, but open-weight models can rival or surpass them. GPT-4. leads with 66,84, followed closely by InternVL at 45.20%, the top open-weight model. Qwen (41.63%) and GPT-4o (38.93%) follow in the second tier, with InstructBLIP (35.59%), Llava (3%,, and Claude (33.49%) at the lower end. The performance gap between top and bottom is jsjustt 13.35% These results show that while closed closed-weight models typically excel, well-designed open-weight models can sometimes outperform proprictary models
LogiCAM enhances CoT and achieves the highest overall performance, with especially strong gains in complex symbolic logic. Integrating LogiCAM into GPT-4.1 results in a substantial performance boost, increasing the average accuracy by 14.13%. When examined by logic type, the improvements are consistent yet differ in scale: FOL accuracy increases by 48.93 %. PL by 31.93 %, and NM by 26. 16.17% %. This pattern indicates that the advantage of LogiCAM grows with the complexity of the logic type: the largest relative improvement is observed in FOL., the most structurally demanding form, followed by PL., and then NM, which is more aligned with intuitive human reasoning and less dependent on rigid symbolic structure. These results suggest that LogiCAM not only strengthens general symbolic reasoning but especially effective in complex logical operations.
7 Analysis and Discussion
We conduct additional experiments and perform detailed analysis to gain deeper insights into the multimodal symbolic reasoning capabilities of current VLMs.
7.1 Effects on Different Types of Symbolic Logic
we evaluate the accuracy of each symbolic logic and found that Model accuracy decreases with rising symbolic complexity: VLMMs perform best with non-monotonic reasoning, less well with propositional logic, and struggle most with first-order logic. First-order logic has the lowest average accuracy at 37.04%, due to its strictest formalism and need for precise variable binding and quantifier tracking. Propositional logic fares better with 42.77% as its simpler structure eases syntactic constraints. Non-monotonie reasoning performs best at 46.09%, due to its closer alignment with human cognition and requiring less rigid symbolic manipulation. Overall, as symbolic complexity increases, model accuracy declines, highlighting the challenges of fine- ine-grained logical abstraction in current VLMs.
In Figure [
7.2 Tracability of Reasoning Step
As shown in Figure 6, LogiCAM leads in both ROUGE-L (0.170) and BertScore (0.835), with the highest overall mean (0.590), indicating its outputs closely match human phrasing and meaning. Claude scores highest on ROSCOE (0.784), reflecting strong logical consistency but performs poorly on ROUGE-L (0.084). GPT-4.1 balances phrasing and semantics (ROUGE-L = 0. 166%, BertScore +.833%% but shows moderate stepwise justification (ROSCOE 0.725%), suggesting occasional logical gaps. Llava and GPT-4o have similar profiles (Average 0.570%), demonstrating that strong semantic similarity (0.822%) doesn't guarantee superior inference quality (ROSCOF 0.776%%)
Surface-level or semantic objectives alone don't ensure logical coherence. Future work should include logic-focused training goals. Pearson's correlation analysis reveals a weak correlation between ROUGE-L and ROSCOE (r 0.25) but moderate correlation between BertScore and ROSCOE = 0.65), suggesting that surface-level metrics do little for logical coherence, while semantically rich training helps more. Claude's high ROSCOE but low ROUGE and BertScore
7

---

Ours
GPT-4.1
InternVI.
Accuracy (%)
13%
5%5
9%
139
4%
5%
67%
61%
Depth
Logical Rules
Commonsense Misgen.
Visual Details Object Recog.
Owen
Claude
GPT-4.1
Heuristic Shorts. Figure 7: Panel A reports accuracy across different depths, while Panel illustrates the error distribution across models.
Alignment Errors
LogiCAM
highlights that reasoning-focuus focussed objectives improve logical rigor, often at the cost of natural phrasing. This suggests that optimizing for surface or semantic metries alone isn't enough to improve logical coherence, and future research should target the quality symbolic logic.
7.3 Depth Analysis
As shown in Figure 7A, All models exhibit clear decline in performance as the symbolic reasoning depth inereases, confirming the benchmark's effectiveness in exposing growing complexity of multimodal logical tasks. GPT-4.1 emerges the strongest baseline, with the highest accuracy after LogiCAM and a moderate 16% drop from 2-3 to 8-9 steps. However, still struggles at greater depths, revealing limits in complex multi-hop reasoning. GPT-4o and Llava maintain stable performance with minor 3-4% drops, but their overall accuracy is much lower, indicating a trade-off between robustness and reasoning capacity. In contrast, Claude suffers sharp 20% decline, highlighting poor generalization on longer symbolic chains.
In contrast, LogiCAM not only delivers superior average performance but also scales more effectively when reasoning chains grow, demonstrates the strongest overall performance and robustness, consistently outperforming other models across all reasoning depths. It achieves 71.91 % accuracy at the shallowest level and maintains a solid 54.61 % even at the deepest. Notably, surpasses the strongest baseline GPT-4.1 by 13% at depths 8-9, highlighting substantial advantage in handling extended reasoning chains. While LogiCAM exhibits a larger absolute drop across depths, its high performance all levels indicates strong generalization to both moderate and complex symbolic reasoning tasks. This drop. however, suggests there is still room to improve long-chain reasoning robustness.
7.4 Ablation Study
We conduct an ablation study, which demonstrates that each module is indispensable, as shown in Figure Removing the symbolic reasoning module produces the largest performance reduction (5.14%), underscoring the importance of adhering to formal logical rules. Omitting heuristic reasoning yields 3.45% degradation, indicating that heuristics serve as an effective complement when strict logical rules are inapplicable. Disabling premise selection results in a 3.27% drop, reflecting its crucial role identifying critical information and simplifying subsequent inference. Collectively, these findings highlight that each module plays critical and non-redundant role, underscoring the necessity the full design for achieving strong overall performance.
7.5 Error Analysis
We conduct thorough error analysis by randomly selecting domain- and symbol-balanced subset of 100 examples for each model. We identify six major error types: incorrect application logical rules, failure to supplement with heuristic commonsense knowledge, overlooking critical visual details, logical misalignment between visual and textual context, improper reliance on heuristic shortcuts where symbolic reasoning is required, and misperception of objects in the image. Details of each error type are discussed in Appendix
Error distribution across different models. As shown in Figure BB failures to logically align and integrate visual with textual premises overwhelmingly dominate (67% for LogiCAM, 74% for GPT-4. 1, and 63% for Intern VI.), demonstrating that cross-movt modal grounding remains the principal hurdle. Looking specifically at each model:

---

A
B
PL
POL
NM
70%
15
2%
10%
17%
56.20%
57.89%
0000%%
IP
30%
LopCCM
tiio symbolic
wio Heurintic
Premice selection-4o
Logical Rules Aligoment Erross
Communneeteo Misgen.
Visual Details Object Recng.
Heuristic Shorts.
Figure 8: Panel A shows the ablation study results bar plots, while Panel B presents pie charts illustrating the error distribution across different logical types.
LogiCAM is designed to blend symbolic deduction with heuristic inference; it exhibits a high rate of heuristic shortcuts (13%), indicating difficulty in discerning when to apply formal logic versus commonsense reasoning.
GPT-4.1 shows minimal reliance on heuristics (3%) and almost no failures supplement with commonsense (1)). yet overlooks visual details in 13% of cases and misapplies formal logical rules 9% of the time. The latter aligns with known Chain-of-Thought behavior, where outputs can seem plausible but contain subtle logical errors 37
InternVI. suffers the highest proportion of pure perception errors (18%), reflecting weaker object recognition than GPT-4.1, and relies heuristic shortcuts in 11% of cases.
Notably, models suffer major logical misalignment between modalities and visual oversight errors, underscoring critical need for advances in vasion-language fusion. Future work should focus on improving cross moddll fusion and incorporating logic-based training objectives, enabling more accurate symbolic reasoning across modalities.
Error distribution across different logical types. We further analyze the error by logical types as shown in Fig and have the following findings:
Consistent Alignment Issues Across Logic Types. A primary source of failure in PL, FOL, and NM arises from logical misalignment between text and image, with this problem being particularly severe NM (79%) and PL (68%). This aligns with our broader finding that mapping formal logical structures onto multimodal contexts remains fundamental challenge for current vision-language models (VLMs).
FOL is Most Prone to Overlooking and Logical Errors. Overlooking errors most frequent FOI. (16%), where models often miss details in multi-entity, nested, or quantified reasoning. Logical rule errors are also highest (17%), reflecting the symbolic complexity of quantifier binding, variable tracking, and relational reasoning compared to PL or NM.
PL's Dependence on Symbolic Alignment. Although PL avoids many deep logical errors, its performance is highly dependent on accurate logical text-image alignment, as reflected in the 68% rate of alignment errors. Once alignment achieved, the relatively simple structure PL facilitates more reliable rule application by the models.
NM's High Alignment Difficulty but Low Logical Error Rates. Despite exhibiting the highest rate of alignment errors (79%). NM shows the lowest incidence of incorrect logical rule application (5%) and commonsense supplementation errors (0%). This pattern suggests that once alignment successfully established, NM reasoning is more consistent with the model's intuitive understanding or default interpretive patterns, which may partly explain its comparatively strong raw performance.
7.6 Case Study
To illustrate the limitations of existing VLMs and how LogiCAM addresses them, we present case study comparing the reasoning of GPT-4.1 (with CoT prompting) and LogiCAM in Figure
GPT-4.1's CoT reasoning exhibits form of "nearsightedness".. As the reasoning chain grows longer, gradually loses the thread that connects image cues to abstract premises, defaulting in- stead to surface-level judgments (e.g., "I can't I see a predator, so unknown"). Without a system- atic Premise Selection process, fails to ground observations like "on grass" relevant tex- tual logical rules (c.g.. (not the grass (A) or searching for food (B)) A (on the grass (-A)) +

---

Image
Text
Centri plaite one he hanc eut bh gras, tound conding te bed lesed pooburring rag sumoesinng. ten Rack in mnal robyts
propetig
good
cauigg arunad, and this lound Bale nans, the Durlyy lrvag anead this loord desccs prodaae la vida siaayy then cnntist
Quesice Dioemiee lobsk vikenve fotee ceed detert producie
GPT4.1 CoT
Ledding dede leaai suc iand apae preo gread Acconding iy unternation uris leaat desccu prcdade tees cemmooemuu: gand Die bubdig tith picar ye aeteeted sdd vistbe 1.....
kunda suttaaas deger desuce Alao oeee saspeet yoa prae d poo caneldd bee aailable mmaaaa basi provatt Thodtrr,
fubos ibaud vhus prodaces
baac utingl Baramaki
oringh kaury
LogiCAM
Select oritasi intormat
ressoning
Perform reseaning
Comp aftor Check
STEP 1
Thes love oe the (ive the incepe
eaaraing dhaditts
Thaj flliiios
liveed inverringg
Newe A kandi en Ri grem (1A) tivn
Cencitace The loor rancinng
Chock hanc rescting
diweriigg
STEP 2
The heed sound liwd.
Homsibi Commusseee
carmediags sgs. solut sliga with image cencuneecis
Conclakas aaa Loud obsariag
the loen covurrnn sereandigg in koch otheri of octyon (D) propetig auve
Maver th kand cheriing samouat
Syobelk ravaaag:
ranc cutye (1f) on peopuiig movee
Midas sceaar WC--tbrllssss--aaoe
Mikee: Landls cbsering to sarmandiig
de (Di torona
wave ou eolyeis (ily ai we
Sydteac ming: Dupactec syflogio (Dwttw Ownnt
Sie it kecs en aldi reiuve 20% propeing
tttmm ttep rmmiknionn
STEP
dudddng
aodyat
Ri mmgge.
Creck sie The Kuas paeper
City imagp..
STEP 5
souther thaa surtyy
ar iaard pripiing prappiinggg so (E% Basil
Symbeli rasstig::
Madaa paine
Ma laadle paras ag nave paoniiee
Losali propring i
1r lausd dun our delect prodace Kicutty
Syobel Modas sillar 0T Phd8lye!
Mi th Eaad dass datac prdaas viiidy Banil
STEP
Bue ddec cat persiii dinat auary
Flad
Orockiiee The kone cacet
Figure 9: A Case Study Comparing CoT and LogiCAM
searching for food (B). Moreover, lacking step-by-step formal inference, it eventually abandons the deeper reasoning chain altogether, falling back to superficial pattern matching.
In contrast, LogiCAM systematically derives new knowledge and reaches the correct answer by integrating three tightly-couples mechanisms at every inference step. Its Premise Selection module continuously extracts and logically maps image features into textual element (e.g., "on grass" + food search; "no shedding" - eedyysis), demonstrating its advantages in multimodal fusion. The Reasoning Type Identifier then selects the appropriate reasoning type, formal logic for structured inferences (e.g.. C - (D V E)) or heuristics to complement symbolic logic. thereby balancing the rigor of formal deduction with the flexibility to incorporate knowledge beyond the scope of logic. Finally, the Symbolic Reasoner rigorously applies formal inference rules (c.g., disjunctive syllogism, modus ponens, modus tollens) derive each new conclusion in a systematic and reliable way. This disciplined, iterative process ensures robustness in handling long reasoning chains.
Conclusion and Future Work
We have pionecred the Multimodal Symbolic Logical Reasoning (MuSLR) task, challenging models to perform precise, rigorous formal logic inferences over combined visual and textual inputs, thereby filling critical gap in existing benchmarks. To support this research direction, release MuSLR-Bench,, a rigorously annotated dataset of 1,093 instances spanning seven application domains, featuring 35 atomic reasoning units and 976 composite logic combinations with depths ranging from 2 to 9. We also propose a strong baseline LogiCAM, a novel modular framework that systematically decomposes the reasoning process into premise selection, reasoning-typp identification, and formal inference, demonstrating substantial performance gains over prior methods.
Looking forward, our diagnostic analyses reveal two key opportunities for advancing multimodal symbolic reasoning. First, integrating dedicated symbolic modules is essential: the LogiCAM outperforms base VLMs precisely because extracts multimodalities based on logic and embeds explicit symbolic reasoning steps. Second, existing VLMs struggle to align and fuse visual and textual information when performing formal logic; Future work should explore tighter multimodal integration, such as cross moosl architectures trained with logic-grounded objectives, to bridge this gap. By making MuSLR and benchmark publicly available, we hope to catalyze research on these challenges and bring truly rigorous, multimodal symbolic reasoning within reach.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Ministry of Education, Singapore, under its MOE AcRF TIER Grant (M0E-MOET3302--000)
References
[1] Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lec. nocaps: novel object captioning scalc. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November
2, 2019, pages 8947-8956, 2019.
[2] Anthropic. Claude 3.7 sonnet. 2025-05-14.
2025. Accessed:
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jiangiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv, 2502:13923, 2025.
[4] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructhlip: Towards general-purposs vision-language models with instruction tuning. CoRR, abs/2305.06333 00000000000000,....000 2023.
[5] Hao Fei, Yuan Zhou, Juncheng Li, Xiangtai Li, Qingshan Xu, Bobo Li, Shenggiong Wu, Yaoting Wang. Junbao Zhou, Jiahao Meng, al. On path multimodal generalist: General-level and general-bench. In Proceedings of the International Conference on Machine Learning, 2025.
[6] Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarande and Asli Celikyilmaz. Roscoe: A suite of metrics for scoring step-by-ster reasoning. CoRR, abs/2212.07919, 2022.
[7] Simeng Han, Hailey Schoelkopf. Yilun Zhao, Zhenting Qi. Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, David Peng. Jonathan Fan, Yixin Liu, Brian Wong. Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang. Shafiq Joty, Alexander Fabbri, Wojciech Kryscinski. Xi Victoria Lin, Caiming Xiong, and Dragomir Radev. FOLIO: natural language reasoning with first-order logic. CoRR, abs/2209 .00400, 2022.
[8] Adam W. Harley, Alex Ufkes, and Konstantinos G. Derpanis. Evaluation of deep convolutional nets for document image classification and retrieval. In Proceedings of the 13th International Conference on Document Analyxis and Recognition (ICDAR), volume 2015, pages 991-995, 2015.
[9] Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Proceedings of the ACI. pages 1049-1065, 2023.
[10] Alistair F. W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lchman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Ccli, and Roger G. Mark. Mimic-iii, a freely accessible critical care database. Sci Data, 3:160035 2016.
[11] Shashank Kirtania, Priyanshu Gupta, and Arjun Radhakrishna. LOGIC-LM++: Multi-step refinement for symbolic formulations. In Proceedingy of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (AL 2024), pages 56-63, Bangkok, Thailand, 2024.
[12] Haoming Li, Zhaoliang Chen, Jonathan Zhang. and Fei Liu. LASP: surveying the ofate-of-the-art in large language model-assisted Al planning. CoRR, abs/2409.01 2024.
[13] Chin- Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74 81, Barcelona, Spain, 2004. Association for Computational Linguistics. URL https://accaathhooggg www.llloyoyoooroogo wrw//o/-/10000 1013/
[14] Tsung- Yi Lin, Michael Maire. Serge J. Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects context. Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, volume 8693, pages 740-755, 2014.
[15] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. CoRR, abs/2310.03744, 2023.
11

---

[16] Jiacheng Liu, Wenya Wang. Dianzhuo Wang. Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. Vera: A general-purpose plausibility estimation model for commonsense statements. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1264-1287, December 2023.
[17] Pan Lu, Swaroop Mishra, Tanglin Xia, I Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28. -Deember 9, 2022, 2022.
[18] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In Proceedings of the International Conference on Learning Representations, 2024.
[19] Theo X. Olausson, Alex Gu. Ben Lipkin, Cedegao E. Zhang, Armando baum, and Roger Levy. LINC: A neurosymbolic approach for logical reasoning by combining language models with first-order logic provers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5153-5176, 2023.
Joshua B. Tenen-
[20] OpenAL. Gpt-40 system card. CoRR, abs/2410.21276,, 2024.
[21] OpenAI. Gpt-4.1. https://openaa:: comt:/ae..om////////e/ee/ www.om/iindxxeeee gptpee/xxppt--////// 2025. Accessed: 2025-05-14.
[22] Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages
2023.
[23] Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. LogicBench: Towards systematic evaluation of logical reasoning ability of large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. pages 13679 13707, 2024.
(24] Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Logii-LogiEval: Towards evaluating multi-step logical reasoning ability of large language models. In Proceedingy of the Annual Meeting of the Association for Computational Linguistics, pages 20856-20879, 2024.
[25] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- ragion-to-phee to-phrase correspondences for richer models. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2641 2641-2649, 2015.
[26] Hyun Ryu, Gyeongman Kim. Hyemin Lee, and Eunho Yang. Divide and translate: Compositional first- order logic translation and verification for complex logical reasoning. In Proceedings of the International Conference on Learning Representations, pages 236-265, 2025.
[27] Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of thoouh.. In Proceedings of the International Conference on Learning Representations, 2023.
[28] Oyvind Tafjord, Bhavana Dalvi, and Peter Clark. ProofWriter: Generating implications, proofs, and abductive statements over natural language. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 3621-36334 2021.
[29] Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song. Mingjic Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning. In Proceedings of the International Conference on Learning Representations, 2024.
[30] Weiqi Wang, Tianqing Fang. Chunyang Li. Haochen Shi, Wenxuan Ding, Baixuan Xu, Zhaowei Wang. Jiaxin Bai, Xin Liu, Cheng Jiayang, Chunkit Chan, and Yangqiu Song. CANDLE: Iterative conceptualiza- tion and instantiation distillation from large language models for commonsense reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 2351-2374, 2024.
[31] Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning: A comprchensive survey. arXiv preprint ar 250:3 / 2025.
[32] Jason Wei. Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou Chain-of of thought prompting elicits reasoning in large language models. In Proceedings of the Annual Conference Neural Information Processing Systems, pages 24824 24837, 2022.
12

---

[33] Shenggiong Wu, Hao Fei, Leigang Qu. Wei Ji. and Tat-Seng Chua. NEXT-GPT: Any-to-any multimodal LLM. In Proceedings of the International Conference Machine Learning, pages 53366-53397, 2024.
[34] Xiaoqian Wu, Yong-Lu Li, Jianhua Sun, and Cewu Lu. Symbol-llm: Leverage language models for symbolic system in visual human activity reasoning. In Proceedings of the Annual Conference Neural Information Processing Systems, 2023.
[35] Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal LLM logical reasoning benchmark visual contexts. CoRR, abs/2407, 04973, 2024.
[36] Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang. Preslav Nakov, Mong- Li Lee. and Wynne Hsu, Aristotle: Mastering logical reasoning with A logic-complete decompose-search resolve framework. CoRR, abs/2412.166955 2024.
[37] Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic ohain-of-thoughh In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 13326- 13365, 2024.
[38] Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. Visulogic: A benchmark for evaluating visual reasoning in multi modal large language models. CoRR, 0os/2504.11277 2025.
[39] Xiang Yue, Yuansheng Ni, Tianyu Zheng. Kai Zhang, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang. Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang. Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. MMMU: massive multi- discipline multimodal understanding and reasoning benchmark for expert AGL In IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9556-9567, 2024.
[40] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi, Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL URI.https: //open/eview.nee//// http://rrmnni-sSeettitreee
[41] Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, and William Yang Wang. Rulearena: A benchmark for rule-guided reasoning with Ilms in real-world scenarios. CoRR, abs/2412.0899 2024.
[42] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Yc, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, Zhangwei Gao, Erfei Cui, Yue Cao, Yangzhou Liu, Weiye Xu, Hao Li, Jiahao Wang, Han Lv, Dengnian Chen, Songze Li, Yinan He, Tan Jiang. Jiapeng Luo, Wang. Conghui He, Botian Shi, Xingcheng Zhang, Wenqi Shao, Junjun He, Yingtong Xiong, Wenwen Qu, Peng Sun, Penglong Jiao, Lijun Wu, Kaipeng Zhang, Huipeng Deng, Jiaye Ge. Kai Chen, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. CoRR, abs/2504.11444 104799 2025.
13

---

Appendix
In the appendix, we provide detailed descriptions of each error type (Section A]. the complete workflow of the MuSLR construction pipeline (Section [B), the full quality control process, including both automatic and manual filtering strategies (Section the details of the LogiCAM framework (Section D) the collection of atomic symbolic logic used in our study (Section F. and an ethics statement (Section G).
A Error Analysis
We provide detailed explanations of each error type below.
This erorr occurs when the model attempts apply formal logical rules but does so incorrectly. Typical mistakes include reversing implications, confusing necessary and sufficient conditions, or failing to properly follow multi-step deductions. While the model recognizes that logical reasoning is needed, the specific application is flawed, leading to invalid conclusions.
Incorrect Application of Logical Rules
Failure to Supplement with Commonsense/ Rule Misgeneralization In some cases, the given input lacks complete information, requiring the model draw commonsense knowledge fill in gaps. This error happens when the model fails to do so, resulting in halted or incomplete reasoning. Alternatively, the model may overgeneralize formal rule, applying it too broadly or narrowly, which also leads to incorrect outcomes.
Overlooking Visual Details This error reflects the model's inability notice correctly interpret critical visual elements in the image, such as small objects, specific colors, or spatial relationships. Missing these details prevents the model from correctly progressing in its reasoning chain, despite the necessary information being present in the visual input.
Premise Integration / Alignment Errors Even when the model successfully extracts information from both text and image, it sometimes fails to align them correctly. This happens when visual entities are mismatched with their textual references (e.g.. linking "the red triangle" to the wrong object in the image). Such misalignment breaks the reasoning process and leads to incorrect answers.
Heuristic Shorteuts over Formal Logic Rather than following precise logical reasoning, the model occasionally defaults to heuristic-based shortcuts, relying superficial patterns associations learned during training. While this may sometimes produce plausible answers, it undermines the rigor required for formal logical tasks, resulting in systematic errors when heuristics misapplied.
Visual Perception Object Recognition Errors This error type stems from failures basic visual perception, such misidentifying objects, misclassifying shapes, colors, spatial positions. When the model starts reasoning from an incorrect visual premise, all subsequent deductions are built on a faulty foundation, leading to incorrect conclusions.
B MuSLR Construction Process
We collect images from multiple sources, including COCO 14]
Flickr30k 25 nocaps MР. Mimic [10, RVL CDIP (). ScienceQA [17] and Traffic Report collected manually. For each image I, visual details V extracted using GPT-4o ensure diverse and fine- fineggrained descriptions.
Step 1: Systematic Rule Selection
We begin by examining a broad
logical inference rules drawn from propositional logic (PL), first-order logic (FOL), and non monotonti logic (NM). We utilize the complete set of logical rules collected by [24]. denoted as = (r1, 72. )m). which comprehensively covers standard inference patterns. Rather than selecting rules randomly, we carefully curate a subset Reeleeted C R. that is both formally sound and frequently encountered in real-world reasoning. This subset includes classical patterns such Modus Ponens, Hypothetical Syllogism, Modus Tollens, and Disjunctive Syllogism. Details about the logical rules are provided in the Appendix.
Step 2: Meaningful Rule Composition:
We select meaningful rule combinations, denoted as Rua (RI, R2,...), ..., to construct logically coherent reasoning chains C (C1, C2....) .} by rule-based substitution. Each reasoning chain C.
14

---

consists of an ordered sequence of rules from Rex and is manually composed by experts in formal logical reasoning ensure coherence and meaningfulness.
Step 3: Grounding in World Contexts:
The meaningful rule composition step produces an abstract, context-independent symbolic rule set Rui (R1, R2.... (e.g., "If A, then B"). During grounding. visual features V from an image guide the retrieval of relevant textual information Tetrieved from sources like healthcare reports, Wikipedia, or traffic incident summaries. Abstract rules from Ree are instantiated using real-world information from Trerieved, creating the grounded rule set Reeal (e.g.. "If someone blowing out candles, they might be celebrating a birthday").
The adapted rule set Reeal will used to construct the instantiated reasoning chain Crea- When the symbolic reasoning rule Reeal alone is insufficient to capture the real-world context retrieved. we incorporate commonsense reasoning to supplement formal logic. This combination forms hybrid reasoning structure Chybeid (11, ........ .gk). where each r, Raym U Res. Here, Ryym comprises rules instantiated from Rga, and Res denotes commonsense reasoning steps. Commonsense reasoning is incorporated only in Chybrid and not explicitly represented in Ryeal- This reflects human cognitive processes, where not all necessary information is always available, and intuitive reasoning is often used fill in the gaps. The Reeal populates the hybrid reasoning template Chybrid yielding the fully grounded reasoning chain Creat. Then we use the conclusion of the Creal to construct questions and ground-truth answers based rule-based substitution.
Step 4: Question Generation
Based the ground-truth reasoning chain Cp. and answer Agt. we generate corresponding questions Q that require multi-step reasoning for solution, following rule-based substitution templates.
Step 5: Automatic and Manual Quality Verification
Finally, both automatic verification procedures and manual expert review are employed to ensure the overall quality, consistency, and correctness of the generated dataset.
C MuSLR Quality Check
To ensure the high quality, relevance, and correctness of the constructed dataset, we implement multi-layered quality control procedure combining both automatic and manual verification steps. Automatic Quality Control: We apply two automatic filtering strategies to enforce logical soundness and diversity:
We compute the lexical similarity between each pair of reasoning steps within reasoning chain using Jaccard Similarity. Chains with similarity score above 0.5 are discarded promote step diversity and minimize redundancy.
Lexical Similarity Filtering:
Commonsense Plausibility Filtering: Each reasoning step is assessed using Vera [6], a T5 model fine-tuned on commonsense reasoning tasks. If any step receives a plausibility score below 0.5, the entire instance is removed ensure logical soundness and realism.
As result of the automatic filtering, the original sample size was reduced from 1,956 to 1,464. Manual Quality Control: Given that the extraction of visual details (V) leverages GPT-4o, which may have hallucinations, we implement a rigorous manual validation stage:
Visual Detail Verification: Human annotators confirm that the extracted visual details accurately reflect the content of the corresponding image, explicitly checking for hallucinated objects, actions, or attributes.
Context and Question Evaluation: Annotators evaluate whether the generated context (Tcontext) and associated questions (Q) are plausible and relevant to real-world scenarios.
Annotation Process and Training All instances were independently reviewed by three trained anno- tators with STEM backgrounds. In total, six annotators were recruited to assess the 1 1,464 instances, with each annotator reviewing 732 instances. For each check, annotators provided judgments using a three-option scale: Yes, No, or Not Sure.
To prepare annotators and ensure consistent application of quality standards, we provided a dedicated training session. This session covered task definitions, annotation guidelines, and hands-on practice with feedback. To further support annotators and minimize cognitive load, we developed custom annotation interface prototype (see Figure 10.. which streamlined the annotation process by integrating
15

---

image previews, visual details, and context input fields for both checks. This tool helped reduce annotation errors and improve task efficiency.
Annotators also underwent calibration phase involving 30 examples, followed by iterative discussion sessions refine annotation guidelines and resolve disagreements. We measured aiter-annotato agreement using Fleiss' Kappa, achieving average score of 0.92 for visual detail verification (substantial agreement) and 0.71 for context alignment (moderate agreement), which consistent with the subjective complexity of evaluating real-world plausibility.
Annotation Results. Visual detail verification exhibited a high level of agreement, with an initial inter-annotator agreement rate of approximately 0.90, reflecting the objective nature of the task. In contrast, context alignment showed lower agreement, with an initial rate of around 0.70, due to its inherently more subjective nature. Instances were initially retained if they received three Yes votes for both checks.
Conflict Resolution and Filtering.
Instances that received unanimous No judgments from all annotators in either check were directly discarded.
For cases with conflicting judgments (e.g., one No, two Yes or any instance with at least one Not Sure), second round of annotation was conducted. During this phase, annotators collaboratively revisited the flagged cases, discussed discrepancies, and reached consensus decision to ensure consistent quality standards.
If, after discussion, the final decision still resulted in a No for either the visual detail correctness context plausibility, the instance was removed.
Across the dataset, 492 instances were filtered by auto- matic checks. 371 by manual annotation, resulting in the final sample size of 1093. Common errors detected included hallucinated objects implausible contexts, further emphasizing the necessity of both automated and human oversight to ensure dataset validity.
Filtering Statisties and Error Examples:
D Detailed LogiCAM Reasoning Process
Below, we present the step-by-step reasoning workflow of LogiCAM.
Step 1: Initial Premise Selection. Given a context set Rreat, an image I, and access to a VLM, we prompt the model to initiate the reasoning process by selecting relevant information Irelevant C C U V. The VLM is instructed to prioritize selecting pair (Ø, v) such that formal inference rule (e.g.. Modus Ponens) can be applied. If no such pair exists, the model selects the information it judges most critical for solving the task.
Step 2: Identify Reasoning Type. For each selected pair Isetvannt we determine the type of reasoning. Symbolic reasoning is applied if the Lyetvant contain a pair (Ø, v) such that a formal inference rule (e.g., Modus Ponens) can be applied, i.e. A (Ø x) x. Otherwise, commonsense reasoning is used.
Step 3: Perform Reasoning. Depending the reasoning type identified in the previous step, the VLM performs inference to derive new knowledge K. For symbolic reasoning, the system applies syllogistic inference, form of deductive reasoning. Specifically, given two selected premises Trelevamt = {Ø) vb), ), the VIM applies formal logical rules to derive a conclusion. For commonsense reasoning, the VLM generates semantically and contextually plausible implication X. such that (Aretevant + x). grounded in real-world commonsense knowledge using a VLM. The result of either reasoning process is recorded K.
Step 4: Check for Completion. We evaluate whether the current knowledge K is sufficient to determine an answer to the given question. For truth evaluation (True/False/Unknowww questions involving a single hypothesis H, if K }= H or K -= -H, the process terminates with the correspond- ing label (True or False); otherwise, it continues. For multiple-choice questions with candidate hypotheses [H, H2, Hs. H4), we apply the reasoning process to each H, individually and select the one for which K -= H; holds, if exactly one such H, exists. If no hypothesis is entailed, or more than one is, we continue the reasoning process. In all cases, the set of relevant information is updated as Frelevant & Tretevant U K, and the procedure is repeated from Step 1. The reasoning loop is bounded by a predefined number of maximum iterations. If no conclusive answer is reached within this limit, the final output is labeled as Unknown for truth evaluation questions, or deemed incorrect for multiple-choice questions.
16

---

E
Additional Experiments
E.1
Using Symbolic Prover on MuSLR
Most existing LLM+solver approaches (e.g., Logic-LM, Logic-LM++, LINC) are designed for text-only reasoning tasks and cannot directly process visual inputs. Extending them multimodal settings typically requires vision-lange vision-language model (VLM), such 1PT-4.1, to translate images into textual descriptions. However, this translation often omits subtle
to-e----erbaliz visual cues.
To illustrate this limitation, we adapted representative LLM+solver method, Logic-LM [22], by pairing with VLM (GPT-4.1) to convert images into text, and compared its performance against LogiCAM on propositional logic (PL) and first-order logic (FOL). (Logic-LM does not support natural language with modalities, NM.) The results are summarized below:
Model
PL (%)
FOL (%)
Logic-LM + VLM
35.14
32.65
LogiCAM
60.44
42.55
Table 2: Performance comparison of Logic-LM with VLM versus LogiCAM on MuSLR.
These findings demonstrate that simply translating visual information into text is insufficient for effective symbolic reasoning. LogiCAM. which natively built on VLMs, achieves significantly higher performance since it can directly access visual content. Nonetheless, LLM+solver approaches remain important, and we propose exploring more integrated multimodal I.LM frameworks as promising directions for future work.
Atomic Symbolic Logic
Below, we present the atomic symbolic rules used to construct MuSLR.
Propositional and First-order Logic
Modus Ponens (MP)
Propositional:
()-)))p)
First-order:
((vr (p(x) (p)))
^ p(a)) + {(a)
If "p implies q" and holds, may conclude q.
Modus Tollens (MT)
Propositional:
( A -q) + p
First-order:
((Vx (p(x) - ((x))) ~p(aa) pp(a)
From p > and mq infer -p
Hypothetical Syllogism (HS)
Propositional:
(p==g)^ +r)) + r))
First-order:
(Vz - ((x)) (())))((()))) (g(x) r())))) + (p(a) - r(a))
Chaining two implications into one.
Disjunctive Syllogism (DS)
Propositional:
((Dv))A-p) H
First-order:
((Vz (p(x) v g(z))) A -p(a)) + g(a)
Eliminate disjunct once the other is shown false.
17

---

Constructive Dilemma (CD)
Propositional:
(1 - (r ( ()) F (g
First-order:
((vx ((p()) + ((x)) A (r(a) + s(z))) A (o(a) Vr(a)))) + (a(a) v s(())
From conditionals and choice of antecedents, infer choice of consequents.
Destructive Dilemma (DD) Propositional:
(IP 9) + ( -s)) + (rp
First-order:
((Vr ((p(z) + A (r=z) (()+
A
-a()))) + (~pla) V -r(a))
The "dual" of the constructive dilemma.
Biconditional Dilemma (BD)
Propositional:
(r
Vr
First-order:
((Vx ((p(x) - ((x)) A (())) v -s(a))) + (a(a) v -r(a))
mix constructive and destructive patterns.
Commutativity of V (CT)
Propositional:
(V (gVp)
First-order:
Vo (p(x) V ((x)) + Vx (x(xx) V v(x))
Order of disjunction doesn't matter.
De Morgan's Transformation (DMT)
Propositional:
(p + (>p V -q)
First-order:
-Vx (p(x) A ((x)) + 3x (~p(x) V -q(z))
Pushing negation inside a conjunction (or quantifier).
Conjunction of Conclusions (CO)
Propositional:
a) (p (p
First-order:
(x((p<>) - q(x)) (p(x) r(=)) + Ya (p(z) + (q(z) A r(x)))
From two implications with the same antecedent, fuse their consequents.
Implication Conjunction (IM)
Propositional:
(p (p- (q +r) r)) + ((p)g) q) r)
First-order:
Yz (p(x) (q(x) r(x))) + Vor ((p()) - r(x))
Currying/un-curyiing of implication.
Material Implication (MT)
Propositional:
(p 4)) + (-DVg)
(No direct first-order analogue listed.)
Existential Generalization (EG) First-order only:
p(a) } D pxp(
From a particular instance infer an existential claim.
Universal Instantiation (UT) First-order only:
Vx p(a)
From universally quantified claim infer it for arbitrary constant.
18

---

Extended Multi-variable FOL Rules
MV1
Va ((p(a) ((x)) - r(z,y)) A Po (p(u) or(u,)) By mgy)
every p A yields r, but there an instance of p where r fails, then that instance must lack q.
MV2
Vevy ((p(z) A ((x)) + ss(x,)) u)) A V2 (r(2) - p(2)) A r(a) (a, + -q(b)
Combines two universally quantified conditionals and a counter-example force (b()).
MV3
Vo By (p(x) + ((x,)) u)) VuVu ((q(u, v)Ar(u, u)) + s(v)) A 323k (p(=)Ar(z, k))
Ei s(w)
Chaining existential-conditionaal universal rule, and an example to derive an existential.
MV4
VrVyV= (p(ar, (1(2, r(v))) Fa (p(u, v, w) A -q(u, w)) T
Es rs)(s)
p always gives or r, and for some triple holds but fails, then some r must hold.
MV5
Yx (p(x)) + By (y, A p(a) + (2, From a universal "p implics an r" and one example p. infer the corresponding existential.
MV6
Vaky (p(z, y) V g(x, A BuEv qitte v) 3:3w p(z, w) quantified disjunction plus a counter-example to one disjunct forces the other.
MV7
Vz (p(z, y) - (a(x) Ar())) A b(a, g(a) A r(b)
From a universal conditional that yields a conjunction, plus an instance, you get both conjuncts.
Non-monotonic Default- Reaaulnnn Patterns
DRS (Default Reasoning with Several Defaults) Manages cases where multiple default rules apply at once and may conflict, by finding a consistent combination.
DRI (Default Reasoning with Irrelevant Information) Ensures that adding facts unrelated default does not block that default's usual conclusion.
DRD (Default Reasoning with Disabled Default) Shows how the presence of an exception can "turn off" default that would otherwise fire.
DRO (Default Reasoning in an Open Domain) Adapts defaults to settings where not all individuals are known or named.
REI (Reasoning about Unknown Expectations Allows inferring default property in the absence of any information to the contrary.
REH (Reasoning about Unknown Expectations II) Refines REI by handling the situation where conflicting expectations might arise.
REII (Reasoning about Unknown Expectations III) Extends the previous patterns to nested or higher-order expectations.
RAP (Reasoning about Priorities) Introduces a priority ordering among defaults to resolve conflicts in favor the pribority rule.
G Ethics Statement
G.1
Statement
This study adheres to rigorous ethical framework to ensure the responsible development, evaluation, and deployment of multimodal general-purpoct AI models. The key ethical considerations are outlined below. These measures ensure that MuSLR, as responsible and inclusive framework, continuously contributes the fair, sustainable, and accountable development of multimodal artificial intelligence.
19

---

G.2 Privacy and Data Protection
The benchmarking and evaluation processes strictly comply with privacy regulations, All tasks and datasets used in MuSLR are carefully curated to exclude any personally identifiable information (PII). Any data obtained from publicly available sources is anonymized and filtered to remove privacy- sensitive content. We are committed fully adhering relevant data protection standards, including the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA), thereby upholding the highest standards of ethical research practices.
G.3 Data Collection
All data included in the MuSLR dataset sourced exclusively from publicly available resources. The data collection protocol is designed to prioritize ethical sourcing, ensuring that contributors' rights are respected, including the right withdraw their data where applicable. This approach ensures transparency and fairness throughout the dataset construction process.
G.4 Annotator Compensation
We fully recognize the critical role human annotators play in creating the high-quality MuSLR dataset. All six annotators involved in the project are trained professionals, and they received fair compensation for their work. Annotators were compensated with cash payments upon completion of their assigned tasks. Each annotator was committed contributing their best efforts data annotation and quality assurance, ensuring the integrity and reliability of the dataset.
G.S Bias and Fairness
We proactively implemented measures to analyze and mitigate potential biases related to gender, ethnicity, language, and other sociocultural factors present in the datasets and evaluation tasks. Our goal is to reduce the risk of perpetuating biases in Al development. While completely eliminating bias remains an ongoing challenge, our commitment identifying and addressing bias throughout the benchmark development process remains steadfast.
20

---

-002% 1I the deceent lucK (ar Riler SEGRSA qalrioation THE an ESier BI
It CMER t are co Bartu au soution title Passiste foorress
vafastionl spprianc the 1471 y 2iller 3s is
cotrrt ussoonnn 3e be by burinoto as VTR77STS tee te the
miller's stati nalitiontins
Sotee of the do soceon is the a regeess for arst
ceeerree attlopp.
sn way "I the on coue is ouell eill und ore
gusideen sette reneett miliass
200 Sol the devenns mosll dur
nitter anu oot lard operin seae
Oecument 100 RESTOO D4
"per eileee the eoowe Nter
F1 Tim decumen secumms la Docaptec
baan if car Mille wile
the dee gall
Informtiio
Car x:
Visual
140
Stea Meat
Back
pooe w The and te Peca Peter and heaeer Dostn Piai nor Adrio UN plece wig Srooeer Th
10/9 1yg
OF STU0
Lainnsy Departieas of Husan Oneiior sad Dentlipone Cilantaa
Dipatiie Nusiveene the 2017--2
Beak bobili Ch 7. Murr O Stinter It Nilrr CV. Schnio It
Outain (DUA) Lecns is Ruzzs 772, 130.
Undre aaniiin of Sociite Oltin copinaiiia aEMA K30 cal., Dioeratiee Colantia
Bookhenn
Remeek Anioai Dipurtsan of Nennaggi Cilambia Univentty
MOK CV. Yeats Demetll Alb Sheiia de Baon Spoilee Goble
Culffrn,,
fdlawdip
10 511-0 10, 198).
1974
1912
1983
BIOGRAPHICAL BKE CH
Adboost Romarske
GRUATT IUSAEE
81
Pb.0
Renderaarr, Depurnacn Madicin,
Bis Thaurria His
Ston Kongeen
Nat
Gue the Sitteing interation for oe
Bark Ciaztii
SHE Grddase Tustenlo
Adound Models XSS. cittee
Qaston Dws.te Homeeln elggn Image?
redvvr soveriee
Ruscino ent Sartraa
Yeur Olleg,, New Yook
D0 adderte PACIS
Grem Hisoneee
Chtl Millr
Saddr Miker
to
1983 Pio-PPo
1944-9977
Maona
1855-196
1977 19
Dititation
Mill:
Comments (:
Quedeo Doeitte eanbel
RESEAROO
Tua
12
Figure 10: Annotation Interface. We developed custom interface to streamline the annotation
process and reduce annotator effort.
21

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We explicitly include the contribution sessions and link that to our main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It fine include aspirational goals as motivation as long as it clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We provide detailed error analysis to discuss the limitations of our proposed method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- ers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not have theoretical results.
22

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We discussed the data construction and method in the main content, and provided more details in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others replicate the model with the same dataset, provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. If the contribution be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., large language model), then there should either
(d) We recognize that reproducibility may be tricky some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, may that access to the model limited in some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will open-source all the data and codes.
Guidelines:
23

---

The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPSS code and data submission guidelines (tttp::/hhips.cccppblls
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for
open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurIPS code and data submission guidelines (https://nips cc/public/gun http:/ggdddeee/eeeeeeee
to
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include the details in the experiment setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Our benchmark large so will be too expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).

---

If eπror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification:
We include the details in the experiment setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer:
[Yes]
Justification:
We fully conduct the NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Not applicable. The proposed benchmark is a technical contribution intended for
research evaluation and does not have direct societal impacts, either positive negative.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
25

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
There is such risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification:
We cite the original paper and specify the version we used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperrwi thcode. com/ /ntaastsssts has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide sufficient details about the benchmark in Section 4.
Guidelines:
The answer NA means that the paper does not release new assets.
26

---

Researchers should communicate the details of the dataset/ccdeemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [Yes]
Justification:
We include the details of how we instruct the human annotators in appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification:
We discuss how we minimize human subjects appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We discuss the details of LLM usage in the paper.
Guidelines:
The answer NA means that the method development in this research does not involve
LLMs as any important, original, or non non-standard components.
httpp:////eurppp..ccooffeeeeeeee L0t0://0f000rr000////////////111 for what should should not be described.
Please refer to our LLM
27


Reviews:
[{"review_id": "iO04F3jkuc", "reviewer": "Reviewer_PLuU", "summary": "This paper introduces a task requiring models to perform formal logical reasoning over combined visual and textual inputs. The authors create (1) MuSLR-Bench, a dataset of ~1k instances spanning 7 domains with ground-truth reasoning chains using propositional logic, first-order logic, and non-monotonic logic. They (2) evaluate 7 VLMs, finding that GPT-4.1 achieves 46.8% accuracy. They (3) propose LogiCAM, a prompt-driven framework that contains premise selection, reasoning type identification, and formal inference components, which improves GPT-4.1's performance by 14%.", "questions": "1. Dataset composition says 35+976 ≠ 1,093. May you please elaborate on the composition?\n2. Are you using a solver?\n3. Figure 7 shows accuracy performance degradation at higher depths. Why?\n4. Could you please elaborate on Weakness 3.\n5. I am willing to revise my scores upwards if you can address weakness 1.\n6. Could you please elaborate on your dataset construction methodology. I find this very interesting. More specifically, I want to understand how you deal with the following\n\t1. How do you do semantic disambiguation in the autoformalization process.\n\t2. Is this quantifier free? How do you deal with quantifiers, which may add significant complications to autoformalization.\n\t3. How do you formalize inherently imprecise natural language statements? How do you deal with implicit meaning in language queries? Did you face any of these issues\n\t4. How do you go about determining the necessary entities, properties, and relations for the problem's domain, defining the universe of discourse. How do you go about choosing the appropriate level of abstraction? How do you frame invariants?", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. (Originality and Significance) The task genuinely important, and the dataset contribution is timely. As mentioned by the authors, prior work like LogicVista and VisuLogic don't test multimodal formal logical rules, but just do logical reasoning in visual contexts. The novelty here comes from the explicitness of symbolic rule annotation in the dataset (compared to implicit logical reasoning in many VQA datasets).  The benchmark will likely be valuable for the community.\n2. (Quality) The experimental design is sound, with multiple evaluation metrics. There are some actionable insights, particularly the finding that 70% of failures stem from cross-modal alignment issues. \n3. (Clarity) The paper contains helpful visualizations, and workflow diagrams to communicate the approach. \n\n**Weaknesses**\n1. (Quality) I think the paper has a glaring gap in comparing LogiCAM against techniques presented in the literature. LLMs+Solver paradigms already exist. For example : Logic-LM (cited) and many more papers (not cited) ; already combines LLMs with symbolic solvers, decomposing reasoning into symbolic steps. Multimodal symbolic reasoning has also been shown to work in papers like Proof-of-Thought (not cited). If you have a new framework to replace these, you should compare it against adapted versions of other frameworks previously proposed, and not just the base language models.\n2. The distinction between \"symbolic\" and \"heuristic\" reasoning in LogiCAM seems arbitrary and inadequately defined.\n3. Why certain logic types (FOL vs PL vs NM) have different difficulty levels needs better understanding. Lines 233-240 do not line up with standard understanding of complexity. In non-monotonic reasoning, you may invalidate previous conclusions, and the solver would need to handle belief revision and consistency maintenance, which would make it more complex than first order logic.\n4. There are a number of typos strewn across the paper, which could use thorough proof-reading. Here are some examples:\n\t1. Fig 1 : Step 4 \"pposing\" instead of \"opposing\"; Step 3: Minor Premise 1: \"woad\" instead of \"road\"\n\t2. Some references use \"CoRR, abs/XXXX\" while others use full venue names\n\t3. Table 1 Formatting InstructBlip row: \"33,33\" should be \"33.33\" (comma instead of period)\n\t4. Figure 9 : Context text says \"arounod\", should be \"around\".", "was_revised": true, "final_justification": "The authors have benchmarked against LogicLM, and will clarify in their final manuscript that their LogiCAM technique is not using a solver, does not provide guarantees, and is a strong prompting based baseline. My concerns are resolved.", "rebuttal": "Thanks to the reviewer for acknowledging the significance, originality and quality of the work. We would like to address your concerns and questions in detail below.\n\n---\n\n**W1: Comparing LogiCAM against techniques presented in the literature**\n\nWe appreciate the reviewer’s suggestion to compare LogiCAM with LLM+solver methods like Logic-LM. We will revise the manuscript to better acknowledge this line of work and expand the related work section.\n\nWe did not include these methods originally because most LLM+solver approaches (e.g., Logic-LM, Logic-LM++, LINC) are designed for text-only tasks and cannot handle visual inputs directly. Adapting them to multimodal settings requires a VLM (e.g., GPT-4.1) to convert images into text, which can omit subtle or hard-to-verbalize visual cues.\n\nTo illustrate this, we adapted a typical LLM+solver approach, Logic-LM, using VLM (GPT4.1), to transform images into text and compared it with LogiCAM on PL and FOL (Logic-LM does not support NM):\n\n| Model             | PL (%)  | FOL (%) |\n|------------------|---------|---------|\n| Logic-LM + VLM   | 35.14%  | 32.65%  |\n| LogiCAM          | 60.44%  | 42.55%  |\n\nThese results support our view that translating visuals into text is not enough for effective symbolic reasoning. LogiCAM, built on VLMs, performs significantly better since it can directly access the image. However, we agree that LLM+solver approaches are important, and we propose a more integrated multimodal LLM+solver framework as future work.\n\nIn the revision, we will 1) Discuss challenges in adapting LLM+solver pipelines to multimodal symbolic reasoning tasks; 2) Cite additional relevant works (e.g., LINC, Logic-LM++, TIC, Divide-and-translate; happy to include others if you believe there are important and relative methods we may have missed.); 3) Expand the related work section (see also our response to Reviewer oQKt, Comment W1).\n\nRegarding **Proof-of-Thought**, we acknowledge its relevance to multimodal symbolic reasoning and will include and cite it in our revised manuscript. However, due to the lack of a public codebase, we were unable to replicate its results within the rebuttal timeframe.\n\n---\n\n**W2. Distinction between \"symbolic\" and \"heuristic\"**\n\nWe thank the reviewer for raising this important point. To address this, we would like to point out that a formal definition of these two types of reasoning is already provided in the Appendix (Lines 583–594) of our manuscript. In brief:\n\n● Symbolic reasoning follows formal logic, using rules like modus ponens to derive conclusions from explicit premises.\n\n● Heuristic reasoning, by contrast, encompasses informal, intuitive, or commonsense reasoning strategies. These are employed in situations where formal logical structure is unavailable or insufficient, such as dealing with ambiguous visual information, implicit information, or assumptions. \n\nThis distinction reflects our goal of combining structured symbolic inference with the flexible reasoning capabilities of VLMs, as not all information can be explicitly encoded in formal logic in real-world situations. We will clarify this distinction earlier in the main text and better reference the definitions from the appendix in the revision.\n\n---\n\n**W3. Why certain logic types (FOL vs PL vs NM) have different difficulty levels needs better understanding**\n\nThank you for highlighting this. Our analysis focuses on empirical model performance, not formal complexity. While NM is theoretically more complex than FOL, we observe that VLMs perform better on NM. This reflects model behavior, not a contradiction of theoretical complexity.\n\nWe believe this is because NM aligns better with heuristic reasoning, which VLMs handle well, whereas FOL requires precise logic (e.g., quantifiers, variable binding), which remains challenging for VLMs due to its probabilistic nature.\n\nThis trend aligns with existing work like Multi-LogiEval [2], which shows that in complex scenarios (e.g., depth >= 5), NM shows higher accuracy than PL and FOL due to the richer rule combinations that enhance contextual reasoning. These results suggest that NM’s empirical advantage stems from how models leverage its structure, not its theoretical complexity.\n\nWe appreciate the reviewer’s comment for helping us improve the clarity of this distinction and will add this to the revision.\n\n[2] Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models. EMNLP 2024.\n\n---\n\n**W4. Typos in the paper**\n\nThank you so much for your careful reading of our paper. We will thoroughly proofread the entire manuscript and correct all identified issues.\n\n---\n\n**Q1: Elaborate on the dataset composition**\n\nThank you for raising this question. To clarify:\n\n●The 35 atomic rules form the basis of our symbolic reasoning tasks.\n\n●We then create permutations and combinations of rules manually. (e.g., applying one rule after another) to make sure it is meaningful.\n\n●However, during dataset construction, we filtered out semantically invalid or redundant combinations (see Appendix C). After this filtering, we retain 976 valid rule combinations.\n\n●Importantly, the total number of instances in the dataset is not equal to the number of rule combinations, because each rule combination can appear in multiple distinct grounded scenarios. That is, different images and premises may instantiate the same abstract logic pattern, resulting in a total of 1,093 instances in the dataset.\n\nWe will revise the dataset section of the paper to clarify the distinction between atomic rules, combinations, and grounded instances, and we thank the reviewer again for catching this ambiguity.\n\n---\n\n**Q2: Are u using a solver?**\n\nNo, LogiCAM is a prompt-guided framework without an external symbolic solver. It serves as a baseline for the MuSLR benchmark to explore how current VLMs handle multimodal symbolic reasoning. Integrating external solvers is a valuable direction, which we will highlight as future work (see W1 for the reason).\n\n---\n\n**Q3: Figure 7 shows accuracy performance degradation at higher depths. Why?**\n\nAs reasoning depth increases, more inference steps are needed, making it harder for the model to stay coherent and avoid errors. This added complexity naturally leads to performance degradation.\n\n---\n\n**Q4: Elaborate on Weakness 3.**\n\nPlease see W3 in detail.\n\n---\n\n**Q5: I am willing to revise my scores upwards if you can address weakness 1.**\n\nThank you for your consideration. We appreciate your openness. Please see our response to W1, and let us know if you have any further questions.\n\n---\n\n**Q6.1: How do you do semantic disambiguation in the autoformalization process.**\n\nFirst, in Step 3, we align abstract logical rules with grounded instances by using GPT-4o to extract fine-grained visual details and relevant textual context. Second, all grounded rules are manually curated and verified to ensure they preserve the intended semantics of the original logical form; any ambiguous or misaligned cases are flagged and discarded.\n\n---\n\n**Q6.2: Is this quantifier free? How do you deal with quantifiers, which may add significant complications to autoformalization.**\n\nWe do not limit ourselves to quantifier-free logic but acknowledge that quantifiers, especially nested ones, add significant complexity. Our dataset includes simple universal and existential quantifiers (e.g., \"All X are Y\", \"There exists an X such that Y\"), balance logical complexity and maintain meaningful, human-interpretable reasoning. Quantifiers are handled through verified templates with substitution schemas, ensuring correct alignment between natural language and formal logic.\n\n---\n\n**Q6.3: How do you formalize inherently imprecise natural language statements? How do you deal with implicit meaning in language queries? Did you face any of these issues?**\n\nYes, we encountered this challenge, as imprecise or underspecified language is common in real-world contexts. That is why we incorporate commonsense/heuristic reasoning alongside formal rules in a hybrid format (see W2), allowing the model to handle ambiguity more effectively. Instances with problematic implicit assumptions are revised or removed through rule-based and manual review (see Quality Control), ensuring soundness in noisy multimodal settings.\n\n---\n\n**Q6.4: How do you go about determining the necessary entities, properties, and relations for the problem's domain, defining the universe of discourse. How do you go about choosing the appropriate level of abstraction? How do you frame invariants?**\n\nWe use a context-driven approach to define the logical structure and abstraction level for each instance, grounded in both visual and textual information. Entities, properties, and relations are extracted using GPT-4o from images and complemented by retrieved text to build a dynamic universe of discourse and verified by trained students afterwards. The abstraction level is chosen to balance logical expressiveness with interpretability, favoring mid-level logic that supports patterns like Modus Tollens or non-monotonic reasoning without requiring deep ontologies or complex quantifiers, as this allows current vision-language models (VLMs) to reason effectively while keeping the logic human-interpretable and tractable for evaluation.\n\nInvariants are implicitly modeled as conditions expected to remain true throughout the reasoning process unless explicitly negated. For example, if a scene establishes that “the road is dry,” this holds across reasoning steps unless a new fact (e.g., “it started raining”) overrides it. We ensure such consistency through expert-designed rules and manual verification.\n\nThis approach ensures reasoning instances are logically sound, semantically grounded, and aligned with real-world multimodal contexts.\n\n---\n\nThank you again, and please don't hesitate to reach out if you have any further questions."}, {"review_id": "6ttrwKUcwA", "reviewer": "Reviewer_oQKt", "summary": "This paper studies VLMs on multimodal symbolic logical reasoning tasks. First, the authors propose a new benchmark MuSLR for this task and evaluate multiple state-of-the-art VLMs on MuSLR, finding that they all struggle with multimodal symbolic reasoning. To this end, the authors then propose LogiCAM to improve the performance of existing VLMs on this task.", "questions": "What is the differences between the proposed MuSLR task and a task that receives multimodal input and includes a symbolic reasoning process?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The contribution of this work is solid. After introducing of the background and motivation, the paper 1) raises a new task MuSLR with a clear definition, 2) introduces MuSLR-Bench as a benchmark for MuSLR task including detailed data statistics and construction process, and 3) proposes LogiCAM as a framework to adress challenging issues in MuSLR task with a clear flowchart and process description. Exhaustive experiments and in-depth analyses offer insights of this task.\n2. The writing of this work is clear and easy to follow.\n3. Part of the code and dataset is provided.\n\nWeaknesses:\n1. The MuSLR seems not entirely new. Some previous work like [1] also uses multimodal input and includes a symbolic reasoning process. The authors should better discuss about their differences with such prior studies.\n\n[1] Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning", "was_revised": true, "final_justification": "The contribution of this work is solid, featuring a carefully designed benchmark and a novel framework to address key challenges in the MuSLR task. However, since similar tasks have been explored in prior literature, this may somewhat diminish the novelty of the paper.", "rebuttal": "Thank you for acknowledging the contributions of our work, including the exhaustive experiments and in-depth analyses. We would like to address your questions in detail below.\n\n---\n\n**W1: Differences with prior studies**\n\nThank you for highlighting the relevance of prior work such as Symbol-LLM. We appreciate the opportunity to clarify the novelty of MuSLR and how it differs from existing studies. While Symbol-LLM and similar works incorporate multimodal input and symbolic reasoning, MuSLR introduces a benchmark, not a specific method, focused on systematically evaluating multistep symbolic reasoning across grounded multimodal contexts. Unlike Symbol-LLM, which centers on visual activity reasoning, MuSLR provides a task suite covering diverse logic types (PL, FOL, NM) and varying reasoning depths.\n\nMuSLR also formalizes reasoning through explicit logical structures and rule compositions (e.g., Modus Ponens, Syllogism, Non-Monotonic Reasoning), each grounded in real-world contexts. In contrast, Symbol-LLM emphasizes scene understanding without modeling or benchmarking formal logic chains.\n\nAnother key distinction is MuSLR’s hybrid reasoning design, combining formal logic with commonsense heuristics to handle ambiguity, something not explored in Symbol-LLM. Finally, MuSLR offers a broader range of logic types and controlled complexity to study model limitations in a structured way.\n\nWe will revise the related work section to include Symbol-LLM and clarify these differences in scope, formulation, and goals (as also noted in our response to Reviewer PLuU, Comment W3). Thank you again for drawing this connection.\n\n---\n\n**Q1: Differences between MuSLR and other symbolic reasoning tasks**\n\nWe would like to clarify the distinction between MuSLR and general tasks that involve multimodal input with symbolic reasoning:\n\n1. Narrowed Focus on Formal Logical Reasoning: \"Symbolic reasoning\" is indeed a broad term, encompassing domains such as mathematical problem solving, program synthesis, and abstract planning. While many existing multimodal tasks include symbolic elements in a loose sense, MuSLR explicitly focuses on formal logical reasoning, grounded in classical logic types such as propositional logic (PL), first-order logic (FOL), and non-monotonic logic (NM). This aspect is largely underexplored in prior multimodal benchmarks, which tend to focus on either mathematical or implicit logical reasoning without a clearly defined logical structure.\n\n2. Hybrid Symbolic + Heuristic Reasoning Paradigm: Another key difference is that MuSLR explicitly requires a combination of symbolic and heuristic reasoning. This reflects human cognitive behavior: we use formal logic when precise conclusions are needed, and rely on heuristics or intuitive reasoning when information is ambiguous or incomplete. MuSLR is designed to evaluate this hybrid reasoning ability by incorporating tasks that cannot be solved purely by logic or intuition alone, something that existing benchmarks rarely capture systematically.\n\nTogether, these aspects make MuSLR not just another multimodal symbolic task, but a structured benchmark for studying how vision-language models handle formal, compositional, and flexible reasoning in grounded, real-world contexts.\nWe will revise the manuscript to better articulate these differences and highlight the novelty of MuSLR’s focus and design.\n\n---\n\nThank you again, and please don't hesitate to reach out if you have any further questions."}, {"review_id": "evR02eTBB5", "reviewer": "Reviewer_CiCb", "summary": "This paper introduces Multimodal Symbolic Logical Reasoning (MuSLR), a novel task requiring Vision Language Models (VLMs) to perform formal logical reasoning by integrating visual and textual inputs. The authors create MuSLR-Bench, a benchmark dataset with 1,093 instances across 7 domains featuring reasoning depths from 2-9 steps, and propose LogiCAM, a modular framework that improves GPT-4.1's performance by 14.13%.", "questions": "1. Is it a typo in line 24 for the word \"text\"? Should it be \"visual\" or anything else?\n2. Seems that all the rules are first order logic rules right? Did you consider a higher-order case to represent a more natural real-world case?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts: I think this paper is really great and deserves a publication. The proposed benchmark addresses a key challenge that most previous works just focused on symbolic reasoning in a single text modality. The proposed method is also reasonable to me. Furthermore, the code is published for reproducibility. \n\nWeaknesses: Although authors have done a lot of experiments, a deeper analysis of the results will better help the community.", "was_revised": true, "final_justification": "The authors have provided further explanations of their experiment results. Although they didn't include higher-order logic into their benchmark, i think this work is good as an pioneer. I confirm my rating.", "rebuttal": "Thank you very much for your positive feedback and support. We're glad you find the benchmark meaningful and the method and reproducibility valuable. We will address your concerns below.\n\n---\n\n**W1: Deeper analysis of the results**\n\nRegarding your suggestion for deeper analysis, we plan to include additional analysis  in the revision to better support the community, including:\n\n1. Error breakdown by logic type (e.g., PL, FOL, NM), rather than by method alone, to better understand where models struggle across different logical structures (see Q1.3 response to reviewer L4xj in detail).\n\n2. Deeper analysis on why models perform differently on different logic types (see W3 response to reviewer PLuU in detail).\n\n3. Ablation study on LogiCAM with both quantitative and qualitative analysis to pinpoint the effectiveness of why each modular design works in the proposed method (see W1 response to reviewer L4xj in detail).\n\n4. Comparison with the LLM+solver paradigm with both quantitative and qualitative analysis to explain the limitations of LLM+solver in the MuSLR task and what we can do in the future (see W1 response to reviewer PLuU in detail).\n\nWe appreciate your constructive input and will ensure the revised version incorporates these enhancements.\n\n---\n\n**Q1: Regarding line 24 – possible typo for \"text\"**\n\nThis is not a typo. Our intent was to emphasize that most existing symbolic reasoning benchmarks are designed for a single modality, text, without integrating visual information. However, we agree that the current phrasing may be unclear, and we will revise that sentence in the final version to more explicitly convey this point.\n\n---\n\n**Q2: Are all the rules first-order logic? Have you considered higher-order logic?**\n\nWhile many of the rules used are from first-order logic (FOL), we also include reasoning patterns from propositional logic (PL) and non-monotonic logic (NM). Our choice of these three categories follows the precedent set by existing textual symbolic reasoning benchmarks (e.g., LogicBench [1], Multi-LogiEval [2]), and we extend them to the multimodal setting for the first time.\n\nRegarding higher-order logic (HOL), we agree it can represent more expressive and natural reasoning structures. However, we chose not to include higher-order logic in this initial benchmark for several reasons. First, HOL introduces significantly higher representational and computational complexity, which may overwhelm current vision-language models and make systematic benchmarking difficult. Second, grounding higher-order reasoning in real-world visual scenarios is much more ambiguous and difficult to verify, both for humans and machines. Besides, our goal is to provide a controlled, interpretable benchmark that can facilitate meaningful comparisons across methods. Including HOL would introduce another layer of abstraction that could obscure insights at this early stage.\n\nThat said, we see higher-order multimodal reasoning as a valuable future direction. A promising extension would involve structured tasks requiring reasoning over sets of entities, predicates over predicates, or temporal/event-based logic. We hope MuSLR can serve as a foundational benchmark to eventually support such higher-order capabilities.\n\nWe will clarify these design choices and propose HOL as a future direction in the revised manuscript. Thank you again for highlighting this point.\n\n[1] LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models. ACL 2024.\n\n[2] Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models. EMNLP 2024.\n\n---\n\nThank you again, and please don't hesitate to reach out if you have any further questions."}, {"review_id": "nyoi4ulDGA", "reviewer": "Reviewer_L4xj", "summary": "This paper proposes MuSLR, the first multimodal, multi-domain benchmark for VLMs to perform symbolic reasoning using both textual and visual information. MuSLR requires VLMs to either 1) decide the truth value of an argument based on the given information or 2) select the argument that best corresponds to the input.\n\nIn addition, this paper introduces the LogiCAM framework, which is a multi-round CoT procedure that guides in-depth reasoning iteratively. In each step, LogiCAM first attends to the most valuable information at the current step, then decides whether to apply formal symbolic reasoning or commonsense heuristics. Next, it performs reasoning and finally checks if the derived results are sufficient to answer the question.\n\nEmpirical evaluations show that MuSLR is a difficult benchmark even for current SOTA VLMs (e.g., GPT-4.1), which achieve an average accuracy of less than 50%. The proposed LogiCAM method significantly boosts model accuracy by around 14.13% on average, demonstrating its effectiveness in iteratively guiding VLMs to solve complex multimodal reasoning problems.", "questions": "1.  It is interesting that model performance varies significantly with different families of logic, since they are all used to describe the same scenarios and, therefore, should not produce significantly different results. IMHO this point is not thoroughly discussed in the current manuscript.\n    -   **Is the entire dataset within the expressive reach of PL?** Specifically, when constructing the dataset, were there any instances that could not be represented with PL alone, thus requiring the introduction of FOL to capture all necessary information?\n        -   If the entire dataset can be expressed within PL, it would be understandable for FOL to compromise model performance, as it introduces many redundant concepts and axioms.\n    -   From my understanding, Figure 8 (Error Distribution) is summarized regardless of logic type. **Could the authors provide some quantitative, or at least qualitative, comments on the error distribution for different logic types? From your perspective, why might these differences occur?**\n    -   It is worth noting that after applying LogiCAM, the performance divergence between different logic types is greatly reduced. In some cases, FOL even achieves the best results. **Could the authors briefly explain why this happens and what kinds of errors LogiCAM helps to avoid?**\n2.  Following up on weakness 1, **is there any way to formally or informally analyze the effectiveness of each module in LogiCAM?** This would make it easier for other researchers to make future improvements and reference the work. I understand that a complete ablation study may not be feasible, so any comments on this would be greatly appreciated.\n3.  **What is the relationship between LogiCAM and other CoT frameworks in prior art, for example, for visual or textual reasoning?** Is it a straightforward application of existing methods, or is it tailored for multimodal tasks?", "limitations": "The proposed MuSLR dataset is valuable; however, the choice of different logic types (PL, FOL, NM) seems somewhat arbitrary and may not be fully informative for downstream VLM evaluation.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1.  The proposed benchmark is highly relevant to VLM research and could serve as a valuable resource for VLM researchers seeking to improve the general reasoning abilities of VLMs, or for neural-symbolic researchers seeking benchmark tasks for multimodal models.\n2.  The proposed LogiCAM approach is intuitive and effective.\n3.  The writing is clear, and the paper contains a rich analysis of different aspects of the dataset and the CoT framework, making the paper informative for readers in related research areas.\n\n**Weaknesses**\n\n1.  The design of the proposed LogiCAM framework is largely heuristic. There is no clear ablation study to pinpoint the effectiveness of each module or to determine how it might be improved.\n2.  LogiCAM uses VLMs directly to perform reasoning. While this is a natural choice for heuristic commonsense reasoning, it is less reliable for formal symbolic reasoning. This approach leads to the \"Incorrect Application of Logical Rules\" errors reported in the paper. Given that a symbolic representation of facts arises naturally in the CoT process, the authors might consider incorporating a formal symbolic reasoner to aid this process and provide verifiable results.", "was_revised": true, "final_justification": "After reading the feedback from the other reviewers, I share some of their concerns regarding the novelty of LogiCAM in the broader context of the field. As I'm not very familiar with the LLM-solver paradigm, I will maintain my current score for now. I remain open to the opinions of the other reviewers on this matter.", "rebuttal": "Thank you for acknowledging the contributions of our work, the effectiveness of our approach, and the clarity of our writing. Please see below for our response to your concerns.\n\n---\n\n**W1: Ablation study for LogiCAM**\n\nThank you for raising this important point. We agree that understanding the contribution of each component in LogiCAM is essential for both interpretability and future extensibility.\n\nTo address this, we have now conducted an ablation study to evaluate the impact of the three core components in LogiCAM:\n\n| Model Variant         | Accuracy (%) |\n|-----------------------|--------------|\n| LogiCAM               | 61.34        |\n| w/o Symbolic          | 56.20        |\n| w/o Heuristic         | 57.89        |\n| w/o Premise selection | 58.07        |\n\nKey findings from our ablation study show that removing the symbolic module drops performance by over 5 points, omitting heuristic reasoning also hurts results, and disabling premise selection impairs focus and accuracy. These results confirm that LogiCAM’s design, where symbolic reasoning provides rigor, heuristics add flexibility, and premise selection boosts efficiency, works as a coherent, modular system. We will include this analysis in our revision.\n\n---\n\n**W2: The author might consider incorporating a formal symbolic reasoner**\n\nThank you for raising this important point. We fully agree that using VLMs directly for formal symbolic reasoning has limitations, given their probabilistic nature and lack of deterministic guarantees.\n\nWe did not include these the symbolic reasoner originally because they are designed for text-only tasks and cannot handle visual inputs directly. Adapting them to multimodal settings requires a VLM (e.g., GPT-4.1) to convert images into text, which can omit subtle or hard-to-verbalize visual cues.\n\nTo illustrate this, we adapted a typical LLM+solver approach, Logic-LM, using VLM (GPT4.1), to transform images into text and compared it with LogiCAM on PL and FOL (Logic-LM does not support NM):\n\n| Model             | PL (%)  | FOL (%) |\n|------------------|---------|---------|\n| Logic-LM + VLM   | 35.14%  | 32.65%  |\n| LogiCAM          | 60.44%  | 42.55%  |\n\nThese results support our view that translating visuals into text is not enough for effective symbolic reasoning. LogiCAM, built on VLMs, performs significantly better since it can directly access the image. However, we agree that LLM+solver approaches are important, and we propose a more integrated multimodal LLM+solver framework as future work.\n\nIn the revision, we will 1) Discuss challenges in adapting LLM+solver pipelines to multimodal symbolic reasoning tasks; 2) Cite additional relevant works (e.g., LINC, Logic-LM++, TIC, Divide-and-translate); 3) Expand the related work section (see also our response to Reviewer oQKt, Comment W1).\n\n---\n\n**Q1.1 Why does model performance vary across logic types despite describing the same scenarios**\n\nWe would like to first clarify that the scenarios are not exactly the same. Each instance in the benchmark includes a unique image and text. While a small subset may share similar abstract logical structures, this is not the norm.\n\nMoreover, the expressive power and syntactic complexity of the logic types differ, which can significantly impact model performance, even when the underlying semantics are comparable. For example, PL involves simple atomic statements, making it easier for models to match surface patterns. FOL introduces quantifiers and variable binding, which require structured reasoning and symbol tracking, areas where current VLMs often struggle. NM involves defaults and exceptions, which can sometimes align more closely with the heuristic or commonsense reasoning patterns of pretrained models.\n\nAs a result, even with similar scenario content, the reasoning demands and representational challenges vary across logic types, leading to performance differences.\n\n---\n\n**Q1.2: Is the entire dataset within the expressive reach of PL?**\n\nNo, the entire dataset cannot be fully expressed using PL alone. During dataset construction, we encountered many instances where quantification over objects or entities was essential. For example:\n\n● \"If all the people are standing, then the room is full.\"\n\n● \"There exists a vehicle approaching the crosswalk.\"\n\nThese naturally require first-order constructs, such as universal (∀) or existential (∃) quantifiers and predicate-argument structure, which are not available in PL. Thus, the inclusion of FOL is not redundant. It is required for the semantic coverage of real-world grounded scenarios in the dataset.\n\nWe will clarify this point in the revised manuscript and give a few representative examples in the appendix to illustrate cases that necessitate FOL.\n\n---\n\n**Q1.3: Can you provide error breakdown by logic type?**\n\nThank you for this suggestion. In the revised version, we will provide error distribution analysis broken down by logic type to complement the aggregate summary in Figure 8. This will include both quantitative differences in error rates and qualitative descriptions of typical failure cases for PL, FOL, and NM.\n\n| Logic | Visual Perception   | Logical Alignment | Heuristic Shortcuts  | Overlook Visual Details | Incorrect Logic | Commonsense Misgen. |\n|-------|------|-------|------|------|---------|---------|\n| PL    | 0.05 | 0.68  | 0.07 | 0.10 | 0.08    | 0.01    |\n| FOL   | 0.05 | 0.44  | 0.11 | 0.16 | 0.17    | 0.06    |\n| NM    | 0.07 | 0.79  | 0.02 | 0.07 | 0.05    | 0.00    |\n\nWe have the following observations:\n\n1. Consistent Alignment Issues Across All Logic Types:\n A major source of failure across PL, FOL, and NM is logical misalignment between text and image, particularly prominent in NM (79%) and PL (68%). This is consistent with our broader finding that mapping formal logical structures onto multimodal context remains a core challenge for current VLMs.\n\n2. FOL is the Most Prone to Overlooking and Logical Errors: Overlooking errors are highest in FOL (16%), suggesting the model frequently misses critical details, especially in scenarios requiring reasoning over multiple entities, nested structures, or quantifiers. Incorrect logical rule application is also most common in FOL (17%), which we attribute to its greater symbolic complexity. Unlike PL or NM, FOL involves quantifier binding, variable tracking, and relational reasoning, making it more error-prone for models with less symbolic rigor.\n\n3. PL Relies Heavily on Symbolic Alignment:\n While PL avoids many deep logic flaws, its performance heavily depends on correct logical text-image mapping (68% alignment errors). Once aligned, its simpler structure makes it easier for models to apply correct reasoning rules.\n\n4. NM Reflects High Alignment Difficulty but Few Logical Errors:\n Despite the highest alignment error rate (79%), NM exhibits the fewest incorrect logic application (5%) or commonsense supplement (0%) errors. This suggests that once alignment is successful, NM reasoning tends to align with the model’s intuitive understanding or default patterns, which may explain its relatively good raw performance.\n\n---\n\n**Q1.4: Why does LogiCAM reduce the performance gap between logic types?**\n\nLogiCAM narrows the performance gap by imposing a unified, scaffolded reasoning workflow on every problem, regardless of logic type. Instead of allowing the model to rely on its uneven internal heuristics, it breaks each task into the same sequence of steps, identifying premises, selecting rules, instantiating variables, handling exceptions or defaults, and drawing a conclusion. This consistent template forces explicit variable binding in FOL, eliminating scope errors; isolates defaults and overrides in NM, preventing overgeneralization; and mandates multi‑step decomposition even for simple PL tasks, deterring one‑shot pattern matching. By channeling PL, FOL, and NM through the same structured pipeline, LogiCAM standardizes the model’s reasoning process, reducing variance and closing the performance gap between logic types.\n\nWe will revise our manuscript to include this analysis.\n\n---\n\n**Q2: Ablation Study**\n\nPlease see details in W1.\n\n---\n\n**Q3: Is LogiCAM a straightforward application or tailorder for multimodal tasks?**\n\nLogiCAM builds on Chain‑of‑Thought prompting but is specially adapted for multimodal symbolic reasoning. It grounds each step in both visual and textual context, integrates formal inference rules (e.g., Modus Ponens, exception handling) alongside heuristics, and uses logic‑aware prompt templates to enforce structured rule application. These adaptations go well beyond standard CoT methods.\n\n---\n\n**Response to Limitations**\n\nFirst, we would like to clarify that the choices of PL, FOL, and NM are in line with text-only symbolic reasoning benchmarks such as LogicBench [1] and Multi‑LogiEval [2] to ensure consistency and comparability. While other logic types like Higher‑Order Logic (HOL) offer greater expressiveness, they bring representational, computational, and annotation challenges beyond current VLM capabilities. Focusing on PL, FOL, and NM lets us deliver a controlled, interpretable benchmark; we plan to explore HOL in future iterations once baseline performance is better understood. We will clarify this rationale in our revision.\n\n[1] LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models. ACL 2024.\n\n[2] Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models. EMNLP 2024.\n\n---\n\nThank you again, and please don't hesitate to reach out if you have any further questions."}]

Author rebuttal:

Meta-review: This paper introduced an important benchmark for multimodal symbolic logical reasoning. Four reviewers unanimously agree with that this paper made significant contributions and recommend an acceptance. After a careful check, the AC agrees with the reviewers and recommend an acceptance.
Decision: Accept (poster)