NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
Authors: Danial Kamali
Parisa Kordjamshidi
Venue: ICLR 2026 Poster (2026)
Keywords: Neuro-Symbolic
Vision and Language
Compositional Reasoning
Primary area: neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)

Abstract: Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer a promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, a neuro-symbolic framework that overcomes these limitations through a hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning over VLM-generated uncertainty. Operating in a training-free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable composition operations support fine-tuning. We evaluate NePTune on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.

Published as a conference paper at ICLR 2026
A NEURO-PYTHONIC FRAMEWORK FOR TUNABLE COMPOSITIONAL REASONING ON VISION-LANGUAGE
NEPTUNE:
Danial Kamali
Parisa Kordjamshidi
Michigan State University
fkamalida, kordjams 3 2msu. edu
ABSTRACT
Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, neuro-symbolic framework that overcomes these limitations through hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning uncertainty. Operating in training- free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable composition operations support fine-tuning. evaluate NePTunee on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.
VLM-generated
Query:
Compositional Reasoning
is there Brrown object Behing every sahere?
belvnd
Composer Eilenwed
Decompaser
Scortng
Figure A natural language query ("Is there a brown object behind every red sphere?") is decomposed into symbolic concepts, such as red, and sphere. These concepts are then composed to enable explicit reasoning over objects and their relations. This illustrates how complex queries can be mapped into structured logical forms.
INTRODUCTION
One key aspect of intelligence is the ability to compose known basic components to generalize and solve novel, complex problems. In vision-language reasoning, this capacity for compositional generalization is crucial. Humans easily decompose complex query like "Is there brown object behind every red sphere?" into its constituent concepts and compose them in order to reason about their relationships (Partee et al. 1984), illustrated in Figure However, modern vision-language models with transformer based architectures often fail dealing with novel compositions, revealing gap between their pattern-matching skill and robust, human-like understanding (Zhu et al. 2022

---

Published as a conference paper at ICLR 2026
Recent studies have demonstrated the superiority of neuro-symbolic (NeSy) methods monolithic neural architectures for reasoning beyond their observed data (Kamali et al. 2025 Yun et al. 2023). In recent years, methods such VisProg (Gupta & Kembhavi 2023 and ViperGPPT Surís et al., 2023) have leveraged Large Language Models (LLLMs) to produce programs from language queries, breaking down complex questions into sequence of executable steps. These programs are then run using pre-trained vision models for perceptual grounding. Although powerful, this paradigm has key shortcomings. First, these systems often rely on a sequence of crisp, intermediate decisions, creating brittle pipelines that are sensitive to perceptual errors. A single mistake in an early step (e.g., misidentifying an object) can cause the entire reasoning chain to fail. Second, these pipelines are inference-only and non-differentiable which prevents them from adapting to new domains where pre-trained models may not perform well. On the other hand, methods such as LEFT (Hsu et al. 2024) and NeSyCoCo (Kamali et al. (025) employ differentiable declarative symbolic reasoning, but their predicates are limited to the set of learned predicates from training on the target domain, which limits their zero-shot applicability. Table summarizes this landscape and highlights where NePTune differs.
Table I: Comparison of recent neuro-symbolic frameworks. We analyze the reasoning scope (local global), types of supported predicates, and predicate source. VFM: vision foundation models, e.g. XVLM; VLM: vision-language models, e.g. Qwen2VI.
Predicate Types
Predicates
Model
Reasoning
Supported Predicates
Class
Attr.
Rel.
Spatial
Pretrained
Trainable
VisProg
Local
Predefined Modules
x
Modules
x
ViperGPP
Local
Dynamic
VFMs
LEFT
Global
Limited to Training Data
NeSyCoCo
Limited Similar Data Training
NAVER
Global
Dynamic
VFMs
X
NePTume
Hybrid
Dynamic
VLMS
To address the challenges of expressive compositional inference and domain adaptation, we propose NePTune, a neuro-symbolic visual reasoning framework. NePTune leverages LLM to generate expressive Python programs where its exccution does not solely rely on crisp decisions. Instead, it performs both soft compositional reasoning the continuous scores provided by VLM under uncertainty and imperative sequential reasoning. The result is framework that decouples reasoning from the perception of atomic concepts, leading remarkable generalization.
Contributions.
A Hybrid Neuro-Symbolic Execution Model: We propose a novel framework that seamlessly combines the imperative control flow of Python with soft compositional logic, enabling complex reasoning under uncertainty.
2. Domain Adaptable Framework: We present modular system that uses an LLM to generate programs on the fly, eliminating the need for predefined predicates and enabling zero-shot generalization well neuro-symbolic fine-tuning for domain adaptation.
3. Strong Compositional Generalization: We demonstrate through extensive experiments that our approach significantly outperforms existing methods, particularly challenging domain-shift benchmarks, showing highly robust results for vision-language reasoning.
RELATED WORK
NePTune framework that integrates multi-moda. foundation models and logical reasoning to achieve compositional vision- languae reasoning. Therefore, we focus on the three topics below.
2.1
COMPOSITIONAL VISION-LANGUACA REASONING
Compositional reasoning is central to building reliable vision- -ision-language systems Sinha et al.
2024 Hupkes al. 2020), enabling models represent and understand multi-step relational
structures (Ontañón et al. 2021). l). While VLMs achieve strong results across broad tasks (Xiao al. 2023 Wang et al. 2024 Liu al. 2024), diagnostic evaluations like CLEVR Johnson al.
2

---

Published as a conference paper at ICLR 2026
2017a) and ReaSCAN et. 2021) expose persistent gaps in compositional generalization (Zhu
et al. 2022) and brittleness under distribution shift (Yun et al.]
[023]
Li et al. 2025) in pure
neural methods. To address this challenge, methods such meta lerminn (Xu et al. (023). soft
prompting (uu et al. 2024), modified neural architectures Qiu et al. (202))) :; Kamali & Kordjamshidi
(202)), and symbolic reasoning (Hsu al. 2024; Kamali et al. 2025) have been explored.
Among these methods, neuro-symbolic methods have been shown to be effective for compositional challenges (Kamali al. (025). Motivated by these findings, we adopt a neuro-symbolic approach
that performs explicit compositional reasoning to achieve more robust and generalizable reasoning.
2.2 NEURO-SYMBOLIC APPROACHES
Neuro-symbolic methods integrate neural perception with symbolic reasoning. Recent approaches utilize LLMS semantic parsers interpret linguistic queries (Gupta & Kembhavi 2023 Faghihi iet al. 2024). There are two common strategies for such methods. One line of work, including VisProg (uupta & Kembhavi, 2023) and ViperGPT (Surís et al. 2023), generates imperative code for a sequence of local reasoning steps that rely on crisp decisions. While powerful, the sequential nature of these methods limits global reasoning under the uncertainty of these local decisions. The second strategy focuses on generating a declarative, symbolic representation of the query for robust symbolic reasoning. This line of work, as developed NS- VQA (Yi et al., 2018), maps questions to functional programs executed over a structured scene representation in CLEVR-style synthetic worlds, using a fixed, sataset-specific inventory of attributes and relations and separately trained concept modules and semantic parser. NS-CL (Mao al. 2019) follows the same program-execution paradigm, but is trained end-to-end from question answer supervision without annotated programs attribute labels. LEFT (Hsu al. (202) extends this declarative approach with L.M--generted first-order logic programs and a differentiable executor, but still relies on domain-speciiic training data for concept learning and grounding,
its performance is tied to the concepts seen during training. NeSyCoCo (Kamali et al.) 2025) improves lexical coverage with distributed predicate representations, but its generalization remains limited to concepts semantically close to those in the training distribution. NAVER (Cai et al.) 2025). on the other hand, emphasizes orchestration through a finite-state controller that manages neural perception and utilizes ProbLog for global reasoning in referring expression grounding. However, our findings indicate that generating ProbLog queries more error-prone for LLMs compared to Python code. Our aim in this work is to synthesize the advantages of both approaches. NePTune is neuro-symbolic framework that performs both imperative and soft compositional reasoning over atomic predicates. Unlike works like NeSyCoCo and LEFT, the training not necessary since the concept scores can be obtained zero-shot by harnessing the power of VL.Ms. However, with differentiable composition functions, it offers optional fine-tuning for domain adaptability by providing the capacity for both compositional inference as well as neuro-symbolic training.
2.3 PREDICATE LEVEL CONCEPT UNDERSTANDING
A critical distinction between reasoning frameworks lies in how they derive and utilize concept-level understanding. General- purppss VLMs (Wang et al. 2024 Liu al. 2024) learn concepts implicitly through end-to-end training, embedding this knowledge directly into the model's weights. In contrast, neuro-symbolic methods externalize this process. program generating framework, such as VisProg, utilizes the concept grounding through a library of specialized, trained, hard-coded vision APIs, which return discrete labels or values. Similarly, ViperGPT utilizes foundation lasion-language models, such as XVLM (Zeng et al. 2022). to obtain labels and discrete decisions. Another class of methods, including LEFT and NeSyCoCo, trains an MLP to generate continuous concept scores, representing the relations or attributes of an object in the visual regions. Our approach utilizes visual prompting, highlighting parts of the image via bounding boxes, elicit referential concept scores from VLMs, which then serve as scores within our framework for reasoning under uncertainty.
3 METHODOLOGG
The NePTune framework performs visual reasoning via three core components, as shown in Figure The process begins with the I.LM-based Program Generator, which translates natural
3

---

Published as a conference paper at ICLR 2026
language query into both Python program and set of relevant object names. These names passed to the Perceptual Grounding to detect all relevant candidate objects in the scene. Finally, the Symbolic Executor runs the generated program. During execution, interacts with the Grounding Interface to obtain atomic concept scores and reason over them to get the final answer.
LLMi-based Frogram Generation
ctjec the anadit
How many solors small spheres are there?
Large Language Model
Prog
Answer:
queryrhhor cscerit the the hrundingheeh
(3)
marcadkniion
Symbolic Executor
Ded maaee Reswiis
Perceptual Grounding
(olck..)))
Object Detection
Vision-Language Model
Aoes barry
Figure 2: NePTune overview. Given image and query, the (I) LLM-based Program
Generation converts the natural language query to a Pythonic program. Then (2) Perceptual Grounding extracts the object bounding boxes. The (3) Symbolic Executor then runs the Python code to reason over (concepts extracted from the VLM using both ssoft composition and
imperative logic to derive the final answer.
3.1 COMPONENT 1: LLM-BASED PROGRAM GENERATION
The first component of NePTune addresses semantic parsing. We leverage an LLM as a powerful few-shot parser to convert a natural language query into formal, executable Python program. Given a query like "Is there a big brown dog?". the LLM prompted decompose it into multi-step program that first identifies set of candidate "dogs", and then reasons over the composition of atomic concepts such as "big", "dog", and "brown" get the final answer. the same L.LM call, also extracts object names for the region proposals. We chose Python as the target symbolic language for two key reasons. First, its Turing-cone cornng-complete nature, including rich control flow structures like loops and conditionals, is essential for expressing complex procedural reasoning. Second, the prevalence open-source Python code makes LLMs particularly adept at generating it.
3.2 COMPONENT 2: PERCEPTUAL GROUNDING
This component connects the symbolic program the visual world. It consists of two main parts: an object proposal module and concept grounding module.
3.2. OBJECT PROPOSAL GENERATION
To ground objects in the scene to the concepts such as blue in the program, we must first identify all potentially relevant objects in the scene. We use the object names extracted by the LLM. For example, from "Are there more cats to the left of the vase or dogs?", the LLM extracts "dog", "vvss", and "cat". We then feed these object names into Grounding DINO (Liu et al. 2023). zero-shot object detection model that takes the image and the list of object names as input and outputs a set of bounding boxes for all matching objects. We note that while this textual query-based proposal is chosen for computational efficiency real-world applications, our framework can operate with any reasonable region proposal method.
3.2.2 CONCEPT GROUNDING INTERFACE
Once query is translated into program, must ground its atomic predicates to the visual content of the image, such as the objects in the scene. We handle this perceptual grounding using a VLM through two primary simplified interfaces score and query, where the image and bounding boxes are abstracted away with global variables for simplicity. The formal definitions these grounding functions are follows:
4

---

Published as a conference paper at ICLR 2026
score (query: num.objects:ingt This function is the core of reasoner. It takes a natural language question, an image, and related bounding boxes, and returns the concept probabilities for related object bounding boxes. This score is the VLM's normalized confidence for a "Yes" answer, computed from the logits of the "Yes" and "No" tokens. Formally, given an image 7, set of N detected objects, visual prompt e, and atomic query Ga. the score is:
clogn("Yec")
= p("
elogi(("Yss" elogu("No")
We treat these outputs as probability scores. To generate an object-centric concept score or answer via a VLM, we represent a symbolic predicate with natural language question. For example, for the symbolic predicate blue, we use "Is the object in the red bounding box blue?". The bounding box b provided to the functions will be drawn on target objects visual prompt, and it is controlled by the num_objects parameter, which specifies the predicate type:
num objects= For queries about the entire image (e.g.. "Is the photo taken indoors?"). No extra visual prompt is used, and the output is a scalar probability.
num num objects=1: For single-object queries, we mark the target object with a red bounding box around it as extra visual prompt (e.g.. "Is the object in the red bounding box brown?"). The output is a vector of size N.
num_ aam.obbectss=2. For multi-object (relational) queries, we mark the main object with red bounding box, and the secondary object with green bounding box, as the extra visual prompt (e.g.. "Iy the person the red box talking to the person in the green box?"). The output is an N x N matrix.
query (query: str, object : This query function is used for tasks requiring open-ended answers. It takes natural language question, an image, and an optional visual prompt, and returns a natural language string. Formally, given an image I, visual prompt v, and atomic query ya the generated answer à is:
à = arg max maxp(a p(a (,v,ga).
aeA
The answer à serves two purposes. First, it can provide the final output for questions that ask "whar" or "which" (e.g., 'return query("Whal color is the object the red bounding box?")").. Second, use query to retrieve symbolic attributes as intermediate variables inside programs. For example, count how many distinct shapes appear in scene, the program first ask for the shape of each object and then reason over the set of answers:
unique unique_shapes seaapsshpess_seeee set ()
for i in object_ids:
shape_i query ("What shape is object in red bounding box?", i) unriquesshapss.. add (shape_i) num aum_unique_shaal shapes = len (unique_shapes)
Here query returns short textual answers (typically single token such as "cube" "sphere"), which are then composed using standard Python control flow and data structures.
NePTune builds directly on top of VLMs that have been pretrained in an end-to-end fashion; our contribution is orthogonal to them and only changes how these backbones are used by exposing them as atomic concept scorers inside a symbolic executor instead of direct prompting answerers.
3.3 COMPONENT'S 3: SYMBOLIC EXECUTOR
The final component of NePTune is the Symbolic Executor, which runs the LLM--geeraad program. A key innovation of our framework is its hybrid execution model, which integrates two distinct reasoning modes.
To reason about visual concepts themselves, we employ a set of soft logical operations based fuzzy logic principles. Instead of operating binary true/false values, these operations work directly on the uncertainty scores obtained from the VLM. This is implemented through our custom data structures, which encapsulate the scores for given predicate
1) Soft Compositional Reasoning:
5

---

Published as a conference paper at ICLR 2026
and overload standard Python operators (& for AND, for OR) to perform the corresponding logical operations. Details of these operations are shown in Table For example, when the program executes brown & dog. our framework takes the element wise minimum of the scores in the two corresponding tensors, implementing the fuzzy t-norm for conjunction.
Table 2: Soft compositional operators used in symbolic execution. Here. a represents an object-centric ID score vector, / relational kD (k 2 2) score matrix, = 0.25 is temperature parameter, ? = 0.25 a margin. We report default operator modes and training- formulations.
Syntax
Logical Form
Description
Differentiable Implementation
ex.existss
O2 Vir
Existential quantification
max(ox)
forfarlll))
Universal quantification
min(a.)
Og &
0z A Og
Logical conjunction
minica Og)
og &
dx Bay
Relational conjunction
min( Bey)
0x los
Og
Logical disjunction
max maxx
Og (m.imllss(()))
cxyy + ciyy -ay (var, a))
Logical implication
not Og
Logical negation
ao.iota(yar) Og count())
Best-match
count(a.)
Count
2 81 82
82
51 $2
Equality
2nT
2
2
Inequality
By
2) Imperative Reasoning:
Our symbolic executor leverages standard Python interpreter to handle the program's 's overall structure and control flow. This is possible since we have defined iteration and Boolean operations on the concept objects, allowing for complex, procedural logic, including conditionals (iflelse), loops (for), and variable assignments, giving our framework the full expressive power of a general-purpose programming language. While some of these operations such as counting over string sets (as shown in Figure 2) can break the computation graph. This hybrid design enables the system to reason fluidly under uncertainty while fully leveraging the expressive power of general- general-purpose purpose programming language.
EXPERIMENTS
We structure our experiments around four research questions. RQ1: How does NePTune perform on zero-shot compositional reasoning on synthetic data? (Experiment 1), RQ2: How does NePTune perform on complex human-generaaed questions? (Experiment 2). RQ3: How does NePTune perform in grounding referring expressions in natural images? (Experiment 3), and RQ4: How does NePTunee perform under domain shift and adapt to an unseen environment? (Experiment 4).
4.1 EXPERIMENTAL SETTING
We evaluate NePTune on four benchmark groups to cover diverse reasoning aspects. First, to assess zero-shot compositional reasoning on synthetic data, we employ CLEVR (Johnson et al.) 2017a) and its extensions (Hsu al. 2024) (Ref, Puzzles, and RPM), comparing NePTune against monolithic VLMs (Qwen2VL. (Wang et al. 2024,, Intern (Chen et al. 2025), Ovis1.6 (Lu et al. )) and symbolic methods such as LEFT. NeSyCoC,, and ViperGPT.. Second, to evaluate (Johnson et al.)
performance on complex human-genme generated questions, we use the CLEVR-
2017b) dataset and compare against similar baseline synthetic CLEVR. We also include the trained neural model MDETR (Kamath et al., 202T) to compare NePTune against neural methods under linguistic diversity. Going beyond VQA, we evaluate NePTune on the task of grounding of referring expressions in natural images, we use SErcooO-Adveesartti (Akula et al. 2020) (Ref-Adv), with baselines including direct VL. grounding, specialized systems such as Grounding DINO (Liu et al.) 2023) and Florence2 (Xiao et al.) (023), ViperGPT, and NAVER. Finally, to measure performance under domain shift and the ability to adapt to unseen environments, we use the photo-realistic gaming dataset Ref-GTA (Tanaka et al. (2019), comparing NePTune against both zero-shot and ine-tuned backbone VLMs, specialized grounding models, ViperGPT, and NAVER. gains these backbones across benchmarks, provides the details on all the datasets and preprocessing. The code is publicly available at httpt:::s/.......... / / wwg.ggttuuu...oo com / /LR/ /.../N.NNNTeeee.
Table high-level summary of NePTune's 's and Appendix D

---

Published as a conference paper at ICLR 2026
4.2 EXPERIMENT 1: CORE COMPOSITIONAL REASONING
Table Accuracy comparison across CLEVR question categories. Results are shown by reasoning type and model paradigm, including zero-shot, neural, and neuro-symbolic approaches.
Intern V1.2.5
NePTune
ViperGPT
NeSyCoCo
LEFT
Training
Zero-Shot Neural
Zero-Shot NeSy
Zero-Shot NeSy
Trained NeSy
Trained NeSy
Category
Final Accuracy
90.25
92.68 (1 2.40)
36,05
99.68
99.50
Exist
87.10
93.19 ( 6.09)
48.75
99.28
98.92
Query Attribute
98.26
96.81 (6 1.4))
29.42
100.00
99.86
Compare Attribute
98.61
91.94 6.67)
53.06
99.45 44
99.72
Count
74.60
87.10 (t 12:50)
21.37
99.79
98.99
Compare Number
90.86
92.57 (1 1.71)
48.57
100.00
100.00
To evaluate NePTune's compositional reasoning capabilities (RQ1), evaluate its performance on the CLEVR benchmark. CLEVR is a standard benchmark for compositional reasoning that features synthetic 3D-rendered images and questions testing compositional visual reasoning. Details of the question categories are provided in Appendix [D] The results in Table ( show that, within the family of neuro-symbolic systems, NePTune establishes itself the strongest zero-shot method, achieving 92.65% accuracy compared to only 36.05% for ViperGPT. While trained approaches such as NeSyCoCo and LEFT nearly saturate CLEVR (99%), NePTune demonstrates that competitive compositional reasoning can be achieved without dataset-specific supervision. Compared to its backbone VLM, InternVL2.5. NePTune still yields improvements raising overall accuracy from 90.25% to 92.65% (1 2.40%). The largest gains appear in quantitative categories where explicit compositional structure is most useful: Count rises from 74.60% to 87. 10% (f 12.50%), and Compare Number increases from 90.86% to 92.57% (1 1.71%). We also see notable improvement on Exist from 87.10% to 93.19% ( 6.09%). consistent with the executor reducing spurious correlation when filtering by attributes and relations. In contrast, attribute-heav categories regress: Query Attribute drops to 96.81% 81% (1 1.45%) points and Compare Attribute drops to 91 91.94% (L 6.67%) points. A closer look at concept-level accuracy revealed that analogical concepts such same color or same shape remain among the most challenging to capture in this benchmark. More discussion of atomic concept evaluation provided in Section 5
Table 4: Accuracy on CLEVR extension tasks. Methods marked with use ground-truth programs. Improvements from the backbone VLM (Intem are marked with the arrow sign (1).
Method
Refi %)
Puzzles(%
RPM(%)
Trained
NeSyCoCol
100.00
95.00
100,00
NeSyCoCo
94.00
94.00
74.00
LEFT
94.00
85.00
87.00
Qwen2VL-7B
21.00
43.00
53,00
InternVL2.5-8B
27.00
52.00
47.00
Zero-shot
Ovis1.6-9B
4.00
47.00
49.00
ViperGPT
8.00
34.00
4.00
NePTunel
99.00 72)
85.00
33)
97.00
50)
NePTune 91.00 64)
81.00 29)
80.00
33)
In addition to the original CLEVR benchmark, we evaluate method on various compositional challenges based on the CLEVR environment introduced in LEFT, including referring expressions (CLEVR-Ref), visual puzzles (CLEVR-Puzzles), and Raven's Progressive Matrices (CLEVR-RPM). As shown in Table 4 NePTune shows the highest performance among the zero-shot methods and outperforms the direct prompting VLMs. In addition, compared to NeSyCoCo and LEFT with trained concepts, we show competitive performance.
4.3 EXPERIMENT 2: COMPLEX HUMAN QUERIES
To evaluate NePTune on more complex and diverse human-generated language (RQ2), we utilize the CLEvR-Humans Johnson et al.) 2017b) benchmark to compare NePTune against other neuro-symbolic and pure neural models. As shown Table 5
7

---

Published as a conference paper at ICLR 2026
NePTune significantly outperforms prior declarative neuro-symbolic methods such as LEFT and NeSyCoCo by a large margin of f 31.55% and performs better than the MDETR trained neural method. It also surpasses the imperative ViperGPT. Furthermore, improves upon its powerful monolithic backbone (IntemVV.2.5588B) by t 1.72%, illustrating its effectiveness on complex, human-generated questions. In addition, to demonstrate the added value of NePTune's expressivity, we conducted an experiment with NeSyCoCo + VLM, which leverages the same Intern VL2.5 as the concept-sco ring module. While this experiment yields t 12.36% improvement over NeSyCoCo, it leaves a large gap behind NePTune. This gap indicates the advantage of our expressivity gained by our method.
Table Aceuracy on the CLeaR-Humans (CH).
Method
(%)
LEFT NcSyCoCo
56,69
56,12
MDETR
Qwen2VL-7T
81.12
InterVVL2.S43
85.95
Ovis1.6-B
79.96
VigerGPT
31.05
NeSyCoCo
NePTone
87.67
4.4 EXPERIMENT 3: REAL-WORLD IMAGES
While various versions of CLEVR used in the aforementioned experiments are strong testbeds for compositional reasoning, after all, these are toy environments with limited diversity. To demonstrate NePTune's ability operate in realistic environments (RQ3), we evaluate it on Referring Expression Grounding (RFG) of real-world images using the RefCOCO-AAvvesariia (Akula) let al.] (020) (Ref-Adv) benchmark. On Ref-Adv, NePTune is the strongest zero-shot reasoner among symbolic baselines such as ViperGPT and NAVER. To ensure fair comparison with NAVER, we follow setup and use the same backbones (Grounding DINO XVLM Intern VL.2-8B), denoted as NePTunet. Under this setting, NePTuun reaches 63.71 vs. 36.45 for NAVER (execution only), and with a simple verification (similar to NAVER) attains 75.54 vs. 65.13 for NAVER. Compared to pure neural methods, NePTTune surpasses all the specialized grounding methods, such Grounding DINO and Florence2 (Xiao et al.) 2023). Additionally, compared their VLM backbones, both NePTune and NePTune* with verification surpass their respective backbones. Details on verification are available in Appendix
Table 6: Ref-Adv dataset.
Accuracy on
Method
Ref-Adv (%)
Grounding DINO-B Florence2-L
60.85
71.73
Ovisl 66
58.47
InternVL.2-8B
72.92
Intern VL2.5-88
76.13
ViperGPT
60.66
NAVER
36.45
NAVER
65.13
NePTune
63.71
+ Verification
75.54
NePTune
71.57
+ Verification
78.08
NePTune (IB)
60.69
Fine-tuning
68.06
0.56
Verification
74.59 0.12
4.5 EXPERIMENT 4: GENERALIZATION AND ADAPTATION
When using popular benchmarks, we did in Experiments 1-3, the possible data contamination makes zero-shot performance less reliable. To answer RQ4, we test NePTune's ability to generalize and adapt to novel environments using a less popular photo-realistic gaming environment that is Ref-GTA (Tanaka et al. 2019) benchmark. Ref-GTA a challenging REG benchmark with images from game simulation, which creates significant domain shift from the natural images on which most VLMs are pre-trained. Based
Table 7:
Accuracy on
Ref-GTA benchmark.
Method
Ref-GTA(%
GroundingDINO-B
27.90
Florence2-L
Ovis1.6-9p
2.78
Intern VL2.5-8B
6.95
Intern VL2.5-1B
1.64
the reports of 0 and In2ern this source is not included in their training data. As shown in Table 7 this domain shift causes significant performance drop for most methods. The powerful pure neural Intern VL225-8BB for example, fails catastrophically. with its accuracy dropping to 6.95%. In contrast, NePTune, when paired with the same VLM, achieves a remarkable 69.69% accuracy. This demonstrates that by utilizing our global symbolic reasoner and atomic-level concept understanding, our framework achieves robustness and compositional generalization that monolithic models lack. Furthermore, while NePTune demonstrates strong zero-shot robustness, its soft operations also enable fine-tuning. As shown in Table 7 by fine-tuning a smaller VLM (Intern VirrmLL2.3 VL2.5- using we can further improve its performance and obtain 69% accuracy. While fine-tuning VLM using original neural training, it only reaches 32% accuracy. These findings are promising for future
+ Fine-tuning
32.61 +: 0.35
ViperGPT
1,40
NAVER!
54.84
NAVER
58.73
NcPTune!
62.73
NePTune
69.69
NePTune (1B)
34.92
+ Fine-tuning
1.16
neuro-symbolic differentiable computations with only 1000 samples,
8

---

Published as a conference paper at ICLR 2026
research and using neuro-symbolic reasoning source of supervision for larger-scale VLM training. More details on fine-tuning are presented in Appendix F
DISCUSSION
VLM for Concept Understanding. central idea of NePTune is employ VLMs as underlying perception modules and concept grounders. The basic assumption here is that while VLMs are prone to fail at reasoning over complex compositions, they should perform better perceiving basic concepts. However, basic perception questions require isolating parts of the visual input, a process known as visual prompting, which has been shown to be challenging in complex questions (Cai al. 2024). In this section, aim to evaluate the hypothesis that modern VLMs guided by visual prompts are effective as concept grounders for our purpose. We utilize ground-truth scene graphs from the CLEVR (Johnson al. (017a) and Visual Genome (Krishna al.] 2016) datasets to automatically generate a set of atomic templated questions. For CLEVR, we create benchmark using scene graphs of 200 sampled scenes, and for Visual Genome, we randomly sample 1000 questions from 200 scene graphs. We generate simple Yes or No questions about their class, attributes, and relations, such as, "Is the object in the bounding box a bird?". We then use our score function that employs the underlying VLM to answer these atomic questions.
Table 8: Backbone VLMs performance on CLEVR and VG scene graphs (Micro F1-Score 100)
Category
Intern Vl2.--L2..588
Ovis1.6-9B
Qwen2VL-7B
CLEVR
VG
CLEVR
VG
CLEVR
VG
Attribute
97.25
89.37
95.87
86.75
89.27
83.45
Class/Object
90.00
81.76
91.98
80.42
82.21
83.28
Relation
90.94
82.35
87.91
80.00
80.04
83.02
Spatial 89.82
93.42
86.83
93.39
89.53
94.41
Overall 94.54
90.41
90.35
88.19
86.59
90.77
The results shown in Table g Our analysis reveals that VLMs are particularly strong identifying objects, classes, properties, and spatial relations. However, the performance is weaker on more complex relational concepts, especially for analogical comparisons such same size. This problem is more severe in the Qwen2VL. model. Although visual prompting can be problematic for general VQA (Cai et al., 2024). our results show this not as challenging in our setting for grounding atomic queries. These results demonstrate that VLMs perform significantly better in answering atomic queries compared complex multi-step queries, reported in Tables 4nd For example, there is performance gap of up to 67% when comparing CLEVR- (27%) compared to CLEVR average atomic evaluation (94%) using Intern Detailed concept-level results of the experiment are presented in the Appendix B Also, the results in Table 9 show the performance of different VLMs NePTune, where they show an average improvement up to 1 32%.
Table 9: Performance of different VLM baselines with direct prompting compared to NePTune across benchmarks. A values indicate gains or losses relative to the raw VLMs.
Model
CLEVR (%)
Ref-Adv (5)
Ref-GTA (%)
Ref (%)
Puzzles (%)
RPM (%)
Avg. (%)
Intem VL.2.588
90.25
76.13
6,95
47.00
NePTune
92,65 2.40
78.08 1.95
69.69 1 62.74
91.00 h 64.00
81.00 19.00
80.00 33.00
82.07 (1.32.18)
Ovis1.6-9B
85.00
58.47
2.78
4.00
47.00
49.00
41.04
NNPPTune
88.80 t 3.80
63.25 4.78
56.32
53.54
85.00 81.00
69.00
22.00
80.00
31.00
73.73 (1 32.69)
Qweu2VL-7B
93.55
81.07
1.75
21.00
43,00
$3,00
48.90
NeFPunee
82.55 11.00
80.93 1 0.14
68.87 1 67.12
81.00 h 60.00
65.00
22.00
71.00 f 18.00
74.89 (1 25.99)
Choice of Symbolic Program. We analyze program execution success rates (no syntax or runtime errors) 500 Ref-Adv and CLeaR- using the GPT-4o (OpenAI-Team) 2024) LLM. As shown in Table 10] frameworks that rely on specialized, non-Pythonic syntax, such LEFT, NeSyCoCo, and NAVER (which uses

---

Published as a conference paper at ICLR 2026
ProbLog), frequently suffer from generation failures. struggles to correctly translate natural language into In contrast, methods that utilize Python, such ViperGPT and NePTune, demonstrate significantly higher rates of successful program execution. However, our analysis of ViperGPT's failures reveals that its purely imperative approach, which often relies on selecting objects by fixed index, is a primary source of error. The NePTune hybrid reasoner proves be the most robust. Our analysis shows that the main source of error for NePTune shifts from low-level syntax errors to higher-level logical errors, where the LLM fails to compose the correct sequence of predicates. Qualitative examples are shown in Appendix A
We found that the LLM often their specific, strict formalisms.
Table 10: Execution success rate (%) of different neuro-symbolic methods on Ref-Adv and CLEVR-Humans.
Method
Ref-Adv
CLEVR
NeSyCoCo
N/A
70.3311.47
LEFT
N/A
64.33++3.4
ViperGPT NAVER
48.67=1.77
95,42:10.30
23.02=4.71
INAA
NePTune
0..82
97.24+0.85
To quantify the correctness of NePTune's programs, we conducted a human evaluation of 600 generated programs across the Ref-Adv, Ref-GTA, and CLEVR- datasets (see Appendix The results indicate that syntax errors are rare in the generated Python programs. Instead, most failures arise from higher-level logical composition mismatches rather than low-level execution issues. On real-world referring expression datasets, the dominant errors occur when atomic prompts fail to preserve plural or group-level concepts from the original natural language query. On the CLEVR-Humane VQA dataset, the few observed errors primarily stem from incorrect application of normalization operators such as iota. Overall, evaluators judged at least 87% of the programs to be logically sound, indicating that NePTune not only mitigates brittle execution failures but also that its first-order logic interface facilitates the generation of logically sound reasoning programs.
Latency and Computational Efficiency. The primary computational cost of NePTune arises from the calls to the VLM for concept grounding, which scales (RRN2) for scene with N candidate objects and R binary relations. To mitigate this overhead, we employed batched prefix caching with the vLLM (Kwon al. (023) tool. Since image tokens and most textual prefixes are shared across atomic queries, this caching mechanism eliminates redundant computations. As detailed in Appendix G this optimization reduces the average execution time on the Ref-Adv dataset. While long tail of highly complex, multi-object queries exists that increases the overall average, the majority of queries are resolved in under 5s for the 8B model and under 2s for the IB model. This demonstrates that NePTune introduces controlled, acceptable overhead compared to single VLM forward pass, offering a favorable trade-off for gains in compositional robustness.
CONCLUSION AND FUTURE WORK
In this work, we introduce NePTune, a novel neuro-symbolic framework for compositional vision-language reasoning. Our approach leverages novel hybrid reasoner that combines the imperative control flow of Python with a declarative, probabilistic soft logic that operates directly on the uncertain outputs of a VLM. This design enables our framework to be both highly expressive and robust in the face of perceptual uncertainty. Our extensive experiments demonstrate the effectiveness of this approach. NePTune demonstrates clear improvement over strong baselines, as well as the underlying VLMs, on complex compositional reasoning benchmarks, in visual question answering, and referring expression, exhibiting remarkable generalization capabilities. While the performance of NePTune is dependent on the quality of its underlying components, such as the LLM for program generation and the VLM for concept grounding, our work demonstrates a flexible and powerful paradigm for building more robust and generalizable Al systems, leveraging these models to elevate their capabilities to a comparatively higher level of performance. Future work could explore methods for more efficient grounding by incorporating external knowledge graphs to resolve complex, non-visual commonsense relations. Additionally, integrating NePTune's 's hybrid symbolic reasoning into broader declarative neuro- symbolic frameworks (Paghihi et al. 2021 sophisticated orchestrations (Cai et al. 2025) could enable the enforcement of explicit logical constraints during fine-tuning (Guo et al. 2020) and further elevate the robustness of visual reasoning.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This project is supported by the Office of Naval Research (ONR) grant N00014-23-1-2417. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views the Office of Naval Research.
ETHICS STATEMENT
This work complies with the ICLR Code of Ethics. NePTune builds on publicly available datasets and open-source VLMs, without collecting new human-subject data. Large Language Models were used solely for grammar polishing and minor text improvements.
REFERENCES
Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, and Siva Reddy. Words aren't t enough, their order matters: On the robustness of grounding visual referring expressions. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 3555-6563 Online, July 2020. Association for Computational Linguistics. doi: 5867
URL.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. In IEEE Conference on Computer Vision and Pattern Recognition, 2024.
Zhixi Cai, Fucai Ke, Simindokht Jahangard, Maria Garcia de la Banda, Reza Haffari, Peter Stuckey, and Hamid Rezatofighi. Naver: A neuro-symbolic compositional automaton for visual grounding with explicit logic reasoning. arXiv preprint arXiv:2502.00372. 2025.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shengiong Ye. Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li. Yimin Ren, Zixuan Chen. Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui Hc, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang. Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang. Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025.
URL http:///xxxvvooooo wwwp://gg//bb/e/eeeeoo absa 2412. 05271
DeepSeek-AI-Team. Deepseek-v3 technical report, 2025. URL http:::/aax....rr////ooo///
241 2212. 19437
Hossein Rajaby Faghihi, Quan Guo, Andrzej Uszok, Aliakbar Nafar, Elaheh Raisi, and Parisa Kordjamshidi. DomiKnowS: a library for integration of symbolic domain knowledge in deep learning, November 2021.
Hossein Rajaby Faghihi, Aliakbar Nafar, Andrzej Uszok, Hamid Karimian, and Parisa Kordjamshidi. Prompt2demodel: Declarative neuro-symbolic modeling with natural language. In Tarek R. Besold, Artur d' Avila Garcez, Ernesto Jimenez-Ruie Ruiz, Roberto Confalonieri. Pranava Madhyastha, and Benedikt Wagner (eds.), Neural-Symbolic Learning and Reasoning, pp- 315-327, Cham, 2024. Springer Nature Switzerland. ISBN 978-9-031-71177-11
Quan Guo, Hossein Rajaby Faghihi, Yue Zhang, Andrzej Uszok, and Parisa Kordjamshidi. dnference-masseee loss for deep structured output learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence (IJCAI 2020), 2020.
Tanmay Gupta and Aniruddha Kembhavi. Visual Programming: Compositional visual reasoning
without training pp. 14953-14962, June 2023. doi: 10.1 10.100090CP55527222223.0001
URL.
https://dot
http://di..eeeccoom...o. www.eeeetteeeoeciieeee........ee wwweeett...rr...reeee org/10........ htt.://....tt...r/......./////// 2023. 2043.01436
Matthew Honnibal and Ines Montani. spacy 2: Natural language understanding with bloom
embeddings, convolutional neural networks and incremental parsing, 2017. To appear.

---

Published as a conference paper at ICLR 2026
Joy Hsu, Jiayuan Mao, Josh Tenenbaum, and Jiajun Wu. What's left? concept grounding with logic-enhanced foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Drew A. Hudson and Christopher D. Manning. Gqa: A new dataset for real- visual reasoning and compositional question answering, 2019. URL htttp::/aarxiv..rrr/bbbb//
09506
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? (extended abstract). In Christian Bessiere (ed.), Proceedings
of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 5065-5069. International Joint Conferences on Artificial Intelligence Organization, 7 2020. doi: 2.2244663/jcai Journal track.
URL tttp:://a...oog/11222085822
2020/ 708
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017a.
Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 0008-3017, 2017b.
URL https:
//api. www.seetmnnnnoolaa...... Crw.aruussoCCoooCD......
Danial Kamali and Parisa Kordjamshidi. Syntax- guided transformers: Elevating compositional generalization and grounding in multimodal environments. In Dieuwke Hupkes, Verna Dankers, Khuyagbaatar Batsuren, Koustuv Sinha, Amirhossein Kazemnejad, Christos Christodoulopoollss,
Ryan Cotterell, and Elia Bruni (eds.), Proceedings of the 1st GenBench Workshop (Benchmarking) Generalisation in NLP, pp. 130-142, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 10//33/v1/2233pe/m http2/222..nncn....1ee
URL https://
ac antho logy. /trgg/222...22 http::3.eeebbhcch...m 1.10
Danial Kamali, J. Barezi, Elham, and Parisa Kordjamshidi. Nesycoco: A neuro-symbollc concept composer for compositional generalization. Proceedings of the AAAI Conference on Artificial Intelligence, 39(4):41844499, Apr. 2025. doi: 10. 1...16699aaai30000 33944332333 URL Inttps:// ojs. aaai org/ / wnwe.. wnn.inndepppp...io php/ MAAII/A aarariiiilee wat.iceeeeeeweeeeeee.e. 32439
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr modulated detection for end-to-end multi-modal understanding. pp. 1760-1778 1770, 2021. URL bttp:://blp..nntttiiee de/ /b/ http://eee..eeee///o//////////// http:///ooo///cc//cc///22
html
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Fei-Fei Li. Visual genome: Comnecting language and vision using crowdsourced dense image annotations, 2016.
602.00020000 07332
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng. Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Chuanhao Li, Wenbo Ye, Zhen Li, Yuwei Wu, and Yunde Jia. Multi-sourced compositional generalization in visual question answering, 2025. /
23045
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
1lava-vl.gie htt.:/ggtt....oo.oooooooooo 2024
Shilong Liu, Zhaoyang Zeng, Tianhe Ren. Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 200.:233300599 2023.
12

---

Published as a conference paper at ICLR 2026
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20797, 2024.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The
Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations, 2019. URL https:
//openzeview.no umr Mage-idd--MMMRttttt
Santiago Ontañión, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher. Making transformers solve 04378 OpenAl-Team. Gpt-4o system card, 2024. UL/Atppsr//axxrr.r////////////
compositional tasks. 1:3591-3607, 2021.
Barbara Partee al. Compositionality. Varieties of formal semantics, 3:281-311, 1984.
Linlu Qiu, Hexiang Hu, Bowen Zhang, Peter Shaw, and Fei Sha. Systematic generalization on What is nearly solved and what is next? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2180-2188, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linnpuutttionallnnuustttttt Linguistics. doi: 10.18653/
gSCAN:
v1/2021 1egolleemnlp-maen
URL https::/aclantholleee org/2011 emn
166
Sania Sinha, Tanawan Premsri, and Parisa Kordjamshidi. A survey on compositional learning of ai models: Theoretical and experimental practices, 2024. URL WRRlhttss://axxi abs/ 24466 08787
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning, 2023.
Mikihiro Tanaka, Takayuki Itamochi, Kenichi Narioka, Ikuro Sato, Yoshitaka Ushiku, and Tatsuya Harada. Generating -o-d-to--ndeestand referring expressions for target identifications. 2019. URL
https://arxiv wwg. /18111/111111 12104
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- lision-language model's perception of the world at any resolution. arXiv preprint arXh::2409.12/99 2024.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: ofate-of-the-ar natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https: wwwww//www///.//////////////ww./ ac web oeeeb.rrg/aaa aatt/ htttthhollly/2000...m
man.pp-eemos.6
Zhengxuan Wu, Elisa Kreiss, Desmond C. Ong, and Christopher Potts. ReaSCAN: Compositional reasoning in language grounding. NewrPPS 202/ Datasets and Benchmarks Track, 2021. URL
https:://openrevier net
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for variety of vision tasks, arXiv preprint arXiv:2311.066 2023.
Guangyue Xu, Parisa Kordjamshidi, and Joyce Chai. MetaReVision: Meta-learning with retrieval for visually grounded compositional concept acquisition. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 02224-12238 Singapore, December 2023. Association for Computational Linguisties. doi: 10. 118655/112222/ 100.66551 findings-emmlp.816 URL httts:://aaaatthllloooooooo www.thhhoooo...o//222222222 findi ngs-iing-aee emn 818
13

---

Published as a conference paper at ICLR 2026
Guangyue Xu. Parisa Kordjamshidi, and Joyce Chai. Gipcol: Graph-injected soft prompting for compositional zero-shot learning. In /EEE/CVF Winter Conference on Applications of Computer Vision, 2024.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406.094 2024.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-Symbon VQA: Disentangling Reasoning from Vision and Language Understanding. In Advances in Neural Information Processing Systems (NIPS), 2018.
Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun. Do vision- pretrained models learn composable primitive concepts? Transactions on Machine Learning Research, 2023. ISSN
2835-8856. URL https:/ http///pnnneewwweeeeeeee hrt5ssoo__...ttaaa/////peeeeeees worum?iroooo YwNrPL
Yan Zeng, Xinsong Zhang, and Hang Li. Multi Multi-grainedd vision language pre-training: Aligning texts with visual concepts. arXiv preprint 1 2021.
Wang Zhu, Jesse Thomason, and Robin Jia. Generalization differences between end-to-end and neuro-symbolic vision-language reasoning systems. pp. 4697-4711, December 2022. doi: 10. 10.18653/v1/2022. 10.16655/110222.inniinnnccciimi Findings emnlp. 345
URL /RL/tttps////caattolllooy wwtp:/yy......2222
A QUALITATIVE ANALYSIS
A.1 REFERRING EXPRESSIONS GROUNDING
Expressions 1 and 2 show the superiority of compositional reasoning compared to the VLM and object detection backbones. They demonstrate how the VLM fails to correctly locate the object in the new environments, while atomic concepts correctly scored by the VLM.
Examples of NePTune on Ref-GTA are shown in Figure 3
A.2 VISUAL QUESTIONS ANSWERING
Examples of NePTune on CLEVR-Human are shown in Figure 4 Example 1 illustrates a case where reasoning over meta-concepts such as distinct shapes required. NeSyCoCo is unable to resolve the query due to this limitation to declarative, while ViperGPT struggles to identify the correct anchor object because does not perform global reasoning across the object set. NePTune, by contrast, is able to provide the correct interpretation. Example 2 presents more complex scenario where reflection reasoning and calibration are critical. Both the sphere and the cylinder reflected in the cyan cube, with the cylinder's 's reflection being more prominent and thus the more likely answer. NePTune does not fully capture this subtle distinction and produces an incorrect response. The competing baselines, however, fail for different and more fundamental reasons: VLM score calibration issues lead to incorrect selection, and ViperGPT relies on bounding box size comparisons that ignore perspective, which causes it treat equally sized cubes different. In this case, the code selects the shiny brown cube in the foreground and concludes that there reflection. Although NePTune also errs, this example highlights the inherent difficulty of fine-grained reflection reasoning and the challenges posed by subtle visual cues.
A.3 FAILURE EXAMPLE.
Example shown Figure 5 illustrates an error caused by an incorrect LLM-generated program in NePTune. Two issues occur simultancously:
1. Incorrect predicate arity:
The program invokes the function is Lis_aayygg girarffff fae with num_object=2 instead of This leads dimensional mismatch when the resulting score tensor is composed with variable x2, causing execution failure.
2. Incorrect bounding box reference: The program refers to the bounding box with the color green instead of red, which is the intended single-object reference.
14

---

Published as a conference paper at ICLR 2026
Expression 1: Man wearing a black suit
0.99
(.30
property))
mwardng stack 121
wartin
Expression 2: Man wearing white jacket with blue stripes and white shorts
J.0
0.99
1e, tan scora(*te the ents sssace tactde of the red hrunetigg hox sarte, ran, 2bwecte)), tape "claee") aacat jecuet with stas striase scanel"te
the ree toureing unering white sezeet with stae st-tows?", objecttre. "pp"" "property") untasing sideustie?, aadecteni, "contect"
rideunte t98, ahtte jeceet.ecttt actn ston, stripes, and whiteeene wwwrtshttttteeee milkdnng { 16 Ha.n as")
instde
stite atortalied
st.anlsing, wtw../s...aaaaaaee (iaraala"'ss"
Figure 3: Qualitative examples of NePTune on the RefGTA dataset. Green boxes indicate objects detected by Grounding DINO, blue boxes show objects selected by the VLM VL2.5-B)). and red boxes highlight the final selections made by NePTune.
While errors such as incorrect bounding box colors can be alleviated a certain extent with regex-based matching, the dimensional inconsistency in predicate scoring is unrecoverable without regeneration.
B
VLM CONCEPT ANALYSIS
B.1
CONCEPT-LEVEI PERFORMANCE
An analysis of the atomic concept grounding performance on the CLEVR dataset shown in Table [] reveals clear patterns of weakness, primarily centered around analogical and complex relational concepts.
B.1.1 ANALOGICAL RELATIONS
The most significant challenge for the VLMs concept is abstract, analogical comparisons. The concept same size is clear example, yielding the lowest F1
in the entire table for Qwen2V1.-7B at 0.604. Ovis1.6-9P also struggles significantly with it, scoring just 0.794. While Intemn performs better, same color notable weak point for it at 0.840. This indicates systemic difficulty in performing comparative judgments with visual prompting compared to simple identification.
B.1.2 3D SPATIAL RELATIONS ARE A CLOSE SECOND
The next most difficult category is 3D spatial relationships. Concepts like behind and front consistently score lower than basic attributes across all models. For instance, the Fl scores for behind 0.836 (0vis1.6), 0.877 (Qwen2VL), and 0.868 (InterVV22.5) This demonstrates weaker understanding of object relations in ttree-dimensiona space compared to intrinsic properties like shape material.
15

---

Published as a conference paper at ICLR 2026
Example
Example 2
How material
GT:3
GT: Cylinder
NePTune
No sncettty wnto adeen hocniing (OWENN))
Sphere
more Peaare de cimer
NeSyCoCo
Dechher Litit stactiole ini per, bacis soleet prodcit
Cube
NeSyCoCo
ViperGPT
Eitchpe Ladio browa Stsntall
Lano CaCc www.incciiion Lavon
ViperGPT
No reflection
cannet determine
thep 2eppraatiee
1iae fram putch:
Ueneerrr shope aer oateria shepat) SUME in
reture (TrGGiu))
Figure 4: Qualitative examples of NePTune compared to ViperGPT and NeSyCoCo CLEVR-Humans.
atitt plnafie acoref"ts miin object drodde the red bouncine scote giretfe?" abfectari, teas""ctass" glraifee tnstie of rivat anedicate i in aduit gloathuihaa beby gireffuz (a11 lassangl 52", (22)) naturs fladilaawdde www.ioatt......... lnta
Lockiing ssanettts
irazde
Figure 5: Qualitative example of wrong NePTune program generation. Mistakes are highlighted with red boxes.
B.1.3 COLOR CONFUSION
Certain colors also pose challenges, particularly for the Qwen2VL model. The color cyan shows notable performance dip for Qwen2VI. (0.805). Further analysis suggests this due confusion with visually similar colors like blue (0.806) and green (0.838). This pattern highlights that while models can identify common colors well, they are less robust with more specific shades and can struggle to differentiate them.
B.1.4 REAL-WORLD WORLD IMAGE CHALLENGES IN VISUAL GENOME
Transitioning from the synthetic CLEVR environment to the complex, real-world images of the Visual Genome (Krishna et al. 2016) dataset reveals notable shift in performance dynamics. We evaluate our models on atomic queries generated from the cleaned VG scene graphs in GQA (Hudson & Manning 2019) using these templates:
16

---

Published as a conference paper at ICLR 2026
Table 11: Fl Scores for atomic concept grounding on CLEVR using visual prompting.
Concept
Ovis1.6
Qwen2VI.
Intern
blue
0.972
0.806
0.986
brown
0.989
0.921
0.989
cube
0.915
0.952
0.991
cyan
0,961
0,805
1.000
cylinder
0.964
0.960
1.000
gray
0.958
0.911
0.980
green
0.989
0.838
0.988
large
0.956
0.975
1.000
metal
0.961
0.965
0.997
purple
0.988
0.927
1.000
red
0.973
0.829
1.000
rubber
0.980
0.924
1.000
small
0.976
0.868
1.000
sphere
0.980
0.980
0.996
yellow
0.974
0.871
0.982
behind
0.836
0.877
0.868
front
0.890
0.850
0.850
left
0.919
0.885
0.943
right 0.934
0.863
0,930
same.color
0.982
0.957
0.840
same shape
0.862
0.855
0.936
same size
0.794
0.604
0.951
Macro F1 0.943
0.874
0.964
Micro F1
0.903
0.866
0.945
Relation Prompt:
Is the /classy/object] } in the red bounding box (classz/objects) the {relation) the green bounding box?
Class Prompt:
Is the object (inside of/in} the red bounding box {article] {class}?
Attribute Prompt:
Is the (class) in the red bounding box {attribute}?
As shown in Table models remain highly proficient at identifying spatial relations and attributes, their performance in the "Class/Object" category is consistently lower than CLEVR. Ovis1.6-9P drops to an Fi-score of 0.804, with Intern -BB and Qwen2VL-7B at 0.818 and 0.833, respectively. This suggests that the visual complexity, clutter, and vast number of fine-graines categories in real-world images make the foundational task of object recognition a more significant hurdle than in the controlled CLEVR environment.
key challenge emerges in fundamental object identification. While the
B.2 TOD-TO-END PERFORMANCE
These findings align with the paper's broader conclusion that errors in the final reasoning pipeline often originate from the VLM's poor performance on these specific types of relational and analogical concepts. Based on the results of this analysis, in addition to the score perceptual interface, generate a set of the most common spatial predicates, such as left, right, behind, front, etc., using the position and depth estimation of bounding boxes for the sake of efficiency and accuracy.
The results Table 13 clearly show that NePTune enhances the compositional reasoning abilities of quantitative
strong base VLMs such 0 and Intern VL2.5. The most significant gains compositional tasks, with accuracy on counting questions improving by over 12% for Intern VL2.5 and counting comparison questions by nearly 11% for 0vis1.6. This highlights the value of
17

---

Published as a conference paper at ICLR 2026
Table 12: Top 20 common concepts F1-Score comparison on real-world scene graphs.
Concept Ovis1.6-9B Qwen2VI.-7B
right 0.940
0.971
0.962
left of
0.951
0.970
0,980
man
0.906
0.914
0.892
shirt
0.840
0.893
0.963
person
0.821
0.792
0.824
window
0.857
0.894
0.957
pole
0.842
0.900
0.927
building
0.621
0.839
0.889
0.750
0.833
0.839
wall
0.815
0.857
0,800
on
0.636
0.727
0.696
sign
0,800
0,815
0.833
car
0.870
0,960
1,000
hand
0.929
0,929
0,923
in
0.444
0.421
0,500
white
0.750
0.889
0.846
near
0.923
0.880
0.923
sky 0.500
0.667
0.800
ear 0,818
0.800
0.909
woman
0.952
0.952
0.952
Table 13: Comparison of model accuracies across various categories. Maximum values for each NePTune and backbone VLMs are bolded. Changes compared to normal models are marked with colors and arrows.
Metric
Qwen2VL
Ovis1.6
InternVL2.5
Qwen-NeSy
Ovis-Nesy
Intern-Nesy
Final Accuracy
93.55
85.00
90.25
82.55 11.00)
88.80 3.80)
92.65 2.40)
Exist (%)
98.21
86.74
87.10
84.59
13.62)
88.53 (1 1.79)
93.19 6.09)
Query Attribute (%)
93.77
90.14
98.26
91.30 2.47)
96.23 6.09)
96.81 1.45)
Compare Altribute (%)
99.44
90.28
98.61
78.33 21.11)
88.33 1.95)
91.94 6.67)
Count (5)
87.50
75.40
74.60
70.16 (1 17.34)
78.83 (1 3.43)
87.10 (1 12,50)
Compare Number (%)
90.29
78.29
90.86
88.57 (L 1.72)
89.14 (1 10.85)
92.57 (1 1.71)
structured, programmatic approach for tasks that pure neural models find challenging. Conversely, our framework degrades the performance of the already accurate model Qwen2VL. A deeper look reveals that the primary source of this degradation is the analogical questions, which the most apparent in the "Compare Attribute" task, where performance drops by 21%. This aligns perfectly with our findings in Experiment 1, which showed that this specific VLM struggles with analogical reasoning (e.g., same color, same shape). This demonstrates both the power of our framework when paired with a reliable grounding VL. and its sensitivity the underlying VL.M's weaknesses, as errors in perception propagated by the logical executor.
C EXPERIMENTAL SETUP
experiments were conducted on a server running Ubuntu OS, equipped with an AMD EPYC 7413 24-core CPU, 700GB of system RAM, and an NVIDIA A6000 GPU (48GB). Unless specified otherwise, our core models included DeepSeekV3 (DeepSeck-AI-Teaul)
2025) as the backbone LLM with greedy sampling, Grounding DINO-B for object detection, and DepthAnything YYngg 2024) for depth estimation. We utilized the Hugging Face Transformers library (Wolf et al. 2020 for VLM interaction and the Intern VL-2.5 codebase for finetuning To supplement the LLM's output, we also used spaCy (Honnibal & Montani] 2017) Named Entity Recognition to extract keywords from the query.
et al.
18

---

Published as a conference paper at ICLR 2026
D DATASETS
Here, we provide a detailed breakdown of the datasets used to evaluate the NePTune framework across range of visual reasoning challenges.
D.1 CLEVR
We evaluate core compositional reasoning using the CLEVR (Compositional Language and Elementary Visual Reasoning) dataset (Johnson et 2017a). It contains 3D-rendered synthetic images of simple objects and is used for the task of Visual Question Answering (VQA), where the model must answer complex, programmatically generated questions about object attributes, counts, and relations. We measure performance using Accuracy (%) and, for answers requiring semantic equivalence matching (e.g., matching "2" with "two"). For CLEVR dataset benchmarks where deterministic evaluation did not capture the correctness of the answer, such as matching "2" with "two", we use GPT-4o as an Al judge for evaluation. The prompt used for this evaluation is shown in Figure 7 Since the CLEVR evaluation set large (700K), used the first 2000 samples for evaluating our method. We used the best available program generation, similar to Hsu et al. (2024).
D.1.1 CLEVR QUERY CATEGORIES
Our fine-grained analysis on CLEVR, presented in Table 3 following five distinet types of reasoning skills using the programs in the dataset:
breaks down performance across the
Exist
These questions test for the presence of an object with specific properties, requiring a "Yes/No" answer. For example. "Is there large green cube behind the small red sphere?"
Query Attribute These questions ask for specific property (e.g., color, material) of uniquely identified object, such as, "What color the small shiny cylinder?"
Compare Attribute This category involves "Yes/No" questions that compare single property between two objects. For example, "Does the large sphere have the same material as the small cube?"
Count These questions require the model to return the total number of objects matching description, such as, "How many red metallic objects are there?"
Compare Number These questions involve comparing the quantities of different sets of objects, also resulting in "Yes/No" answer. For example, "Are there more spheres than cubes?"
D.2
The Humaas dataset (Johnson et al., 2017b) allows us to evaluate performance on more natural language. uses the same synthetic images as CLEVR but features more complex and linguistically diverse questions written by humans. Similar to CLEVR, performance is measured by Accuracy (%).
D.3 CLEVR EXTENSIONS
To probe wider range of reasoning skills, collection of compositional challenges from Hsu et al. (2024), built on the CLEVR environment. Each task tests a unique aspect of complex reasoning:
CLEVR-Ref
This task tests referring expression grounding, where the model must identify target object based on its relationship to other objects. The evaluation is conducted by measuring the IOU value greater than 0.5.
Example: "There is sphere that is front the gray cylinder. find the small cylinder that left of it."
CLEVR-Puzzles This benchmark evaluates the model's ability solve visual constraint satisfaction problems by finding a set of objects that simultaneously satisfy multiple attribute and relational constraints.
19

---

Published as a conference paper at ICLR 2026
"Can you find four objects from the image such that: object 1 is large metal object; object a metal object; object is a small rubber cylinder: object 4 is small yellow metal cylinder; object I is front object 2; object 1 is behind object 4; object is behind object 3."
Example:
CLEVR-RPM This task tests abstract relational reasoning by mimicking Raven's Progressive Matrices. The model presented with objects in a grid that follow a pattern and must identify candidate object from the scene that correctly completes that pattern.
"There are 9 objects, ordered in a 3x3 grid: row col is a small rubber object; row I col 2 is small rubber object; row I col 3 is a large rubber object; row 2 col is a small metal object; row col a small metal object; row 2 col 3 is large metal object; row col I is small metal object; now 3 col is a small metal object; am missing one object at row col Can you find an object in the scene that can fit there?"
Example:
D.4 REF-ADVERSARIAL (REF-ADV)
To test performance real-world scenarios, we use the Ref-Adversarial benchmark (Akula et al. 2020). This dataset consists of real-world images with visually similar distractor objects, making the task of Referring Expression Grounding particularly challenging. REG requires the model to locate the specific object corresponding to text description, and evaluate the model using Grounding Accuracy (%), where a prediction considered correct the Intersection over Union (IoU) with the ground-truth bounding box is 0.5 greater.
D.5 REF-GTA
To measure generalization under significant domain shift, we use the Ref-GTA benchmark (Tanaka al. 2019). This is REG benchmark where images sourced from photo-realistic game simulation, creating an out-of-distribution challenge for models pre-trained on natural images. Performance is measured by Grounding Accuracy (%), defined by Intersection over Union (IoU) threshold of 0.5 between the predicted and ground-truth bounding boxes.
E VERIFICATION
You're an image analyst designed to check if the highlighted objects in the image meet the query description, and which one more likely to meet the query description. The query is: "(query)" Please check the highlighted object "0" [in the red bounding box] and "1" [in the green bounding box] in the image and the question: Which object more likely meet the query description? Your answer should be "0", "1", Answer with one word or phrase.
Figure 6: Prompt used for the Pairwise Arbiter verification strategy.
To recover from occasional symbolic execution failures, we add lightweight verification stage that operates after reasoning has produced a candidate answer. We explore two simple strategies:
1. Pairwise Arbiter:
The model is presented with two candidates, the backbone prediction and the symbolic reasoning output, along with the query. It acts arbiter, directly judging which option better satisfies the description. Our prompt, similar to the one used by NAVER for this process, is shown in Figure 6
2. Confidence Gating:
We compute a softmax distribution over the symbolic executor's scores with temperature T, and use threshold to decide whether to trust the executor. If max < т, we fallback to the backbone prediction; otherwise, we keep the executor's output. Both and 7 are tuned 500-example held-out set for each backbone.
20

---

Published as a conference paper at ICLR 2026
Table 14 summarizes the performance of both light- verification strategies across different backbones. Overall, we observe that the Pairwise Arbiter tends to outperform the Confidence Gating method, highlighting the benefit of leveraging VLM arbitrate between backbone and symbolic predictions.
Table 14: Post-verification aceuracy Ref-Adv.
Backbone VLM
Verification
Accuracy (%)
T
T
Symbolic Share( %
Confidence Gating
80.91
0.70
0.40
73.59
Qwen2VL-7B
Pairwise Arbiter
80.93
N/A
N/A
80.23
Confidence Gating
60.49
0,30
0.10
86.79
Ovis1.6-9B
Pairwise Arbiter
63.25
N/A
N/A
29.73
Confidence Gating
76.65
0.60
0.50
71.06
InternVL2.558B
Pairwise Arbiter
78.08
N/A
N/A
77.86
You are an automatic answer checker! I will give you a question and answer, and a generated response, and you will tell me if the response is correct given the ground truth answer. If the response is correct, you will say < Yes>>, otherwise the same. Shiny and Metal or Metallic are the same. Square and Cube are the same. Yellow and Golden colors are similar in the images, too. Balls and spheres are similar. Check if the response is correct given the questions and the ground truth answer, deeply thinking about the context. Answer with Yes or <No>> and don't 't mention them other than for the final answer. Important! Ignore the red bounding box reference in the generated response. Question: " "(Question)" Ground Truth Answer: "[Answer)" Generated Response: "( "(Prediction)"
will say <No>. Rubber and Matte are
Figure 7: Visual Question Answering judge LLM prompt.
F HYPERPARAMETERS
The fine-tuning hyperparameters for our experiments are detailed in Table 15 These settings are based on the configurations defined in our training script.
During our experiments, we found that applying LoRA fine-tuning to the vision backbone, in addition to the language model, was highly effective for domain adaptation. To illustrate, without any neuro-symbolic fine-tuning, the base NePTune framework achieved 56.49% accuracy on Ref-GTA. In contrast, standard fine-tuning approach on the base VLM only reached 4.19% accuracy. This performance gap underscores the effectiveness of our neuro-symbolic method and justifies fine-tuning the vision components to achieve optimal results in novel environments.
G LATENCY ANALYSIS
The primary computational cost of NePTune stems from the calls to the VLM for concept grounding. For scene with N candidate objects, R binary relations, and C unary concepts, scoring unary concept (such shape, attribute) requires N VLM calls (overall OCCN) across all unary concepts), and scoring a binary relation requires 22 VLM calls (overall O(RN2) across all relations used). This can be slow for scenes with many candidate objects. To reduce this computational overhead, we apply a simple optimization. Before execution, we collect the set of required atomic concepts from the generated program and issue them in single batched call, with prefix caching enabled, using the vLLM (Kwon et al.] 2023) tool. Since the image tokens and most textual prefixes are shared across these queries, prefix caching eliminates redundant computations.
21

---

Published as a conference paper at ICLR 2026
Table 15: Hyperparameters for Fine-tuning Experiments
Standard Fine-Tuning Tiee-Tuning
NePTune Neuro-Symbolic Fine- Fine-Tuning
Hyperparameter
on Ref-Adv
on Ref-GTA
General Training
Learning Rate (Ir)
4 x 10-5
x 10-5
4 10-5
Batch Size
1
Gradient Accumulation
4
4
4
Weight Decay
0.01 Cosine
0.01 Cosine
0.01
LR Scheduler
Cosine
Warmup Ratio
0.03 Cross-Entropy
0.03
0.03
Loss Function
Binary Cross-Entropy Binary Cross-Entropy
LoRA Configuration
Rank (r)
16 (LMM,, 8 (Vision) 32 (LM). 16 (Vision)
16 (I.M)
16 (L.M), 8 (Vision) 32 (LM). 16 (Vision)
Alpha (ar)
32 (LM)
Dropout
0.05
0.05
0.05
Figure 8 summarizes the average latency of NePTune Ref-Ady, using the experimental setup described in Appendix C on a single GPU.
For Intern VL2.5-1B: Grounding with VLM forward pass takes 0.90 s, while the average non-cached NePTune query takes 3.63 s. Our caching optimization reduces this average to 1.30 s, revealing a small 0.4 overhead.
For InternVL2.5-4 Grounding with VLM forward pass takes 1.48 the non-cached average is 12.35 s, and the cached average drops to 8.19 s.
In other words, NePTune with caching adds 0.4 of overhead for the 1B backbone and 6.7 for the 8B backbone, remaining small multiple of a VLM forward pass.
Execution Times by Backbons Impimentation
25
Vanilla
20-
Optimized
Parrage Seeculon Time
12,55s
B319%
3.635
1.50s
0.90s
S-111
Inter/ 5-88
Figure 8: Latency Analysis of NePTune over Ref-Adv dataset. "Vanilla" refers to non-cached execution, "Optimized" is our optimized approach, and "VLM" is single forward pass.
The latency averages reported in Figure g are heavily affected by few computationally intensive queries. Figure[]showe the full execution time distribution for both models. For the 1B model, the vast majority queries are extremely fast, with 3,030 queries completing in under 2 seconds. For the 8B model, most queries (2,143) are resolved in under 5 seconds. However, we also observe longer tail, with 229 complex queries taking longer than 25 seconds, which increases the average.
Figure providess detailed breakdown of why certain queries are slow, plotting time function of the Number of Proposal Objects (x-axis, N) and the Number of Concepts (y-axis, = R+C) for both Intern VL.2.5-BB and 8B.
These heatmaps confirm that latency increases with both N and K, but the strongest spikes correlate with large N in relational predicates, which matches the O ORRN2) cost of relational scoring. For the IB backbone, average latency increases from about 0.43 s at N = 2 to about 2.56 s at For the 8B backbone, average latency increases from about 2.51 at N 2 to about 16.70 N 6.
22

---

Published as a conference paper at ICLR 2026
Dirritution Vumberde Oipecca
Coecation Time Ciatibuttoo
Concution Time Ciamnbution
3350
1330
3991
3999
3000
590
084
500
123
Dendicttime linckedd
limelte an
(a) Object proposals distribution.
(b) Intern 1B
(c) AntemvVL2.5
Figure 9: Number of object proposals (a) and latency distribution for Intern Intemv VL2.5-1B (b) and
Intern VL2.5-8B (c) on Ref-Adv.
As shown in Figure 9a. on Ref-Adv roughly 90% of queries have N < 6, most points lie in this low to moderate regime. Only about 1% of queries have N 2 10, and these high-N relational cases account for the few cells that reach very large runtimes, including maximum of about 257 (K = 6, N = 18) for the 8B model.
This analysis shows that NePTune introduces controlled overhead compared to single VLM pass, with large costs concentrated in rare queries that involve many proposal objects and multiple relational concepts. We view this favorable trade-off given the substantial gains in compositional robustness and NePTune's 's ability to solve complex queries that the backbone models alone fail to answer.
Jaabe Priees Cajeste
(a) IL2.51
(b) IntemVLL..58BB
Figure 10: Heatmap of time for Intern VL2.5-IB (a) and VLte5nVL2.5-8B (b). Bright yellow cells indicate high-latency queries, which correlate strongly with a high number of proposal objects. Cells show the average execution time of the bin in seconds and are colored by log(time) for better visualization.
H HUMAN EVALUATION OF PROGRAM GENERATION
In this section, we quantify how often NePTune's L.M program generator produces incorrect programs and what kinds of errors it makes, using small but carefully controlled human evaluation across three benchmarks: Ref-Adv, Ref-GTA, and CLEVR-Humans.
H.1 EXPERIMENTAL SETUP
For each dataset, we randomly sampled 200 questions from the corresponding evaluation sets used in our main experiments. For each question, we ran NePTune's program generator and obtained a Python program. This program, together with the question and its output, was then evaluated by human raters. We used three evaluators with knowledge of machine learning and Python programming for each dataset. Before starting evaluation, we held a session which we explained the NePTune framework (bounding boxes, score and query calls, soft logic operators) and walked through several example questions and their programs.
Each evaluator then independently labeled all 200 programs for each dataset. During the evaluation, one of the authors was available online to answer questions about ambiguities and border cases for the evaluators.
23

---

Published as a conference paper at ICLR 2026
Program evaluation outcomes per dataset (200 questions each). The Correct column shows the number of programs marked correct by all evaluators; the error columns count programs marked with that error by least one evaluator; and k is the average pairwise Cohen's s for the binary correctness decision.
Table 16:
Dataset
Correct
Syntax
Atomic
Logic
Other
Accuracy
K
Ref-Adv 175
87.50%
0.76
Ref-GTA
187
0
12
1
93.50%
0.73
CLEVR-Human
195
0
0
0
97.50%
0.71
H.2 EVALUATION SCHEME
For each question and generated program, evaluators followed two-step decision rule.
Step 1: correctness. Evaluators first decided whether the program is fully correct:
Mark as correct. The program is syntactically valid and logically faithful to the question and implements the intended reasoning steps.
Step 2: error tags. If the program was not marked correct, evaluators selected from the following error labels that applied:
Syntax error. The program has Python syntax error fails runtime due an obvious coding mistake, for example, a wrong number of arguments to score, misspelled function name, an indexing error.
Wrong atomic question. One or more atomic predicates in the program incorrectly decomposed from the natural language query. This includes mis-specifying the question passed to score or query. For example, dropping or adding attributes, using singular instead of plural (or vice versa), selecting the wrong bounding-bow reference or color, or assigning the wrong arity (using a unary predicate where a binary relation required). These errors reflect mismatch between the intended atomic meaning of the query and the atomic predicates used in the program. For instance, if the query talking about "two men sitting table" but the code decomposes that to score("... man?") the query is asking about "dining table" but the atomic query decomposition just says "table".
Wrong logic. The program executes but implements the question (for example, using disjunction instead of conjunction, mis-handling quantifiers such as "exactly one", or applying a condition to the wrong set objects).
wrong logical structure for
Other issue. Any remaining problem that does not the above categories, including ambiguous cases or issues clearly caused by the underlying VLM output rather than by the program structure.
H.3 LABEL DISTRIBUTION AND AGREEMENT
Table I6 summarizes the label distribution across evaluators for each dataset. For each dataset, we report the number of programs marked at least once with each emor label (union over the three evaluators), accuracy (fraction of programs marked correct by all evaluators), and the average pairwise Cohen's и computed for the binary correctness decision (Correct vs. Incorrect). Inter-evaluator agreement, measured as average pairwise Cohen's K, was consistently high across datasets.
H.4 DATASET SPECIFIC ERROR PATTERNS
The human study also reveals qualitatively different error patterns across the three datasets.
CLEvR-Humans. Typical issues include using iota normalization for indefinite descriptions, or miscounting when the reference object should be excluded from a set and subtle logical composition.
24

---

Published as a conference paper at ICLR 2026
In these cases, the atomic score calls usually reasonable, but the higher-level program does not faithfully capture uniqueness and distinctness. For instance, to answer "Is there small red sphere next to a large red sphere", the program applies small red sphere iota ("x1"), "), which can lead to a wrong answer if there is no red sphere. As we expected, CleaR-Humans is dataset with simple closed set of concepts; therefore, wrong atomic queries didn't exist.
Ref-Adv and Ref-GTA. In these two datasets, the main failure mode concerns atomic prompting for plural and group expressions. Many referring expressions in these datasets refer groups of objects (e.g., "the children in the background" versus "the child on the left"). Our evaluators found that while the overall program structure is usually correct, the generator often translates these into atomic prompts that implicitly assume single object drop the plurality constraint (e.g., asking about "the child" inside score when the query is about "children"). This may lead to wrong region selection even if the tensor logic and local perception subset of atomic prompting errors involving simple template issues, specifically red/green bounding box mismatches. Because these followed consistent patterns, were able to detect and fix them automatically using rule-based pattern matching. Finally, cases marked with other generally concern ambiguity. For instance, given the expression "The telephone closest to the screen," the reference to "screen" is unclear, it could mean the camera viewpoint or a physical screen in the image. Using images alongside the query in future work might help mitigate this issue. Together with the low number of syntax and logic errors, this suggests that NePTune's s main remaining bottlenecks lie in subtle specification mismatches (especially plurality) rather than gross program failures.
otherwise sound. We also identified
H.S SUMMARY
Overall, the human study shows that NePTune's program generator is reliable in practice: most programs are judged correct, syntax errors are rare, and inter-evaluator agreement is high. The remaining issues are localized specification errors rather than gross logical failures. On CLEVR-Humans, using the normalization function iota was the most common error, while on Ref-Adv and Ref-GTA, the dominant emrors come from atomic prompts that drop plural group cues in the natural language passed to scor score/query. Several of these atomic prompt bugs follow simple patterns that can be addressed with some prompt tuning.
25


Reviews:
[{"review_id": "upv74M0Jzl", "reviewer": "Reviewer_Nt6P", "summary": "This paper introduces NePTune, a neuro-symbolic framework for vision-language reasoning. NePTune translates natural language queries into executable Python programs that use a hybrid execution model. This model combines imperative Python control flow with differentiable, soft logic operators that reason over uncertainty scores from a Vision-Language Model (VLM). The framework operates zero-shot by using an LLM to generate programs and a VLM to ground atomic concepts. It also supports fine-tuning for domain adaptation, demonstrating strong compositional generalization, especially on domain-shift benchmarks.", "questions": "Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.\n\n1. How frequent are logical program generation failures, and does NePTune have any mechanism to detect or recover from them?\n2. Could the authors provide a rough estimate of the end-to-end wall-clock time for a complex query? How does this multi-step pipeline's latency compare to a single forward pass of the backbone VLM?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results.\n\n1. On the Ref-GTA domain-shift benchmark, the VLM backbone (InternVL2.5) achieves only 6.95% accuracy, while NePTune achieves 69.69%. This demonstrates a highly robust decoupling of reasoning from perception.\n2. The framework integrates imperative Python with soft, differentiable logical operators. This is more expressive than declarative-only systems and more robust than crisp-logic pipelines.\n3. The soft-logic operators make the pipeline differentiable, enabling fine-tuning.", "weaknesses": "A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.\n\n1. Performance is fundamentally limited by the VLM's ability to ground atomic concepts.\n2. While more robust, the LLM-based program generator is still a point of failure. The authors provide a clear example of this in Appendix A.3, where the LLM generates a program with incorrect logic and arity, causing an execution failure.\n3. The framework is complex, involving multiple serial steps. This multi-component pipeline is likely to have significantly higher latency than a single VLM forward pass.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uNslfBCl59", "reviewer": "Reviewer_9EoX", "summary": "This paper introduces NePTune, a neuro-symbolic framework for vision-language (VL) reasoning that addresses limitations in prior work, such as brittle pipelines and a lack of adaptability. NePTune translates a natural language query into an executable Python program, similar to models like ViperGPT. However, it makes two key contributions:\n\n- Hybrid Execution Model: The generated Python program combines standard imperative control flow (loops, conditionals) with \"soft logic\" operators (based on fuzzy logic, e.g., t-norms for AND) that operate directly on the continuous uncertainty scores provided by a Vision-Language Model (VLM).\n\n- Differentiability: These soft logic operations are differentiable, allowing the entire framework (specifically the VLM backbone) to be fine-tuned via \"neuro-symbolic fine-tuning\" for domain adaptation.\n\nOperating in a training-free manner, NePTune already outperforms strong baselines. When fine-tuned, it shows remarkable adaptation capabilities, especially on domain-shift benchmarks like Ref-GTA.", "questions": "- The ablation study (Table 10) shows a massive +19.19% improvement from adding \"Imperative Reasoning\" on top of declarative logic. Could the authors provide a concrete example of a query from CLEVR-Humans that requires this imperative logic (e.g., loops or complex conditionals) and which a purely declarative approach fails to solve?\n\n- Regarding the $N \\times N$ matrix for relational queries: What is the practical limit for N (number of objects) before the VLM grounding and subsequent reasoning become computationally prohibitive? Have the authors tested this on more cluttered datasets, like COCO?\n\n- The fine-tuning results on Ref-GTA are a key strength. To clarify: what exactly is being fine-tuned? Is it only the VLM backbone? Or are the parameters of the soft-logic operators (e.g., the temperature $\\tau$ and margin $\\gamma$ from Table 2) also learned during the neuro-symbolic fine-tuning process?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core contribution—a symbolic executor that blends imperative Python with differentiable, soft-logic operators (Table 2)—is highly novel and effectively tackles the problem of uncertainty propagation in multi-step reasoning.\n\n- The framework's differentiability is a key advantage. The ability to fine-tune the system for a new domain (as shown on Ref-GTA, where neuro-symbolic fine-tuning achieved 69.9% vs. 32.6% for standard VLM fine-tuning) is a significant step forward from inference-only models.\n\n- By operating on continuous scores rather than crisp decisions, NePTune avoids the cascading error problem that plagues many multi-step reasoning pipelines.\n\n- NePTune demonstrates state-of-the-art zero-shot performance against comparable methods (e.g., 92.65% vs 36.05% for ViperGPT on CLEVR) and massive gains on challenging domain-shift benchmarks (Ref-GTA), proving the practical value of its design.\n\n- The paper is extremely well-written, and its contributions are clearly articulated and well-situated against prior work.", "weaknesses": "- The entire system still hinges on an LLM's ability to correctly parse the natural language query into a logically sound Python program. The paper notes that errors shift from \"syntax\" to \"logic\" (Sec 5), but a logical error in the generated program is just as fatal as a syntax error. A deeper analysis of the types of logical errors the LLM still makes would be valuable.\n\n- The score function for relational queries (num_objects=2) involves VLM queries that result in an $N \\times N$ matrix, where N is the number of detected objects. This approach seems to scale quadratically with the number of objects in the scene. It's unclear how the system would perform in highly cluttered real-world scenes with hundreds of object proposals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OVTRTRVfQZ", "reviewer": "Reviewer_fRHk", "summary": "The paper presents NePTune, a neuro-symbolic framework aimed at enhancing compositional reasoning with Vision-Language Models (VLMs). NePTune consists of three main components: an LLM-based program generator, a perceptual grounding module, and a symbolic executor. The program generator converts natural language queries into executable Python programs that can query a VLM to extract object-specific information. The resulting perceptual outputs are then combined using soft logic operators, allowing the system to reason over uncertain predictions from the VLM. The framework operates in a training-free manner but remains differentiable for fine-tuning.", "questions": "- Q1: Is the score function currently limited to pairs of objects, or can it be extended to higher-arity relations?\n- Q2: What is the number of tokens that the query function can return? \n- Q3: Line 241/242 seems to have a missing or incomplete expression \"…’).\". Could you clarify?\n- Q4: For program generation (component 1), do you use greedy decoding or a sampling-based strategy? Same question for the baseline VLMs\n- Q5: Regarding Table 4, could you clarify what \"ground-truth programs\" refers to, how they are obtained, and why the dataset \"Puzzles\" can not benefit as strongly from it?\n- Q6: What motivated the choice of VLMs for NePTune and comparisons? Do you expect similar zero-shot results for larger models (e.g., >70B or GPT-5 etc.) as in Tables 5 and 6?\n- Q7: Can you estimate the computational or token cost of NePTune compared to direct prompting? Given the small amount of tokens for the score and query functions, it seems low, but quantitative insight or discussions would be helpful.\n\nGenerally, I like the proposed method and am willing to raise my score if my concerns and questions are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper introduces a novel neuro-symbolic design that effectively integrates the power of VLMs, symbol grounding, and explicit compositional reasoning under uncertainty.\n- The score-based perceptual grounding interface elegantly connects neural perception with symbolic reasoning.\n- The modular, differentiable design is appealing as it supports both zero-shot use and potential fine-tuning without dataset-specific supervision.", "weaknesses": "- While the breadth of experiments is commendable, the sheer number of evaluations can make the presentation somewhat overwhelming. In particular, the new experiments introduced in the discussion section are confusing, as they could be more appropriately integrated into the main experimental section. For example, Table 9 provides a helpful overview of the different datasets and could be placed earlier to illustrate NePTune’s general effectiveness. More broadly, adding a concise overview of the datasets, models, and baselines at the beginning of the experimental section would substantially improve readability. It might also be worth considering focusing on a smaller subset of key experiments in the main paper and moving some of the less central results to the appendix. For instance, Table 5 appears useful as supplementary evidence but is not discussed in much detail, suggesting it could be moved without loss of clarity.\n\n- The paper would benefit from a more detailed discussion of how the framework scales when applied to larger VLMs or more complex perceptual inputs, as well as an analysis of its computational overhead compared to direct prompting. It would also be valuable to elaborate on potential limitations that may arise despite high perceptual quality in the VLM or strong program generation capabilities from the LLM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZ3IgZqKkf", "reviewer": "Reviewer_vXFY", "summary": "This paper investigates integrating both the coding and perception abilities of foundation models within a near-symbolic framework to improve the reasoning failures of the foundation models alone. The authors propose a three stage processing pipeline including structured and targeted queries to VLMs which are integrated within code executions obtained from an LLM. The goal of the approach is to tackle zero-shot processing. The authors evaluate their Neptune approach on several, mainly CLEVR-based datasets.", "questions": "What exactly is a \"response\" in ll 236? \n\nHow does Neptune decide when to use a \"query\" \n\nWhat is the intuition/benefit of having soft compositional reasoning components? The authors seemed quite brief on this.\n\nI don't quite understand the intuition/explanation for the weaker \"query attribute\" and \"compare attribute\" results in table 3? Could the authors please elaborate on this?\n\nTab. 7 how is the VLM fine-tuned? What for is it fine-tuned?\n\nWhy not integrate table 9 with the other evaluations around RQ 1?\n\nWhat is the importance of repeatedly naming models end-to-end when they are not actually trained end-to-end for the evaluations in the paper? Moreover, if Neptune is based on these end-to-end models, also Neptune can profit from end-to-end pretraining.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The topic of this work tackles many relevant aspects of current foundation models. The experimental evaluation is generally well structured and presented. The results show promising boosts particularly over the pure VLM/LLM baselines.", "weaknesses": "Generally, I don't have any fundamental issues, just some information is missing and a little unclear. I think if the authors could clarify these/ integrate them into the paper, it would be easier to understand the contribution of the work.\n\nWhat I am definitely missing is a section specifying the exact model setups and high-level implementation information regarding Neptune, but also the baselines. I would place this before section 4.1. I.e. what models were used for the Neptune results? What models are being compared to? Currently all of this information is spread out, if at all, across section 4. \n\nThe authors provide good RQs in Section 4, however these are never referenced after being introduced. For improved readability I suggest to reference them within each experimental subsection.\n\nI am a little confused about which baselines are compared to for which evaluations. E.g. why is MDETR used in Table 5, but not for the evaluations for Table 3 or 4? Same goes for Naver.\n\nWhy don't the authors discuss and compare to older NeSy approaches such as [1]?\n\nI find the Discussion particularly around table 8 somewhat confusing. What are the authors trying to show? In my understanding the point is to show that the basic \"perception\" performance are quite high?\n\n[1] Yi, Kexin, et al. \"Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.\" Advances in neural information processing systems 31 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)