NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language

## Abstract

Abstract Modern Vision-Language Models (VLMs) have achieved impressive performance in various tasks, yet they often struggle with compositional reasoning, the ability to decompose and recombine concepts to solve novel problems. While neuro-symbolic approaches offer a promising direction, they are typically constrained by crisp logical execution or predefined predicates, which limit flexibility. In this work, we introduce NePTune, a neuro-symbolic framework that overcomes these limitations through a hybrid execution model that integrates the perception capabilities of foundation vision models with the compositional expressiveness of symbolic reasoning. NePTune dynamically translates natural language queries into executable Python programs that blend imperative control flow with soft logic operators capable of reasoning over VLM-generated uncertainty. Operating in a training-free manner, NePTune, with a modular design, decouples perception from reasoning, yet its differentiable operations support fine-tuning. We evaluate NePTune on multiple visual reasoning benchmarks and various domains, utilizing adversarial tests, and demonstrate a significant improvement over strong base models, as well as its effective compositional generalization and adaptation capabilities in novel environments.

## 1 Introduction

One key aspect of intelligence is the ability to generalize and compose known basic components to solve novel, complex problems. In vision-language reasoning, this capacity for compositional generalization is crucial. Humans easily decompose a complex query like “Is there a brown object behind every red sphere?” into its constituent concepts and reason about their relationships (Partee et al., 1984), as illustrated in Figure [1](https://arxiv.org/html/2509.25757v1#S0.F1). However, modern vision-language models with transformer-based architectures often fail at dealing with novel compositions, revealing a gap between their pattern-matching skill and robust, human-like understanding (Zhu et al., 2022).

Recent studies have demonstrated the superiority of neuro-symbolic (NeSy) methods over end-to-end neural architectures for reasoning beyond their observed data (Kamali et al., 2025; Yun et al., 2023). In recent years, methods such as VisProg (Gupta & Kembhavi, 2023) and ViperGPT (Surís et al., 2023) have leveraged Large Language Models (LLMs) to produce programs from language queries, breaking down complex questions into a sequence of executable steps. These programs are then run using pre-trained vision models for perceptual grounding. Although powerful, this paradigm has key shortcomings. First, these systems often rely on a sequence of crisp, intermediate decisions, creating brittle pipelines that are sensitive to perceptual errors. A single mistake in an early step (e.g., misidentifying an object) can cause the entire reasoning chain to fail. Second, these pipelines are inference-only and non-differentiable, which prevents them from adapting to new domains where pre-trained models may not perform well. On the other hand, methods such as LEFT (Hsu et al., 2024) and NeSyCoCo (Kamali et al., 2025) employ differentiable declarative symbolic reasoning, but their predicates are limited to the set of learned predicates from training on the target domain, which limits their zero-shot applicability. Table [1](https://arxiv.org/html/2509.25757v1#S1.T1) summarizes this landscape and highlights where NePTune differs.

**Table 1: Comparison of recent neuro-symbolic frameworks. We analyze the reasoning scope (local vs global), types of supported predicates, and predicate source. VFM: vision foundation models, e.g. XVLM; VLM: vision language models, e.g. Qwen2VL.**
|  |  |  | Predicate Types | Predicates |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Reasoning | Supported Predicates | Class | Attr. | Rel. | Spatial | Pretrained | Trainable |
| VisProg | Local | Predefined Modules | ✓ | ✓ | ✗ | ✓ | Modules | ✗ |
| ViperGPT | Local | Dynamic | ✓ | ✓ | ✗ | ✓ | VFMs | ✗ |
| LEFT | Global | Limited to Training Data | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| NeSyCoCo | Global | Limited to Similar Data as Training | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |
| NAVER | Global | Dynamic | ✓ | ✓ | ✓ | ✓ | VFMs | ✗ |
| NePTune | Hybrid | Dynamic | ✓ | ✓ | ✓ | ✓ | VLMs | ✓ |

To address the challenges of expressive compositional inference and domain adaptation, we propose NePTune, a neuro-symbolic visual reasoning framework. NePTune leverages an LLM to generate expressive Python programs where its execution does not solely rely on crisp decisions. Instead, it performs both soft compositional reasoning on the continuous scores provided by a VLM under uncertainty and imperative sequential reasoning. The result is a framework that decouples reasoning from the perception of atomic concepts, leading to remarkable generalization.

## 2 Related Work

NePTune is a framework that integrates multi-modal foundation models and logical reasoning to achieve compositional vision-language reasoning. Therefore, we focus on the three topics below.

### 2.1 Compositional Vision-Language Reasoning

Compositional reasoning is central to building reliable vision–language systems (Sinha et al., 2024; Hupkes et al., 2020), enabling models to represent and execute multi-step, relational structures (Ontañón et al., 2021). While VLMs achieve strong results across broad tasks  (Xiao et al., 2023a; Wang et al., 2024; Liu et al., 2024), diagnostic evaluations like CLEVR (Johnson et al., 2017a) and ReaSCAN (Wu et al., 2021) expose persistent gaps in compositional generalization (Zhu et al., 2022) and brittleness under distribution shift (Yun et al., 2023; Li et al., 2025) in end-to-end neural methods. Although improved architectures show promise (Qiu et al., 2021), explicit and structured neuro-symbolic methods have been particularly effective on these challenges (Kamali et al., 2025). Motivated by these findings, we adopt a neuro-symbolic approach that performs explicit compositional reasoning to achieve more robust and generalizable visual reasoning.

### 2.2 Neuro-Symbolic Approaches

Neuro-symbolic methods integrate neural perception with symbolic reasoning. Recent approaches utilize LLMs as semantic parsers to interpret linguistic queries. There are two common strategies for such methods. One line of work, including VisProg (Gupta & Kembhavi, 2023) and ViperGPT (Surís et al., 2023), generates imperative code for a sequence of local reasoning steps that rely on crisp decisions. While powerful, the sequential nature of these methods limits global reasoning under the uncertainty of these local decisions. The second strategy, employed by models such as NSCL (Mao et al., 2019), LEFT (Hsu et al., 2024), and NeSyCoCo (Kamali et al., 2025), focuses on generating a declarative, symbolic representation of the query to enable global reasoning. However, these frameworks rely on training data for concept learning and grounding in each domain, and are therefore constrained by the observed predicates in the training data. While NeSyCoCo improves concept generalization by utilizing a predicate embedding model and covers the lexical variety of natural language, it remains limited to concepts similar to those in its training data. NAVER (Cai et al., 2025), on the other hand, emphasizes orchestration through a finite-state controller that manages neural perception and utilizes ProbLog for global reasoning in referring expression grounding. However, our findings indicate that generating ProbLog queries is more error-prone for LLMs compared to Python code. Our aim in this work is to synthesize the advantages of both approaches. NePTune is a neuro-symbolic framework that performs both imperative and soft compositional reasoning over atomic predicates. Unlike works like NeSyCoCo and LEFT, the training is not necessary since the concept scores can be obtained zero-shot by harnessing the power of VLMs. However, with differentiable composition functions, it offers optional fine-tuning for domain adaptability by providing the capacity for both compositional inference as well as neuro-symbolic training.

### 2.3 Predicate Level Concept Understanding

A critical distinction between reasoning frameworks lies in how they derive and utilize concept-level understanding. General-purpose VLMs (Wang et al., 2024; Liu et al., 2024) learn concepts implicitly through end-to-end training, embedding this knowledge directly into the model’s weights. In contrast, neuro-symbolic methods externalize this process. A program generating framework, such as VisProg (Gupta & Kembhavi, 2023), utilizes the concept grounding through a library of specialized, trained, hard-coded vision APIs, which return discrete labels or values. Similarly, ViperGPT (Surís et al., 2023) utilizes foundation vision-language models, such as XVLM (Zeng et al., 2021), to obtain labels and discrete decisions. Another class of methods, including LEFT and NeSyCoCo, trains an MLP to generate continuous concept scores, representing the relations or attributes of an object in the visual regions. Our approach utilizes visual prompting, highlighting parts of the image via bounding boxes, to elicit referential concept scores from VLMs, which then serve as scores within our framework for reasoning under uncertainty.

## 3 Methodology

The NePTune framework performs visual reasoning via three core components, as shown in Figure [2](https://arxiv.org/html/2509.25757v1#S3.F2). The process begins with the LLM-based Program Generator, which translates a natural language query into both a Python program and a set of relevant object names. These names are passed to the Perceptual Grounding to detect all relevant candidate objects in the scene. Finally, the Symbolic Executor runs the generated program. During execution, it interacts with the Grounding Interface to obtain atomic concept scores and reason over them to get the final answer.

Figure: Figure 2: NePTune overview. Given an image and a query, the (1) LLM-based Program Generation converts the natural language query to a Pythonic program. Then (2) Perceptual Grounding extracts the object bounding boxes. The (3) Symbolic Executor then runs the Python code to reason over concepts extracted from the VLM using both soft composition and imperative logic to derive the final answer.
Refer to caption: x2.png

### 3.1 Component 1: LLM-based Program Generation

The first component of NePTune addresses semantic parsing. We leverage an LLM as a powerful few-shot parser to convert a natural language query into a formal, executable Python program. Given a query like “Is there a big brown dog?”, the LLM is prompted to decompose it into a multi-step program that first identifies a set of candidate “dogs”, and then reasons over the composition of atomic concepts such as “big”, “dog”, and “brown” to get the final answer. In the same LLM call, it also extracts object names for the region proposals. We chose Python as the target symbolic language for two key reasons. First, its Turing-complete nature, including rich control flow structures like loops and conditionals, is essential for expressing complex procedural reasoning. Second, the prevalence of open-source Python code makes LLMs particularly adept at generating it.

### 3.2 Component 2: Perceptual Grounding

This component connects the symbolic program to the visual world. It consists of two main parts: an object proposal module and a concept grounding module.

#### 3.2.1 Object Proposal Generation

To ground objects in the scene to the concepts such as blue in the program, we must first identify all potentially relevant objects in the scene. We use the object names extracted by the LLM. For example, from “Are there more cats to the left of the vase or dogs?”, the LLM extracts “dog”, “vase”, and “cat”. We then feed these object names into Grounding DINO (Liu et al., 2023), a zero-shot object detection model that takes the image and the list of object names as input and outputs a set of bounding boxes for all matching objects. We note that while this textual query-based proposal is chosen for computational efficiency in real-world applications, our framework can operate with any reasonable region proposal method.

#### 3.2.2 Concept Grounding Interface

Once a query is translated into a program, we must ground its atomic predicates to the visual content of the image, such as the objects in the scene. We handle this *perceptual grounding* using a VLM through two primary simplified interfaces score and query, where the image and bounding boxes are abstracted away with global variables for simplicity. The formal definitions of these grounding functions are as follows:

score(query:str, num_objects:int): This function is the core of our reasoner. It takes a natural language question, an image, and related bounding boxes, and returns the concept probabilities for related object bounding boxes. This score is the VLM’s normalized confidence for a ”Yes” answer, computed from the logits of the ”Yes” and ”No” tokens. Formally, given an image $I$, set of $N$ detected objects, a visual prompt $v$, and atomic query $q_{a}$, the score $s$ is:

$$ $s=p(\text{"Yes"}|I,v,q_{a})=\frac{e^{\text{logit}(\text{"Yes"})}}{e^{\text{logit}(\text{"Yes"})}+e^{\text{logit}(\text{"No"})}}$ $$

We treat these outputs as probability scores. To generate an object-centric concept score or answer via a VLM, we represent a symbolic predicate with a natural language question. For example, for the symbolic predicate blue, we use “Is the object in the red bounding box blue?”. The bounding box $b$ provided to the functions will be drawn on target objects as a *visual prompt*, and it is controlled by the num_objects parameter, which specifies the predicate type:

- •
num_objects=0: For queries about the entire image (e.g., “Is the photo taken indoors?”). No visual prompt is used, and the output is a scalar probability.
- •
num_objects=1: For single-object queries, we mark the target object with a red bounding box around it as a visual prompt (e.g., “Is the object in the red bounding box brown?”). The output is a vector of size $N$.
- •
num_objects=2: For multi-object (relational) queries, we mark the main object with a red bounding box, and the secondary object with a green bounding box, as the visual prompt (e.g., “Is the person in the red box talking to the person in the green box?”). The output is an $N\times N$ matrix.

query(query:str, object_id:int):
This query function is used for tasks requiring open-ended answers. It takes a natural language question, an image, and an optional visual prompt, and returns a natural language string. Formally, given an image $I$, visual prompt $v$, and atomic query $q_{a}$, the generated answer $A$:

$$ $A=\operatorname*{argmax}_{\text{response}}p(\text{response}|I,v,q_{a})$ $$

The answer $A$ serves two purposes. First, it can provide the final output for questions that ask ”what” or ”which” (e.g., ‘return query(”What color is the object in the red bounding box?”)‘). Second, it can be used as an intermediate variable to enable complex conditional constructs within the generated Python program, allowing the reasoning path to change based on the VLM’s perception (e.g., ‘if query(”What shape is the object in the red bounding box?”) == ”cube”: …‘).

### 3.3 Component 3: Symbolic Executor

The final component of NePTune is the Symbolic Executor, which runs the LLM-generated program. A key innovation of our framework is its hybrid execution model, which integrates two distinct reasoning modes.

1) Soft Compositional Reasoning: To reason about visual concepts themselves, we employ a set of soft logical operations based on fuzzy logic principles. Instead of operating on binary true/false values, these operations work directly on the uncertainty scores obtained from the VLM. This is implemented through our custom data structures, which encapsulate the scores for a given predicate and overload standard Python operators (& for AND, | for OR) to perform the corresponding logical operations. Details of these operations are shown in Table [2](https://arxiv.org/html/2509.25757v1#S3.T2). For example, when the program executes brown & dog, our framework takes the element-wise minimum of the scores in the two corresponding tensors, implementing the fuzzy t-norm for conjunction.

**Table 2: Mathematical Expressions: Logical Forms, Descriptions, and Differentiable Implementations. Here $\alpha$ represents an object-centric or scalar probabilistic score, $\beta$ represents a relation probability score, $\tau=0.25$ is a temperature parameter, and $\gamma=0.25$ is a margin.**
| Syntax | Logical Form | Description | Differentiable Implementation |
| --- | --- | --- | --- |
| $\alpha_{x}.\texttt{exists}()$ | $\exists x\,\alpha_{x}$ | Existential quantification | $\max(\alpha_{x})$ |
| $\alpha_{x}.\texttt{forall}()$ | $\forall x\,\alpha_{x}$ | Universal quantification | $\min(\alpha_{x})$ |
| $\alpha_{x}\,\&\,\alpha_{y}$ | $\alpha_{x}\land\alpha_{y}$ | Logical conjunction | $\min(\alpha_{x},\alpha_{y})$ |
| $\alpha_{x}\,\&\,\beta_{xy}$ | $\alpha_{x}\land\beta_{xy}$ | Relational conjunction | $\sum_{y}\alpha_{x}\cdot\beta_{xy}$ |
| $\alpha_{x}\,|\,\alpha_{y}$ | $\alpha_{x}\lor\alpha_{y}$ | Logical disjunction | $\max(\alpha_{x},\alpha_{y})$ |
| $\alpha_{x}.\texttt{implies}(\alpha_{y})$ | $\alpha_{x}\rightarrow\alpha_{y}$ | Logical implication | $\max(1-\alpha_{x},\alpha_{y})$ |
| $\texttt{not }\alpha_{x}$ | $\neg\alpha_{x}$ | Logical negation | $1-\alpha_{x}$ |
| $\alpha_{x}.\texttt{iota}(\text{var})$ | $\iota(\text{var},\alpha_{x})$ | Best match | $\operatorname{softmax}(\alpha_{x})$ |
| $\alpha_{x}.\texttt{count}()$ | $\text{count}(\alpha_{x})$ | Counting elements | $\sum\alpha_{x}$ |
| $s_{1}==s_{2}$ | $s_{1}=s_{2}$ | Scalar equality | $\sigma\left(\frac{\tau(\gamma-|s_{1}-s_{2}|)}{\gamma}\right)$ |
| $s_{1}>s_{2}$ | $s_{1}>s_{2}$ | Scalar inequality | $\sigma\left(\tau(s_{1}-s_{2}-1+\gamma)\right)$ |

2) Imperative Reasoning: Our symbolic executor leverages a standard Python interpreter to handle the program’s overall structure and control flow. This is possible since we have defined iteration and Boolean operations on the concept objects, allowing for complex, procedural logic, including conditionals (*if*/*else*), loops (*for*), and variable assignments, giving our framework the full expressive power of a general-purpose programming language. While some of these operations such as counting over string sets (as shown in Figure [2](https://arxiv.org/html/2509.25757v1#S3.F2)) can break the computation graph. This hybrid design enables the system to reason fluidly under uncertainty while fully leveraging the expressive power of a general-purpose programming language.

## 4 Experiments

We structure our experiments around four research questions. RQ1: How does NePTune perform on zero-shot compositional reasoning on synthetic data? (Experiment 1), RQ2: How does NePTune perform on complex human-generated questions? (Experiment 2), RQ3: How does NePTune perform in grounding referring expressions in natural images? (Experiment 3), and RQ4: How does NePTune perform under domain shift and adapt to an unseen environment? (Experiment 4).
To address these questions, we evaluate NePTune across a diverse set of datasets covering synthetic, human-annotated, and realistic environments, spanning both question answering and referring expression grounding tasks. Details of the datasets and tasks are provided in Appendix [D](https://arxiv.org/html/2509.25757v1#A4).

### 4.1 Experiment 1: Core Compositional Reasoning

**Table 3: Accuracy comparison across CLEVR question categories. Results are shown by reasoning type and model paradigm, including zero-shot, end-to-end, and neuro-symbolic approaches.**
|  | InternVL2.5 | NePTune | ViperGPT | NeSyCoCo | LEFT |
| --- | --- | --- | --- | --- | --- |
| Training | Zero-Shot | Zero-Shot | Zero-Shot | Trained | Trained |
| Category | End-to-End | NeSy | NeSy | NeSy | NeSy |
| Final Accuracy | 90.25 | 92.65 ($\uparrow$ 2.40) | 36.05 | 99.68 | 99.50 |
| Exist | 87.10 | 93.19 ($\uparrow$ 6.09) | 48.75 | 99.28 | 98.92 |
| Query Attribute | 98.26 | 96.81 ($\downarrow$ 1.45) | 29.42 | 100.00 | 99.86 |
| Compare Attribute | 98.61 | 91.94 ($\downarrow$ 6.67) | 53.06 | 99.44 | 99.72 |
| Count | 74.60 | 87.10 ($\uparrow$ 12.50) | 21.37 | 99.79 | 98.99 |
| Compare Number | 90.86 | 92.57 ($\uparrow$ 1.71) | 48.57 | 100.00 | 100.00 |

To evaluate NePTune’s reasoning capabilities, we evaluate its performance on the CLEVR benchmark. CLEVR is a standard benchmark for compositional reasoning that features synthetic 3D-rendered images and questions testing compositional visual reasoning. Details of the question categories are provided in Appendix [D](https://arxiv.org/html/2509.25757v1#A4). The results in Table [3](https://arxiv.org/html/2509.25757v1#S4.T3) show that, within the family of neuro-symbolic systems, NePTune establishes itself as the strongest zero-shot method, achieving 92.65% accuracy compared to only 36.05% for ViperGPT. While trained approaches such as NeSyCoCo and LEFT nearly saturate CLEVR (99%), NePTune demonstrates that competitive compositional reasoning can be achieved without dataset-specific supervision. Compared to its backbone VLM, InternVL2.5, NePTune still yields improvements raising overall accuracy from 90.25% to 92.65% ($\uparrow$ 2.40%). The largest gains appear in quantitative categories where explicit compositional structure is most useful: Count rises from 74.60% to 87.10% ($\uparrow$ 12.50%), and Compare Number increases from 90.86% to 92.57% ($\uparrow$ 1.71%). We also see a notable improvement on Exist from 87.10% to 93.19% ($\uparrow$ 6.09%), consistent with the executor reducing spurious correlation when filtering by attributes and relations. In contrast, attribute-heavy categories regress: Query Attribute drops by $\downarrow$ 1.45% points and Compare Attribute drops by $\downarrow$ 6.67% points. A closer look at concept-level accuracy revealed that analogical concepts such as same color or same shape remain among the most challenging to capture in this benchmark. More discussion of atomic concept evaluation is provided in Section [5](https://arxiv.org/html/2509.25757v1#S5).

**Table 4: Accuracy on CLEVR extension tasks. Methods marked with ^† use ground-truth programs. Improvements from the backbone VLM (InternVL2.5) are marked with the arrow sign ($\uparrow$).**
|  | Method | Ref(%) | Puzzles(%) | RPM(%) |
| --- | --- | --- | --- | --- |
| Trained | NeSyCoCo^† | 100.00 | 95.00 | 100.00 |
| NeSyCoCo | 94.00 | 94.00 | 74.00 |  |
| LEFT | 94.00 | 85.00 | 87.00 |  |
| Zero-shot | Qwen2VL.5-8B | 21.00 | 43.00 | 53.00 |
| InternVL2.5-8B | 27.00 | 52.00 | 47.00 |  |
| Ovis1.6-9B | 4.00 | 47.00 | 49.00 |  |
| ViperGPT | 8.00 | 34.00 | 4.00 |  |
| VisProg | 35.00 | 27.00 | 51.00 |  |
| NePTune^† | 99.00 ($\uparrow$ 72) | 65.00 ($\uparrow$ 13) | 99.00 ($\uparrow$ 52) |  |
| NePTune | 91.00 ($\uparrow$ 64) | 60.00 ($\uparrow$ 8) | 80.00 ($\uparrow$ 33) |  |

In addition to the original CLEVR benchmark, we evaluate our method on various compositional challenges based on the CLEVR environment introduced in LEFT (Hsu et al., 2024), including referring expressions CLEVR-Ref, visual puzzles CLEVR-Puzzles, and Raven’s Progressive Matrices CLEVR-RPM. As shown in Table [4](https://arxiv.org/html/2509.25757v1#S4.T4), NePTune shows the highest performance among the zero-shot methods and outperforms the end-to-end VLMs. In addition, compared to NeSyCoCo and LEFT with trained concepts, we show competitive performance.

### 4.2 Experiment 2: Complex Human Queries

**Table 5: Accuracy on the CLEVR-Humans (CH) dataset.**
|  | Method | CH (%) |
| --- | --- | --- |
| Trained | LEFT | 56.69 |
| NeSyCoCo | 56.12 |  |
| MDETR | 81.73 |  |
| Zero-shot | Qwen2VL-7B | 84.12 |
| InternVL2.5-8B | 85.95 |  |
| Ovis1.6-9B | 79.96 |  |
| ViperGPT | 31.05 |  |
| NePTune | 87.67 |  |

Here, we evaluate NePTune in a more complex and diverse human-generated language. We utilize the CLEVR-Humans (Johnson et al., 2017b) benchmark to evaluate and compare NePTune against other neuro-symbolic and end-to-end models. As shown in Table [5](https://arxiv.org/html/2509.25757v1#S4.T5), NePTune significantly outperforms prior neuro-symbolic methods such as LEFT and NeSyCoCo by a large margin of $\uparrow$ 30.98% even compared to end-to-end methods such as MDETR (Kamath et al., 2021). Furthermore, it improves upon its powerful end-to-end backbone (InternVL2.5-8B) by $\uparrow$ 1.72%, demonstrating its effectiveness on complex, human-generated questions.

### 4.3 Experiment 3: Real-world Images

**Table 6: Accuracy on Ref-Adv dataset.**
| Method | Ref-Adv (%) |
| --- | --- |
| Grounding DINO-B | 60.85 |
| Florence2-L | 71.73 |
| Ovis1.6-9B | 30.70 |
| InternVL2-8B | 72.92 |
| InternVL2.5-8B | 76.13 |
| ViperGPT | 60.66 |
| NAVER^† | 36.45 |
| NAVER | 65.13 |
| NePTune^‡ | 63.71 |
| + Verification | 75.54 |
| NePTune | 71.57 |
| + Verification | 78.08 |
| NePTune (1B) | 60.69 |
| + Fine-tuning | 68.06 $\pm$ 0.56 |
| + Verification | 74.59 $\pm$ 0.12 |

While various versions of CLEVR used in the aforementioned experiments are strong testbeds for compositional reasoning, after all, these are toy environments with limited diversity. To demonstrate NePTune’s ability to operate in realistic environments, we evaluate it on Referring Expression Grounding (REG) of real-world images using the RefCOCO-Adversarial (Akula et al., 2020) (Ref-Adv) benchmark. On Ref-Adv, NePTune is the strongest zero-shot reasoner among symbolic baselines such as ViperGPT and NAVER. To ensure a fair comparison with NAVER, we follow its setup and use the same backbones (Grounding DINO + XVLM + InternVL2-8B), denoted as NePTune^‡. Under this setting, NePTune^‡ reaches 63.71 vs. 36.45 for NAVER^† (execution only), and with a simple verification (similar to NAVER) attains 71.57 vs. 65.13 for NAVER. Compared to end-to-end methods, NePTune surpasses all the specialized grounding methods such as Grounding DINO and Florence2 (Xiao et al., 2023b). Additionally, compared to their VLM backbones, both NePTune and NePTune^‡ with verification surpass their respective backbones. Details on verification are available in Appendix [E](https://arxiv.org/html/2509.25757v1#A5).

### 4.4 Experiment 4: Generalization and Adaptation

When using popular benchmarks, as we did in Experiments 1-3, the possible data contamination makes zero-shot performance less reliable. In this experiment, we test NePTune’s ability to generalize and adapt to novel environments using a less popular photo-realistic gaming environment that is Ref-GTA (Tanaka et al., 2019) benchmark. Ref-GTA is a challenging REG benchmark with images from a game simulation, which creates a significant domain shift from the natural images on which most VLMs are pre-trained.

**Table 7: Accuracy on Ref-GTA benchmark.**
| Method | Ref-GTA(%) |
| --- | --- |
| GroundingDINO-B | 27.90 |
| Florence2-L | 58.65 |
| Ovis1.6-9B | 2.78 |
| InternVL2.5-8B | 6.95 |
| InternVL2.5-1B | 1.64 |
| + Fine-tuning | 32.61 $\pm$ 0.35 |
| ViperGPT | 1.40 |
| NAVER^† | 54.84 |
| NAVER | 58.73 |
| NePTune^‡ | 62.73 |
| NePTune | 69.69 |
| NePTune (1B) | 34.92 |
| + Fine-tuning | 69.90 $\pm$ 1.16 |

Based on the reports of Ovis1.6 (Lu et al., 2024) and InternVL2.5 (Chen et al., 2025), this source is not included in their training data. As shown in Table [7](https://arxiv.org/html/2509.25757v1#S4.T7), this domain shift causes a significant performance drop for most methods. The powerful end-to-end InternVL2.5-8B model, for example, fails catastrophically, with its accuracy dropping to 6.95%. In contrast, NePTune, when paired with the same VLM, achieves a remarkable 69.69% accuracy. This demonstrates that by utilizing our global symbolic reasoner and atomic-level concept understanding, our framework achieves robustness and compositional generalization that monolithic models lack. Furthermore, while NePTune demonstrates strong zero-shot robustness, its soft operations also enable fine-tuning. As shown in Table [7](https://arxiv.org/html/2509.25757v1#S4.T7), by fine-tuning a smaller VLM (InternVL2.5-1B) using our neuro-symbolic differentiable computations with only 1000 samples, we can further improve its performance and obtain 69% accuracy.
While fine-tuning VLM using original neural training, it only reaches 32% accuracy. These findings are promising for future research and using neuro-symbolic reasoning as a source of supervision for larger-scale VLM training. More details on fine-tuning are presented in Appendix [F](https://arxiv.org/html/2509.25757v1#A6).

## 5 Discussion

##### VLM for Concept Understanding.

A central idea of NePTune is to employ VLMs as underlying perception modules and concept grounders. The basic assumption here is that while VLMs are prone to fail at reasoning over complex compositions, they should perform better in perceiving basic concepts. However, basic perception questions require isolating parts of the visual input, a process known as visual prompting, which has been shown to be challenging in complex questions (Cai et al., 2024). In this section, we aim to evaluate the hypothesis that modern VLMs guided by visual prompts are effective as concept grounders for our purpose. We utilize ground-truth scene graphs from the CLEVR (Johnson et al., 2017a) and Visual Genome (Krishna et al., 2016) datasets to automatically generate a set of atomic templated questions. For CLEVR, we create a benchmark using scene graphs of 200 sampled scenes, and for Visual Genome, we randomly sample 1000 questions from 200 scene graphs. We generate simple Yes or No questions about their class, attributes, and relations, such as, “Is the object in the bounding box a bird?”. We then use our score function that employs the underlying VLM to answer these atomic questions.

**Table 8: Backbone VLMs performance on CLEVR and VG scene graphs (Micro F1-Score x 100)**
| Category | InternVL2.5-8B | Ovis1.6-9B | Qwen2VL-7B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | CLEVR | VG | CLEVR | VG | CLEVR | VG |
| Attribute | 97.25 | 89.37 | 95.87 | 86.75 | 89.27 | 83.45 |
| Class/Object | 90.00 | 81.76 | 91.98 | 80.42 | 82.21 | 83.28 |
| Relation | 90.94 | 82.35 | 87.91 | 80.00 | 80.04 | 83.02 |
| Spatial | 89.82 | 93.42 | 86.83 | 93.39 | 89.53 | 94.41 |
| Overall | 94.54 | 90.41 | 90.35 | 88.19 | 86.59 | 90.77 |

The results are shown in Table [8](https://arxiv.org/html/2509.25757v1#S5.T8). Our analysis reveals that VLMs are particularly strong at identifying object, classes, properties, and spatial relations. However, the performance is weaker on more complex relational concepts, especially for analogical comparisons such as same size. This problem is more severe in the Qwen2VL model. Although visual prompting can be problematic for general VQA (Cai et al., 2024), our results show this is not as challenging in our setting for grounding atomic queries. These results demonstrate that VLMs perform significantly better in answering atomic queries compared to complex multi-step queries, as reported in Tables [4](https://arxiv.org/html/2509.25757v1#S4.T4) and [5](https://arxiv.org/html/2509.25757v1#S4.T5). For example, there is a performance gap of up to 67% when comparing CLEVR-Ref (27%) compared to CLEVR average atomic evaluation (94%) using InternVL2.5. Detailed concept-level results of the experiment are presented in the Appendix [B](https://arxiv.org/html/2509.25757v1#A2). Furthermore, the results in Table [9](https://arxiv.org/html/2509.25757v1#S5.T9) demonstrate the end-to-end performance of different VLMs in NePTune, where backbones show an average improvement of up to $\uparrow$ 29%.

**Table 9: Performance of different VLM backbones (end-to-end) and with NePTune across benchmarks. $\Delta$ values indicate gains or losses relative to the raw VLMs.**
| Model | CLEVR (%) | Ref-Adv (%) | Ref-GTA (%) | Ref (%) | Puzzles (%) | RPM (%) | Avg. (%) |
| --- | --- | --- | --- | --- | --- | --- | --- |
| InternVL2.5-8B | 90.25 | 76.13 | 6.95 | 27.00 | 52.00 | 47.00 | 49.89 |
| + NePTune | 92.65 $\uparrow$ 2.40 | 78.08 $\uparrow$ 1.95 | 69.69 $\uparrow$ 62.74 | 91.00 $\uparrow$ 64.00 | 60.00 $\uparrow$ 8.00 | 80.00 $\uparrow$ 33.00 | 78.57 ($\uparrow$ 28.68) |
| Ovis1.6-9B | 85.00 | 58.47 | 2.78 | 4.00 | 47.00 | 49.00 | 41.04 |
| + NePTune | 88.80 $\uparrow$ 3.80 | 63.25 $\uparrow$ 4.78 | 56.32 $\uparrow$ 53.54 | 75.00 $\uparrow$ 71.00 | 57.00 $\uparrow$ 10.00 | 80.00 $\uparrow$ 31.00 | 70.06 ($\uparrow$ 29.02) |
| Qwen2VL-7B | 93.55 | 81.07 | 1.75 | 21.00 | 43.00 | 53.00 | 48.90 |
| + NePTune | 82.55 $\downarrow$ 11.00 | 80.93 $\downarrow$ 0.14 | 68.87 $\uparrow$ 67.12 | 71.00 $\uparrow$ 50.00 | 53.00 $\uparrow$ 10.00 | 71.00 $\uparrow$ 18.00 | 71.23 ($\uparrow$ 22.33) |

**Table 10: Ablation study on CLEVR-Humans (CH).**
| Ablation Setting | CH (%) |
| --- | --- |
| Declarative + Trained Concepts | 56.12 |
| + VLM Concepts | 68.48 ($\uparrow$ 12.36) |
| + Imperative Reasoning | 87.67 ($\uparrow$ 19.19) |

##### Ablation Study.

Table [10](https://arxiv.org/html/2509.25757v1#S5.T10) presents an ablation study of NePTune on CLEVR-Humans, starting from a declarative reasoning backbone. Incorporating InternVL2.5 as the concept scoring module yields a $\uparrow$ 12.36 improvement, showing that strong VLMs provide more robust and generalizable concept grounders than trained concept grounders on CLEVR. Building on this, adding NePTune’s imperative reasoning further boosts accuracy by $\uparrow$ 19.19, closing the gap left by purely declarative systems.

##### Choice of Symbolic Program.

We analyze program execution success rates (no syntax or runtime errors) on 500 Ref-Adv and CLEVR-Humans using the GPT-4o (OpenAI-Team, 2024) LLM. As shown in Table [11](https://arxiv.org/html/2509.25757v1#S5.T11), frameworks that rely on specialized, non-Pythonic syntax, such as LEFT, NeSyCoCo, and NAVER (which uses ProbLog), frequently suffer from generation failures.

**Table 11: Execution success rate (%) of different neuro-symbolic methods on Ref-Adv and CLEVR-Humans. Incompatible models are excluded.**
| Method | Ref-Adv | CLEVR |
| --- | --- | --- |
| NeSyCoCo | N/A | 70.33$\pm$3.47 |
| LEFT | N/A | 64.33$\pm$3.94 |
| ViperGPT | 48.67$\pm$1.70 | 95.42$\pm$0.30 |
| NAVER | 23.02$\pm$4.71 | N/A |
| NePTune | 98.66$\pm$0.82 | 97.24$\pm$0.85 |

We found that the LLM often struggles to correctly translate natural language into their specific, strict formalisms. In contrast, methods that utilize Python, such as ViperGPT and NePTune, demonstrate significantly higher rates of successful program execution.
However, our analysis of ViperGPT’s failures reveals that its purely imperative approach, which often relies on selecting objects by a fixed index, is a primary source of error. The NePTune hybrid reasoner proves to be the most robust. Our analysis shows that the main source of error for NePTune shifts from low-level syntax errors to higher-level logical errors, where the LLM fails to compose the correct sequence of predicates. Qualitative examples are shown in Appendix [A](https://arxiv.org/html/2509.25757v1#A1).

## 6 Conclusion

In this work, we introduce NePTune, a novel neuro-symbolic framework for compositional vision-language reasoning. Our approach leverages a novel hybrid reasoner that combines the imperative control flow of Python with a declarative, probabilistic soft logic that operates directly on the uncertain outputs of a VLM. This design enables our framework to be both highly expressive and robust in the face of perceptual uncertainty.
Our extensive experiments demonstrate the effectiveness of this approach. NePTune demonstrates clear improvement over strong baselines, as well as the underlying VLMs, on complex compositional reasoning benchmarks, in visual question answering, and referring expression, exhibiting remarkable generalization capabilities. While the performance of NePTune is dependent on the quality of its underlying components, such as the LLM for program generation and the VLM for concept grounding, our work demonstrates a flexible and powerful paradigm for building more robust and generalizable AI systems, leveraging these models to elevate their capabilities to a comparatively higher level of performance. Future work could explore methods for more efficient grounding and the integration of our work into visual reasoning orchestration.

## Acknowledgments

This project is supported by the Office of Naval Research (ONR) grant N00014-23-1-2417. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Office of Naval Research.

## Ethics Statement

This work complies with the ICLR Code of Ethics. NePTune builds on publicly available datasets and open-source VLMs, without collecting new human-subject data. Large Language Models were used solely for grammar polishing and minor text improvements.

## Reproducibility Statement

We detail model design, training, and evaluation in the paper and appendix, including hyperparameters and dataset preprocessing. Ablations and failure analyses clarify component contributions. Anonymized code and scripts will be released to enable reproduction of all reported results.

## References

- Akula et al. (2020)
Arjun Akula, Spandana Gella, Yaser Al-Onaizan, Song-Chun Zhu, and Siva Reddy.
Words aren’t enough, their order matters: On the robustness of
grounding visual referring expressions.
In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault
(eds.), *Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics*, pp. 6555–6565, Online, July 2020. Association
for Computational Linguistics.
doi: 10.18653/v1/2020.acl-main.586.
URL [https://aclanthology.org/2020.acl-main.586/](https://aclanthology.org/2020.acl-main.586/).
- Cai et al. (2024)
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai,
Dennis Park, and Yong Jae Lee.
Making large multimodal models understand arbitrary visual prompts.
In *IEEE Conference on Computer Vision and Pattern Recognition*,
2024.
- Cai et al. (2025)
Zhixi Cai, Fucai Ke, Simindokht Jahangard, Maria Garcia de la Banda, Reza
Haffari, Peter J. Stuckey, and Hamid Rezatofighi.
Naver: A neuro-symbolic compositional automaton for visual grounding
with explicit logic reasoning.
*arXiv preprint arXiv:2502.00372*, 2025.
- Chen et al. (2025)
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo
Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li,
Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang, Tan Jiang, Bo Wang, Conghui
He, Botian Shi, Xingcheng Zhang, Han Lv, Yi Wang, Wenqi Shao, Pei Chu,
Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng
Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin,
Yu Qiao, Jifeng Dai, and Wenhai Wang.
Expanding performance boundaries of open-source multimodal models
with model, data, and test-time scaling, 2025.
URL [https://arxiv.org/abs/2412.05271](https://arxiv.org/abs/2412.05271).
- DeepSeek-AI-Team (2025)
DeepSeek-AI-Team.
Deepseek-v3 technical report, 2025.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
- Gupta & Kembhavi (2023)
Tanmay Gupta and Aniruddha Kembhavi.
Visual Programming: Compositional visual reasoning without training
.
pp. 14953–14962, June 2023.
doi: 10.1109/CVPR52729.2023.01436.
URL
[https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.01436](https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.01436).
- Honnibal & Montani (2017)
Matthew Honnibal and Ines Montani.
spacy 2: Natural language understanding with bloom embeddings,
convolutional neural networks and incremental parsing, 2017.
To appear.
- Hsu et al. (2024)
Joy Hsu, Jiayuan Mao, Josh Tenenbaum, and Jiajun Wu.
What’s left? concept grounding with logic-enhanced foundation
models.
*Advances in Neural Information Processing Systems*, 36, 2024.
- Hudson & Manning (2019)
Drew A. Hudson and Christopher D. Manning.
Gqa: A new dataset for real-world visual reasoning and compositional
question answering, 2019.
URL [https://arxiv.org/abs/1902.09506](https://arxiv.org/abs/1902.09506).
- Hupkes et al. (2020)
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni.
Compositionality decomposed: How do neural networks generalise?
(extended abstract).
In Christian Bessiere (ed.), *Proceedings of the Twenty-Ninth
International Joint Conference on Artificial Intelligence, IJCAI-20*, pp. 5065–5069. International Joint Conferences on Artificial Intelligence
Organization, 7 2020.
doi: 10.24963/ijcai.2020/708.
URL [https://doi.org/10.24963/ijcai.2020/708](https://doi.org/10.24963/ijcai.2020/708).
Journal track.
- Johnson et al. (2017a)
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei,
C Lawrence Zitnick, and Ross Girshick.
Clevr: A diagnostic dataset for compositional language and elementary
visual reasoning.
In *CVPR*, 2017a.
- Johnson et al. (2017b)
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman,
Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick.
Inferring and executing programs for visual reasoning,
2017b.
- Kamali et al. (2025)
Danial Kamali, J. Barezi, Elham, and Parisa Kordjamshidi.
Nesycoco: A neuro-symbolic concept composer for compositional
generalization.
*Proceedings of the AAAI Conference on Artificial Intelligence*,
39(4):4184–4193, Apr. 2025.
doi: 10.1609/aaai.v39i4.32439.
URL [https://ojs.aaai.org/index.php/AAAI/article/view/32439](https://ojs.aaai.org/index.php/AAAI/article/view/32439).
- Kamath et al. (2021)
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and
Nicolas Carion.
Mdetr - modulated detection for end-to-end multi-modal understanding.
pp. 1760–1770, 2021.
URL
[http://dblp.uni-trier.de/db/conf/iccv/iccv2021.html#KamathSLSMC21](http://dblp.uni-trier.de/db/conf/iccv/iccv2021.html#KamathSLSMC21).
- Krishna et al. (2016)
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua
Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma,
Michael S. Bernstein, and Fei-Fei Li.
Visual genome: Connecting language and vision using crowdsourced
dense image annotations, 2016.
URL [https://arxiv.org/abs/1602.07332](https://arxiv.org/abs/1602.07332).
- Li et al. (2025)
Chuanhao Li, Wenbo Ye, Zhen Li, Yuwei Wu, and Yunde Jia.
Multi-sourced compositional generalization in visual question
answering, 2025.
URL [https://arxiv.org/abs/2505.23045](https://arxiv.org/abs/2505.23045).
- Liu et al. (2024)
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and
Yong Jae Lee.
Llava-next: Improved reasoning, ocr, and world knowledge, January
2024.
URL [https://llava-vl.github.io/blog/2024-01-30-llava-next/](https://llava-vl.github.io/blog/2024-01-30-llava-next/).
- Liu et al. (2023)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan
Li, Jianwei Yang, Hang Su, Jun Zhu, et al.
Grounding dino: Marrying dino with grounded pre-training for open-set
object detection.
*arXiv preprint arXiv:2303.05499*, 2023.
- Lu et al. (2024)
Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and
Han-Jia Ye.
Ovis: Structural embedding alignment for multimodal large language
model.
*arXiv:2405.20797*, 2024.
- Mao et al. (2019)
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu.
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and
Sentences From Natural Supervision.
In *International Conference on Learning Representations*, 2019.
URL [https://openreview.net/forum?id=rJgMlhRctm](https://openreview.net/forum?id=rJgMlhRctm).
- Ontañón et al. (2021)
Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek, and Zachary Fisher.
Making transformers solve compositional tasks.
1:3591–3607, 8 2021.
URL [http://arxiv.org/abs/2108.04378](http://arxiv.org/abs/2108.04378).
- OpenAI-Team (2024)
OpenAI-Team.
Gpt-4o system card, 2024.
URL [https://arxiv.org/abs/2410.21276](https://arxiv.org/abs/2410.21276).
- Partee et al. (1984)
Barbara Partee et al.
Compositionality.
*Varieties of formal semantics*, 3:281–311, 1984.
- Qiu et al. (2021)
Linlu Qiu, Hexiang Hu, Bowen Zhang, Peter Shaw, and Fei Sha.
Systematic generalization on gSCAN: What is nearly solved and
what is next?
In *Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing*, pp. 2180–2188, Online and Punta Cana,
Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.166.
URL [https://aclanthology.org/2021.emnlp-main.166](https://aclanthology.org/2021.emnlp-main.166).
- Sinha et al. (2024)
Sania Sinha, Tanawan Premsri, and Parisa Kordjamshidi.
A survey on compositional learning of ai models: Theoretical and
experimental practices, 2024.
URL [https://arxiv.org/abs/2406.08787](https://arxiv.org/abs/2406.08787).
- Surís et al. (2023)
Dídac Surís, Sachit Menon, Carl Vondrick, and .
Vipergpt: Visual inference via python execution for reasoning, 2023.
- Tanaka et al. (2019)
Mikihiro Tanaka, Takayuki Itamochi, Kenichi Narioka, Ikuro Sato, Yoshitaka
Ushiku, and Tatsuya Harada.
Generating easy-to-understand referring expressions for target
identifications, 2019.
URL [https://arxiv.org/abs/1811.12104](https://arxiv.org/abs/1811.12104).
- Wang et al. (2024)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin
Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du,
Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang
Lin.
Qwen2-vl: Enhancing vision-language model’s perception of the world
at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wolf et al. (2020)
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe
Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien
Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest,
and Alexander M. Rush.
Transformers: State-of-the-art natural language processing.
In *Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing: System Demonstrations*, pp. 38–45, Online,
October 2020. Association for Computational Linguistics.
URL [https://www.aclweb.org/anthology/2020.emnlp-demos.6](https://www.aclweb.org/anthology/2020.emnlp-demos.6).
- Wu et al. (2021)
Zhengxuan Wu, Elisa Kreiss, Desmond C. Ong, and Christopher Potts.
ReaSCAN: Compositional reasoning in language grounding.
*NeurIPS 2021 Datasets and Benchmarks Track*, 2021.
URL [https://openreview.net/forum?id=Rtquf4Jk0jN](https://openreview.net/forum?id=Rtquf4Jk0jN).
- Xiao et al. (2023a)
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael
Zeng, Ce Liu, and Lu Yuan.
Florence-2: Advancing a unified representation for a variety of
vision tasks.
*arXiv preprint arXiv:2311.06242*, 2023a.
- Xiao et al. (2023b)
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael
Zeng, Ce Liu, and Lu Yuan.
Florence-2: Advancing a unified representation for a variety of
vision tasks.
*arXiv preprint arXiv:2311.06242*, 2023b.
- Yang et al. (2024)
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and
Hengshuang Zhao.
Depth anything v2.
*arXiv:2406.09414*, 2024.
- Yun et al. (2023)
Tian Yun, Usha Bhalla, Ellie Pavlick, and Chen Sun.
Do vision-language pretrained models learn composable primitive
concepts?
*Transactions on Machine Learning Research*, 2023.
ISSN 2835-8856.
URL [https://openreview.net/forum?id=YwNrPLjHSL](https://openreview.net/forum?id=YwNrPLjHSL).
- Zeng et al. (2021)
Yan Zeng, Xinsong Zhang, and Hang Li.
Multi-grained vision language pre-training: Aligning texts with
visual concepts.
*arXiv preprint arXiv:2111.08276*, 2021.
- Zhu et al. (2022)
Wang Zhu, Jesse Thomason, Robin Jia, and .
Generalization differences between end-to-end and neuro-symbolic
vision-language reasoning systems.
pp. 4697–4711, December 2022.
doi: 10.18653/v1/2022.findings-emnlp.345.
URL [https://aclanthology.org/2022.findings-emnlp.345](https://aclanthology.org/2022.findings-emnlp.345).

## Appendix A Qualitative Analysis

### A.1 Referring Expressions Grounding

Examples of NePTune on Ref-GTA are shown in Figure [3](https://arxiv.org/html/2509.25757v1#A1.F3). Expressions 1 and 2 show the superiority of compositional reasoning compared to the VLM and object detection backbones. They demonstrate how the VLM fails to correctly locate the object in the new environments, while atomic concepts are correctly scored by the VLM.

Figure: Figure 3: Qualitative examples of NePTune on the RefGTA dataset. Green boxes indicate objects detected by Grounding DINO, blue boxes show objects selected by the VLM (InternVL2.5-8B), and red boxes highlight the final selections made by NePTune.
Refer to caption: x3.png

### A.2 Visual Questions Answering

Examples of NePTune on CLEVR-Humans are shown in Figure [4](https://arxiv.org/html/2509.25757v1#A1.F4).
Example 1 illustrates a case where reasoning over meta-concepts such as distinct shapes is required.
NeSyCoCo is unable to resolve the query due to this limitation to declarative, while ViperGPT struggles to identify the correct anchor object because it does not perform global reasoning across the object set.
NePTune, by contrast, is able to provide the correct interpretation. Example 2 presents a more complex scenario where reflection reasoning and calibration are critical.
Both the sphere and the cylinder are reflected in the cyan cube, with the cylinder’s reflection being more prominent and thus the more likely answer.
NePTune does not fully capture this subtle distinction and produces an incorrect response.
The competing baselines, however, fail for different and more fundamental reasons: VLM score calibration issues lead to incorrect selection, and ViperGPT relies on bounding box size comparisons that ignore perspective, which causes it to treat equally sized cubes as different.
In this case, the code selects the shiny brown cube in the foreground and concludes that there is no reflection.
Although NePTune also errs, this example highlights the inherent difficulty of fine-grained reflection reasoning and the challenges posed by subtle visual cues.

Figure: Figure 4: Qualitative examples of NePTune compared to ViperGPT and NeSyCoCo on CLEVR-Humans.
Refer to caption: x4.png

### A.3 Failure Example.

Example shown in Figure [5](https://arxiv.org/html/2509.25757v1#A1.F5) illustrates an error caused by an incorrect LLM-generated program in NePTune. Two issues occur simultaneously:

- 1.
Incorrect predicate arity: The program invokes the function is_baby_giraffe with num_object=2 instead of 1. This leads to a dimensional mismatch when the resulting score tensor is composed with variable x2, causing execution failure.
- 2.
Incorrect bounding box reference: The program refers to the bounding box with the color *green* instead of *red*, which is the intended single-object reference.

While errors such as incorrect bounding box colors can be alleviated to a certain extent with regex-based matching, the dimensional inconsistency in predicate scoring is unrecoverable without regeneration.

Figure: Figure 5: Qualitative example of wrong NePTune program generation. Mistakes are highlighted with red boxes.
Refer to caption: x5.png

## Appendix B VLM Concept Analysis

### B.1 Concept-Level Performance

**Table 12: F1 Scores for atomic concept grounding on CLEVR using visual prompting.**
| Concept | Ovis1.6 | Qwen2VL | InternVL2.5 |
| --- | --- | --- | --- |
| blue | 0.972 | 0.806 | 0.986 |
| brown | 0.989 | 0.921 | 0.989 |
| cube | 0.915 | 0.952 | 0.991 |
| cyan | 0.961 | 0.805 | 1.000 |
| cylinder | 0.964 | 0.960 | 1.000 |
| gray | 0.958 | 0.911 | 0.980 |
| green | 0.989 | 0.838 | 0.988 |
| large | 0.956 | 0.975 | 1.000 |
| metal | 0.961 | 0.965 | 0.997 |
| purple | 0.988 | 0.927 | 1.000 |
| red | 0.973 | 0.829 | 1.000 |
| rubber | 0.980 | 0.924 | 1.000 |
| small | 0.976 | 0.868 | 1.000 |
| sphere | 0.980 | 0.980 | 0.996 |
| yellow | 0.974 | 0.871 | 0.982 |
| behind | 0.836 | 0.877 | 0.868 |
| front | 0.890 | 0.850 | 0.850 |
| left | 0.919 | 0.885 | 0.943 |
| right | 0.934 | 0.863 | 0.930 |
| same_color | 0.982 | 0.957 | 0.840 |
| same_shape | 0.862 | 0.855 | 0.936 |
| same_size | 0.794 | 0.604 | 0.951 |
| Macro F1 | 0.943 | 0.874 | 0.964 |
| Micro F1 | 0.903 | 0.866 | 0.945 |

An analysis of the atomic concept grounding performance on the CLEVR dataset shown in Table [12](https://arxiv.org/html/2509.25757v1#A2.T12) reveals clear patterns of weakness, primarily centered around analogical and complex relational concepts.

#### B.1.1 Analogical Relations

The most significant challenge for the VLMs concept is abstract, analogical comparisons. The concept *same size* is a clear example, yielding the lowest F1 score in the entire table for Qwen2VL-7B at 0.604. Ovis1.6-9B also struggles significantly with it, scoring just 0.794. While InternVL2.5-8B performs better, *same color* is a notable weak point for it at 0.840. This indicates a systemic difficulty in performing comparative judgments with visual prompting compared to simple identification.

#### B.1.2 3D Spatial Relations are a Close Second

The next most difficult category is 3D spatial relationships. Concepts like *behind* and *front* consistently score lower than basic attributes across all models. For instance, the F1 scores for *behind* are 0.836 (Ovis1.6), 0.877 (Qwen2VL), and 0.868 (InternVL2.5). This demonstrates a weaker understanding of object relations in three-dimensional space compared to intrinsic properties like shape or material.

#### B.1.3 Color Confusion

Certain colors also pose challenges, particularly for the Qwen2VL model. The color *cyan* shows a notable performance dip for Qwen2VL (0.805). Further analysis suggests this is due to confusion with visually similar colors like *blue* (0.806) and *green* (0.838). This pattern highlights that while models can identify common colors well, they are less robust with more specific shades and can struggle to differentiate them.

#### B.1.4 Real-World Image Challenges in Visual Genome

Transitioning from the synthetic CLEVR environment to the complex, real-world images of the Visual Genome (Krishna et al., 2016) dataset reveals a notable shift in performance dynamics. We evaluate our models on atomic queries generated from the cleaned VG scene graphs in GQA (Hudson & Manning, 2019) using these templates:

- •
Relation Prompt:
Is the {class1/object1} in the red bounding box {class2/object2} the {relation} in the green bounding box?
- •
Class Prompt:
Is the object {inside of/in} the red bounding box {article} {class}?
- •
Attribute Prompt:
Is the {class} in the red bounding box {attribute}?

As shown in Table [13](https://arxiv.org/html/2509.25757v1#A2.T13), a key challenge emerges in fundamental object identification. While the models remain highly proficient at identifying spatial relations and attributes, their performance in the ”Class/Object” category is consistently lower than CLEVR. Ovis1.6-9B drops to an F1-score of 0.804, with InternVL2.5-8B and Qwen2VL-7B at 0.818 and 0.833, respectively. This suggests that the visual complexity, clutter, and vast number of fine-grained categories in real-world images make the foundational task of object recognition a more significant hurdle than in the controlled CLEVR environment.

**Table 13: Top 20 common concepts F1-Score comparison on real-world scene graphs.**
| Concept | Ovis1.6-9B | Qwen2VL-7B | InternVL2.5 |
| --- | --- | --- | --- |
| right of | 0.940 | 0.971 | 0.962 |
| left of | 0.951 | 0.970 | 0.980 |
| man | 0.906 | 0.914 | 0.892 |
| shirt | 0.840 | 0.893 | 0.963 |
| person | 0.821 | 0.792 | 0.824 |
| window | 0.857 | 0.894 | 0.957 |
| pole | 0.842 | 0.900 | 0.927 |
| building | 0.621 | 0.839 | 0.889 |
| tree | 0.750 | 0.833 | 0.839 |
| wall | 0.815 | 0.857 | 0.800 |
| on | 0.636 | 0.727 | 0.696 |
| sign | 0.800 | 0.815 | 0.833 |
| car | 0.870 | 0.960 | 1.000 |
| hand | 0.929 | 0.929 | 0.923 |
| in | 0.444 | 0.421 | 0.500 |
| white | 0.750 | 0.889 | 0.846 |
| near | 0.923 | 0.880 | 0.923 |
| sky | 0.500 | 0.667 | 0.800 |
| ear | 0.818 | 0.800 | 0.909 |
| woman | 0.952 | 0.952 | 0.952 |

### B.2 End-to-End Performance

These findings align with the paper’s broader conclusion that errors in the final reasoning pipeline often originate from the VLM’s poor performance on these specific types of relational and analogical concepts. Based on the results of this analysis, in addition to the score perceptual interface, we generate a set of the most common spatial predicates, such as left, right, behind, front, etc., using the position and depth estimation of bounding boxes for the sake of efficiency and accuracy.

**Table 14: Comparison of model accuracies across various categories. Maximum values for each NePTune and end-to-end are bolded. Changes compared to normal models are marked with colors and arrows.**
| Metric | Qwen2VL | Ovis1.6 | InternVL2.5 | Qwen-NeSy | Ovis-Nesy | Intern-Nesy |
| --- | --- | --- | --- | --- | --- | --- |
| Final Accuracy (%) | 93.55 | 85.00 | 90.25 | 82.55 ($\downarrow$ 11.00) | 88.80 ($\uparrow$ 3.80) | 92.65 ($\uparrow$ 2.40) |
| Exist (%) | 98.21 | 86.74 | 87.10 | 84.59 ($\downarrow$ 13.62) | 88.53 ($\uparrow$ 1.79) | 93.19 ($\uparrow$ 6.09) |
| Query Attribute (%) | 93.77 | 90.14 | 98.26 | 91.30 ($\downarrow$ 2.47) | 96.23 ($\uparrow$ 6.09) | 96.81 ($\downarrow$ 1.45) |
| Compare Attribute (%) | 99.44 | 90.28 | 98.61 | 78.33 ($\downarrow$ 21.11) | 88.33 ($\downarrow$ 1.95) | 91.94 ($\downarrow$ 6.67) |
| Count (%) | 87.50 | 75.40 | 74.60 | 70.16 ($\downarrow$ 17.34) | 78.83 ($\uparrow$ 3.43) | 87.10 ($\uparrow$ 12.50) |
| Compare Number (%) | 90.29 | 78.29 | 90.86 | 88.57 ($\downarrow$ 1.72) | 89.14 ($\uparrow$ 10.85) | 92.57 ($\uparrow$ 1.71) |

The results in Table [14](https://arxiv.org/html/2509.25757v1#A2.T14) clearly show that NePTune enhances the compositional reasoning abilities of strong base VLMs such as Ovis1.6 and InternVL2.5. The most significant gains are on quantitative compositional tasks, with accuracy on counting questions improving by over 12% for InternVL2.5 and counting comparison questions by nearly 11% for Ovis1.6. This highlights the value of our structured, programmatic approach for tasks that end-to-end models find challenging. Conversely, our framework degrades the performance of the already accurate model Qwen2VL. A deeper look reveals that the primary source of this degradation is the analogical questions, which are the most apparent in the ”Compare Attribute” task, where performance drops by 21%. This aligns perfectly with our findings in Experiment 1, which showed that this specific VLM struggles with analogical reasoning (e.g., same color, same shape). This demonstrates both the power of our framework when paired with a reliable grounding VLM and its sensitivity to the underlying VLM’s weaknesses, as errors in perception are propagated by the logical executor.

## Appendix C Experimental Setup

All experiments were conducted on a server running Ubuntu OS, equipped with an AMD EPYC 7413 24-core CPU, 700GB of system RAM, and an NVIDIA A6000 GPU (48GB). Unless specified otherwise, our core models included DeepSeekV3 (DeepSeek-AI-Team, 2025) as the backbone LLM, GroundingDINO-B for object detection, and DepthAnythingV2 (Yang et al., 2024) for depth estimation. We utilized the Hugging Face Transformers library (Wolf et al., 2020) for VLM interaction and the InternVL-2.5 codebase for finetuning(^1^11https://internvl.readthedocs.io/en/latest/internvl2.5/finetune.html). To supplement the LLM’s output, we also used spaCy (Honnibal & Montani, 2017) Named Entity Recognition to extract keywords from the query.

## Appendix D Datasets

Here, we provide a detailed breakdown of the datasets used to evaluate the NePTune framework across a range of visual reasoning challenges.

### D.1 CLEVR

We evaluate core compositional reasoning using the CLEVR (Compositional Language and Elementary Visual Reasoning) dataset (Johnson et al., 2017a). It contains 3D-rendered synthetic images of simple objects and is used for the task of Visual Question Answering (VQA), where the model must answer complex, programmatically generated questions about object attributes, counts, and relations. We measure performance using Accuracy (%) and, for answers requiring semantic equivalence matching (e.g., matching “2” with “two”). For CLEVR dataset benchmarks where a deterministic evaluation did not capture the correctness of the answer, such as matching ”2” with ”two”, we use GPT-4o as an AI judge for evaluation. The prompt used for this evaluation is shown in Figure [7](https://arxiv.org/html/2509.25757v1#A5.F7). Since the CLVER evaluation set was large (700K), we used the first 2000 samples for evaluating our method.

#### D.1.1 CLEVR Query Categories

Our fine-grained analysis on CLEVR, presented in Table [3](https://arxiv.org/html/2509.25757v1#S4.T3), breaks down performance across the following five distinct types of reasoning skills using the programs in the dataset:

These questions test for the presence of an object with specific properties, requiring a “Yes/No” answer. For example, “Is there a large green cube behind the small red sphere?”

These questions ask for a specific property (e.g., color, material) of a uniquely identified object, such as, “What color is the small shiny cylinder?”

This category involves “Yes/No” questions that compare a single property between two objects. For example, “Does the large sphere have the same material as the small cube?”

These questions require the model to return the total number of objects matching a description, such as, “How many red metallic objects are there?”

These questions involve comparing the quantities of two different sets of objects, also resulting in a “Yes/No” answer. For example, “Are there more spheres than cubes?”

### D.2 CLEVR-Humans

The CLEVR-Humans dataset (Johnson et al., 2017b) allows us to evaluate performance on more natural language. It uses the same synthetic images as CLEVR but features more complex and linguistically diverse questions written by humans. Similar to CLVER, performance is measured by Accuracy (%).

### D.3 CLEVR Extensions

To probe a wider range of reasoning skills, we use a collection of compositional challenges from Hsu et al., 2024, built on the CLEVR environment (Hsu et al., 2024). Each task tests a unique aspect of complex reasoning:

This task tests referring expression grounding, where the model must identify a target object based on its relationship to other objects. The evaluation is conducted by measuring the IOU value greater than 0.5.

> Example: “There is a sphere that is front the gray cylinder, find the small cylinder that is left of it.”

This benchmark evaluates the model’s ability to solve visual constraint satisfaction problems by finding a set of objects that simultaneously satisfy multiple attribute and relational constraints.

> Example: “Can you find four objects from the image such that: object 1 is a large metal object; object 2 is a metal object; object 3 is a small rubber cylinder; object 4 is a small yellow metal cylinder; object 1 is front object 2; object 1 is behind object 4; object 1 is behind object 3.”

This task tests abstract relational reasoning by mimicking Raven’s Progressive Matrices. The model is presented with objects in a grid that follow a pattern and must identify a candidate object from the scene that correctly completes that pattern.

> Example: “There are 9 objects, ordered in a 3x3 grid: row 1 col 1 is a small rubber object; row 1 col 2 is a small rubber object; row 1 col 3 is a large rubber object; row 2 col 1 is a small metal object; row 2 col 2 is a small metal object; row 2 col 3 is a large metal object; row 3 col 1 is a small metal object; row 3 col 2 is a small metal object; I am missing one object at row 2 col 2. Can you find an object in the scene that can fit there?”

### D.4 RefCOCO-Adversarial (Ref-Adv)

To test performance in real-world scenarios, we use the RefCOCO-Adversarial benchmark (Akula et al., 2020). This dataset consists of real-world images with visually similar distractor objects, making the task of Referring Expression Grounding particularly challenging. REG requires the model to locate the specific object corresponding to a text description, and we evaluate the model using Grounding Accuracy (%), where a prediction is considered correct if the Intersection over Union (IoU) with the ground-truth bounding box is 0.5 or greater.

### D.5 Ref-GTA

To measure generalization under a significant domain shift, we use the Ref-GTA benchmark (Tanaka et al., 2019). This is a REG benchmark where images are sourced from a photo-realistic game simulation, creating an out-of-distribution challenge for models pre-trained on natural images. Performance is measured by Grounding Accuracy (%), defined by an Intersection over Union (IoU) threshold of 0.5 between the predicted and ground-truth bounding boxes.

## Appendix E Verification

Figure: Figure 6: Prompt used for the Pairwise Arbiter verification strategy.

To recover from occasional symbolic execution failures, we add a lightweight *verification* stage that operates after reasoning has produced a candidate answer. We explore two simple strategies:

- 1.
Pairwise Arbiter: The model is presented with two candidates, the backbone prediction and the symbolic reasoning output, along with the query. It acts as an arbiter, directly judging which option better satisfies the description. Our prompt, similar to the one used by NAVER for this process, is shown in Figure [6](https://arxiv.org/html/2509.25757v1#A5.F6).
- 2.
Confidence Gating: We compute a softmax distribution over the symbolic executor’s scores with temperature $T$, and use a threshold $\tau$ to decide whether to trust the executor. If $\max(\operatorname{softmax})<\tau$, we fallback to the backbone prediction; otherwise, we keep the executor’s output. Both $\tau$ and $T$ are tuned on a 500-example held-out set for each backbone.

Table [15](https://arxiv.org/html/2509.25757v1#A5.T15) summarizes the performance of both light-weight verification strategies across different backbones. Overall, we observe that the *Pairwise Arbiter* tends to outperform the *Confidence Gating* method, highlighting the benefit of leveraging a VLM to arbitrate between backbone and symbolic predictions.

**Table 15: Post-verification accuracy on Ref-Adv.**
| Backbone VLM | Verification | Accuracy (%) | $\bm{\tau}$ | $\bm{T}$ | Symbolic Share(%) |
| --- | --- | --- | --- | --- | --- |
| Qwen2VL-7B | Confidence Gating | 80.91 | 0.70 | 0.40 | 73.59 |
| Pairwise Arbiter | 80.93 | N/A | N/A | 80.23 |  |
| Ovis1.6-9B | Confidence Gating | 60.49 | 0.30 | 0.10 | 86.79 |
| Pairwise Arbiter | 63.25 | N/A | N/A | 29.73 |  |
| InternVL2.5-8B | Confidence Gating | 76.65 | 0.60 | 0.50 | 71.06 |
| Pairwise Arbiter | 78.08 | N/A | N/A | 77.86 |  |

Figure: Figure 7: Visual Question Answering judge LLM prompt.

## Appendix F Hyperparameters

The fine-tuning hyperparameters for our experiments are detailed in Table [16](https://arxiv.org/html/2509.25757v1#A6.T16). These settings are based on the configurations defined in our training script.

During our experiments, we found that applying LoRA fine-tuning to the vision backbone, in addition to the language model, was highly effective for domain adaptation. To illustrate, without any neuro-symbolic fine-tuning, the base NePTune framework achieved 56.49% accuracy on Ref-GTA. In contrast, a standard fine-tuning approach on the base VLM only reached 4.19% accuracy. This performance gap underscores the effectiveness of our neuro-symbolic method and justifies fine-tuning the vision components to achieve optimal results in novel environments.

**Table 16: Hyperparameters for Fine-Tuning Experiments**
| Hyperparameter | Standard | NePTune Neuro-Symbolic Fine-Tuning |  |
| --- | --- | --- | --- |
| Fine-Tuning | on RefCOCO-Adv | on Ref-GTA |  |
| General Training |  |  |  |
| Learning Rate (lr) | $4\times 10^{-5}$ | $4\times 10^{-5}$ | $4\times 10^{-5}$ |
| Batch Size | 1 | 1 | 1 |
| Gradient Accumulation | 4 | 4 | 4 |
| Weight Decay | 0.01 | 0.01 | 0.01 |
| LR Scheduler | Cosine | Cosine | Cosine |
| Warmup Ratio | 0.03 | 0.03 | 0.03 |
| Loss Function | Cross-Entropy | Binary Cross-Entropy | Binary Cross-Entropy |
| LoRA Configuration |  |  |  |
| Rank (r) | 16 (LM), 8 (Vision) | 16 (LM) | 16 (LM), 8 (Vision) |
| Alpha ($\alpha$) | 32 (LM), 16 (Vision) | 32 (LM) | 32 (LM), 16 (Vision) |
| Dropout | 0.05 | 0.05 | 0.05 |

## Peer Reviews

confidence: 2
contribution: 2
initial_rating: 4
presentation: 3
questions: Please list up and carefully describe any questions and suggestions for the authors. Think of the things where a response from the author can change your opinion, clarify a confusion or address a limitation. This is important for a productive rebuttal and discussion phase with the authors.

1. How frequent are logical program generation failures, and does NePTune have any mechanism to detect or recover from them?
2. Could the authors provide a rough estimate of the end-to-end wall-clock time for a complex query? How does this multi-step pipeline's latency compare to a single forward pass of the backbone VLM?
soundness: 2
strengths: A substantive assessment of the strengths of the paper, touching on each of the following dimensions: originality, quality, clarity, and significance. We encourage reviewers to be broad in their definitions of originality and significance. For example, originality may arise from a new definition or problem formulation, creative combinations of existing ideas, application to a new domain, or removing limitations from prior results.

1. On the Ref-GTA domain-shift benchmark, the VLM backbone (InternVL2.5) achieves only 6.95% accuracy, while NePTune achieves 69.69%. This demonstrates a highly robust decoupling of reasoning from perception.
2. The framework integrates imperative Python with soft, differentiable logical operators. This is more expressive than declarative-only systems and more robust than crisp-logic pipelines.
3. The soft-logic operators make the pipeline differentiable, enabling fine-tuning.
summary: This paper introduces NePTune, a neuro-symbolic framework for vision-language reasoning. NePTune translates natural language queries into executable Python programs that use a hybrid execution model. This model combines imperative Python control flow with differentiable, soft logic operators that reason over uncertainty scores from a Vision-Language Model (VLM). The framework operates zero-shot by using an LLM to generate programs and a VLM to ground atomic concepts. It also supports fine-tuning for domain adaptation, demonstrating strong compositional generalization, especially on domain-shift benchmarks.
weaknesses: A substantive assessment of the weaknesses of the paper. Focus on constructive and actionable insights on how the work could improve towards its stated goals. Be specific, avoid generic remarks. For example, if you believe the contribution lacks novelty, provide references and an explanation as evidence; if you believe experiments are insufficient, explain why and exactly what is missing, etc.

1. Performance is fundamentally limited by the VLM's ability to ground atomic concepts.
2. While more robust, the LLM-based program generator is still a point of failure. The authors provide a clear example of this in Appendix A.3, where the LLM generates a program with incorrect logic and arity, causing an execution failure.
3. The framework is complex, involving multiple serial steps. This multi-component pipeline is likely to have significantly higher latency than a single VLM forward pass.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - The ablation study (Table 10) shows a massive +19.19% improvement from adding "Imperative Reasoning" on top of declarative logic. Could the authors provide a concrete example of a query from CLEVR-Humans that requires this imperative logic (e.g., loops or complex conditionals) and which a purely declarative approach fails to solve?

- Regarding the $N \times N$ matrix for relational queries: What is the practical limit for N (number of objects) before the VLM grounding and subsequent reasoning become computationally prohibitive? Have the authors tested this on more cluttered datasets, like COCO?

- The fine-tuning results on Ref-GTA are a key strength. To clarify: what exactly is being fine-tuned? Is it only the VLM backbone? Or are the parameters of the soft-logic operators (e.g., the temperature $\tau$ and margin $\gamma$ from Table 2) also learned during the neuro-symbolic fine-tuning process?
soundness: 3
strengths: - The core contribution—a symbolic executor that blends imperative Python with differentiable, soft-logic operators (Table 2)—is highly novel and effectively tackles the problem of uncertainty propagation in multi-step reasoning.

- The framework's differentiability is a key advantage. The ability to fine-tune the system for a new domain (as shown on Ref-GTA, where neuro-symbolic fine-tuning achieved 69.9% vs. 32.6% for standard VLM fine-tuning) is a significant step forward from inference-only models.

- By operating on continuous scores rather than crisp decisions, NePTune avoids the cascading error problem that plagues many multi-step reasoning pipelines.

- NePTune demonstrates state-of-the-art zero-shot performance against comparable methods (e.g., 92.65% vs 36.05% for ViperGPT on CLEVR) and massive gains on challenging domain-shift benchmarks (Ref-GTA), proving the practical value of its design.

- The paper is extremely well-written, and its contributions are clearly articulated and well-situated against prior work.
summary: This paper introduces NePTune, a neuro-symbolic framework for vision-language (VL) reasoning that addresses limitations in prior work, such as brittle pipelines and a lack of adaptability. NePTune translates a natural language query into an executable Python program, similar to models like ViperGPT. However, it makes two key contributions:

- Hybrid Execution Model: The generated Python program combines standard imperative control flow (loops, conditionals) with "soft logic" operators (based on fuzzy logic, e.g., t-norms for AND) that operate directly on the continuous uncertainty scores provided by a Vision-Language Model (VLM).

- Differentiability: These soft logic operations are differentiable, allowing the entire framework (specifically the VLM backbone) to be fine-tuned via "neuro-symbolic fine-tuning" for domain adaptation.

Operating in a training-free manner, NePTune already outperforms strong baselines. When fine-tuned, it shows remarkable adaptation capabilities, especially on domain-shift benchmarks like Ref-GTA.
weaknesses: - The entire system still hinges on an LLM's ability to correctly parse the natural language query into a logically sound Python program. The paper notes that errors shift from "syntax" to "logic" (Sec 5), but a logical error in the generated program is just as fatal as a syntax error. A deeper analysis of the types of logical errors the LLM still makes would be valuable.

- The score function for relational queries (num_objects=2) involves VLM queries that result in an $N \times N$ matrix, where N is the number of detected objects. This approach seems to scale quadratically with the number of objects in the scene. It's unclear how the system would perform in highly cluttered real-world scenes with hundreds of object proposals.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - Q1: Is the score function currently limited to pairs of objects, or can it be extended to higher-arity relations?
- Q2: What is the number of tokens that the query function can return? 
- Q3: Line 241/242 seems to have a missing or incomplete expression "…’).". Could you clarify?
- Q4: For program generation (component 1), do you use greedy decoding or a sampling-based strategy? Same question for the baseline VLMs
- Q5: Regarding Table 4, could you clarify what "ground-truth programs" refers to, how they are obtained, and why the dataset "Puzzles" can not benefit as strongly from it?
- Q6: What motivated the choice of VLMs for NePTune and comparisons? Do you expect similar zero-shot results for larger models (e.g., >70B or GPT-5 etc.) as in Tables 5 and 6?
- Q7: Can you estimate the computational or token cost of NePTune compared to direct prompting? Given the small amount of tokens for the score and query functions, it seems low, but quantitative insight or discussions would be helpful.

Generally, I like the proposed method and am willing to raise my score if my concerns and questions are addressed.
soundness: 3
strengths: - The paper introduces a novel neuro-symbolic design that effectively integrates the power of VLMs, symbol grounding, and explicit compositional reasoning under uncertainty.
- The score-based perceptual grounding interface elegantly connects neural perception with symbolic reasoning.
- The modular, differentiable design is appealing as it supports both zero-shot use and potential fine-tuning without dataset-specific supervision.
summary: The paper presents NePTune, a neuro-symbolic framework aimed at enhancing compositional reasoning with Vision-Language Models (VLMs). NePTune consists of three main components: an LLM-based program generator, a perceptual grounding module, and a symbolic executor. The program generator converts natural language queries into executable Python programs that can query a VLM to extract object-specific information. The resulting perceptual outputs are then combined using soft logic operators, allowing the system to reason over uncertain predictions from the VLM. The framework operates in a training-free manner but remains differentiable for fine-tuning.
weaknesses: - While the breadth of experiments is commendable, the sheer number of evaluations can make the presentation somewhat overwhelming. In particular, the new experiments introduced in the discussion section are confusing, as they could be more appropriately integrated into the main experimental section. For example, Table 9 provides a helpful overview of the different datasets and could be placed earlier to illustrate NePTune’s general effectiveness. More broadly, adding a concise overview of the datasets, models, and baselines at the beginning of the experimental section would substantially improve readability. It might also be worth considering focusing on a smaller subset of key experiments in the main paper and moving some of the less central results to the appendix. For instance, Table 5 appears useful as supplementary evidence but is not discussed in much detail, suggesting it could be moved without loss of clarity.

- The paper would benefit from a more detailed discussion of how the framework scales when applied to larger VLMs or more complex perceptual inputs, as well as an analysis of its computational overhead compared to direct prompting. It would also be valuable to elaborate on potential limitations that may arise despite high perceptual quality in the VLM or strong program generation capabilities from the LLM.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: What exactly is a "response" in ll 236? 

How does Neptune decide when to use a "query" 

What is the intuition/benefit of having soft compositional reasoning components? The authors seemed quite brief on this.

I don't quite understand the intuition/explanation for the weaker "query attribute" and "compare attribute" results in table 3? Could the authors please elaborate on this?

Tab. 7 how is the VLM fine-tuned? What for is it fine-tuned?

Why not integrate table 9 with the other evaluations around RQ 1?

What is the importance of repeatedly naming models end-to-end when they are not actually trained end-to-end for the evaluations in the paper? Moreover, if Neptune is based on these end-to-end models, also Neptune can profit from end-to-end pretraining.
soundness: 3
strengths: The topic of this work tackles many relevant aspects of current foundation models. The experimental evaluation is generally well structured and presented. The results show promising boosts particularly over the pure VLM/LLM baselines.
summary: This paper investigates integrating both the coding and perception abilities of foundation models within a near-symbolic framework to improve the reasoning failures of the foundation models alone. The authors propose a three stage processing pipeline including structured and targeted queries to VLMs which are integrated within code executions obtained from an LLM. The goal of the approach is to tackle zero-shot processing. The authors evaluate their Neptune approach on several, mainly CLEVR-based datasets.
weaknesses: Generally, I don't have any fundamental issues, just some information is missing and a little unclear. I think if the authors could clarify these/ integrate them into the paper, it would be easier to understand the contribution of the work.

What I am definitely missing is a section specifying the exact model setups and high-level implementation information regarding Neptune, but also the baselines. I would place this before section 4.1. I.e. what models were used for the Neptune results? What models are being compared to? Currently all of this information is spread out, if at all, across section 4. 

The authors provide good RQs in Section 4, however these are never referenced after being introduced. For improved readability I suggest to reference them within each experimental subsection.

I am a little confused about which baselines are compared to for which evaluations. E.g. why is MDETR used in Table 5, but not for the evaluations for Table 3 or 4? Same goes for Naver.

Why don't the authors discuss and compare to older NeSy approaches such as [1]?

I find the Discussion particularly around table 8 somewhat confusing. What are the authors trying to show? In my understanding the point is to show that the basic "perception" performance are quite high?

[1] Yi, Kexin, et al. "Neural-symbolic vqa: Disentangling reasoning from vision and language understanding." Advances in neural information processing systems 31 (2018).


Decision: Accept (Poster)