DeiSAM: Segment Anything with Deictic Prompting
Authors: Hikaru Shindo
Manuel Brack
Gopika Sudhakaran
Devendra Singh Dhami
Patrick Schramowski
Kristian Kersting
Venue: NeurIPS 2024 poster (2024)
Keywords: neuro-symbolic reasoning
object segmentation
deictic representation
large language models
differentiable logic programming
Primary area: other
TL;DR: Segment objects from complex textual prompts using neuro-symbolic reasoning with large-scale neural networks.

Abstract: Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects in complex scenes, humans instinctively rely on deictic descriptions in natural language, i.e., referring to something depending on the context such as "The object that is on the desk and behind the cup.". However, deep learning approaches cannot reliably interpret such deictic representations due to their lack of reasoning capabilities in complex scenarios. To remedy this issue, we propose DeiSAM — a combination of large pre-trained neural networks with differentiable logic reasoners — for deictic promptable segmentation. Given a complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning on generated scene graphs. Subsequently, DeiSAM segments objects by matching them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is a substantial improvement over purely data-driven baselines for deictic promptable segmentation.

DeiSAM: Segment Anything with Deictic Prompting
Hikaru Shindo Manuel Brack 1.2
Gopika Sudhakaran' 1.3
Devendra Singh Dhami Patrick Schramowski
12,3,
Kristian Kersting 122.3
'Techiical University of Darmstadt "German Research Center for AI (DFKI)
'Hessian Center for AI (hessian.Al) "Eindhoven University of Technology
"Center for European Research in Trusted Artificial Intelligence (CERTAIN)
Abstract
Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects complex scenes, humans instinctively rely on deictic descriptions natural language, i.e., referring to something depending on the context, such as "The object that is on the desk and behind the cup". However, deep learning ap- proaches cannot reliably interpret such deictic representations they have limited reasoning capabilities, particularly in complex scenarios. Therefore, we propose DeiSAM combination of large pre-trained neural networks with differentiable logic reasoners- for deictic promptable segmentation. Given complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning generated scene graphs. Subsequently, DeiSAM segments objects by match- ing them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is substantial improvement over purely data-driven baselines for deictic promptable segmentation.
Introduction
Recently, large-scale neural networks have substantially advanced various tasks at the intersection of vision and language. One such challenge is grounded image segmentation, wherein objects within scene are identified through textual descriptions. For instance, Grounding Dino (Liu et al.) (023c). combined with the Segment Anything Model (SAM) (Kirillov al., 2023), excels at this task provided with appropriate prompts. However, a well-ddoumentted limitation of data-driven neural approaches their lack of reasoning capabilities (Shi et al. [2023] Huang et al., 2024a). Consequently, they often fail to understand complex prompts that require high-level reasoning on relations and
attributes of multiple objects, as demonstrated in Fig. T
In contrast, humans identify objects through structured descriptions of complex scenes referring an object, e.g., "An object that is on the boat and holding an umbrella". These descriptions referred to as deictic representations and were introduced to artificial intelligence research motivated by linguistics (Agre & Chapman 1987), , and subsequently applied in reinforcement learning (Finney et al. 2002). A deictic expression refers to an object depending on the agent using it and the overall context. Although deictic representations play central role human comprehension of scenes, current approaches fail to interpret them faithfully due to their poor reasoning capabilities.
To remedy these issues, propose AM, which is combination of large pre-trained neural networks with differentiable logic reasoners for deictic promptable object detection and segmentation.
"corresponding author: hikaru.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input
Output
Visual Scene
Deictic "An object that is on the boat, Prompt and that is holding an umbrella"
GroundedSAM, SKEM (neural baselines)
DeiSAM (ours)
DeiSAM segments objects with deictic prompting. Shown are segmentation masks with an input textual prompt. DeiSAM (right) correctly segments the people on the boat holding umbrellas, whereas the neural baselines (left) incorrectly segment the boat instead (Best viewed in color).
Figure I:
The DeiSAM pipeline is highly modular and fully differentiable, sophisticatedly integrating large pre-trained networks and neuro-symbolic reasoners. Specifically, we leverage Large Language Models (LLMs) to generate logic rules for given deictic prompt and perform differentiable forward reasoning (Shindo et al. 2023 2024) with scene graph generators (Zellers et al.) 2018). The reasoner is efficiently combined with neural networks by leveraging forward propagation on computational graphs. The result of this reasoning step is used to ground a segmentation model that reliably identifies the objects best matching the imput.
In summary, we make the following contributions: 1) We propose DeiSAM a modular, neuro- symbolic framework using LLMs and scene graphs for object segmentation with complex textual prompts. 2) We introduce novel Deictic Visual Genome (DeiVG) benchmark that contains visual scenes paired with deictic representations, i.e., complex textual identifications of objects in the scene. To further investigate the challenging nature of abstract prompts, we curate new DeiRefCOCO+ benchmark. It is deictic variant of RefCOCO+, an established reference object detection benchmark. 3) We empirically demonstrate that DeiS AM strongly outperforms neural baselines for deictic segmentation. 4) We showcase that DeiSAM can perform end-to-end training via differentiable reasoning to improve the segmentation quality adapting to complex downstream reasoning tasks.
Related Work
Multi-modal Large Language Models. The recent achievements of large language models (LLMMs) (Brown et al., 2020) have led to the development of multi-modal models, including vision- language models (Radford et al. 2021 Alayrac et al. 2022 Li al. 2022a Liu et al., (023b), which take visual and textual inputs. However, these large models' reasoning capabilities are limited (Huang et al. 2024a)) often inferring wrong conclusions when confronted with complex reasoning tasks. DeiSAM addresses these issues by combining large models with (differentiable) reasoners.
Additionally, DeiSAM is related to prior work using LLMs for program generation. For example, LLMs have been applied to generate probabilistic programs (Wong et al. 2023). Answer Set Pro- grams (Ishay al. [2023] Yang et al., 2023), and programs for visual reasoning (Surís al. 2023) Stanié et al. (024). These works have demonstrated that LLMs are powerful program generators and outperform simple zero-shot reasoning. With DeiSAM we propose the usage of LLMs to generate differentiable logic programs for image segmentation and object detection.
Scene Graph Generation. Scene Graph Generators (SGGs) encode complex visual relations to
summary graph using the comprehensive contextual knowledge relation encoders (Lu et al. 2016
Zellers et al. 2018 Tang et al. 2019). Recently, the focus has shifted to transformer-based SGGs that
use attention to capture global context while improving visual and semantic fusion (Lin et al. 2020 Lu et al. | Dong et al. 2022). Lately, attention has also been used to capture object-level re ralation cues using visual and geometric features (Sudhakaran et al.] 2023). The modularity of Ac allows for using any SGG to obtain graph representations of input visual scenes. Scene graphs are essential for segmentation models to be faithful reasoners. Without them, models may develop shorteuts, resulting in apparent answers through flawed scene understanding (Marconato et al. 2023).
Visual Reasoning and Segmentation. Visual Reasoning has been fundamental problem in machine learning research, resulting in multiple benchmarks (Antol al. 2015 Johnson et al. 2017 Yi et al.
Code: https://gitti com/ml htt.s//www..ooooeeeoeeeoereaeaaa

---

2020) to address this topic and subsequent frameworks (Yi et al. 2018; Mao et al. 2019 Amizadeh al. 2020 Hsu et al., 2023) that perform reasoning using symbolic programs and multi-modal transformers (Tan & Bansal 2019). These benchmarks are primarily developed to answer queries written in natural language texts paired with visual inputs. Our proposed dataset, DeiVG, the first to integrate complex textual prompts into the task of image segmentation with natural images. In a similar vein, to tackle visual reasoning tasks, neuro-symbolic rule learning frameworks have been proposed, where discrete rule structures are learned via backpropagation (Evans & Grefenstette 2018, Minervini et al. 2020] Shindo et al. [02T] 2023 2024] Zimmer et al. 2023). These works have primarily been tested visual arithmetic tasks synthetic environments for reasoning (Stammer et al. 202I). To this end, LASER (Huang et al. 2024b) leverages logical specifications to learn properties from videos. DeiSAM is unique suuro--ymbolic framework that addresses image segmentation in natural images and utilizes differentiable reasoning for program learning.
segmentation masks given visual input (Wang) jt al. 2018, Guo et al., 2018). Multiple datasets and tasks have been proposed that assess a model's reasoning ability to identify objects (Kazemzadeh et al. 2014 Yu et al. 2016). Recently, Segment Anything Model (SAM) (irillov et | (023) has been released, achieving strong results on zero-shot
Semantic segmentation aims to generate objects'
(Liu image segmentation tasks. Grounded SAM (Ren et al. 2024) combines Grounding DINO
ct al. 2023c) with SAM, allowing for objects described by textual prompts. Moreover, LISA La
et al. 2023) fine-tunes multi-modal LLMs to perform low-level reasoning over image segmentation. However, LISA still requires strong prior information on the type target object (e.g. "the person that is
wearing green shoes") and breaks down for more abstract tasks (cf. Sec. [.5). In contrast, DeiSAM encodes the reasoning process explicitly as a differentiable function, thus avoiding spurious neural networks' behavior. Consequently, DeiSAM is capable of high-level reasoning on arbitrarily abstract prompts (e.g. "an object") utilizing structured representation of scene graphs. To this end, frameworks that enhance the transformer (or attention) architecture for various segmentation tasks have been proposed (Liu et al. 2023a Wu et al. b))44ab). These approaches rely on transformers (or attentions) as their core reasoning pipeline. In contrast, DeiSAM explicitly encodes logical reasoning processes to guarantee accurate and faithful interpretation of abstract and complex prompts.
3 DeiSAM - The Deictic Segment Anything Model
DeiSAM uses first-order logic as its language, and we provide its formal definition in App. A Let us start by outlining the DeiSAM pipeline with brief overview of its modules, before describing essential components in more detail.
3.1 Overview: Deictic Segmentation
We show schematic overview of the proposed DeiSAM workflow in Fig. 2 First, an input image is transferred into graphical representation using (1) Scene Graph Generator. Specifically, a scene graph comprises a set of triplets (n1, e, n22). where entities n1 and n2 have relation e. For example, a person (nz) is holding (e) an umbrella (n2). Consequently, each triplet (Mz e, 122) in a scene graph can be interpreted as fact, e(n,, D2), where e 2-ary predicate and n1 and n2 are constants in first-order logic. The textual deictic prompt needs to be interpreted as structured logical expression to perform reasoning on these facts.
For this step. DeiSAM leverages (2) Large Lan- guage Models, which can generate logic rules for deictic descriptions, given sufficiently restrictive prompts as we demonstrate. In our example, the LLM would translate "An object that is on the boat,
// Program cond1 (X) -on (X, type(Y, boott).
cond2 -holding type uubrelll). target (X) :-cond (X), condd((()
Listing 1: Rules generated by LLMSs and that is holding an umbrella" into the rules Program 1 in Listing [ The first two rules define the conditions described in the prompt, and the last rule identifies corresponding objects. However, users often use terminology different from that of the SGG, e.g., boat and barge target the same concept but will not be trivially matched. To bridge the semantic gap, we introduce a (3) semantic unifier. This module leverages word embeddings of labels, entities, and relations in the generated scene graphs and rules to match synonymous terms by modifying rules accordingly. The semantically unified rules are then compiled to a (4) forward reasoner, which computes logical entailment using forward chaining (Shindo al. 2023). The reasoner identifies the targeted objects and their bounding
3

---

person
Scene Graph Generator
Differentiable
foot dinng
Semantic Unifier
Forward
boat
Reasoner
barge
umbrella
Visual Input
Da barge
Rules
Segmentation Model
"An object that is on the boat, and that is holding an umbrella"
Large Language Model
tazget (31 m-oond1 180, sendz (X).
condi (X0 zmon (X, type boat)
cond2 (X) s-holding DX, 4 type bi, unbrellat
Output
Deictic Prompt
Large-scale Neural Networks
Other Modules
Figure 2: DeiSAM architecture. An image paired with a deictic prompt is given as input. We parse the image into scene graph (1) and generate logic rules (2) corresponding to the deictic prompt using a large language model. The generated scene graph and rules are fed to the Semantic Unifier module (3), where synonymous terms unified. For example, barge in the scene graph and boat in the generated rules will be interpreted as the same term. Next, the forward reasoner (4) infers target objects specified by the textual deictic prompt. Lastly, perform object segmentation (5) on extracted cropped image regions of the target objects. Since the forward reasoner is differentiable (Shindo al. 2023), gradients can be passed through the entire pipeline (Best viewed in color).
boxes from the graph. Lastly, we segment the object by feeding the cropped images (5) segmentation model.
Now, let investigate the two modules of DeiSAM in detail: rule generation and reasoning.
3.2 LLMs Logic Generators
To perform reasoning on textual prompts, we need to identify corresponding rules. We use LLMs to parse textual descriptions to logic rules using the system prompt specifying the rule format to be generated. The complete prompt is provided in App. B DeiSAM uses specific rule format describing object and attribute relations. For example, fact on (person, boat) in a scene graph would be decomposed into multiple facts on (X, Y), type (X, person), and type (Y, boat) to account for several entities with the same attribute in the scene.
The computational and memory cost of forward reasoning is determined by the number of variables over all rules and the number of conditions. Naive formatting of rules (Shindo et al. 2024) leads to an exponential resource increase with the growing complexity of deictic prompts. Since the representations used in the forward reasoner are pre-computed and kept in memory, non-optimized approaches will quickly lead to exhaustive memory consumption (Evans & Grefenstette 2018). In our format, however, we restrict the used variables to and and only increase the number of rules with growing prompt complexity. Thus resulting in linear scaling of computational costs instead.
3.3 Reasoning with Deictic Prompting
DeiSAM performs differentiable forward reasoning follows. We build reasoning function freason x R + T where generated by LLM, and T is set of facts representing identified target objects in the scene.
set facts representing a scene graph, R is set rules
(Differentiable) Forward Reasoning. For visual input æ R.2, DeiSAM utilizes scene graph generators (Zellers et al., (0118) to obtain logical graph representation G. where each fact rel(obj1, obj2) E G represents an edge in the scene graph. Each fact in given set G is mapped to confidence score using a valuation vector
[0, 1 A SGG is function sgg RR2 + [0. [| that produces a valuation vector out of a visual input. DeiSAM builds on the neuro-symbolic message- passing reasoner (NEUMANN) (Shindo al. (024) to perform reasoning. For a given set of rules R. DeiSAM constructs forward reasoning gruph, which is bi-directional graph representation of a logic program. Given an initial valuation vector produced by an SGG, DeiSAM computes logical consequences in differentiable manner by performing bi-directional message passing

---

the constructed reasoning graph using soft-logic operations (cf. A.P. DeiSAM identifies target objects be segmented using confidence scores over facts representing targets, e.g., target(obj1) and extracts the corresponding bounding boxes from the scene graph.
A.I.
Semantic Unifier. DM unifies diverging semantics in the generated rules and scene graph using concept embeddings similar to neural theorem provers (Rocktäschel & Riedel, 2017). We rewrite the corresponding rules R of prompt by identifying the most similar terms in the scene graph for each predicate and constant. If rule R E R contains a term I, which does not appear in scene graph G. we compute the most similar term as arg max encoder embedding model for texts. We apply this procedure to terms and predicates individually.
encoder (y), where encoder an
The Deictic Visual Genome
To facilitate a thorough evaluation of the novel deictic object segmentation tasks, we introduce the Deictic Visual Genome (DeiVG) dataset. Building on Visual Genome (Krishna et al. 2017). we construct pairs of deictic
object that has handle and that is on a bench prompts and corresponding object annotations for real-world images, as shown in Fig. 3 Our analysis of the scene graphs in Visual Genome found the annotations often be noisy and ambiguous, which aligns with observations from previous research (Hudson & Manning 2019). Consequently, we substantially filtered and cleaned potential candidates to produce a sound dataset.
cooler
Figure 3: An exam-
First, we restricted ourselves to 19 commonly occurring relations and en- sured that prompts were unambiguous, with only one kind of target object satisfying the prompt. Specifically, DeiVG contains prompts requiring the correct identification of multiple objects, but these are guaranteed to be the same type according to Visual Genomes synset annotations. We automatically synthesize prompts from the filtered scene graphs using textual templates, egg. the relations has (cooler, handle) and on ( (cooler, (enoch) bench) would yield a prompt "An object that has handle and that is on bench" targeting the cooler. Entries in the DeiVG dataset can be categorized by the number of relations they use in their object description. We introduce three subsets with 1-3 relations, which we denote as DeiVG,, DeiVGg, and DeiVGs, respectively. Each dataset is distinct, e.g., DeiVG; contains only prompts using 2 relations. For each set, we randomly select 10k samples that we make publicly available to encourage further research.
ple from Deictic Visual Genome (DeiVG2).
5 Experimental Evaluation
With the methodology of DeiSAM and our novel evaluation benchmark DeiVG established, now provide empirical and qualitative experiments. Our results outline AMiS benefits over purely neural approaches, supplemented by ablation studies of each module. Additionally, we investigate RefCOCO Yu et al. 2016), a low-level reasoning benchmark for segmentation tasks, and demonstrate the robustness of DeiSAM for abstract prompts. Lastly, we show that DeiSAM is end-to-end trainable and can thus be leveraged to improve the performance of the neural components in the pipeline.
5.1 Experimental Setup
We base our experiments the three subsets of Dei As evaluation metric, we use mean average precision (mAP) over objects. Since the object segmentation quality largely depends on the used segmentation model, we focus on assessing the object identification preceding the segmentation step. The default DeiSAM configuration for the subsequent experiments uses the ground truth scene graphs from Visual Genome (Krishna et al., 2017). gpt-3. 5 5-turbe as LLM for rule generation, ada-0022 -02 as embedding model for semantic unilication, and SAM (Kirillov et al. (022)) for object segmentation. Additionally, we provide few-shot examples of deictic prompts and paired rules in the input context of the LLM, which improves performance (cf. App. E. We present detailed ablations on each component of the DeiSAM pipeline Sec. 5.4
//oppaa ctt/bboo//ntrrouuutaa chatept: and whicser apis
bttpo://opeaa. coa/blog//ee /aes improved cubedding xodel
5

---

Table I: DeiSAM handles deictic prompting. Mean Average Precision (mAP) of DeiSAM and neural base- lines on Ve datasets are shown. Subscript numbers indicate the complexity of prompts.
An object on the large chair and wears tie.
An object in front of the cart and has shadow.
Mean Average Precision (%) f
Method
DeiVG,
DeiVG₂
DeiVG3
SEEM
1.58
4.44
7.54
in Front of cearriage
OFA-SAM
3,37
9.01
15.38
Leddy bea
bench
GLIP-SAM
2.32
0.03
0.00
Gr.Dino-SAM
10.48
32.33
46.04
Figure 4: DeiSAM handles ambiguous prompts. Results with prompts (top) with scene graphs (bottom).
LISA
14.90
56.03
75.79
DeiSAM (ours)
65.14
85.40
87.83
We compare DeiSAM to multiple purely neural approaches, both empirically as well as qualitatively, We include three baselines that use one model for object identification and subsequently segment the grounded image using SAM (Kirillov et al. (023). similar to the grounding DeiSAM. Our comparison includes the following models for visual grounding: 1) One-For-All (OFA) (Wang et al. 2022), a unified transformer- based grruence-to-seqquee
model for vision and language tasks of which we use dedicated visual grounding checkpoin [1 2) Grounded Language- Image Pre-training (GLIP) (Li et al. 2022b) a model for specifically designed for object-aware and semantically-ric object detection and grounding. 3) GroundingDino (Liu et al.. (023c) open-set object detector combining transformer- based detection with grounded pre-training. Moreover, we compare to an end-to-end semantic segmentation model supporting textual prompts with SEEM (Zou et al. (2033). Lastly, we compare to LISA (Lai et al. 2023), a state-of-the-art neural reasoning segmentation model.
5.2 Empirical Evidence
The results on DeiVG of all baselines compared to DeiSAM are summarized in Tab. DeiSAM clearly outperforms all purely neural approaches by a large margin on all splits of DeiVG. The perfor- mance of most methods improves with more descriptive deictic prompts, ie., more relations being used. We attribute this effect to two distinct causes. For one, additional information describing the target object contributes to higher accuracy object detection. On the other hand, DeiVG, contains DeiVGa. Consequently,
significantly more samples with multiple target objects than DeiVG2 cases in which a method identifies only one out of multiple objects will have a higher impact on the overall performance. Overall, the large gap between DeiSAM and all baselines highlights the lack of complex reasoning capabilities in prevalent models and DeiSAM's large advantage. We further provide runtime comparison and its analysis in App. F showcasing that DeiSAM's runtime is comparable to the baselines, and the bottleneck is in the LLMs, not in the reasoning pipeline.
5.3 Qualitative Evaluation
After empirically demonstrating DeiSAM's capabilities, look into some qualitative examples. In Fig. we demonstrate the efficacy of the semantic unifier. All examples use terminology in the deictic prompt diverging from the scene graph entity names. Nonetheless, the unification step successfully maps synonymous terms and still produces the correct segmentation masks, overcoming the limitation of off-the-shelf symbolic logic reasoners.
In Fig.5. [] we further compare DeiSAM with the purely neural baselines. DeiSAM produces the correct segmentation mask for complicated shapes (e.g., partially occluded cable) complex scenarios (e.g., multiple people, only some holding umbrellas). All baseline methods, however, regularly fail to identify the correct object. A common failure mode is confounding nouns in the deictic prompt. For example, when describing an object in relation to a 'boat', the boat itself is identified instead of the target object. These examples strongly illustrate the improvements of DeiSAN AM the pure neural approach on abstract reasoning tasks.
httpp://sooeleccpp.o http://aaaelaop..cccaaaaaaaaaaaa grounding.re www.ooo Liscoooo_aase Largo__aa_acc satgg/ooee//suserree ha/sumaary

---

DeñSAM (Ours)
GroundedDino SAM
SEEM
QFA-SAM
GAIP-SAM
An object that is the table and that behind mug.
An object that on the boat and that is holding an umbrella.
An object that on the car and that has
Figure 5: DeiSAM segments objects with deictic prompts. Segmentation results the DeiVG dataset using DeiSAM and baselines are shown with deictic prompts. DeiSAM correctly identifies and segments objects given deictic prompts (left-most column), while the baselines often segment wrong object. More results are available in App. G (Best viewed color).
5.4 Ablations
The modular nature of the DeiSAM pipeline enables easy component variations. Next, we investigate the performance of key modules in isolation and their overall influence on the pipeline.
LLM Rule Generation. One of the key steps for DeiSAM is the translation of deictic prompts posed in natural language into syntactically and semantically sound logic rules. We ob- served that the performance of instruction-tuned LLMs this task heavily depends the em- ployed prompting technique. Consequently, we leverage the well-known methods of few-shot prompting (Brown et al. 2020) and chain-of- thought (CoT) (Wei et al. 2022).
Ablations on prompting techniques for rule generation w/ Llama-2-13BB 2-13B-Chat Few-shot examples are imperative for rule generation with ofain-of-thought (CoT) prompting providing addi-
Table 2:
tional improvements for complex deictic prompts.
Prompting Technique Instruct Only
Overall Success (%)
DeiVG,
DeiVGa
DeiVGa
0.00
0.00
0.00
CoT
0.00
0.00
0.00
Few-shot
94.04
92.52
90.17
To that end, we first let the model extract all predicates from deictic prompt, which we sub- sequently provide as additional context for the rule generation. For both cases, we provide multiple few-shot examples. We evaluate all prompting approaches with -Lama-22113 1-B (Touvron al. 2023) in Tab. 2 Clearly, few-shot examples are imperative to perform rule generation successfully. Addi- tionally, CoT for predicate decomposition further improves the rule generation for complex prompts.
Few-shot + CoT
91.00
95.17
93.45
With the best prompting technique identified, we additionally evaluated multiple open and closed- source language models of different sizes (ef. App. (F. In general, all instruction -uuned models can generate logic rules from deictic prompts. However, larger models strongly outperform smaller ones, especially for more complex inputs. The overall best-performing model was gpt-3.5-turbo producing correct rules for DeiVG for 93.65% of all samples.
Semantic Unification. Next, we take a more detailed look into the semantic unification module. At this step, we bridge the semantic gap between differing formulations in the deictic prompt and the scene graph generator. To evaluate this task, we created an exemplary benchmark based on synonyms in the visual genome. For 2.5k scenes in DeiVG, we considered all objects in the scene graph and identified one object name that differed from its synset entry. Based on that synonym, the task is to identify the one, unique, synonymous object in the scene. For example, in an image containing "table', 'couch', "chair", and 'cupboard' the query 'sofa' should identify the 'couch" as most likely synoynm. Overall, the task is considerably more challenging than may appear at first glance, with the best model only achieving success rate of 72% (cf. Fp. We observed, for example, the query 'sofa' is matched with 'pillow' instead of the targeted 'couch" or 'trousers' with 'jacket' instead of 'pants'. These results motivate further research into the semantic unification process.
7

---

Table 3: Comparison on RefCOCO+.
Table Comparison on DeiRefCOCO+.
Mean
Average Precision (9))
Mean Average Precision (%)
Method
val
testA
testB
Method
val
testA
testB
LISA
67.55
74.86
63.03
LISA
44.92
47.60
43.23
GroundedSAM
55.09
66.21
44.21
GroundedSAM
30.06
31.75
28.12
DeiSAM
71.72
77.29
64.98
DeiSAM
71.56
79.51
66.43
5.5 Solving Reference Expression
In addition to experiments on DeiVG, we also consider the RefCOCO dataset Yuu et al. 2016), which comprises referring expressions for object segmentation. Thus, this dataset is the most similar setup to the deictic segmentation task in prior work. The key difference between RefCOCO and DeiVG is that the latter is built to evaluate models' abstract reasoning capabilities complex visual scenes. In contrast, RefCOCO mainly evaluates descriptive object identifications, i.e., objects with names and properties, e.g. "old man or child green short", compared to e.g. "an object on table and next to a computer" in DeiVG. Consequently, deictic prompts are more challenging than the reference texts in RefCOCO, since DeiVG prompts do not include explicit names and properties of target objects.
Since there were no publicly available scene graphs (or SGGs) for MSCOCO images, we used GPT3 convert the reference text to structured scene graph representation with additional annotations. Tab. [ shows the mAP of LISA, Arr and DeiSAM on RefCOCO, LISA achieved better overall performances than GroundedSAM, showing its strong capability on the reference task. DeiSAM, however, remains competitive with LISA and achieves better results on all splits.
To further investigate the challenging nature of abstract prompts, we curated DeiRefCOCO+ benchmark that contains more abstract textual references. Specifically, we turned the reference texts in RefCOCO+ into deictic prompts by removing any description of the target object. For example, the prompt "kid wearing navy shirt" is modified to "an object that is wearing navy shirt". Tab. again shows the mAP of DeiSAM, GroundedSAM, and LISA on the modified DeiRefCOCO+ dataset. Importantly, DeiSAM retains a similar performance on both types of prompt formulations. In comparison, we observe strong drop performance for GroundedSAM and LISA with the absence of confounding object descriptions. These results further highlight the strength and abstraction level of the performed reasoning performed by DeiSAM
5.6 DeiCLEVR Abstract Reasoning Segmentation
DeiSAM excels in high-level abstract reasoning, where purely neural pipelines often struggle. To demonstrate, developed DeiCLEVR, abstract reasoning segmentation task based CLEVR (Johnson et 2017). This task challenges models with abstract concepts and relationships.
Task. The task is to segment objects given prompts where the answers are derived by the reasoning over abstract list operations. We consider operations: delete and sort. The input is a pair of image and prompt, e.g. "Segment the second left-most object after deleting gray object?". Examples are shown in Fig. 6 To solve this task, models need understand the visual scenes and perform high-level abstract reasoning segment.
Dataset. (Image) Each scene contains at most 3 objects with different attributes: (i) colors of cyan, gray, red, and yellow, (ii) shapes of sphere, cube, and cylinder, (iii) materials of metal and matte. We excluded color duplications in a single image. (Prompts) We generated prompts using a templates: "The [Position] object after [Operation]?" !"" where [Position] can take either of: left-most first, second, or third. [Operation] can take either of: (I) delete an object, and (II) sort the objects in the order of: cyan gray red < yellow (alphabetical order). We generated 10k examples for each operation.
Models. We used DeiSAM with Slot Attention (Locatello et 2020) pretrained the visual Inductive Logic Programming (ILP) dataset (Shindo et al. (024), which contains positive and negative visual scenes for list operations. We used GroundedSAM and LISA for neural baselines (cf. App. E
'In App. F we demonstrate experiments on DeiVG with prompts in the RefCOCO's reference format, highlighting the robustness of DeiSAM against neural baselines.

---

Prompt
Input
DeiSAM
GroundedSAM
LISA
The 2nd leftmost object afler deleting red object.
The leftmost object after sorting objects the order Syan Srddy-redyyllloww
Figure 6: DeiSAM performs abstract reasoning segmentation. When presented with visual scene paired with an abstract, complex prompt (left), DeiSAM effectively identifies and segments the object specified by the prompt, while neural baselines frequently fail to deduce the target object (right).
Result. In Table we present the mean Average Precision (mAP) for each baseline evaluated. The purely neural baselines struggle to accurately deduce segmentations in response to abstract reasoning prompts, while DeiSAM excels at identifying and segmenting the object specified by the prompt. Moreover, Fig. 6 provides qualitative examples illustrat- ing that DeiSAM effectively segments objects requiring high-level reasoning. In contrast, the neural baselines fre- quently fail to segment the correct target object. These findings indicate that existing neural basclines are inad- equate for addressing abstract reasoning prompts. We demonstrate that integrating differentiable logic reasoners can significantly enhance reasoning capabilities.
Table DeiSAM handles abstract vi- sual reasoning, mAP on DeiCLEVR.
mAP (1)
Delete
Sort
DeiSAM
99.29
99.57
GroundedSAM
7.6
15.39
LISA
12.88
11.15
5.7 End-to-End Training of DeiSAM
Since DeiSAM employs a differentiable forward reasoner, a meaningful gradient signal can be back-propagated through the entire pipeline. Consequently, DeiSAM enables end-to-end learning on complex object detection and segmentation tasks with logical reasoning explicitly modeled during training. To illustrate this property, show that DeiSAM can learn weighted mixtures of scene graph generators by propagating gradients through the reasoning module.
Task. We consider 2 distinct scene graph generators and compose weighted mix- ture of them. We show an example for the deictic prompt "An object that has hair and that is on surfboard" in Listing 2 The first rules compute the target object for and
Program
targetigg (X, :-cond1 (x,S6) ccond2 (x,d)(( cond1 (x,SG) : (X,,, type (Y, hai, cond2 o-onggg (x,Y,80 Y,S0), onšgg aurfboard,
Compose veighted mixtures.
target :-targetSgg (X,, w_2: target (X) :-targetšgg agg))
each SGG, similarly to Program the last rules produce weighted merge of both predictions. Importantly, Program 2 utilizes different SGGs (i.e. variable SG) and merges the results using learnable weights (i.e.w_ 1 and 2_2). Consequently, the learning task is the optimization of weights w; R for downstream deictic segmentation. The differentiability of the DeiSAM pipeline allows efficient gradient- graddeet-based optimizations.
Listing A program for SGG learning.
pooottannlyyProrraa
Experimental Setup. We used VETO (Sudhakaran al., 2023), which outperforms other SGGs on biased datasets where only some relations appear frequently. As the second 'SGG' for our weighted mixture model, we relied on ground-truth scene graphs from Visual Genome. We consider the following baselines: DeiSAM-VETO that only uses a pre-trained VETO model (Sudhakaran et al. 2023), DeiSAM-Mixture (naive) that uses mixture of VETO and VG scene graphs with randomly initialized weights. We compare those approaches to DeiS DeiSAM-Mixture Mixture*, which uses the trained mixture. We extracted instances from DeiVG datasets not used in VETO training (ca. 2000 samples), which we divided into a training, validation, and test split. For rule generation, we use the same system prompt and models in Sec. S.c. adapting the generated programs for weight learning.
We minimize the binary cross entropy loss with respect to rule weights w_1 and w_2. To calculate this loss, we provide labels for predicted masks in the model, i.e., a binary label y, E 1). For each instance Dei DeiSAM predicts segmentation masks the forward pass, and gradients are backpropagated through the differentiable forward reasoner (ef. App. E.T for details).

---

Table 6: End-to-end training improves DeiSAM. Mean Average Precision on the test split of the task of learning SGGs. DeiSAM- DE-EMOT-ETOOO uses a trained VETO model (Sudhakaran et al., 2023). DeiS AM- Mixture (naive) uses a mixture of a trained VETO model and VG scene graphs with randomly ini- tialized rule weights, DeiSAM-Mixtrre uses the resulted mixture model after the weight learning.
input image
An object that is on a blanket, that has face
D.
0. 96
before learning
after learning
Figure 7: DeiSAM can learn to produce bet- ter masks. Shown are the input image (left) and target segmentation masks together with confidence scores obtained before (middle) and after (right) end-to-end training DeiSAM. DeiSAM improves the quality of segmenta- tion by learning (Best viewed in color).
mAP (%) f
Method
DeiVG1
DeiVG2
DeiSAM- VETO
6.64
15.92
DeiSAM: M-Miixture (naive)
37.61
59.81
DeiS BoiSAM-Mixtun
64.44
86.57
Result. In Tab. 6 compare the mAP on the test split. The trained model DeiSA.M-Mixturee clearly outperforms the naive baseline, demonstrating successful training of the DeiSAM pipeline using gradients via differentiable reasoning. DeiSAM-VETO weak performance can be attributed to objects that appear only on prompts but not its training data (cf. App. D)
Fig 7 shows examples of segmentation masks and their confidence scores produced by DeiSAM- Mixture models before and after training. Before learning, wrong or incomplete regions are segmented with low confidence scores because the reasoner fails to identify correct objects with low-quality scene graphs that miss critical objects and relations. After learning, DeiSAM produces faithful segmentation masks and increased confidence scores. This experiment highlights that DM improves the quality of scene graphs and the subsequent segmentation masks by learning using gradients, i.e., it is fully trainable pipeline with strong capacity for complex logic reasoning.
6 Conclusion
Before concluding, let us discuss the limitations and future research directions. Our investigation of DeiSAM's components highlights some clear avenues for future research. While LLMs perform well at parsing deictic prompts into logic rules with few-shot prompting, their performance could be improved further by, e.g., syntactically constrained sampling" dedicated fine-tuning. Further, the observed challenges in semantic unification could be addressed by querying LLMs instead of using embedding models or providing multiple weighted candidates to the reasoner.
Upon manual inspection of the DeiVG dataset, we identified some inconsistent examples annotated with erroneous scene graphs in Visual Genome that cannot be automatically cleaned up without external object identification (cf. App. D). Our results support the assessment that generating rich scene graphs is key but difficult to achieve differentiable pipeline of AM can be utilized for meaningful training on complex downstream tasks. Thus allowing for the incorporation of real-worid use cases in the training of SGGs in an end-to-end fashion. Further, DeiSAM can provide valuable information on general performance and failure cases of SGGs by investigating deictic segmentation tasks. Furthermore. the modularity of DeiSAM allows for easy integration of potential improvements to any of its components.
zero-shot fashion. However, as we demonstrated, the
To conclude, we proposed DeiSAM perform deictic object segmentation in complex scenes. DeiSAM effectively combines large-scale neural networks with differentiable forward reasoning in modular pipeline. DeiSAM allows users to intuitively describe objects in complex scenes by their relations to other objects. Moreover, we introduced the novel Deictic Visual Genome (De benchmark for segmentation with complex deictic prompts. In our extensive experiments, we demonstrated that DeiSAM strongly outperforms neural baselines highlighting its strong reasoning capabilities on visual scenes with complex textual prompts. To this end, our empirical results revealed open research questions and important future avenues of visual scene understanding.
httpo:// http://ttttubbootteaeaaeeee htw.Teeaccc//Sytttttiaaayy Coustrained Sampling
10

---

Acknowledgements. The authors thank Maurice Kraus and Felix Friedrich for their valuable feedback on the manuscript. This work was partly supported by the EU ICT-48 Network of Al Research Excellence Center "TAILOR" (EU Horizon 2020, GA No 952215), and the Collaboration Lab "AI in Construction" (AICO). The work has also benefited from the Federal Ministry of Education and Research (BMBF) Competence Center for Al and Labour ("KompAKI", FKZ 02L19C150) and from the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK) cluster projects "The Third Wave of AI" and "The Adaptive Mind". We gratefully acknowledge support by the German Center for Artificial Intelligence (DFKI) project "SAINT". This work also benefited the HMWK/ / BMBF ATHENE project "AVSV" and the National High Performance Computing Center for Computational Engineering Science (NHR4CES). The Eindhoven University of Technology authors received support from their Department of Mathematics and Computer Science and the Eindhoven Artificial Intelligence Systems Institute.
References
Agre, P. E. and Chapman, D. Pengi: An implementation of theory of activity. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 1987.
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei. S., Monteiro, M., Menick, J. L.,, Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M. a., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flamingo: visual language model for few-shot learning. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Amizadeh, S., Palangi, H., Polozov, A., Huang, Y., and Koishida, K. Neuro-symbolic visual reasoning: Disentangling "Visual" from "Reasoning". In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2015.
Brown, T. B., Mann, B.., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P. Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C.,, Hesse, C., Chen, M., Sigler. E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
Cropper, A. and Dumancic, S. Inductive logic programming at 30: A new introduction. Journal of Artificial Intelligence Research (JAIR), 2022.
Deiseroth, B., Schramowski, P., Hikaru Shindo, D. S. D., and Kersting, K. Logicrank: Logic induced reranking for generative text-to-image systems. arXiv Preprint:208..1 13518, 2022.
Dong, X., Gan, T., Song, X., Wu, J., Cheng, Y., and Nie, L. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Evans, R. and Grefenstette, E. Learning explanatory rules from noisy data. Journal of Artificial Intelligence Research (JAIR), 2018.
Finney, S., Gardiol, N., Kaelbling, L. P., and Oates, T. The thing that we tried didn't work very well: Deictic representation in reinforcement learning. In Proceedings of Conference on Uncertainty in Artificial Intelligence (UAI). 2002.
Guo, Y., Liu, Y., Georgiou, T., and Lew, M. S. A review of semantic segmentation using deep neural networks. Int. J. Multim. Inf. Retr. 2018
Helff, I.., Stammer, W., Shindo, H., Dhami, S., and Kersting, K. V-lol: diagnostic dataset for visual logical learning. arXiv Preprint:2306.07775 37744000077,,... 2023.

---

Hsu, J., Mao, J.. Tenenbaum, J. B., and Wu, J. What's s left? concept grounding with logic-enhanced foundation models. Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
Huang. J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. Large language models cannot self-correct reasoning yet. In Proceedings of the International Conference on Learning Representations (ICLR), 2024a.
Huang, J., Li, Z.,, Naik, M., and Lim, S.-N. Laser: A neuro-symbolic framework for learning spatial-temporal scene graphs with weak supervision. arXiv Preprint:2304.0766 2024b.
Hudson, D. A. and Manning, C. D. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ishay, A., Yang. Z., and Lee, J. Leveraging large language models to generate answer set programs. In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning (KR), 2023.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C., and Girshick, R. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. ReferltGame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg,
A. C.,, Lo, W.-.,, Dollar, P., and Girshick, R. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2017.
Lai, X., Tian, Z., Chen. Y., Li, Y., Yuan, Y.. Liu, S., and Jia. J. Lisa: Reasoning segmentation via large language model. arXiv Preprint:2308. 200,0002200 2023.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping lunguage-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), 2022a.
Li, L. H., Zhang, P., Zhang, H., Yang. J., Li, C., Zhong, Y., Wang, L., Yuan, L., Zhang, L., Hwang, J., Chang, K., and Gao, J. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Lin, X., Ding, C., Zeng, J., and Tao, D. GPS-Net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Liu, C., C.Ding, H., and Jiang, X. GRES: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023a.
Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference Neural Information Processing Systems (NeurIPS), 2023b.
Liu. S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv 23ep3int:303..5-44 2023c.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2020.
12

---

Lu. C., Krishna, R., Bernstein, M.. and Fei-Fei, L. Visual relationship detection with language priors. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Lu, Y., Rai, H., Chang, J., Knyazev, B., Yu, G., Shekhar, S., Taylor, G. W., and Volkovs, M. Context- aware scene graph generation with Seq2Seq transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Mao, J., Gan, C., Kohli, P., Tenenbaum, J. B., and Wu, The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Marconato, E., Teso, S., Vergari, A., and Passerini, A. Not all neuro-symbolic concepts created equal: Analysis and mitigation of reasoning shortcuts. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
Minervini, P., Riedel, S., Stenetorp, P., Grefenstette, E., and Rocktäschel, T. Learning reasoning strategies in end-to-end differentiable proving. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F., Zeng, Z., Zhang, H., Li, F., Yang, J., Li, H., Jiang, Q., and Zhang, L. Grounded SAM: assembling open- world models for diverse visual tasks. arXiv Preprint: 1411, 2024.
Rocktäschel, T. and Riedel, S. End-to-end differentiable proving. In Proceedings of the Advances in Neural Information Processing Systems: Anmual Conference on Neural Information Processing Systems (NeurIPS), 2017.
Russell, S. and Norvig, P. Artificial Intelligence: A Modern Approach. Prentice Hall, 3 edition, 2010.
Sha, J., Shindo, H., Kersting, K., and Dhami, D. S. Neural-symboll predicate invention: Learning relational concepts from visual scenes. In Proceedings of the 17th International Workshop on Neural-Symbolic Learning and Reasoning (NeSy), 2023.
Sha, J., Shindo, H., Delfosse, Q. Kersting, K., and Dhami, D. S. EXPIL: explanatory predicate invention for learning games. arXiv Preprint:24066.0::00 06007 2024.
Shi, F., Chen, X., Misra, K., Scales, N., Dohan, D., Chi, E. H., Schärli, N., and Zhou, Large language models can be easily distracted by irrelevant context. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
Shindo, H., Nishino, M., and Yamamoto. A. Using binary decision diagrams enumerate inductive logic programming solutions. In Up-and-Coming and Short Papers of the 28th International Conference on Inductive Logic Programming (ILP 2018), 2018
Shindo, H. Dhami, D. S., and Kersting, K. Neuro-symbolic forward reasoning. arXiv Preprint:2110.09388, 2021.
Shindo, H., Pfanschilling, V., Dhami, D. S., and Kersting, K. oilp: thinking visual scenes as differentiable logic programs. Machine Learning (MIJ), 2023.
Shindo, H., Pfanschilling, V., Dhami, D. S.,, and Kersting, K. Learning differentiable logic programs for abstract visual reasoning. Machine Learning (MLJ), 2024.
Stammer, W., Schramowski, P., and Kersting, K. Right for the right concept: Revising neuro-symbol concepts by interacting with their explanations. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Stammer, W., Friedrich, F,, Steinmann, D., Brack, M., Shindo, H., and Kersting, K. Learning by self-explaining. Transactions on Machine Learning Research (TMLR), 2024.
13

---

Stanić, A., Caelles, S., and Tschannen, M. Towards truly zero-shot compositional visual reasoning with Ilms as programmers. Transactions on Machine Learning Research (TMLR), 2024.
Sudhakaran, G., Dhami, D. S., Kersting, K., and Roth, S. Vision relation transformer for unbiased scene graph generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Surís, D., Menon, S., and Vondrick, C. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Tan, H. and Bansal, M. LXMERT: learning cross-modality encoder representations from transformers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Tang, K., Zhang. H., Wu, B., Luo, W., and Liu, W. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and 2023.
efficient foundation language models. arXiv Preprint:2002.3397
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., and Cottrell, G. W. Understanding convolution for semantic segmentation. In Proceedings of IEEE Winter Conference on Applications of Computer Vision (WACV), 2018.
Wang, P., Yang. A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. OFA: unifying architectures, tasks, and modalities through a simple srquence-to-sqqenne learning framework. In Proceedings of the International Conference
Machine Learning (ICML), 2022.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thouud thothhu prompting elicits reasoning in large language models. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2022.
Wong, L.,, Grand, G., Lew, K., Goodman, N. D., Mansinghka, V. K., Andreas, and Tenenbaum. J. B. From word models to world models: Translating from natural language to the probabilistic language of thought. arXiv Preprint: 12672, 2023.
Wu, J., Li, X., Li, X., Ding, H., Tong, Y., and Tao, D. Towards robust referring image segmentation. IEEE-TIP, 2024a.
Wu, T.-H., Biamby, G., Chan, D., Dunlap, L., Gupta, R., Wang, X., Gonzalez, J. E., and Darrell, See say and segment: Teaching imms to overcome false premises. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Xu, D., Zhu, Y., Choy, C. B., and Fei-Fei, L. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-5419, 2017.
Yang, Z., Ishay, A., and Lee, J. Coupling large language models with logic programming for robust and general reasoning from text. In Findings of the Association for Computational Linguistics (ACL), 2023.
Ye, Z., Shindo, H., Dhami, D. S., and Kersting, K. Neural meta-symbolic reasoning and learning. arXiv Preprint: 2211. 1.1 1650, 2022.
Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., and Tenenbaum, J. Neural-symbolic VQA: disen- tangling reasoning from vision and language understanding. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2018.
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A., and Tenenbaum, J. B. Clevrer: Collision events for video representation and reasoning. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
14

---

Yu, L., Poirson, P., Yang, S., Berg, A. C., and Berg, T. L. Modeling context in referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Zimmer, M., Feng, X., Glanois, C., JIANG, Z.., Zhang, J., Weng. P., Li, D., HAO, J., and Liu, W. Differentiable logic machines. Transactions Machine Learning Research (TMLR), 2023.
Zou, X., Yang. J., Zhang. H., F., Li, L., Wang. J., Wang. L., Gao, and Lee. Y. J. Segment every- thing everywhere all at once. In Proceedings of the Advances in Neural Information Processing Systems: Annual Conference on Neural Information Processing Systems (NeurIPS), 2023.
15

---

conjunction nodes
atom nodes
tazget [obj11
cond1 (obj 1]
cond2 [obj]1
en [oa]1,obj]]
Lype [ob]2,
holding [obj1,
type (obj3, umbrella)
Figure 8: Forward reasoning graph for Program 1 Listing 1 reasoning graph consists of atom nodes and conjumction nodes, and is obtained by grounding rules i.e., removing variables by, e.g.. f obj1, + obj2. By performing bi-directional message passing on the reasoning graph using soft-logic operations, DeiSAM computes logical consequences in differentiable manner. Only relevant nodes are shown (Best viewed in color).
A First-Order Logic and Differentiable Reasoning.
We provide a formal definition of first-order logic (FOL). An atom is a formula p(t,..... tq), where p is a predicate symbol (e.g. type)))) and 1,,.... tn are terms. term variable constant. A ground atom or simply a fact is an atom with no variables (e.g. type [bjj. boat)). A literal is an atom (A) or its negation (-4), and a clause is a finite disjunction (V) of literals. definite clause is clause with exactly one positive literal. If A, BI..... Bn are atoms, then A V -B1 V... V VB, ,, is definite clause. We write definite clauses the form of :- ..,..., B,, and refer to them as rules for simplicity in this paper. Forward Reasoning is a data-driven approach of reasoning in FOL. (Russell & Norvig 2010). i.e., given set of facts and a set of rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoners compute logical entailment using tensor representations (Evans & Grefenstette 2018 Shindo et al., 2023) or graph neural networks (Shindo et al., 2024). and perform rule learning using gradients given labeled examples in the form of inductive logic programming (Cropper Dumancie 2022).
DeiSAM employs graph neural network- differentiable forward reasoner (Shindo et al. 20244. and we briefly explain the reasoning process. We represent a of (weighted) rules as a directed bipartite graph. For example, Fig. is a reasoning graph that represents Program 1
A.1 Details of Differentiable Forward Reasoning
We provide the details of differentiable forward reasoning.
Definition A.1. Forward Reasoning Graph a bipartite directed graph (Vg. Vn, Eg Eg) where VG is set of nodes representing ground atoms (atom nodes), Vx is set of nodes representing conjunctions (conjunction nodes), Eg-+A is set of edges from atom conjunction nodes and Ex-ig set of edges from conjunction to atom nodes.
DeiSAM performs forward-chaining reasoning by passing messages on the reasoning graph. Essen- tially, forward reasoning consists two steps: (1) computing conjunctions of body atoms for each rule and (2) computing disjunctions for head atoms deduced by different rules. These two steps can be efficiently computed on bi-directional message- message-passing on the forward reasoning graph. We now describe each step in detail.
(Direction >) From Atom to Conjunction. First, messages are passed the conjunction nodes from atom nodes. For conjunction node Vv, the node features are updated:
V (o.t
(1)
where A is a soft implementation of conjunction, and V is soft implementation of disjunction. Intuitively, probabilistic truth values for bodies of all ground rules are computed softly by Eq. [
(Direction <-) From Conjunction to Atom. Following the first message passing, the atom nodes then updated using the messages from conjunction nodes. For atom node vi E Vg. the node features are updated:
(1+1) 0/
= =V V
(0)
JEN()) Wyr o/
16

---

where Wy is weight of edge ej+i- We assume that each rule C₁ E C has its weight 8R, and wys = 0k edge ej-s; on the reasoning graph is produced by rule Cg. Intuitively, in Eq. 2 new atomss are deduced by gathering values from different ground rules and from the previous step.
We used product for conjunction, and log-sum-exp function for disjunction:
(II,..,, Rn) = log Σ
exp(x1/7),
where 7 > 0 a smooth parameter. Eq. approximates the maximum value given input .1....,, Bn-
Prediction. The probabilistic logical entailment is computed by the bi-directional message-passing,
atoms [0, i]]si be imput node features, which map a fact to scalar value, RG be the reasoning graph, w be the rule weights, B be background knowledge, and T E N be the infer step. For fact
Let x (0)
G2 E G, DeiSAM computes the probability as:
p(G; X xaooms RG, w. B, = Xatoms [].
(4)
where X Xotoms message-passing
E
[0, 1]|61 is the node features of atom nodes after T'steps of the bi-directional
By optimizing the cross-entropy loss, the differentiable forward reasoner can solve Inductive Logic Programming (ILP) problems with propositional encoding (Shindo et al. 2018), where the task is
to find classification rules given positive and negative examples. has been extensively applied
to solve complex visual patterns (Helff et al., [202)). image generation (Deiseroth et al. (002)).
meta-level reasoning and learning (Ye et al.) 2022). predicate invention (Sha et al.) 2024 2023), and
self-explanatory learning (Stammer et al. 2024).
B System Prompts for Rule Generation
To generate logic rules using LLMs, we used the following system prompt.
Given deictic representation available predicates, generate rules the format. target(X) :oodd1 ... condn (X). cond1 - (x, type((Y
condn (X) (X, predicates and ttppe(Ycconat2).
conat2).
that appear in the given Capitalize variables: Y, Z, W,
In practice, this system prompt combined with few-shot examples for downstream task (ef. App.
C DeiVG Datasets
We generated DeiVG dataset using Visual Genome dataset (Krishna et al. (017). We used the entire Visual Genome dataset generate deictic prompts and answers out of scene graphs, and randomly downsampled 10k examples. We only considered the following relations:
'on'
wwearing'
sitting on'
'waars'
'near'
'made of"
'has'
"along'
'above'
parredd on'
'in front of"
'carrying'
behind
att'
'riding'
'holding'
"under'
'over'
"against'
The prompts are synthetically generated by extracting relations that shares the same subject in the scene. For example, with pair VG relations, "person is holding an umbrella" and "person on boat", we generate deictic prompt, "an object that is holding an umbrella, and that is on boat". Subsequently, the corresponding answer extracted from the scene graph. We provide two instances in the generated DeiVG, dataset in Listing 3
17

---

X Example 1
Seictic_promphr "an object that is on a sofa, and that is on book", ansver: [{"name": "paper",
"h": 32,
"synsets": ("paper.n.01]]
"object_idd: 2687751,
"v": 61, "y": 236,
"x": 272 VG_ image_igd 2376540, VG_data_ 44297]
Example
deictic_prompt: object that has shadow, that is wearing black shirt", [{"h": 50, "object_id": 1001275, "#erged__ojject_dd== : [1001270], "synsets"; [*man.n.011)
"u": 21, "y": 333, "x": 47,
"names": ["man"]}), VG_ vG_image_id: 2365153, vo_data_iindxx 55196]
Listing 3: DeiVG examples.
D Additional Analysis on VG Scene Graphs
We provide further investigation of Visual Genome (VG) scene graph annotations. We demonstrate that (1) VG annotations have multiple versions and there is non-trivial discrepancy between their relation distributions, and (2) VG annotations contain incomplete and erroneous scene graphs that cannot be automatically cleaned up without external object identification.
VG Annotation Discrepancy We investigated the scene graph generator module, going beyond the ground truth scene graphs of Visual Genome. One of the key challenges in using pre-trained SGG is the potential mismatch between the set of objects and annotations in DeiVG and the training data of the SGG. While we built the latest version of Visual Genome with extended annotations (VGv1.4). an older version (Krishna er al., 2017) has been commonly used for benchmarking different SGGs by excluding non-frequent object types and relations (Xu al. (017). This preprocessed older version (VG) is still considered the standard benchmark for SGGs, which lacks many crucial objects and attributes in DeiVG built upon VGvL44 We illustrate the discrepancy in Fig. 9 where we compare the Kernel Density Estimate (KDE) of VG,1.4 and VG. It shows that the newer version contains more types of objects in the top-and- middle frequency range. Additionally, many object types of VG,1.4 in the middle-frequent parsing scenes from DeiVG with SGGs pre- pre-trained on VG, the objects in the deictic prompt may not be contained in the generated scene graph at all. For example, for a given deictic prompt "an object that wearing a black shirt", we observed that the pre-trained SGG failed detect black shirt because it was not included in its training data. This discrepancy leads to sub-par performance of DeiSAM with pre-trained SGG, as shown in Tab. 6 While training the SGG specifically on the relevant scene distribution can partially address this issue, it is crucial to highlight that DeiSAM can be leveraged to improve SGGs as well. Our subsequent demonstration in Section 5.7 provides a glimpse of DeiSAM's capabilities, showcasing its differentiable forward reasoner's ability to perform end-to-end training, thereby unlocking new horizons in scene understanding and reasoning.
range are not contained at all in VG. Consequently, when
In Fig. 10 the plot on the left depicts the distribution of the least frequent tail object types on the latest extended version (VG,1.4) in comparison to the older version (VG) (Krishna et al. 2017). Object types are sorted with respect to the number of occurrences (x-axis). In the non-frequent range of 0 to 1000, VG,1. contains object types that never appear on the older version (VG), revealing these tail classes of object types are completely missing in the processed older annotations (Xu et al. 2017)
18

---

Distribution of lop & middle frequent object types
Distribution of middle frequent object types
0.12
1,6r
VGv1.4
VGv1.4
0.10
0.k
Densily
0.00
VG
Density
VG
0.06
0.4
0.04
0.2
0.0
10000
1000
2500
Occurrencess
# Occurrencess
Figure 9: The large discrepancy between DeiVG and standard Visual Genome. DeiVG uses Visual Genome with extended annotations (VG,1.4) but the older version (VG) (Krishna et al. 2017) has been commonly used for SGG training. Object types are sorted w.r.t. the number of occurrences (X-axis) and their corresponding occupancies are shown (y-axis). The left plot is for top-and-middle frequent object types (0-50k #Occ.), and the right plot is for middle frequent object types (0-3k #OCc.). VG+1.4 contains many object types that do not appear in VG (Best viewed in color).
VGVL.4
VC+2 VOJUL LYPO VS5 object typees
1/5 4 total objacts
2,00
VS
VG Eo0 obyects
1.75
1.50
5200
1.25
4000
102
1.00
3200
0.75
52-
2000
0.30
1000
0.25
1000 2000 3000
4000
salo soco 7000
5000
0.00
Figure 10: Further comparisons of VG with VG. The plot the left depicts the distribution of the least frequent tail object types of the latest extended version (VGv1.1) in comparison to the older version (VG) (Krishna et al.] (2017). Object types are sorted w.r.t. the number of occurrences (x-axis). The plot on the right shows the comparison of the number of object types and the number of total objects in the datasets. VG.1.4 contains many more object types than VG, making difficult for SGGs that are pre-trained on the older version to achieve high performance (Best viewed in color).
which are commonly used for benchmarking SGGs. Moreover, the bar plot on the right indicates a notable increase in the number of object types and total object counts in VG,1.4 in comparison to VG, making difficult for SGGs that are pre-trained on the older version to achieve high performance on the De dataset built upon VG
Errors on VG Annotations. Upon manual inspection of the DeiVG dataset, we identified some inconsistent examples resulting from incomplete and erroneous scene graphs in Visual Genome that cannot be automatically cleaned up without external object identification. For example, the annotations shown in Fig. lead to DeiVG sample with two missing target objects and an incorrect one. We plan on building a more consistent deictic segmentation benchmark using a cleanup process similar to GQA (Hudson & Manning 2019).
E Details of Experiments
We provide details of the models used in the evaluation. For all methods using SAM for segmentation including DeiSAM-we use the same publicly available SAM checkpoin
DeiSAM. We used NEUMANN (Shindo et al., 2024) with 7 = 0.01 for soft-logic operations, and the number of inference steps is set to 2. We set the box threshold to 0.3 and the text threshold to 0.25 for the SAM model. All generated rules are assigned a weight of 1.0. If no targets are detected, DeiSAM produces a mask of randomly chosen object in the scene.
For LLMs, we provided few-shot examples of deictic prompts and desired outputs shown in Listing 4 These few-shot examples improved the quality of the rule generation that follows a certain format. These are combined with the system prompt in App. B to generate rules by LLMs.
httpo://huggiggtace. wttp://ccccccccccccaeeeeeaaocaet wttp::/hhhhhhttttttaee//eeaaeeee vit. 0b8939.9pth
19

---

Prompt: 'An object that is wearing a helmet'
inimet
Figure 11: Example of erroneous annotations in Visual Genome leading to inconsistent examples in DeiVG. Here, the annotations for 'helmet' are incomplete and one case, the relation 'wears' linked the wrong person (Best viewed in color).
Examples:
an object that next to keyboard. available predicates: next_to cond1 -nxxt (x,, (), type keyybaaa keyboard). : (X)
deak. available predicates: on cond1 on, Y) (Y,e(k)))sk) (x)get(X): :cond (X).
object that
object that is ground, and that is behind white line. available predicates: on, behind cond1 (X) -x Y) (Y, ground))
cond2 (X) -behind Y) type whiteline).
target condd (X), cond2(X)
an object that is near deak and against wall. available predicates: near aharragainst cond1 (X) :-near (X, Y) ttyee type deed)) cond2 (x): :-gaannst (X, tt,pe(Y, target (X) :-cond (X), cond2
an object that has sides, that is on a pole, and that is above stop sign. available predicates: aboove cond1 (X) :-has (X,Y), (Cxx,Y)tyye (tt.)tyyt.yyyy......... ttype(,, cond2 (X, type(Y,
cond3 -above (X,Y). tyype sttoopsgg))) target :-cond (X), cond2(X) cond3(X)
object that vearing shirt, that has hair, and that wearing shoes. availsble predicates: wearing,hae, wearwn condl (x ttye ehi cond2 (X, type(Y,
cond3 (X) (x Y) type shoes)
target :-cond1 (X), cond2(X), (X).
Listing Few-short examples for rule generation using LLMss.
with GroundedSAM. used a publicly available GroundedDino version with Swin-B backbone, pre- trained on COCO, O365, GoldG, Cap4M, OpenImage, ODinW-35 and RefCOCO, We set the box threshold 0.3 and the text threshold to 0.25 for SAM model.
GLIP We used publicly available version of GLIP-1 10 with Swin-L backbone and pre-trained on FourODs, GoldG, CC3M+ 12M, SBU. We the prediction threshold 0.4.
OFA We used a publicly available version of LaA-Large fine-tuned for visual grounding.
bitoc: //ettbub. con com/10GA http://eee.eeeeeeeoeeeeeeeeeeeee /tw. 1.0 alpha2/ htan://grondiggiin..oo wtt.e/eccbbaoccorrot.ao..aaoccee pth
htts:://uucissttce.coo http:://wwaaaatto...o/////////as
podol. pth
httpo: http:://ooeleccpp. http::ppp..ccoooo/s/////ttte//ee grounding Lorgo sa/sunnary
20

---

SEEM We used a publicly available checkpoint of SEEM with Focal-L backbone.
Evaluation Metric. We used mean average precision (mAP) evaluate segmentation models. Segmentation masks are converted to corresponding bounding boxes by computing their contours, and then mAP is computed by comparing them with the ground truth bounding boxes provided by Visual Genome.
E.1 Learning to Segment Better
We describe the experimental setting in more detail.
Method. Let (a, p) E D be a DeiVG dataset, where æ is an input image, and is a deictic prompt. For each input, DeiSAM produces segmentation masks M, with corresponding confidence score 8i, i... {(M, ==)):=....n f
For each mask M,, we consider a binary label Yi E {0. 1} using its corresponding bounding box and comparing with ground-truth masks with the loU score, which assesses the quality of bounding boxes. For each instance (z. p) D, we compute the binary- cross entropy loss: { = - (M,::)) (.::....)aasswwl fffdsssa (m.pp) (3/ log 8; + (1 y4) log(1 s:)), We minimize the loss É through gradient descent with respect to the rule weights in the differentiable programs.
deessmm (x.,))
Task. Given SGGs s991, 8992: .... s' we compose a DeiSAM model with a mixture of them, i.e. (x)) ΣS w,sgg, (æ) with w; [0, and sgg; IR [0. 1]|s1, where G is a set of facts to describe scene graphs. For example, Program 2 shown on the right represents the mixture of scene graph generators for the deictic prompt "An object that has hair and that is on a surfboard". In contrast to Program the program utilizes different SGGs and merges the results using learnable weights. The learning task is the optimization of weights w; for downstream deictic segmentation. The differentiable reasoning pipeline in DeiSAM allows efficient gradient-based optimizations with automatic differentiation.
Dataset. Let (æc. p) E 'D be DeiVG dataset, where æ is an input image, and is a deictic prompt and (0), (0l,.... 0m) E A be the answers, where 01,... Om. are correct target objects be segmented specified by the prompt. For each input, DeiSAM produces segmentation masks with their confidence:
{M, s:)]==o.... = fdessam (z. p),
(5)
where M, is the i-th predicted segmentation mask and is the confidence score. For each mask M, we consider binary label y E {0. 1) by computing its corresponding bounding box and using the IoU score, which assesses the quality of bounding boxes:
y/ (1 if max, (oU, 0,) otherwise
(6)
where 0 > 0 is a threshold, B, is bounding box computed from mask M,, and O, is mask that represents answer object 0y. We extracted instances from DeiVG datasets not used in VETO training (ca. 2000 samples), which are divided into training, validation, and test splits that contain 1200, 400, and 400 instances, respectively. no targets are detected by the forward reasoner, DeiSAM produces mask of randomly chosen object in the scene, i.e., if the maximum confidence score for targets is less than 0.2, DeiSAM was set to sample a random object in the scene and segment with a confidence score randomly sampled from uniform distribution with a range of [0.1, 0.4].
Optimization, We used the RMSProp optimizer with learning rate of 1e 2, and performed 200 steps of weight updates with a batch size of 1. The reasoners' inference step was set to 4. We used IoU score's threshold = 0.8.
F Ablations
Subsequently, we present detailed results corresponding the ablation studies in Sec. 5.4
LLMs for Rule Generation and Semantic Unification. First, we evaluated multiple open and closed-source language models of different sizes on rule generation. The results on correct predicate identification and rule generation are reported Tab. 7 general, all instruction-tun -tuned models can generate logic rules from deictic prompts. However, larger models strongly outperform smaller
httpo://huggisgtacc.. wttp://oocccccccccc///coooooee
21

---

Table 7: Performance in logic rule generation of various large language models. We report the success rate of extracting the correct predicates and generating the correct rules in isolation since the predicates are supplied as context for rule generation. The combined success rate are the percentage of samples were both are correct.
Curr. Predicates (%)
Corr: Rules (%)
Cumbined (5)
Model
De
DeVV2
DeiVGa
DeiVGz
De
De
DeiVC2
DeiVGg
Mistral-Pp 7B Instruct
75.50
75.52
67.80
86.95
82.56
75.43
65.38
64.63
$1.50
Llama- 2 Chat
75.03
26.91
12:47
98.29
96.36
95.22
74.06
25.92
12.06
Clama-2-11 2-aee----3------ Chas
92.87
97.19
96-05
97.88
97.82
97.13
91:00
95.
9345
96.57
97.43
97.45
95.04
95.36
Table 8: Comparison of various embedding models on the semantic unification step in the DeiSAM pipeline. The task is to identify one synonymous object name out of all objects in a scene, given their embedding similarity. Success rate is reported over ca. 2.5k scenes from Visual Genome.
Model
Unification Success (%) f
Glove-Wiki- Fiik-Gr
52.27
VeedZeee-Googee--
55.03
OpenAI-CLIP ViT-B/32
52.90
DFN5B-CLIP ViT-H/14
66.65
MonarchMixer-Ber
30.15
MPNet-Base-at
71.62
MiniLM--L6-22
62.79
OpenAl-ada-002
66.50
ones, especially for more complex inputs. Interestingly, the types of failure cases differ significantly between models. For example, both Llama models and GPT-3.5-turbo rarely make syntactical errors in rule generation (< 1%), whereas most of Mistral-7B's incorrect rules are already syntactically unsound.
For the semantic unification task we compare various semantic embedding models as shown in Table
Runtime Analysis. We provide comparisons of the runtime of Grounded-SANe LISA, and DeiSAM in Tab. 99] It shows the inference time per instance (a visual input with textual prompt), averaged over 1000 examples for each dataset. GroundedS achieved remarkably faster inferences since it does not encode the reasoning process. Both LISA and DeiSAM approaches require about 6 to 10 seconds per example. For more analysis, we provide the running time for each component of DeiSAM in Tab. T0] It shows the inference time per instance of rule generation, semantic unification, differentiable forward reasoning, and segmentation. We observe that semantic unification is the most time-consuming process in the DeiSAM pipeline. In contrast, the differentiable forward reasoning and segmentation only make up negligible fraction of the runtime. However, as outlined in the paper, the rule generation and semantic unification step rely on the OpenAl API and could be significantly reduced by running a local model instead.
Table 9: Runtime comparison of DeiSAM and basclines.
Table 10: DeiSAM runtime analysis.
Running Time (sec)
Running Time (sec)
Method
DeiVGz
DeiVGg
Method
DeiVGa
DeiVGg
Rule Generation
1.4 0.63
1.76 -: 0.67
GroundedSAM
0.01 1 0.00
0.01 1 0.00
Semantic Unification
6.38 1 1.1
7.03 1.17
LISA
6.38 + 1.1
7.03 + 1.17
Forward Reasoning
0.18 1 0.67
0.18 1 0.31
DeiSAM (ours)
9.98 4.61
10.85 = 3.19
Segmentation
1.22 x 10-6
9.82 x 10-7
DeiVG in the ReCOCO's reference format. We conducted additional experiments by modifying the deictic prompts in DeiVG datasets to the ones with targets' labels, resulting in prompts similar to reference texts in RefCOCOs (Yu et al. 2016). Table 11 shows the mAP on DeiVG datasets with the modified prompts using LISA, GroundedS AM, and AM. + represents the gain compared to the original performance with deictic prompts (shown in Tab. Neural baselines gained the
22

---

performance remarkably by the modified prompts since they contain labels for targets (e.g. r..g.person, kid), on which neural models highly rely to identify targets.
Table 11: Comparison with modified prompts, e.g. "Person on the boat" instead of "An object on the boat". represents the gain compared to the original performance with deictic prompts.
Mean Average Precision (%) 1
Method LISA
DeiVGn
DeiVG2
DeiVGa
26.81 (+11.91)
68.78 (+12.75)
82.20 (+6.4T)
AM
21.87 (+11.39)
47.91 (+15.58)
89.48 (4-43.44)
DeiSAM (ours)
64.78 (-0.36)
84.23 (-1.17)
88.19 (+0.36)
G Additional Segmentation Results
We provide supplementary results of the segmentation on the DeiVG, dataset in Fig. 12
object that street and has smoke.
object the chair and holds a plant.
An object above the table and has black
object that is carrying the person and has tail.
object that side walk ond wears
metal.
DeiSAM (Ours)
Grounded SAM
abject behind girls and has fires.
An object that the metal pole has
An object near microoaae and on fridge.
object that the An object that is the bike and has a purse.
grass has door.
DEISAM (Ours)
Grounded SAM
Figure 12: Segmentation results the DeiVG, dataset using DeiSAM and GroundedSAM with deictic prompts (top).
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our main claim is that the proposed approach can solve deictic prompts to segment objects. In our experiments, we have extensively shown that the proposed model outperforms baselines on deictic segmentation tasks, empirically supporting the main claim made in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the last section the paper, we discuss multiple limitations of the proposed approach. Moreover, in the appendix, we provide empirical results of the runtime of our framework compared to baselines and discuss the computational efficiency.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not include any theoretical results in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe details of the model, including hyperparameters and system prompts. in the paper and in the appendix. Moreover, we provide the link to the anonymous repository that contains all the codes needed run the experiments demonstrated in the paper and it contains the link to the dataset proposed in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
25

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide the link to the anonymous repository that contains all of the codes needed run the experiments demonstrated in the paper and it contains the link to the dataset proposed in the paper. We will make the repository public upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ mublic/guidess https///oooeeubbsssioooooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. http/uullcggggucce Caleee//ooSSuubisse asiionooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the experimental setting in the paper and also more details in the appendix. We present all the necessary to run the experiments together with the code on the anonymous repository.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in the appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We report the error bars for the runtime analysis in App. For main ex- periments in Sec. [ we do not report error bars because most of our experiments are deterministic, iee,, the proposed reasoning pipeline does not involve stochastic computations. The internal usage of LLMs could produce errors by hallucinations, but minimize such
26

---

errors by performing error handling, i.e., we repeat the function call when they produce output that cannot be parsed by the system. Thus, we report numbers from single run for each experiment. The only exception is the experiment of weight learning. We report single run because of the non-trivial financial cost of the L.M usage, i.e. the API needs to be called iteratively in the learning process, and the expected budget significantly increases. To mitigate this, we provide the random seed to reproduce the result for all experiments in our codebase to ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the details the machine where we performed experiments. includes details of the hardware, e.g. CPU, GPU, and RAM. Moreover, we report the running time of the proposed model and baselines for our main experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Guidelines:
27

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This paper presents an abstract methodology that has no effects on the societal impact specifically, such as malicious uses or fairness considerations. These aspects of societal impacts will be attributed to the dataset itself to be used, not our proposed framework.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
This paper presents an abstract methodology that has no effects on the safe- guards specifically, such as the high risk of misuse. The paper does not contain any release of pretrained models or datasets newly curated using web scraping.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
28

---

12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In the paper, we cite and refer to all the papers that our framework uses. Moreover, we provide the link to codebases of the relevant papers in our repository. We also declare the license in the repository.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We include instructions for code and dataset in the repository. It is accessible via an anonymized URL that we provide.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
29

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "PznPiILPuI", "reviewer": "Reviewer_WXVv", "summary": "The manuscript proposes DeiSAM, a framework that integrates large pre-trained neural networks with differentiable logic reasoners to address deictic promptable segmentation. \nDeiSAM leverages large language models (LLMs) to generate first-order logic rules and performs forward reasoning on generated scene graphs for object segmentation based on complex textual prompts. \nThe Deictic Visual Genome (DeiVG) dataset is introduced for evaluation.", "questions": "Detailed questions are referred to as Weaknesses.", "limitations": "The authors have discussed the limitations of the work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-structured and easy to read.\n2. The integration of LLMs with differentiable logic reasoners for deictic prompt-based segmentation is a novel approach, addressing limitations in neural network-based segmentation models.", "weaknesses": "1. Over-simplified Assumptions: The curated datasets DeiVG1, DeiVG2, and DeiVG3 contain prompts using only 1-3 relations and are restricted to avoid multiple objects with ambiguity. However, in real-world applications, object and relation categories can be varied and rich, with inevitable visual-language alignment gaps. For example, in an image with a black cat and a white cat, how would the method handle a prompt to segment the white cat using relative descriptions? Moreover, the dataset size is small (10k), making large models prone to overfitting.\n   \n2. Dependence on Scene Graph Quality: The method heavily relies on the quality of scene graphs. Inaccurate scene graphs can lead to incorrect segmentation results, which is a critical limitation for practical applications. For instance, the large performance gap indicated in Table 5 highlights this dependency.\n\n3. Unclear Philosophy: The use of a semantic unifier (an LLM) to associate objects in the prompt and scene graph seems trivial. The rationale for using a differentiable forward reasoner instead of continuing to use an LLM is not well-justified. The necessity and advantage of having a \"differentiable\" reasoner are unclear. How does this reasoner fundamentally differ from previous neural-based methods like Grounding DINO or LISA?\n\n4. Insufficient Experiments: The experiments are limited to relatively easy cases for prompt-based segmentation on a small-scale image dataset. The paper lacks evaluations on more challenging scenarios, such as multiple objects of the same category or prompts with more complicated relations. Comparisons with previous methods under these conditions would provide a more comprehensive evaluation of DeiSAM’s effectiveness.\n\n5. Unclear Contributions: Modules of DeiSAM are borrowed from other existing works, and there is no novel architecture designed for the prompt-based segmentation. It appears more like a technique report for an engineering project instead of being a research paper suitable for a top-tier conference like NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comment and for acknowledging that the paper addresses limitations in neural segmentation models and is well-structured and easy to read.\nWe address the concerns next.\n\n> Over-simplified Assumptions: How would the method handle a prompt to segment the white cat using relative descriptions?\n\nOur method can handle such cases. Let us describe how to do that.\nGiven prompt: “Segment a white cat”, an LLM can generate rules in our format:\n```\ncond1(X):-type(X,Y),type(Y,white_cat).\ntarget(X):-cond1(X).\n```\nThen, the semantic unifier will consider the embedding of the term `white_cat` and match it to an entity in the scene graph representing a white cat. The embedding-based semantic unification can address the visual-language alignment. If the scene graph misses entities of a white cat and a black cat in the first place, it would be hard for DeiSAM to perform reasoning correctly. A crucial unresolved issue is how to generate scene graphs in a zero-shot manner, akin to large visual-language models.\n\n\n> Moreover, the dataset size is small (10k), making large models prone to overfitting.\n\nThe DeiVG dataset is proposed to evaluate segmentation models, and thus we believe that the dataset with 10k examples is not extremely small for this usage.\n\n>  Inaccurate scene graphs can lead to incorrect segmentation results, which is a critical limitation for practical applications. \n\nWe argue that segmentation models must understand the scenes to be faithful reasoners. Without this assumption, models can learn to *shortcut*, i.e., learn to pretend to answer via an incorrect scene understanding and reasoning [1]. This does not essentially solve the complex reasoning prompts that we aim to address in the paper.\n\n[1] Not All Neuro-Symbolic Concepts Are Created Equal: Analysis and Mitigation of Reasoning Shortcuts, NeurIPS 2023\n\n\n> the large performance gap indicated in Table 5 highlights this dependency.\n\nWe agree with this. The quality of scene graphs significantly impacts the segmentation quality, but DeiSAM demonstrates resilience to noise in scene graphs. If the quality of scene graphs is extremely low, the resulting segmentation quality will also be low, as the reasoning process relies on the scene graph representation. However, DeiSAM manages noise in scene graphs through embedding-based vocabulary matching and by learning a mixture of scene graphs, as demonstrated in Table 5.\n\n\n> Unclear Philosophy:  The rationale for using a differentiable forward reasoner instead of continuing to use an LLM is not well-justified.\n\nThe primary advantage is that segmentation models transform into reliable reasoners. LLMs often hallucinate on complex abstract prompts and scenes. To demonstrate this, we conducted additional experiments on 3D visual scenes with more complex prompts. For more details, refer to the general remark.\n\n> The necessity and advantage of having a \"differentiable\" reasoner are unclear.  How does this reasoner fundamentally differ from previous neural-based methods like Grounding DINO or LISA?\n\nThe advantage of the resulting model lies in its ability to learn through gradients, thereby enhancing its performance. In contrast, using an off-the-shelf discrete logic system (e.g., Prolog) for a segmentation model would render the entire system static by blocking the gradient flow. This static nature would severely limit its applicability, as it would necessitate expert intervention to rewrite rules whenever they are imperfect. To reconcile explicit logical reasoning with gradient-based learning, we employ a differentiable logic reasoner. As illustrated in Table 5, DeiSAM obviates the need for expert efforts to improve segmentation performance.\n\n\n> The paper lacks evaluations on more challenging scenarios, such as multiple objects of the same category or prompts with more complicated relations. \n\nRefer to the general remark.\n\n> There is no novel architecture designed for the prompt-based segmentation. It appears more like a technique report for an engineering project instead of being a research paper suitable for a top-tier conference like NeurIPS.\n\nWe disagree with the premise. DeiSAM addresses segmentation tasks involving abstract, complex prompts by integrating logic reasoners. Experimental results substantiate its effectiveness, demonstrating that it outperforms strong neural baselines.\n\nWe believe that utilizing existing modules and a well-defined computational architecture should not be viewed as a disadvantage. We contend that the criteria for acceptance at a conference should be based on the overall contribution of the paper rather than solely on the introduction of a \"novel architecture.\"\n\n\nThank you for your insightful suggestions. We hope our response effectively addresses your concerns, and we are happy to answer any further questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBZacfr2Wr", "reviewer": "Reviewer_YJtK", "summary": "The paper proposes to study Deictic references/prompts, i.e., phrases/references that describe the role/purpose/context rather than naming the object directly. Firstly, the paper constructs a new dataset with deictic prompts based on Visual Genome; Next, it is shown that existing methods do poorly on this task. Next, a neuro-symbolic method that works on top of a generated scene graph is proposed which is shown to work a lot better on this newly constructed dataset as well as perform comparably on related tasks such as referring expression recognition.\n\n[EDIT: Post rebuttal comments are useful but do not change overall score]", "questions": "Q1. If scene graphs are assumed to be available, should they not be available in some form to every algorithm. E.g., if the object name is easily decodable from deictic expression by looking it up in the scene graph, can we not change the expression to include the name of the object as well? E.g., \"A red furry object\" --> Select all objects that are red and furry from scene graph --> Grab the names of those objects -> use the name instead of deictic expression when prompting SEEM or LISA.\n\nQ2. What is the effect of the quality of scene graphs on the final results of the proposed method. Is it resitant to mistakes in the SGG prediction?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**[S1] Interesting Findings:** While the setup is somewhat artificial/contrived (see W1), the results on existing methods are surprisingly poor for the proposed task. This is an interesting finding and might point to a bigger problem in the way \"referring segmentation\" is done currently. While there are existing datasets with referring expressions, I think the paper studies a more challenging setup which is harder to game so it should provide a clearer picture of true understanding capability of the models.\n\n**[S2] Good clarity:** The presentation is clear and detailed. The experimental designs are meaningful and the results are clearly presented.", "weaknesses": "**[W1] Somewhat contrived setup:** I fully agree with the big premise that deictic expressions are important and common in everyday usage. However, many examples from the constructed dataset are unnatural and forced. The paper also takes deictic phrases to an extreme by removing all nouns from the deictic expressions with very generic prompts such as \"stuff\" or \"object\". It is more common for references to consist of categorial nouns instead of object names (E.g., that toy on top of the chair) instead of teddy bear, etc. So, while a useful tool in diagnostic abilities, I feel that the dataset is not broadly applicable. Similarly, since it is such an unnatural setup; It might also make it unrealistically hard for general purpose baseline methods.\n\n**[W2] Reliance on Scene graphs:** The proposed work relies on access to scene graph prediction model outputs. This is not something that is available to baselines and, in my opinion, does a lot of heavy lifting. I believe this makes it hard to do a fair comparison with the baselines. One could even argue that if the scene graph is comprehensive and accurate the **main** task in decoding deictic expressions (Thought I admit that it is still non trivial). Also see Q1.\n\n[W3] Minor issues on the dataset: The dataset is divided based on how many \"hops\" of reasoning are necessary to decode the expression. E.g., \"A tall object wearing a hat walking in sidewalk\" is a 3 hop expression (tall, wearing a hat, walking on the sidewalk). However, unless there are counterexamples to each. (E.g., other tall things, other stuff wearing the hat, ...) it degenerates into a subset of that. This means a shortcut \"wearing a hat\" alone might be enough to decode the object. I would like to see an analysis of this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed and fruitful feedback and for acknowledging that our findings are insightful, the experimental setups are meaningful, and the paper is well-written.\n\nWe address the concerns next.\n\n> W1: many examples from the constructed dataset are unnatural by removing all nouns from the deictic expressions with \"stuff\" or \"object\".\n>  It is more common for references to consist of categorial nouns instead of object names (E.g., that toy on top of the chair) instead of teddy bear\n\nWe agree with this. Using categorical names (e.g. toy, food, deice) would be very promising in covering more real-world applications of referring expressions. This could be achieved by having an LLM while dataset generation to convert each specific name to a categorical name.\n\nHowever, we do not conduct experiments on this as our primary focus is abstract reasoning in which object and categorical names are absent in the prompt, i.e. the models need to deduce the object name from a given relational specification. \n\nTo mitigate, we reported on Table 10 in the Appendix of performances by making DeiVG datasets with a more referring-expression style, e.g. we provide \"Person on the boat\" instead of “An object on the boat”. The results indicate that neural baselines gain a lot by having object names. We expect that if we provide the categorical names in the prompt, neural baselines will gain some performance but not as much as with object names.\n\n\n> W2:   Reliance on Scene graphs: The proposed work relies on access to scene graph prediction model outputs.\n\nWe answer this later for Q1.\n\n> W3: Shortcut of in the DeiVG dataset. E.g.  \"wearing a hat\" alone might be enough to decode the object given prompt: tall, wearing a hat, walking on the sidewalk.\n\nWe agree with the reviewer that this is often the case in our dataset. We have analyzed how often it happens in the generated DeiVG dataset. \n\nFor example, with a prompt “An object wearing a hat walking on the sidewalk\", we consider the following variations of ways to match objects: \n\n- Relation: We consider only relations to detect matching between objects in the scene, i.e., \"wearing\" or \"walking on,\" disregarding the object names.\n- Object: We consider only objects to detect matching between objects in the scene, i,e.,  \"hat\" or \"sidewalk,\" disregarding the relations.\n- Either: We consider either a partial match from Relation or Object.\n- Complete: We consider complete counterexamples, i.e., \"wearing hat\" or \"walking on the sidewalk\", which is exactly articulated by the reviewer.\n\nThe table below shows the proportion of prompts involved with such shortcuts. \n\n| Proportion (%) | Relation | Object | Either | Complete |\n|----------------|------------------|----------------|----------------|-------------------------|\n| DeiVG$_3$        | 23.01            | 46.00          | 17.35          | 60.56                   |\n| DeiVG$_2$        | 23.39            | 48.22          | 18.13          | 58.65                   |\n\nThe result indicates that the shortcut often happens in the dataset. In DeiVG$_3$, 60\\% of the prompts are involved with such a shortcut, i.e., no other objects in the scene share the same 3 relations. We included this result to the appendix of our manuscript. Thank you for the suggestion to improve the paper.\n\n\nMoreover, to explore more cases with similar objects without shortcuts, we conducted an additional experiment on an abstract 3D environment with more complex prompts. For more details, refer to the general remark.\n\n> Q1: Can we use scene graphs for other baselines? E.g., \"A red furry object\" --> Select all objects that are red and furry from scene graph --> Grab the names of those objects -> use the name instead of deictic expression when prompting SEEM or LISA.\n\nThis approach would not be fair. While it is an interesting suggestion to use scene graphs for neural baselines, the proposed method involves providing an external program for preprocessing the scene graphs so that neural models can process them. This would result in a composite model consisting of a scene graph preprocessor and a separate large segmentation model. Such an approach severely limits further extensions aimed at enhancing scene understanding and reasoning segmentation because it is not straightforward to propagate segmentation errors back to improve scene graph generation.\n\nIn contrast, DeiSAM treats scene graphs as an internal primal representation. It performs segmentation through differentiable reasoning, allowing it to backpropagate errors to the scene graphs.  This capability is crucial for enabling the model to learn to segment more accurately from data and to generate explanations using gradients, indicating which parts of the scene were essential for the segmentation.\n\nAn alternative comparison could be to encode the scene graphs as text and provide them to the neural baselines.\n\n\n> Q2: What is the effect of the quality of scene graphs on the final results of the proposed method. Is it resitant to mistakes in the SGG prediction?\n\nIt significantly impacts the segmentation quality, but DeiSAM demonstrates resilience to noise in SGGs. If the quality of scene graphs is extremely low, the resulting segmentation quality will also be low, as the reasoning process relies on the scene graph representation. However, DeiSAM manages noise in scene graphs through embedding-based vocabulary matching and by learning a mixture of scene graphs.\n\n\nThank you for your valuable suggestions. We hope our response satisfactorily addresses your concerns, and we are happy to answer any additional questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vrtYCUpgWc", "reviewer": "Reviewer_TxaQ", "summary": "This paper proposes DeiSAM to enhance the ability of current state-of-the-art referring segmentation frameworks with deictic prompting. DeiSAM is an innovative approach that combines large pre-trained neural networks with differentiable logic reasoners to perform image segmentation based on complex, deictic textual prompts. The method leverages Large Language Models (LLMs) to generate first-order logic rules from textual descriptions and uses differentiable reasoning on scene graphs to identify and segment objects in images accordingly. The paper also introduces the Deictic Visual Genome (DeiVG) dataset for evaluating deictic segmentation and demonstrates DeiSAM's superiority over neural baselines through empirical results.", "questions": "All are listed in weaknesses. Please see above.", "limitations": "The authors have adequately addressed the limitations in the submitted manuscript.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In general, this paper identifies an important problem in the intersection between referring image segmentation and scene graph generation, and proposes a simple yet effective solution to resolve it. The contribution is comprehensive, from my perspective, including the following aspects:\n\n1. **Novelty and Innovation**: The paper proposes a novel framework that integrates Large Language Models (LLM) with visual logic reasoning for deictic image segmentation, addressing a gap in the field where current methods struggle with complex prompts. The symbolic rules designed in the pipeline look relevant to the visual programming concept, which is novel and technically elegant.\n\n2. **Empirical Validation and Efficacy**: The authors introduce a new dataset, DeiVG, to test the capabilities of DeiSAM, and provide extensive experimental results that validate the effectiveness of their approach, showcasing clear improvements over existing baselines. Concretely, for DeiVG1, DeiVG2 and DeiVG3 respectively, the proposed method outperforms the previous state-of-the-art LISA by 50.24%, 29.37% and 12.04% in terms of Mean Average Precision.\n\n3. **Simplicity and End-to-end Differentiable**: DeiSAM is both simple and effective, which makes it easy to follow. DeiSAM's modular architecture and the use of differentiable reasoning allow for end-to-end training, which is a significant advantage for adapting to complex downstream tasks and improving segmentation quality.", "weaknesses": "1.\nThe clarity of the paper could be further improved. Despite careful reading and familiarity with the field, several aspects remain confusing:\n\n- **Training Process**\n  - **Unclear Tuning**: What components are tuned during the training stage? \n  - **Revision Recommendation**: Annotate which modules are tuned during training.\n\n- **Scene Graph Generator**\n  - **Pre-training Limitations**: The Scene Graph Generator appears pre-trained on Visual Genome, which annotates only a limited set of objects (e.g., persons, boats, cars).\n  - **Generalizability Concerns**: \n    - Will the proposed method work on images containing objects not annotated in Visual Genome? \n    - If not, the title \"Segment Anything with Deictic Prompting\" may be an overstatement.\n\n- **Semantic Unifier and Forward Reasoning**\n  - **Ambiguous Illustration**: The explanation of these components lacks clarity.\n    - What are the inputs and outputs of the Semantic Unifier?\n    - What constitutes the \"background knowledge\" $\\mathcal{B}$ in forward reasoning?\n    - What is learned in the forward reasoning process?\n\n2.\nThe proposed method should also be evaluated on other referring image segmentation benchmarks, such as RefCOCO, RefCOCOg, and RefCOCO+. This evaluation would help determine whether the proposed method is effective in general referring image segmentation settings, or if the integration of scene graph information potentially hinders performance in standard referring segmentation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the fruitful feedback and for acknowledging that the proposed framework is novel and effective and the empirical results are valid.\n\nWe would like to address the concerns raised by the reviewer.\n\n> The proposed method should also be evaluated on other referring image segmentation benchmarks, such as RefCOCO, RefCOCOg, and RefCOCO+\n\nWe already reported this in the paper. Table 3 shows the result on the RefCOCO+ benchmark. We observed that DeiSAM can handle the referring expression task, which is the common benchmark suggested by the reviewer. \n\nMoreover, in Table 4, we demonstrated that neural baselines (LISA and GroundedSAM) significantly degraded their performance if we modified the textual input to the abstract form, e.g., “kid wearing navy shirt” is modified to “an object that is wearing navy shirt.” This indicates the lack of the ability to use abstract high-level reasoning in neural baselines. \n\n\n> What components are tuned during the training stage? Annotate the tuned modules.\n\nThe rule weights in the differentiable forward reasoner. On line 321, we explicitly stated: “We minimize the binary cross entropy loss with respect to rule weights $w_1$ and $w_2$.”\nTo enhance clarity, we added annotations to the figure as the reviewer proposed. Thank you for the suggestion.\n\n\n> Pre-training Limitations: The Scene Graph Generator appears pre-trained on Visual Genome, which annotates only a limited set of objects (e.g., persons, boats, cars).\n\n\nWe agree with the reviewer that good scene graph representations are the key to the proposed DeiSAM framework. Our finding is that if good scene graphs (generators) are available, the segmentation quality on abstract prompts can be much improved by combining logical reasoning. \n\nAs scene graphs can be noisy and low-quality, DeiSAM performs embedding-based vocabulary matching and learning scene graphs in combination. It is a vital open question to handle scene understanding for unseen data in a zero-shot manner.\n\n\n\n> Will the proposed method work on images containing objects not annotated in Visual Genome?\n\nYes, it works. We conducted additional experiments using the CLEVR environment [1] that contains abstract 3D visual scenes, which is largely different from visual genome. We show that DeiSAM can segment abstract 3D objects given more complex textual prompts. \nMoreover, our result on the RefCOCO+ dataset also validates that it is not limited to only visual genome images.\n\n[1] CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR 2017\n\n> If not, the title \"Segment Anything with Deictic Prompting\" may be an overstatement.\n\nAs the premise is false (we clarified above), we believe this is not an overstatement.\n\n> What are the inputs and outputs of the Semantic Unifier?\n\nThe input is a scene graph and a set of rules generated by LLMs from textual prompts. If a term in the rules is absent in the scene graph, the semantic unifier investigates the most likely matching between these two representations using term embedding. \nWe added these technical details to the final version of our manuscript Thank you for pointing this out.\n\n\n> What constitutes the \"background knowledge\"  in forward reasoning?\n\nIt is a set of rules and facts described in first-order logic. It is common for logical reasoners to accept background knowledge from users. For DeiSAM, we provided no background knowledge throughout the experiments.\n\n> What is learned in the forward reasoning process?\n\nIn our experiments, rule weights are learned for composing a mixture of scene graphs to maximize segmentation performance.\n\nTo clarify, our forward reasoner implements forward-chaining reasoning in first-order logic, a method used to deduce all facts given known rules and known premises. This reasoning process is inherently deterministic. The differentiable version of the forward reasoner introduces weighted rules, which can be optimized through backpropagation. Our approach seamlessly integrates forward reasoning with neural modules, thereby enhancing the overall learning performance.\n\n\n\nThank you once again for your valuable suggestions. We hope our response addresses your concerns, and we are happy to answer any further questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MEVQfwtqNA", "reviewer": "Reviewer_9P1c", "summary": "This paper introduces a new reasoning-based segmentation by adding deictic logic as prompts into the SAM. Motivated by the deictic logical analysis, the authors present DeiSAM, a combination of large pre-trained neural networks with differentiable logic reasoners for deictic promptable segmentation. DeiSAM first use LLMs to generate first-order logic rules and perform differentiable forward reasoning on generated scene graphs. In addition, authors present Deictic Visual Genome dataset. Experimental results show DeiSAM works better than pure data-driven baselines.", "questions": "See the weakness part.", "limitations": "Yes, they have checklist.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1, Overall writing is good and easy to follow. Motivation is easy to understand.\n\n2, The performance on proposed DeiVG benchmark looks good.\n\n3, The idea of using first order deictic as prompting for segmentation is interesting.", "weaknesses": "Despite the idea of combining first order deictic prompt into SAM is interesting, there are still lot of limitations in this work.\n\n1, Motivation is not new. Performing reasoning segmentation via LLM is not new. There several previous works exploring the same settings [1]-[5]. The difference lies on the author adopt more visual cues (scene graph) into LLM before sending the language features to SAM.\n\n[1], LISA: Reasoning Segmentation via Large Language Model, CVPR-2024.\n\n[2], An Improved Baseline for Reasoning Segmentation with Large Language Model, arxiv-2023.\n\n[3], GRES: Generalized Referring Expression Segmentation, CVPR-2023.\n\n[4], Towards Robust Referring Image Segmentation, TIP-2023.\n\n[5], See, Say, and Segment: Teaching LMMs to Overcome False Premises, CVPR-2024.\n\n2, The technical novelties are limited. The proposed method just combines previous differentiable logic reasoners with LISA-like segmenter (LLM + SAM) architecture. I find no extra insights or advantages for end-to-end reasoning learning. Moreover, the method needs extra scene graphs as inputs, which introduce more complexities to the pipeline.\n\n3, Related works discussion. Missing lots of works on referring segmentation [3]-[5].\n\n4, Performance and benchmarks. There are only refcoco+ and proposed benchmark results. Several important benchmarks are missing (Please refer to [1]). Moreover, the performance is not strong. This result is misleading since the authors miss lots of previous works for fair comparison.\n\n5, Missing detailed ablation studies for each component. It is hard to know the real benefits on the combined logical reasoner and effects of input scene graph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the fruitful feedback and for acknowledging that the paper is well-written and that combining first-order logic with prompting is interesting.\n\nWe would like to address concerns next.\n\n> Motivation is not new. Performing reasoning segmentation via LLM is not new. There several previous works exploring the same settings [1]-[5].\n\nWe disagree with this. Our claim is to use something other than LLMs for reasoning, as they often hallucinate on complex prompts. Instead, we propose using logic for reasoning. We integrate LLMs and SAM with differentiable reasoners for neuro-symbolic inference on various deictic prompts. To the best of our knowledge, this approach has not been attempted in previous studies.\n\nMost importantly, we demonstrated that LISA [1] significantly degrades its performance on abstract prompts in Tables 3 and 4, indicating a divergence from our problem setting. Our focus does not encompass referring expressions or commonsense reasoning, which differ from high-level abstract reasoning.\n\n\n\n \n> The difference lies on the author adopt more visual cues (scene graph) into LLM before sending the language features to SAM.\n\nNot quite. There is a factual error: we do not feed scene graphs to LLMs, nor do we utilize LLMs directly for reasoning. In DeiSAM, LLMs generate logic programs to reason with scene graphs on the abstract level.\n\n\n> The technical novelties are limited.  The proposed method just combines previous differentiable logic reasoners with LISA-like segmenter (LLM + SAM) architecture.\n\nAs Reviewer YJtK pointed out, our major findings are as follows:\n- Existing strong baselines using transformers for reasoning perform poorly on the segmentation tasks with abstract complex prompts.\n- This indicates that the current transformer-baed models are limited in high-level abstract visual reasoning.\n- DeiSAM is the first framework to successfully extend segmentation models using differentiable reasoning with scene graphs to gain reasoning ability over abstract representations.\n\n\n\n > I find no extra insights or advantages for end-to-end reasoning learning.\n\nWe address the trade-off between logic-based reasoning and model adaptability: while logic provides faithful reasoning capabilities, it forces the model to be deterministic and unlearnable. For instance, employing off-the-shelf logic libraries such as Prolog ensures reasoning but impedes gradient-based learning due to their non-differentiable inference. The key insight of our approach is that it enables the segmentation model to function as both a faithful reasoner and an adaptive learner, as demonstrated in Table 5.\n\n\n>  Moreover, the method needs extra scene graphs as inputs, which introduce more complexities to the pipeline.\n\nNot quite. There is indeed a trade-off between the complexity of the input and the reasoning models employed. Scene-graph representations make the reasoning module highly parameter-efficient. For instance, in DeiSAM, the forward reasoner with $N$ rules requires only $N$ parameters, with one weight per rule. \nIn contrast, purely neural models typically require a large number of parameters in the reasoning model (e.g., experiments in [5] employ LLaVA-v1.5-7B as the LLM backbone) to perform reasoning on pixels.\n\n\n>  4, Performance and benchmarks. There are only refcoco+ and proposed benchmark results. Several important benchmarks are missing (Please refer to [1]). \n\nIn our manuscript, on lines 90-95, we explicitly discuss the reasons for not conducting experiments on the benchmark [1]. The suggested benchmark [1] primarily focuses on low-level commonsense reasoning, whereas our objective is to address high-level abstract reasoning segmentation. This rationale also extends to other referring-expression segmentation benchmarks.\n\n\n> Moreover, the performance is not strong. This result is misleading since the authors miss lots of previous works for fair comparison.\n\nWe respectfully disagree with the suggestion that additional baselines are necessary to validate our experiments. We consistently used the LISA [1] model, a state-of-the-art segmentation method, as the baseline for our experiments. None of the works suggested by the reviewer specifically address high-level abstract reasoning. Therefore, we believe that incorporating these additional baselines is not essential to validate our experimental results.\n\n> Missing detailed ablation studies for each component. It is hard to know the real benefits on the combined logical reasoner and effects of input scene graph.\n\n\nWe refer to the general remark. To answer this, we conducted additional experiments on abstract visual scenes with more complex prompts and demonstrated the gain by having the logical reasoner with scene graphs.\n\n> Missing lots of works on referring segmentation [3]-[5].\n\nWe agree with the reviewer that papers [3]-[5] are relevant to our study. Here, we discuss these works and clarify our paper's contribution.\n\nGRES [3] proposes two distinct attention mechanism to address addresses multi-target segmentation. R-RIS [4] proposes  RefSegformer, a transformer-based model that includes a language encoder, a vision encoder, and an encoder-fusion meta-architecture  for handling incorrectly described textual prompts. See, Say, and Segment [5] propose SESAME, an LMM designed to \"see\" whether objects are present, \"say\" to interact with users, and \"segment\" target objects.\n\nAll these methods rely on transformers (or attentions) as their core reasoning pipeline, and they would inherit the reasoning limitations inherent to purely neural models. In contrast, DeiSAM explicitly encodes logical reasoning processes to guarantee accurate and faithful interpretation of abstract and complex prompts.  We will include these discussions in the final version.\n\n\nThank you once again for your valuable feedback. We hope our response adequately addresses your concerns, and we would be pleased to answer any further questions you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback and insights on the paper. Here, we would like to address concerns shared by several reviewers.

Reviewer WXVv and Reviewer 9P1c:
> What is the benefit of integrating (differentiable) logical reasoners into segmentation models? 

The primary advantage is that segmentation models transform into reliable reasoners. It is well-documented that large language models (LLMs) often underperform on complex reasoning tasks [1,2]. Consequently, depending on LLMs within a segmentation model introduces an inherent bottleneck for abstract reasoning capabilities. This paper aims to propose a segmentation model capable of addressing abstract reasoning. Crucially, this approach is distinct from solving referential tasks, where abstract representations and complex reasoning do not play a central role.

[1] Large language models can be easily distracted by irrelevant context. ICML 2023

[2] Large language models cannot self-correct reasoning yet. ICLR 2024


This argument responds to a comment from Reviewer WXVv:
>  The paper lacks evaluations on more challenging scenarios, such as multiple objects of the same category or prompts with more complicated relations. 

Reviewer 9P1c:

>  It is hard to know the real benefits on the combined logical reasoner and scene graphs.

To answer this,  we conducted additional experiments using the CLEVR environment, wherein multiple abstract objects (e.g., a large cube and a small sphere) are presented in 3D scenes. Our objective is to demonstrate that DeiSAM can handle such abstract objects with complex prompts while neural baselines frequently struggle to reason with them.


**Task.** The  task is to segment objects given prompts where the answers are derived by the reasoning over abstract list operations. We consider 2 operations: delete and sort. The input is a pair of an image and a prompt, e.g. “Segment the second left-most object after deleting a gray object?”.  Examples are shown in Fig. 1 in the attached PDF.  To solve this task, models need to understand the visual scenes and perform high-level abstract reasoning to segment.

**Dataset.**  **(Image)** We generated visual scenes using the CLEVR environment [1]. Each visual scene contains at most 3 objects with different attributes: (i) colors of cyan, gray, red, and yellow, (ii) shapes of sphere, cube, and cylinder, (iii) materials of metal and matte. We excluded color duplications in a single image. **(Prompts)** We generated prompts using a templates: “The [Position] object after [Operation]?”, where [Position] can take either of: left-most first, second, or third. [Operation] can take either of: (I) delete an object, and (II) sort the objects in the order of: cyan < gray < red < yellow (alphabetical order). We generated 10k examples for each operation.

**Models.** We employed DeiSAM with pre-trained Slot Attention [2] to perceive objects. DeiSAM initially learns abstract list operations from the visual Inductive Logic Programming (ILP) dataset [3], which includes positive and negative 3D visual scenes for each list operation, thereby deriving rules to perform these operations. These rules are then applied to segment objects, augmented by those generated from textual prompts. We used GroundedSAM and LISA for neural baselines. Hyperparameters are described in Section E in the Appendix. 


**Results.** 
The table below shows mAP for each baseline. The result indicates that segmentation models relying on neural reasoning pipelines fail to deduce segmentations for abstract reasoning prompts, while DeiSAM effectively identifies and segments the object specified by the prompt.

| mAP ( $\uparrow$) | Delete | Sort |
|---------------|---------|--------|
| DeiSAM | 99.29 | 99.57 | 
| GroundedSAM | 7.6 | 15.39 | 
| LISA | 12.88 | 11.15 | 

Moreover, Fig. 1 in the attached PDF shows qualitative results. DeiSAM successfully segments objects with high-level reasoning, but neural baselines often failed to identify the correct target object. 
We will add these results to the final version of the paper.

Most importantly, our primary goal is not to outperform neural baselines across all types of reasoning tasks. Instead, our aim is to demonstrate that neural baselines are insufficient for solving abstract reasoning prompts and to enhance their capabilities by integrating differentiable logic reasoners.

[1] CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning. CVPR 2017

[2] Object-Centric Learning with Slot Attention. NeurIPS 2020

[3] Learning Differentiable Logic Programs for Abstract Visual Reasoning. Mach. Learn., 2024

Meta-review: Summary of Review Process:
- 4 reviews.
- Mixed scores 3, 4, 6, 6
- Authors submitted rebuttals/responses
- 3 out of 4 reviewers did not engage in discussion after repeated requests from AC. Only Reviewer YJtK (score 6) engaged in discussion.
- The AC has decided that the lack of participation from 3 reviewers should not negatively impact the final recommendation. Therefore the AC will review the questions and responses instead of directly considering the *scores* of the unresponsive reviewers.

Meta Review:
There was consensus from all reviewers regarding the novelty of the approach, clarity of writing, and experimental findings of the paper.  The following is a summary of questions and rebuttal:
- Reviewer 9P1c commented that the motivation was not new and the method used a combination of previous techniques. The AC believes that the response from authors sufficiently addresses this concern.
- Reviewer 9P1c suggested previous articles to be mentioned in related work. The authors have agreed to do so.
- Reviewer 9P1c asked for additional ablation studies. The authors have added results in the general remark. The AC recommends the authors to add these to the main paper.
- Reviewer TxaQ asked the authors to elaborate on some confusing aspects related to the technical approach, illustrations, and some limitations. The authors have responded sufficiently with elaboration/clarification and additional experiments.
- Reviewer WXVv asked the authors to comment on the assumptions and dependence on scene graph, provide additional experiments, and elaborate on the rationale and contributions. The authors have responded to each question. The AC recommends the authors to integrate these responses in the main paper.
- Reviewer YJtK participated in the discussion and gave a rating of Weak Accept. The reviewer states that some of the points still remain unaddressed.

Overall, the AC's evaluation is that the paper is an interesting work on neuro-symbolic approaches using LLMs to generate logical rules and perform differential reasoning on scene graphs.
Decision: Accept (poster)