DeiSAM: Segment Anything with Deictic Prompting

Authors: hikaru  shindo
manuel  brack
gopika  sudhakaran
singh  devendra
  dhami
patrick  schramowski
kristian  kersting
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 13:24:00

Abstract: Large-scale, pre-trained neural networks have demonstrated strong capabilities in various tasks, including zero-shot image segmentation. To identify concrete objects in complex scenes, humans instinctively rely on deictic descriptions in natural language, i.e., referring to something depending on the context such as "The object that is on the desk and behind the cup.". However, deep learning approaches cannot reliably interpret such deictic representations due to their lack of reasoning capabilities in complex scenarios. To remedy this issue, we propose DeiSAM — a combination of large pre-trained neural networks with differentiable logic reasoners — for deictic promptable segmentation. Given a complex, textual segmentation description, DeiSAM leverages Large Language Models (LLMs) to generate first-order logic rules and performs differentiable forward reasoning on generated scene graphs. Subsequently, DeiSAM segments objects by matching them to the logically inferred image regions. As part of our evaluation, we propose the Deictic Visual Genome (DeiVG) dataset, containing paired visual input and complex, deictic textual prompts. Our empirical results demonstrate that DeiSAM is a substantial improvement over purely data-driven baselines for deictic promptable segmentation.

Main paper: 
 introduction: 
 Recently, large-scale neural networks have substantially advanced various tasks at the intersection of vision and language. One such challenge is grounded image segmentation, wherein objects within a scene are identified through textual descriptions. For instance, Grounding Dino (Liu et al., 2023c) , combined with the Segment Anything Model (SAM) (Kirillov et al., 2023) , excels at this task if provided with appropriate prompts. However, a well-documented limitation of data-driven neural approaches is their lack of reasoning capabilities (Shi et al., 2023; Huang et al., 2024) . Consequently, they often fail to understand complex prompts that require high-level reasoning on relations and attributes of multiple objects, as demonstrated in Fig. 1 .In contrast, humans identify objects through structured descriptions of complex scenes referring to an object, e.g., "An object that is on the boat and holding an umbrella". These descriptions are referred to as deictic representations and were introduced to artificial intelligence research motivated by linguistics (Agre & Chapman, 1987) , and subsequently applied in reinforcement learning (Finney et al., 2002) . A deictic expression refers to an object depending on the agent using it and the overall context. Although deictic representations play a central role in human comprehension of scenes, current approaches fail to interpret them faithfully due to their poor reasoning capabilities.To remedy these issues, we propose DeiSAM, which is a combination of large pre-trained neural networks with differentiable logic reasoners for deictic promptable object detection and segmentation. The DeiSAM pipeline is highly modular and fully differentiable, sophisticatedly integrating large pre-trained networks and neuro-symbolic reasoners. Specifically, we leverage Large Language Models (LLMs) to generate logic rules for a given deictic prompt and perform differentiable forward reasoning (Shindo et al., 2023 (Shindo et al., , 2024) ) with scene graph generators (Zellers et al., 2018) . The reasoner is efficiently combined with neural networks by leveraging forward propagation on computational graphs. The result of this reasoning step is used to ground a segmentation model that reliably identifies the objects best matching the input.In summary, we make the following contributions: 1) We propose DeiSAM2 , a modular, neurosymbolic framework using LLMs and scene graphs for object segmentation with complex textual prompts. 2) We introduce a novel Deictic Visual Genome (DeiVG) benchmark that contains visual scenes paired with deictic representations, i.e., complex textual identifications of objects in the scene.To further investigate the challenging nature of abstract prompts, we curate a new DeiRefCOCO+ benchmark. It is a deictic variant of RefCOCO+, an established reference object detection benchmark.3) We empirically demonstrate that DeiSAM strongly outperforms neural baselines for deictic segmentation. 4) We showcase that DeiSAM can perform end-to-end training via differentiable reasoning to improve the segmentation quality adapting to complex downstream reasoning tasks. 2020) to address this topic and subsequent frameworks (Yi et al., 2018; Mao et al., 2019; Amizadeh et al., 2020; Hsu et al., 2023 ) that perform reasoning using symbolic programs and multi-modal transformers (Tan & Bansal, 2019) . These benchmarks are primarily developed to answer queries written in natural language texts paired with visual inputs. Our proposed dataset, DeiVG, is the first to integrate complex textual prompts into the task of image segmentation with natural images.In a similar vein, to tackle visual reasoning tasks, neuro-symbolic rule learning frameworks have been proposed, where discrete rule structures are learned via backpropagation (Evans & Grefenstette, 2018; Minervini et al., 2020; Shindo et al., 2021 Shindo et al., , 2023 Shindo et al., , 2024;; Zimmer et al., 2023) . These works have primarily been tested on visual arithmetic tasks and dedicated synthetic environments for reasoning (Stammer et al., 2021) . DeiSAM is a unique neuro-symbolic framework that addresses image segmentation in natural images and utilizes differentiable reasoning for program learning.Semantic segmentation aims to generate objects' segmentation masks given visual input (Wang et al., 2018; Guo et al., 2018) . Multiple datasets and tasks have been proposed that assess a model's reasoning ability to identify objects (Kazemzadeh et al., 2014; Yu et al., 2016) . Recently, Segment Anything Model (SAM) (Kirillov et al., 2023) has been released, achieving strong results on zero-shot image segmentation tasks. Grounded SAM (Ren et al., 2024) combines Grounding DINO (Liu et al., 2023c) with SAM, allowing for objects described by textual prompts. Moreover, LISA (Lai et al., 2023) fine-tunes multi-modal LLMs to perform low-level reasoning over image segmentation. However, LISA still requires strong prior information on the type target object (e.g."the person that is wearing green shoes") and breaks down for more abstract tasks (cf. Sec. 5.5). In contrast, DeiSAM encodes the reasoning process explicitly as a differentiable function, thus avoiding spurious neural networks' behavior. Consequently, DeiSAM is capable of high-level reasoning on arbitrarily abstract prompts (e.g."an object") utilizing structured representation of scene graphs. To this end, frameworks that enhance the transformer (or attention) architecture for various segmentation tasks have been proposed (Liu et al., 2023a; Wu et al., 2024a,b) . These approaches rely on transformers (or attentions) as their core reasoning pipeline. In contrast, DeiSAM explicitly encodes logical reasoning processes to guarantee accurate and faithful interpretation of abstract and complex prompts. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In the last section of the paper, we discuss multiple limitations of the proposed approach. Moreover, in the appendix, we provide empirical results of the runtime of our framework compared to baselines and discuss the computational efficiency. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory Assumptions and Proofs some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
deisam -the deictic segment anything model: 
 DeiSAM uses first-order logic as its language, and we provide its formal definition in App. A. Let us start by outlining the DeiSAM pipeline with a brief overview of its modules, before describing essential components in more detail.We show a schematic overview of the proposed DeiSAM workflow in Fig. 2 . First, an input image is transferred into a graphical representation using a (1) Scene Graph Generator. Specifically, a scene graph comprises a set of triplets (n 1 , e, n 2 ), where entities n 1 and n 2 have relation e. For example, a person (n 1 ) is holding (e) an umbrella (n 2 ). Consequently, each triplet (n 1 , e, n 2 ) in a scene graph can be interpreted as a fact, e(n 1 , n 2 ), where e is a 2-ary predicate and n 1 and n 2 are constants in first-order logic. The textual deictic prompt needs to be interpreted as a structured logical expression to perform reasoning on these facts.% Program 1 cond1(X):-on(X,Y),type(Y,boat). cond2(X):-holding(X,Y),type(Y,umbrella). target(X):-cond1(X),cond2(X).Listing 1: Rules generated by LLMs.For this step, DeiSAM leverages (2) Large Language Models, which can generate logic rules for deictic descriptions, given sufficiently restrictive prompts as we demonstrate. In our example, the LLM would translate "An object that is on the boat, and that is holding an umbrella" into the rules (Program 1) in Listing 1. The first two rules define the conditions described in the prompt, and the last rule identifies corresponding objects. However, users often use terminology different from that of the SGG, e.g., boat and barge target the same concept but will not be trivially matched. To bridge the semantic gap, we introduce a (3) semantic unifier. This module leverages word embeddings of labels, entities, and relations in the generated scene graphs and rules to match synonymous terms by modifying rules accordingly. The semantically unified rules are then compiled to a (4) forward reasoner, which computes logical entailment using forward chaining (Shindo et al., 2023) . The reasoner identifies the targeted objects and their boundingForward Reasoner Large Language Model"An object that is on the boat, and that is holding an umbrella"Segmentation Model target(X):-cond1(X),cond2(X). cond1(X):-on(X,Y),type(Y,boat) cond2(X):-holding(X,Z),type(Z,umbrella). An image paired with a deictic prompt is given as input. We parse the image into a scene graph (1) and generate logic rules (2) corresponding to the deictic prompt using a large language model. The generated scene graph and rules are fed to the Semantic Unifier module (3), where synonymous terms are unified. For example, barge in the scene graph and boat in the generated rules will be interpreted as the same term. Next, the forward reasoner (4) infers target objects specified by the textual deictic prompt. Lastly, we perform object segmentation (5) on extracted cropped image regions of the target objects. Since the forward reasoner is differentiable (Shindo et al., 2023) , gradients can be passed through the entire pipeline (Best viewed in color).boxes from the scene graph. Lastly, we segment the object by feeding the cropped images to a (5) segmentation model.Now, let us investigate the two core modules of DeiSAM in detail: rule generation and reasoning.To perform reasoning on textual prompts, we need to identify corresponding rules. We use LLMs to parse textual descriptions to logic rules using the system prompt specifying the rule format to be generated. The complete prompt is provided in App. B. DeiSAM uses a specific rule format describing object and attribute relations. For example, a fact on(person,boat) in a scene graph would be decomposed into multiple facts on(X,Y), type(X,person), and type(Y,boat) to account for several entities with the same attribute in the scene.The computational and memory cost of forward reasoning is determined by the number of variables over all rules and the number of conditions. Naive formatting of rules (Shindo et al., 2024) leads to an exponential resource increase with the growing complexity of deictic prompts. Since the representations used in the forward reasoner are pre-computed and kept in memory, non-optimized approaches will quickly lead to exhaustive memory consumption (Evans & Grefenstette, 2018) . In our format, however, we restrict the used variables to X and Y and only increase the number of rules with growing prompt complexity. Thus resulting in linear scaling of computational costs instead.DeiSAM performs differentiable forward reasoning as follows. We build a reasoning function f reason : G × R → T where G is a set of facts representing a scene graph, R is a set of rules generated by an LLM, and T is a set of facts representing identified target objects in the scene.(Differentiable) Forward Reasoning. For a visual input x ∈ R 2 , DeiSAM utilizes scene graph generators (Zellers et al., 2018) to obtain a logical graph representation G, where each fact rel(obj1, obj2) ∈ G represents an edge in the scene graph. Each fact in a given set G is mapped to a confidence score using a valuation vector v ∈ [0, 1] |G| . A SGG is a function sgg : R 2 → [0, 1] |G| that produces a valuation vector out of a visual input. DeiSAM builds on the neuro-symbolic messagepassing reasoner (NEUMANN) (Shindo et al., 2024) to perform reasoning. For a given set of rules R, DeiSAM constructs a forward reasoning graph, which is a bi-directional graph representation of a logic program. Given an initial valuation vector produced by an SGG, DeiSAM computes logical consequences in a differentiable manner by performing bi-directional message passing on the constructed reasoning graph using soft-logic operations (cf. App. A.1). DeiSAM identifies target objects to be segmented using confidence scores over facts representing targets, e.g., target(obj1), and extracts the corresponding bounding boxes from the scene graph.Semantic Unifier. DeiSAM unifies diverging semantics in the generated rules and scene graph using concept embeddings similar to neural theorem provers (Rocktäschel & Riedel, 2017) . We rewrite the corresponding rules R of a prompt by identifying the most similar terms in the scene graph for each predicate and constant. If rule R ∈ R contains a term x, which does not appear in scene graph G, we compute the most similar term as arg max y∈G encoder (x) ⊤ • encoder (y), where encoder is an embedding model for texts. We apply this procedure to terms and predicates individually.4 The Deictic Visual Genome coolerAn object that has a handle and that is on a bench To facilitate a thorough evaluation of the novel deictic object segmentation tasks, we introduce the Deictic Visual Genome (DeiVG) dataset. Building on Visual Genome (Krishna et al., 2017) , we construct pairs of deictic prompts and corresponding object annotations for real-world images, as shown in Fig. 3 . Our analysis of the scene graphs in Visual Genome found the annotations to often be noisy and ambiguous, which aligns with observations from previous research (Hudson & Manning, 2019) . Consequently, we substantially filtered and cleaned potential candidates to produce a sound dataset.First, we restricted ourselves to 19 commonly occurring relations and ensured that prompts were unambiguous, with only one kind of target object satisfying the prompt. Specifically, DeiVG contains prompts requiring the correct identification of multiple objects, but these are guaranteed to be the same type according to Visual Genomes synset annotations. We automatically synthesize prompts from the filtered scene graphs using textual templates, e.g., the relations has(cooler,handle) and on(cooler,bench) would yield a prompt "An object that has a handle and that is on a bench" targeting the cooler. Entries in the DeiVG dataset can be categorized by the number of relations they use in their object description. We introduce three subsets with 1-3 relations, which we denote as DeiVG 1 , DeiVG 2 , and DeiVG 3 , respectively. Each dataset is distinct, e.g., DeiVG 2 contains only prompts using 2 relations. For each set, we randomly select 10k samples that we make publicly available to encourage further research. 
experimental evaluation: 
 With the methodology of DeiSAM and our novel evaluation benchmark DeiVG established, we now provide empirical and qualitative experiments. Our results outline DeiSAM's benefits over purely neural approaches, supplemented by ablation studies of each module. Additionally, we investigate RefCOCO (Yu et al., 2016) , a low-level reasoning benchmark for segmentation tasks, and demonstrate the robustness of DeiSAM for abstract prompts. Lastly, we show that DeiSAM is end-to-end trainable and can thus be leveraged to improve the performance of the neural components in the pipeline.We base our experiments on the three subsets of DeiVG. As an evaluation metric, we use mean average precision (mAP) over objects. Since the object segmentation quality largely depends on the used segmentation model, we focus on assessing the object identification preceding the segmentation step. The default DeiSAM configuration for the subsequent experiments uses the ground truth scene graphs from Visual Genome (Krishna et al., 2017) , gpt-3.5-turbo3 as LLM for rule generation, ada-0024 as embedding model for semantic unification, and SAM (Kirillov et al., 2023) for object segmentation. Additionally, we provide few-shot examples of deictic prompts and paired rules in the input context of the LLM, which improves performance (cf. App. E). We present detailed ablations on each component of the DeiSAM pipeline in Sec. 5.4. We compare DeiSAM to multiple purely neural approaches, both empirically as well as qualitatively.We include three baselines that use one model for object identification and subsequently segment the grounded image using SAM (Kirillov et al., 2023) , similar to the grounding in DeiSAM. Our comparison includes the following models for visual grounding: 1) One-For-All (OFA) (Wang et al., 2022) , a unified transformer-based sequence-to-sequence model for vision and language tasks of which we use a dedicated visual grounding checkpoint5 . 2) Grounded Language-Image Pre-training (GLIP) (Li et al., 2022b) a model for specifically designed for object-aware and semantically-rich object detection and grounding. 3) GroundingDino (Liu et al., 2023c) an open-set object detector combining transformer-based detection with grounded pre-training. Moreover, we compare to an end-to-end semantic segmentation model supporting textual prompts with SEEM (Zou et al., 2023) . Lastly, we compare to LISA (Lai et al., 2023) , a state-of-the-art neural reasoning segmentation model.The results on DeiVG of all baselines compared to DeiSAM are summarized in Tab. 1. DeiSAM clearly outperforms all purely neural approaches by a large margin on all splits of DeiVG. The performance of most methods improves with more descriptive deictic prompts, i.e., more relations being used. We attribute this effect to two distinct causes. For one, additional information describing the target object contributes to higher accuracy in object detection. On the other hand, DeiVG 1 contains significantly more samples with multiple target objects than DeiVG 2 or DeiVG 3 . Consequently, cases in which a method identifies only one out of multiple objects will have a higher impact on the overall performance. Overall, the large gap between DeiSAM and all baselines highlights the lack of complex reasoning capabilities in prevalent models and DeiSAM's large advantage. We further provide a runtime comparison and its analysis in App. F, showcasing that DeiSAM's runtime is comparable to the baselines, and the bottleneck is in the LLMs, not in the reasoning pipeline.After empirically demonstrating DeiSAM's capabilities, we look into some qualitative examples. In Fig. 4 , we demonstrate the efficacy of the semantic unifier. All examples use terminology in the deictic prompt diverging from the scene graph entity names. Nonetheless, the unification step successfully maps synonymous terms and still produces the correct segmentation masks, overcoming the limitation of off-the-shelf symbolic logic reasoners.In Fig. 5 , we further compare DeiSAM with the purely neural baselines. DeiSAM produces the correct segmentation mask even for complicated shapes (e.g., partially occluded cable) or complex scenarios (e.g., multiple people, only some holding umbrellas). All baseline methods, however, regularly fail to identify the correct object. A common failure mode is confounding nouns in the deictic prompt. For example, when describing an object in relation to a 'boat', the boat itself is identified instead of the target object. These examples strongly illustrate the improvements of DeiSAM over the pure neural approach on abstract reasoning tasks.An object that is on the table and that is behind a mug.An object that is on the boat and that is holding an umbrella.An object that is on the car and that has ears. The modular nature of the DeiSAM pipeline enables easy component variations. Next, we investigate the performance of key modules in isolation and their overall influence on the pipeline. LLM Rule Generation. One of the key steps for DeiSAM is the translation of deictic prompts posed in natural language into syntactically and semantically sound logic rules. We observed that the performance of instruction-tuned LLMs on this task heavily depends on the employed prompting technique. Consequently, we leverage the well-known methods of few-shot prompting (Brown et al., 2020) and chain-ofthought (CoT) (Wei et al., 2022) .To that end, we first let the model extract all predicates from a deictic prompt, which we subsequently provide as additional context for the rule generation. For both cases, we provide multiple few-shot examples.We evaluate all prompting approaches with LLama-2-13B (Touvron et al., 2023) in Tab. 2. Clearly, few-shot examples are imperative to perform rule generation successfully. Additionally, CoT for predicate decomposition further improves the rule generation for complex prompts.With the best prompting technique identified, we additionally evaluated multiple open and closedsource language models of different sizes (cf. App. F). In general, all instruction-tuned models can generate logic rules from deictic prompts. However, larger models strongly outperform smaller ones, especially for more complex inputs. The overall best-performing model was gpt-3.5-turbo producing correct rules for DeiVG for 93.65% of all samples.Semantic Unification. Next, we take a more detailed look into the semantic unification module. At this step, we bridge the semantic gap between differing formulations in the deictic prompt and the scene graph generator. To evaluate this task, we created an exemplary benchmark based on synonyms in the visual genome. For 2.5k scenes in DeiVG, we considered all objects in the scene graph and identified one object name that differed from its synset entry. Based on that synonym, the task is to identify the one, unique, synonymous object in the scene. For example, in an image containing a 'table', 'couch', 'chair', and 'cupboard' the query 'sofa' should identify the 'couch' as most likely synoynm. Overall, the task is considerably more challenging than it may appear at first glance, with the best model only achieving a success rate of 72% (cf. App F). We observed, for example, the query 'sofa' is matched with 'pillow' instead of the targeted 'couch' or 'trousers' with 'jacket' instead of 'pants'. These results motivate further research into the semantic unification process. To further investigate the challenging nature of abstract prompts, we curated a DeiRefCOCO+ benchmark that contains more abstract textual references. Specifically, we turned the reference texts in RefCOCO+ into deictic prompts by removing any description of the target object. For example, the prompt "kid wearing navy shirt" is modified to "an object that is wearing navy shirt". Tab. 4 again shows the mAP of DeiSAM, GroundedSAM, and LISA on the modified DeiRefCOCO+ dataset. Importantly, DeiSAM retains a similar performance on both types of prompt formulations. In comparison, we observe a strong drop in performance for GroundedSAM and LISA with the absence of confounding object descriptions. These results further highlight the strength and abstraction level of the performed reasoning performed by DeiSAM6 .DeiSAM excels in high-level abstract reasoning, where purely neural pipelines often struggle.To demonstrate, we developed DeiCLEVR, an abstract reasoning segmentation task based on CLEVR (Johnson et al., 2017) . This task challenges models with abstract concepts and relationships.Task. The task is to segment objects given prompts where the answers are derived by the reasoning over abstract list operations. We consider 2 operations: delete and sort. The input is a pair of an image and a prompt, e.g. "Segment the second left-most object after deleting a gray object?". Examples are shown in Fig. 6 . To solve this task, models need to understand the visual scenes and perform high-level abstract reasoning to segment.Each scene contains at most 3 objects with different attributes: (i) colors of cyan, gray, red, and yellow, (ii) shapes of sphere, cube, and cylinder, (iii) materials of metal and matte. We excluded color duplications in a single image. (Prompts) We generated prompts using a templates: "The [Position] object after [Operation]?", where [Position] can take either of: left-most first, second, or third.[Operation] can take either of: (I) delete an object, and (II) sort the objects in the order of: cyan < gray < red < yellow (alphabetical order). We generated 10k examples for each operation.Models. We used DeiSAM with Slot Attention (Locatello et al., 2020) pretrained on the visual Inductive Logic Programming (ILP) dataset (Shindo et al., 2024) , which contains positive and negative visual scenes for list operations. We used GroundedSAM and LISA for neural baselines (cf. App. E).The 2nd leftmost object after deleting a red object.The leftmost object after sorting objects in the order of cyan<gray<red<yellow.Figure 6 : DeiSAM performs abstract reasoning segmentation. When presented with a visual scene paired with an abstract, complex prompt (left), DeiSAM effectively identifies and segments the object specified by the prompt, while neural baselines frequently fail to deduce the target object (right).Result. In Table 5 , we present the mean Average Precision (mAP) for each baseline evaluated. The purely neural baselines struggle to accurately deduce segmentations in response to abstract reasoning prompts, while DeiSAM excels at identifying and segmenting the object specified by the prompt. Moreover, Fig. 6 provides qualitative examples illustrating that DeiSAM effectively segments objects requiring high-level reasoning. In contrast, the neural baselines frequently fail to segment the correct target object. These findings indicate that existing neural baselines are inadequate for addressing abstract reasoning prompts. We demonstrate that integrating differentiable logic reasoners can significantly enhance reasoning capabilities.Since DeiSAM employs a differentiable forward reasoner, a meaningful gradient signal can be back-propagated through the entire pipeline. Consequently, DeiSAM enables end-to-end learning on complex object detection and segmentation tasks with logical reasoning explicitly modeled during training. To illustrate this property, we show that DeiSAM can learn weighted mixtures of scene graph generators by propagating gradients through the reasoning module.% Program 2 targetSgg(X,SG):-cond1(X,SG),cond2(X,SG). cond1(X,SG):-hasSgg(X,Y,SG),typeSgg(Y,hair,SG). cond2(X,SG):-onSgg(X,Y,SG),onSgg(Y,surfboard,SG). % Compose weighted mixtures. w_1: target(X):-targetSgg(X,sgg1). w_2: target(X):-targetSgg(X,sgg2).Listing 2: A program for SGG learning.Task. We consider 2 distinct scene graph generators and compose a weighted mixture of them. We show an example for the deictic prompt "An object that has hair and that is on a surfboard" in Listing 2. The first 3 rules compute the target object for each SGG, similarly to Program 1, and the last 2 rules produce a weighted merge of both predictions. Importantly, Program 2 utilizes different SGGs (i.e.variable SG) and merges the results using learnable weights (i.e.w_1 and w_2). Consequently, the learning task is the optimization of weights w i ∈ R for downstream deictic segmentation. The differentiability of the DeiSAM pipeline allows efficient gradient-based optimizations.Experimental Setup. We used VETO (Sudhakaran et al., 2023) , which outperforms other SGGs on biased datasets where only some relations appear frequently. As the second 'SGG' for our weighted mixture model, we relied on ground-truth scene graphs from Visual Genome. We consider the following baselines: DeiSAM-VETO that only uses a pre-trained VETO model (Sudhakaran et al., 2023) , DeiSAM-Mixture (naive) that uses a mixture of VETO and VG scene graphs with randomly initialized weights. We compare those approaches to DeiSAM-Mixture*, which uses the trained mixture. We extracted instances from DeiVG datasets not used in VETO training (ca. 2000 samples), which we divided into a training, validation, and test split. For rule generation, we use the same system prompt and models as in Sec. 5.1, adapting the generated programs for weight learning.We minimize the binary cross entropy loss with respect to rule weights w_1 and w_2. To calculate this loss, we provide labels for predicted masks in the model, i.e., a binary label y i ∈ {0, 1}. For each instance in DeiVG, DeiSAM predicts segmentation masks in the forward pass, and gradients are backpropagated through the differentiable forward reasoner (cf. App. E.1 for details). Result. In Tab. 6, we compare the mAP on the test split. The trained model DeiSAM-Mixture * clearly outperforms the naive baseline, demonstrating successful training of the DeiSAM pipeline using gradients via differentiable reasoning. DeiSAM-VETO weak performance can be attributed to objects that appear only on prompts but not its training data (cf. App. D).Fig 7 shows examples of segmentation masks and their confidence scores produced by DeiSAM-Mixture models before and after training. Before learning, wrong or incomplete regions are segmented with low confidence scores because the reasoner fails to identify correct objects with low-quality scene graphs that miss critical objects and relations. After learning, DeiSAM produces faithful segmentation masks and increased confidence scores. This experiment highlights that DeiSAM improves the quality of scene graphs and the subsequent segmentation masks by learning using gradients, i.e., it is a fully trainable pipeline with a strong capacity for complex logic reasoning.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: We provide the link to the anonymous repository that contains all of the codes needed to run the experiments demonstrated in the paper and it contains the link to the dataset proposed in the paper. We will make the repository public upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We provide the experimental setting in the paper and also more details in the appendix. We present all the necessary to run the experiments together with the code on the anonymous repository. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in the appendix, or as supplemental material. 
conclusion: 
 Before concluding, let us discuss the limitations and future research directions. Our investigation of DeiSAM's components highlights some clear avenues for future research. While LLMs perform well at parsing deictic prompts into logic rules with few-shot prompting, their performance could be improved further by, e.g., syntactically constrained sampling 7 or dedicated fine-tuning. Further, the observed challenges in semantic unification could be addressed by querying LLMs instead of using embedding models or providing multiple weighted candidates to the reasoner.Upon manual inspection of the DeiVG dataset, we identified some inconsistent examples annotated with erroneous scene graphs in Visual Genome that cannot be automatically cleaned up without external object identification (cf. App. D). Our results support the assessment that generating rich scene graphs is key but difficult to achieve in a zero-shot fashion. However, as we demonstrated, the differentiable pipeline of DeiSAM can be utilized for meaningful training on complex downstream tasks. Thus allowing for the incorporation of real-world use cases in the training of SGGs in an end-to-end fashion. Further, DeiSAM can provide valuable information on general performance and failure cases of SGGs by investigating deictic segmentation tasks. Furthermore, the modularity of DeiSAM allows for easy integration of potential improvements to any of its components.To conclude, we proposed DeiSAM to perform deictic object segmentation in complex scenes. DeiSAM effectively combines large-scale neural networks with differentiable forward reasoning in a modular pipeline. DeiSAM allows users to intuitively describe objects in complex scenes by their relations to other objects. Moreover, we introduced the novel Deictic Visual Genome (DeiVG) benchmark for segmentation with complex deictic prompts. In our extensive experiments, we demonstrated that DeiSAM strongly outperforms neural baselines highlighting its strong reasoning capabilities on visual scenes with complex textual prompts. A First-Order Logic and Differentiable Reasoning.We provide a formal definition of first-order logic (FOL). An atom is a formula p(t 1 , . . . , t n ), where p is a predicate symbol (e.g.type) and t 1 , . . . , t n are terms. A term is a variable or a constant. A ground atom or simply a fact is an atom with no variables (e.g.type(obj1, boat)). A literal is an atom (A) or its negation (¬A), and a clause is a finite disjunction (∨) of literals. A definite clause is a clause with exactly one positive literal. If A, B 1 , . . . , B n are atoms, then A ∨ ¬B 1 ∨ . . . ∨ ¬B n is a definite clause. We write definite clauses in the form of A :-B 1 , . . . , B n , and refer to them as rules for simplicity in this paper. Forward Reasoning is a data-driven approach of reasoning in FOL (Russell & Norvig, 2010) , i.e., given a set of facts and a set of rules, new facts are deduced by applying the rules to the facts. Differentiable forward reasoners compute logical entailment using tensor representations (Evans & Grefenstette, 2018; Shindo et al., 2023) or graph neural networks (Shindo et al., 2024) , and perform rule learning using gradients given labeled examples in the form of inductive logic programming (Cropper & Dumancic, 2022) .DeiSAM employs a graph neural network-based differentiable forward reasoner (Shindo et al., 2024) , and we briefly explain the reasoning process. We represent a set of (weighted) rules as a directed bipartite graph. For example, Fig. 8 is a reasoning graph that represents Program 1. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We report the error bars for the runtime analysis in App. F. For main experiments in Sec. 5, we do not report error bars because most of our experiments are deterministic, i.e., the proposed reasoning pipeline does not involve stochastic computations. The internal usage of LLMs could produce errors by hallucinations, but we minimize such errors by performing error handling, i.e., we repeat the function call when they produce output that cannot be parsed by the system. Thus, we report numbers from a single run for each experiment. The only exception is the experiment of weight learning. We report a single run because of the non-trivial financial cost of the LLM usage, i.e., the API needs to be called iteratively in the learning process, and the expected budget significantly increases.To mitigate this, we provide the random seed to reproduce the result for all experiments in our codebase to ensure reproducibility. Guidelines:• The answer NA means that the paper does not include experiments. Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: This paper presents an abstract methodology that has no effects on the societal impact specifically, such as malicious uses or fairness considerations. These aspects of societal impacts will be attributed to the dataset itself to be used, not our proposed framework. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper presents an abstract methodology that has no effects on the safeguards specifically, such as the high risk of misuse. The paper does not contain any release of pretrained models or datasets newly curated using web scraping. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide the details of differentiable forward reasoning.Definition A.1. A Forward Reasoning Graph is a bipartite directed graph (V G , V ∧ , E G→∧ , E ∧→G ),where V G is a set of nodes representing ground atoms (atom nodes), V ∧ is set of nodes representing conjunctions (conjunction nodes), E G→∧ is set of edges from atom to conjunction nodes and E ∧→G is a set of edges from conjunction to atom nodes. DeiSAM performs forward-chaining reasoning by passing messages on the reasoning graph. Essentially, forward reasoning consists of two steps: (1) computing conjunctions of body atoms for each rule and (2) computing disjunctions for head atoms deduced by different rules. These two steps can be efficiently computed on bi-directional message-passing on the forward reasoning graph. We now describe each step in detail.(Direction →) From Atom to Conjunction. First, messages are passed to the conjunction nodes from atom nodes. For conjunction node v i ∈ V ∧ , the node features are updated:EQUATIONwhere is a soft implementation of conjunction, and is a soft implementation of disjunction.Intuitively, probabilistic truth values for bodies of all ground rules are computed softly by Eq. 1.(Direction ←) From Conjunction to Atom. Following the first message passing, the atom nodes are then updated using the messages from conjunction nodes. For atom node v i ∈ V G , the node features are updated: EQUATIONWe provide a further investigation of Visual Genome (VG) scene graph annotations. We demonstrate that (1) VG annotations have multiple versions and there is a non-trivial discrepancy between their relation distributions, and (2) VG annotations contain incomplete and erroneous scene graphs that cannot be automatically cleaned up without external object identification.We investigated the scene graph generator module, going beyond the ground truth scene graphs of Visual Genome. One of the key challenges in using a pre-trained SGG is the potential mismatch between the set of objects and annotations in DeiVG and the training data of the SGG. While we built on the latest version of Visual Genome with extended annotations (VG v1.4 ), an older version (Krishna et al., 2017) has been commonly used for benchmarking different SGGs by excluding non-frequent object types and relations (Xu et al., 2017) . This preprocessed older version (VG) is still considered the standard benchmark for SGGs, which lacks many crucial objects and attributes in DeiVG built upon VG v1.4 . We illustrate the discrepancy in Fig. 9 , where we compare the Kernel Density Estimate (KDE) of VG v1.4 and VG. It shows that the newer version contains more types of objects in the top-and-middle frequency range. Additionally, many object types of VG v1.4 in the middle-frequency range are not contained at all in VG. Consequently, when parsing scenes from DeiVG with SGGs pre-trained on VG, the objects in the deictic prompt may not be contained in the generated scene graph at all. For example, for a given deictic prompt "an object that is wearing a black shirt", we observed that the pre-trained SGG failed to detect black shirt because it was not included in its training data. This discrepancy leads to sub-par performance of DeiSAM with a pre-trained SGG, as shown in Tab. 6. While training the SGG specifically on the relevant scene distribution can partially address this issue, it is crucial to highlight that DeiSAM can be leveraged to improve SGGs as well. Our subsequent demonstration in Section 5.7 provides a glimpse of DeiSAM's capabilities, showcasing its differentiable forward reasoner's ability to perform end-to-end training, thereby unlocking new horizons in scene understanding and reasoning.In Fig. 10 , the plot on the left depicts the distribution of the least frequent tail object types on the latest extended version (VG v1.4 ) in comparison to the older version (VG) (Krishna et al., 2017) . Object types are sorted with respect to the number of occurrences (x-axis). In the non-frequent range of 0 to 1000, VG v1.4 contains object types that never appear on the older version (VG), revealing these tail classes of object types are completely missing in the processed older annotations (Xu et al., 2017) , (VG v1.4 ), but the older version (VG) (Krishna et al., 2017) has been commonly used for SGG training. Object types are sorted w.r.t. the number of occurrences (x-axis) and their corresponding occupancies are shown (y-axis). The left plot is for top-and-middle frequent object types (0-50k #occ.), and the right plot is for middle frequent object types (0-3k #occ.). VG v1.4 contains many object types that do not appear in VG (Best viewed in color). VG v1.4 ) in comparison to the older version (VG) (Krishna et al., 2017) . Object types are sorted w.r.t. the number of occurrences (x-axis).The plot on the right shows the comparison of the number of object types and the number of total objects in the datasets. VG v1.4 contains many more object types than VG, making it difficult for SGGs that are pre-trained on the older version to achieve high performance (Best viewed in color).which are commonly used for benchmarking SGGs. Moreover, the bar plot on the right indicates a notable increase in the number of object types and total object counts in VG v1.4 in comparison to VG, making it difficult for SGGs that are pre-trained on the older version to achieve high performance on the DeiVG dataset built upon VG v1.4 .Errors on VG Annotations. Upon manual inspection of the DeiVG dataset, we identified some inconsistent examples resulting from incomplete and erroneous scene graphs in Visual Genome that cannot be automatically cleaned up without external object identification. For example, the annotations shown in Fig. 11 lead to a DeiVG sample with two missing target objects and an incorrect one. We plan on building a more consistent deictic segmentation benchmark using a cleanup process similar to GQA (Hudson & Manning, 2019 ).We provide details of the models used in the evaluation. For all methods using SAM for segmentationincluding DeiSAM-we use the same publicly available SAM checkpoint 8 .DeiSAM. We used NEUMANN (Shindo et al., 2024) with γ = 0.01 for soft-logic operations, and the number of inference steps is set to 2. We set the box threshold to 0.3 and the text threshold to 0.25 for the SAM model. All generated rules are assigned a weight of 1.0. If no targets are detected, DeiSAM produces a mask of a randomly chosen object in the scene.For LLMs, we provided few-shot examples of deictic prompts and desired outputs as shown in Listing 4. These few-shot examples improved the quality of the rule generation that follows a certain format. These are combined with the system prompt in App. B to generate rules by LLMs.Prompt: 'An object that is wearing a helmet' ones, especially for more complex inputs. Interestingly, the types of failure cases differ significantly between models. For example, both Llama models and GPT-3.5-turbo rarely make syntactical errors in rule generation (< 1%), whereas most of Mistral-7B's incorrect rules are already syntactically unsound.For the semantic unification task we compare various semantic embedding models as shown in Table 8 .Runtime Analysis. We provide comparisons of the runtime of Grounded-SAM, LISA, and DeiSAM in Tab. 9. It shows the inference time per instance (a visual input with a textual prompt), averaged over 1000 examples for each dataset. GroundedSAM achieved remarkably faster inferences since it does not encode the reasoning process. Both LISA and DeiSAM approaches require about 6 to 10 seconds per example. For more analysis, we provide the running time for each component of DeiSAM in Tab. 10. It shows the inference time per instance of rule generation, semantic unification, differentiable forward reasoning, and segmentation. We observe that semantic unification is the most time-consuming process in the DeiSAM pipeline. In contrast, the differentiable forward reasoning and segmentation only make up a negligible fraction of the runtime. However, as outlined in the paper, the rule generation and semantic unification step rely on the OpenAI API and could be significantly reduced by running a local model instead. DeiVG in the ReCOCO's reference format. We conducted additional experiments by modifying the deictic prompts in DeiVG datasets to the ones with targets' labels, resulting in prompts similar to reference texts in RefCOCOs (Yu et al., 2016) . Table 11 shows the mAP on DeiVG datasets with the modified prompts using LISA, GroundedSAM, and DeiSAM. ± represents the gain compared to the original performance with deictic prompts (shown in Tab. 1). Neural baselines gained the performance remarkably by the modified prompts since they contain labels for targets (e.g.person, kid), on which neural models highly rely to identify targets. We provide supplementary results of the segmentation on the DeiVG 2 dataset in Fig. 12 .Grounded SAMAn object that is on the street and has smoke. An object that is near the chair and holds a plant.An object that is above the table and has black metal.An object that is on the side walk and wears a coat.An object that is carrying the person and has a tail.An object that is on the metal pole and has a bike.An object near the microwave and on the fridge.An object that is on the bike and has a purse.An object that is on the grass and has a door. An object that is behind girls and has tires. Justification: Our main claim is that the proposed approach can solve deictic prompts to segment objects. In our experiments, we have extensively shown that the proposed model outperforms baselines on deictic segmentation tasks, empirically supporting the main claim made in the abstract and introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Code: https://github.com/ml-research/deictic-segment-anythinghttps://openai.com/blog/introducing-chatgpt-and-whisper-apishttps://openai.com/blog/new-and-improved-embedding-modelhttps://modelscope.cn/models/damo/ofa_visual-grounding_refcoco_large_en/summaryIn App. F, we demonstrate experiments on DeiVG with prompts in the RefCOCO's reference format, highlighting the robustness of DeiSAM against neural baselines.https://github.com/IsaacRe/Syntactically-Constrained-Samplinghttps://huggingface.co/spaces/abhishek/StableSAM/blob/main/sam_vit_h_4b8939.pthhttps://huggingface.co/xdecoder/SEEM/resolve/main/seem_focall_v0.pt 


Decision: Summary of Review Process:
- 4 reviews.
- Mixed scores 3, 4, 6, 6
- Authors submitted rebuttals/responses
- 3 out of 4 reviewers did not engage in discussion after repeated requests from AC. Only Reviewer YJtK (score 6) engaged in discussion.
- The AC has decided that the lack of participation from 3 reviewers should not negatively impact the final recommendation. Therefore the AC will review the questions and responses instead of directly considering the *scores* of the unresponsive reviewers.

Meta Review:
There was consensus from all reviewers regarding the novelty of the approach, clarity of writing, and experimental findings of the paper.  The following is a summary of questions and rebuttal:
- Reviewer 9P1c commented that the motivation was not new and the method used a combination of previous techniques. The AC believes that the response from authors sufficiently addresses this concern.
- Reviewer 9P1c suggested previous articles to be mentioned in related work. The authors have agreed to do so.
- Reviewer 9P1c asked for additional ablation studies. The authors have added results in the general remark. The AC recommends the authors to add these to the main paper.
- Reviewer TxaQ asked the authors to elaborate on some confusing aspects related to the technical approach, illustrations, and some limitations. The authors have responded sufficiently with elaboration/clarification and additional experiments.
- Reviewer WXVv asked the authors to comment on the assumptions and dependence on scene graph, provide additional experiments, and elaborate on the rationale and contributions. The authors have responded to each question. The AC recommends the authors to integrate these responses in the main paper.
- Reviewer YJtK participated in the discussion and gave a rating of Weak Accept. The reviewer states that some of the points still remain unaddressed.

Overall, the AC's evaluation is that the paper is an interesting work on neuro-symbolic approaches using LLMs to generate logical rules and perform differential reasoning on scene graphs.

Reviews: review: paper_summary: paper_summary: The manuscript proposes DeiSAM, a framework that integrates large pre-trained neural networks with differentiable logic reasoners to address deictic promptable segmentation. 
DeiSAM leverages large language models (LLMs) to generate first-order logic rules and performs forward reasoning on generated scene graphs for object segmentation based on complex textual prompts. 
The Deictic Visual Genome (DeiVG) dataset is introduced for evaluation.
strength_weakness: strength_weakness: 1. Over-simplified Assumptions: The curated datasets DeiVG1, DeiVG2, and DeiVG3 contain prompts using only 1-3 relations and are restricted to avoid multiple objects with ambiguity. However, in real-world applications, object and relation categories can be varied and rich, with inevitable visual-language alignment gaps. For example, in an image with a black cat and a white cat, how would the method handle a prompt to segment the white cat using relative descriptions? Moreover, the dataset size is small (10k), making large models prone to overfitting.
   
2. Dependence on Scene Graph Quality: The method heavily relies on the quality of scene graphs. Inaccurate scene graphs can lead to incorrect segmentation results, which is a critical limitation for practical applications. For instance, the large performance gap indicated in Table 5 highlights this dependency.

3. Unclear Philosophy: The use of a semantic unifier (an LLM) to associate objects in the prompt and scene graph seems trivial. The rationale for using a differentiable forward reasoner instead of continuing to use an LLM is not well-justified. The necessity and advantage of having a "differentiable" reasoner are unclear. How does this reasoner fundamentally differ from previous neural-based methods like Grounding DINO or LISA?

4. Insufficient Experiments: The experiments are limited to relatively easy cases for prompt-based segmentation on a small-scale image dataset. The paper lacks evaluations on more challenging scenarios, such as multiple objects of the same category or prompts with more complicated relations. Comparisons with previous methods under these conditions would provide a more comprehensive evaluation of DeiSAM’s effectiveness.

5. Unclear Contributions: Modules of DeiSAM are borrowed from other existing works, and there is no novel architecture designed for the prompt-based segmentation. It appears more like a technique report for an engineering project instead of being a research paper suitable for a top-tier conference like NeurIPS.
questions: questions: Detailed questions are referred to as Weaknesses.
limitations: limitations: The authors have discussed the limitations of the work.
paper_summary: paper_summary: The paper proposes to study Deictic references/prompts, i.e., phrases/references that describe the role/purpose/context rather than naming the object directly. Firstly, the paper constructs a new dataset with deictic prompts based on Visual Genome; Next, it is shown that existing methods do poorly on this task. Next, a neuro-symbolic method that works on top of a generated scene graph is proposed which is shown to work a lot better on this newly constructed dataset as well as perform comparably on related tasks such as referring expression recognition.

[EDIT: Post rebuttal comments are useful but do not change overall score]
strength_weakness: strength_weakness: **[W1] Somewhat contrived setup:** I fully agree with the big premise that deictic expressions are important and common in everyday usage. However, many examples from the constructed dataset are unnatural and forced. The paper also takes deictic phrases to an extreme by removing all nouns from the deictic expressions with very generic prompts such as "stuff" or "object". It is more common for references to consist of categorial nouns instead of object names (E.g., that toy on top of the chair) instead of teddy bear, etc. So, while a useful tool in diagnostic abilities, I feel that the dataset is not broadly applicable. Similarly, since it is such an unnatural setup; It might also make it unrealistically hard for general purpose baseline methods.

**[W2] Reliance on Scene graphs:** The proposed work relies on access to scene graph prediction model outputs. This is not something that is available to baselines and, in my opinion, does a lot of heavy lifting. I believe this makes it hard to do a fair comparison with the baselines. One could even argue that if the scene graph is comprehensive and accurate the **main** task in decoding deictic expressions (Thought I admit that it is still non trivial). Also see Q1.

[W3] Minor issues on the dataset: The dataset is divided based on how many "hops" of reasoning are necessary to decode the expression. E.g., "A tall object wearing a hat walking in sidewalk" is a 3 hop expression (tall, wearing a hat, walking on the sidewalk). However, unless there are counterexamples to each. (E.g., other tall things, other stuff wearing the hat, ...) it degenerates into a subset of that. This means a shortcut "wearing a hat" alone might be enough to decode the object. I would like to see an analysis of this.
questions: questions: Q1. If scene graphs are assumed to be available, should they not be available in some form to every algorithm. E.g., if the object name is easily decodable from deictic expression by looking it up in the scene graph, can we not change the expression to include the name of the object as well? E.g., "A red furry object" --> Select all objects that are red and furry from scene graph --> Grab the names of those objects -> use the name instead of deictic expression when prompting SEEM or LISA.

Q2. What is the effect of the quality of scene graphs on the final results of the proposed method. Is it resitant to mistakes in the SGG prediction?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes DeiSAM to enhance the ability of current state-of-the-art referring segmentation frameworks with deictic prompting. DeiSAM is an innovative approach that combines large pre-trained neural networks with differentiable logic reasoners to perform image segmentation based on complex, deictic textual prompts. The method leverages Large Language Models (LLMs) to generate first-order logic rules from textual descriptions and uses differentiable reasoning on scene graphs to identify and segment objects in images accordingly. The paper also introduces the Deictic Visual Genome (DeiVG) dataset for evaluating deictic segmentation and demonstrates DeiSAM's superiority over neural baselines through empirical results.
strength_weakness: strength_weakness: 1.
The clarity of the paper could be further improved. Despite careful reading and familiarity with the field, several aspects remain confusing:

- **Training Process**
  - **Unclear Tuning**: What components are tuned during the training stage? 
  - **Revision Recommendation**: Annotate which modules are tuned during training.

- **Scene Graph Generator**
  - **Pre-training Limitations**: The Scene Graph Generator appears pre-trained on Visual Genome, which annotates only a limited set of objects (e.g., persons, boats, cars).
  - **Generalizability Concerns**: 
    - Will the proposed method work on images containing objects not annotated in Visual Genome? 
    - If not, the title "Segment Anything with Deictic Prompting" may be an overstatement.

- **Semantic Unifier and Forward Reasoning**
  - **Ambiguous Illustration**: The explanation of these components lacks clarity.
    - What are the inputs and outputs of the Semantic Unifier?
    - What constitutes the "background knowledge" $\mathcal{B}$ in forward reasoning?
    - What is learned in the forward reasoning process?

2.
The proposed method should also be evaluated on other referring image segmentation benchmarks, such as RefCOCO, RefCOCOg, and RefCOCO+. This evaluation would help determine whether the proposed method is effective in general referring image segmentation settings, or if the integration of scene graph information potentially hinders performance in standard referring segmentation tasks.
questions: questions: All are listed in weaknesses. Please see above.
limitations: limitations: The authors have adequately addressed the limitations in the submitted manuscript.
paper_summary: paper_summary: This paper introduces a new reasoning-based segmentation by adding deictic logic as prompts into the SAM. Motivated by the deictic logical analysis, the authors present DeiSAM, a combination of large pre-trained neural networks with differentiable logic reasoners for deictic promptable segmentation. DeiSAM first use LLMs to generate first-order logic rules and perform differentiable forward reasoning on generated scene graphs. In addition, authors present Deictic Visual Genome dataset. Experimental results show DeiSAM works better than pure data-driven baselines.
strength_weakness: strength_weakness: Despite the idea of combining first order deictic prompt into SAM is interesting, there are still lot of limitations in this work.

1, Motivation is not new. Performing reasoning segmentation via LLM is not new. There several previous works exploring the same settings [1]-[5]. The difference lies on the author adopt more visual cues (scene graph) into LLM before sending the language features to SAM.

[1], LISA: Reasoning Segmentation via Large Language Model, CVPR-2024.

[2], An Improved Baseline for Reasoning Segmentation with Large Language Model, arxiv-2023.

[3], GRES: Generalized Referring Expression Segmentation, CVPR-2023.

[4], Towards Robust Referring Image Segmentation, TIP-2023.

[5], See, Say, and Segment: Teaching LMMs to Overcome False Premises, CVPR-2024.

2, The technical novelties are limited. The proposed method just combines previous differentiable logic reasoners with LISA-like segmenter (LLM + SAM) architecture. I find no extra insights or advantages for end-to-end reasoning learning. Moreover, the method needs extra scene graphs as inputs, which introduce more complexities to the pipeline.

3, Related works discussion. Missing lots of works on referring segmentation [3]-[5].

4, Performance and benchmarks. There are only refcoco+ and proposed benchmark results. Several important benchmarks are missing (Please refer to [1]). Moreover, the performance is not strong. This result is misleading since the authors miss lots of previous works for fair comparison.

5, Missing detailed ablation studies for each component. It is hard to know the real benefits on the combined logical reasoner and effects of input scene graph.
questions: questions: See the weakness part.
limitations: limitations: Yes, they have checklist.

Comments: title: Thank you for your response
Post-rebuttal comments
Reviewer 9P1c: please respond to the authors' rebuttal
Reviewer TxaQ: please respond to the authors' rebuttal
Reviewer YJtK: please respond to the authors' rebuttal
Reviewer WXVv: please respond to the authors' rebuttal
comment: Thank you for your response. We agree with the reviewer that the experimental setup suggested in the response for the LLM baselines would work as another naive baseline, on top of the LLM baselines presented in the paper. We will run the experiments with the suggested setup and include the results to the final version of our manuscript. Thank you again for your insightful suggestions to improve the paper.
I have read the responses to my comments as well as all other reviews and their responses carefully. Like me, others also had concerns about the use of scene graphs and limited vocals caused by using a COCO-bases scene graph. There hasn't been a very satisfactory response about this fact. 

Secon point that I raised was about "unfair" use of extra information in form of scene graph. Unfortunately, that also hasn't been very satisfactory.

> This approach would not be fair. While it is an interesting suggestion to use scene graphs for neural baselines, the proposed method involves providing an external program for preprocessing the scene graphs so that neural models can process them. This would result in a composite model consisting of a scene graph preprocessor and a separate large segmentation model. 

I disagree. Scene graph representation in form of scene graph triplets can be natively processed by LLMs and even better with a few in-context examples. Even if needed to make a "program" it can be a very simple one based on semantic parsers (which can be inaccurate but would still provide a sort of a equalization of available information. 

> Such an approach severely limits further extensions aimed at enhancing scene understanding and reasoning segmentation because it is not straightforward to propagate segmentation errors back to improve scene graph generation.

Yes it would, but that is fine and would only serve to differentiate the submitted work with the above approach, which is a strength. We would still have liked to see what the results are compared to a hypothetical naive model that gets the equivalent amount of information.

However, both those points were originally included in my first round review so the scores remain unchanged.
Dear reviewer,

thanks for your participation in the NeurIPS peer review process.  We are waiting for your response to the rebuttal.  You gave a reject rating (3), with 5 weaknesses.

Is the response from authors satisfactory?
- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Dear reviewer,

thanks for your participation in the NeurIPS peer review process. You indicated that you are leaning towards accepting the paper. Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Dear reviewer,

thanks for your participation in the NeurIPS peer review process. You indicated that you are leaning towards accepting the paper. Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Dear reviewer,

thanks for your participation in the NeurIPS peer review process.  We are waiting for your response to the rebuttal.  You gave a borderline rating (4). Is the response from authors satisfactory? Does it address weaknesses that you mentioned in the review?

- If yes, are you planning to increase your score?
- If no, could you help the authors understand how they can improve their paper in future versions?

Thanks,
AC
Dear Reviewer,

We hope we have answered all your questions and resolved the outstanding concerns. As the discussion phase is coming to an end, we would appreciate if the reviewer engages in a discussion so that we can answer any further concerns, if any.

Regards,

Authors
Dear Reviewer,

We hope we have answered all your questions and resolved the outstanding concerns. As the discussion phase is coming to an end, we would appreciate if the reviewer engages in a discussion so that we can answer any further concerns, if any.

Regards,

Authors
Dear Reviewer,

We hope we have answered all your questions and resolved the outstanding concerns. As the discussion phase is coming to an end, we would appreciate if the reviewer engages in a discussion so that we can answer any further concerns, if any.

Regards,

Authors
Dear Reviewer,

We hope we have answered all your questions and resolved the outstanding concerns. As the discussion phase is coming to an end, we would appreciate if the reviewer engages in a discussion so that we can answer any further concerns, if any.

Regards,

Authors