Title: SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Referring Image Segmentation

Referential Ambiguity

Mamba

Multi-Modal Understanding
Abstract: Referring Image Segmentation (RIS) aims to segment the target object in an image given a natural language expression. While recent methods leverage pre-trained vision backbones and more training corpus to achieve impressive results, they predominantly focus on simple expressions—short, clear noun phrases like “red car” or “left girl”. This simplification often reduces RIS to a key word/concept matching problem, limiting the model’s ability to handle referential ambiguity in expressions. In this work, we identify two challenging real-world scenarios: object-distracting expressions, which involve multiple entities with contextual cues, and category-implicit expressions, where the object class is not explicitly stated. To address the challenges, we propose a novel framework, SaFiRe, which mimics the human two-phase cognitive process—first forming a global understanding, then refining it through detail-oriented inspection. This is naturally supported by Mamba’s scan-then-update property, which aligns with our phased design and enables efficient multi-cycle refinement with linear complexity. We further introduce aRefCOCO, a new benchmark designed to evaluate RIS models under ambiguous referring expressions. Extensive experiments on both standard and proposed datasets demonstrate the superiority of SaFiRe over state-of-the-art baselines. Project page: https://zhenjiemao.github.io/SaFiRe/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper critiques existing approaches in Referring Image Segmentation (RIS) for their reliance on simple keyword matching, which often fails in ambiguous situations. To address this limitation, the paper introduces a framework called SaFiRe, inspired by human cognitive processes consisting of a global understanding phase (Saccade) followed by a fine-grained refinement phase (Fixation). The architecture utilizes the Mamba state-space model to effectively manage long-range dependencies, aligning with the cognitive scan-then-inspect paradigm. Furthermore, to evaluate model performance under referential ambiguity, the paper presents a new benchmark, aRefCOCO, specifically designed to include challenging cases such as object-distracting and category-implicit expressions. The proposed method consistently outperforms existing approaches across both standard and ambiguous benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper makes an intriguing contribution to the field, with current strengths and potential suggestions outlined below: \nStrengths: \n1. Methodological Novelty: The concept of mapping the two-phase human cognition-inspired “Saccade” for global coarse grounding and “Fixation” for local refinement provides an interesting and well-conceived framework for addressing RIS challenges.\n2. Architectural Efficiency: Leveraging the SFLayer in existing segmentation pipelines through Mamba’s linear complexity enables scalable refinement, which is reflected in improved inference speed during evaluations.\n3. New Benchmark: The identification of oversimplified referring expressions in existing RIS benchmarks and the creation of a dataset focused on evaluating model performance under ambiguous expressions represent a significant advancement in addressing real-world linguistic challenges.\n4. Evaluations and Experimentations: A clear evaluations across various methods on common benchmarks, along with clear ablations conducted in the paper, enhance understanding of the proposed method's advantages and the components contributing to its results.\nWeakness:\n1. Problem Novelty: The problem addressed in this paper closely resembles that of one of the cited papers (LISA), with insufficient articulation of the differences between the two. Clearly distinguishing between these works is crucial to avoid confusion regarding their focus on “reasoning segmentation” under ambiguous language prompts.\n2. Underexplored comparisons: Given the objective of handling ambiguity in text prompts for segmentation, it is essential to explicitly discuss and evaluate reasoning models like LISA and LISA++ to clarify SaFiRe’s position within the wider community.\n3. Benchmark: The availability of aRefCOCO as a test-only dataset could limit its utility within the community. Providing a clear train/val split would enhance its usefulness for future benchmarking efforts.\n4. Quantification of Speedup: While the proposed method's speedup is specified, quantifying it in relation to computational requirements (GFLOPS) would be beneficial for the community.\n\nQUESTIONS:\n1. Novelty Clarification: How does SaFiRe fundamentally differ from “Reasoning Segmentation” approaches such as LISA? Is the distinction only architectural, or is there a deeper conceptual or functional divergence?\n\n2. Failure Case Insights: While ambiguous expressions are addressed, what patterns emerge in failure cases (e.g., pronoun chains, conflicting attributes)? Are there categories of ambiguity where SaFiRe still struggles?\n\n3. Reasoning Model Benchmarking: Have the authors considered evaluating SaFiRe against approaches that share similar conceptual (i.e., reasoning-centric baselines like LISA or LISA++)? A clearer empirical comparison could help position this work more accurately within the reasoning segmentation landscape.\n\nLIMITATIONS:\nYES.\nThe authors have adequately outlined both the methodological and ethical limitations of their work. A suggestion to enhance this section would be to include failure case visualizations, which would complement the limitations by providing concrete examples along with appropriate explanations. This addition could help illustrate the specific challenges encountered and further clarify the contexts in which the proposed method may struggle.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

comments: SUMMARY:
The paper presents a novel framework, called SaFiRe, to address the challenges in referring image segmentation: object-distracting expressions and category-implicit expressions. It is based on a Mamba’s scan-then-update function. A new benchmark called aRefCOCO is also introduced for evaluating on ambiguous expressions. Experimental results are promising when compared to existing RIS methods.

STRENGTHS AND WEAKNESSES:
Strengths:
+ Experiment results are promising.
+ The utilization of state-space sequence modeling for RIS is reasonable.

Weaknesses:
- The motivation of the paper is kind of vague and the explanation of SaFire’s design is questionable. Even though the authors claim that their `Saccade` and `Fixation` come from cognitive science, it can not convince me that such operation aligns with their original aspects.
- Also, the novelty of the proposed framework is questionable, as it borrows largely similar designs from VMamba, like the scanning and folding techniques in other SSM works.
- It is unclear whether the samples selected for the aRefCOCO are valid or not when including VLM (i.e., Qwen-VL-Max) for curation due to the absence of further discussion of it. This would also cause a potential cherry-picking by selecting challenging samples from RefCOCO where the proposed method outperforms and the baselines do not. A good practice is to provide more samples in the Supp. Materials for evaluation.

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a referring image segmentation framework that tackles two challenges: object-distracting expressions and category-implicit expressions. Built upon Mamba, the model first forms a global understanding and then refines it through region-wise inspection. A new benchmark is introduced to evaluate RIS models under ambiguous settings. Experiments show the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n1. The paper is overall well-written and easy to follow.\n2. The identification of object-distracting and category-implicit expression types is well-motivated and grounded in practical RIS limitations.\n3. The introduction of the new benchmark evaluates RIS models under ambiguous referring expressions, which better reflects real-world challenges.\n4. The experiments show the superiority of the proposed method in accuracy and inference speed.\n\n[Weaknesses]\n1. While the Saccade-Fixation design is novel, the rest of the model relies heavily on standard components (Swin-B, BERT). The innovation is primarily in the fusion mechanism rather than in the overall pipeline, which may limit its extensibility to other tasks or modalities without further architectural adjustments.\n\n2. Although the introduced benchmark aRefCOCO increases ambiguity, it is still generated by a VLM, which could inadvertently introduce biases, even with meticulous filtering. The paper lacks evaluations on real-world, human-annotated ambiguous datasets or interactive grounding scenarios, limiting its external validity.\n\n3. The paper could benefit from more analysis on failure cases of the proposed method. This would help clarify its limitations, especially in handling complex descriptions.\n\nQUESTIONS:\n1. How is the inference speed compared to ReMamber [9]?\n2. What are the failure cases of the proposed method?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SaFiRe, a novel framework for Referring Image Segmentation (RIS) that addresses referential ambiguity in natural language expressions. Inspired by human two-phase cognitive processes, SaFiRe integrates Saccade and Fixation operations using Mamba’s state-space modeling. The Saccade operation establishes coarse cross-modal alignment, while Fixation refines it via region-wise text reiteration. The authors also introduce aRefCOCO, a benchmark for ambiguous expressions, and demonstrate SaFiRe’s superiority over SOTA methods on both standard and aRefCOCO datasets. Key contributions include modeling cognitive mechanisms, efficient multi-cycle refinement with linear complexity, and a new benchmark for ambiguous RIS.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1.The paper introduces SaFiRe, a novel framework that simulates human two-phase cognitive processes: global understanding via Saccade and detailed refinement via Fixation, to address referential ambiguity in RIS. By leveraging Mamba’s state-space modeling, the framework achieves efficient multi-cycle refinement with linear complexity, enabling robust alignment between visual regions and ambiguous language expressions. This design achieves significant performance gains on standard benchmarks and superior accuracy on ambiguous expressions.\n\n2.The authors present aRefCOCO, a test-only benchmark specifically designed to evaluate RIS models under object-distracting and category-implicit scenarios. With an average of 12.6 words per expression, 3.1 same-category distractors per image, and frequent pronoun usage, aRefCOCO pushes the field to tackle real-world referential ambiguity. \n\nWeaknesses:\n1.Unfair, Marginal and Incomplete Comparative Evaluation\nThe experimental validation lacks rigor due to suboptimal benchmarking choices. SaFiRe uses Swin Transformer as the visual backbone rather than Mamba, making direct comparisons with ReMamber (a pure Mamba-based model) inherently unfair, as the latter’s architecture is tailored for state-space modeling . When compared to Magnet (a Swin-BERT-based method more aligned with SaFiRe’s backbone), performance gains are marginal, failing to demonstrate substantial superiority. Additionally, the evaluation omits recent SOTA methods(e.g., EEVG[1], C3VG[2], OneRef[3], and PromptRIS[4]). This absence hinders the paper’s ability to validate SaFiRe’s competitiveness in the evolving RIS landscape, casting doubt on the robustness of its claimed advancements.\n\n2.Inadequate Transparency in aRefCOCO Dataset Generation\nThe reliability of aRefCOCO, the proposed benchmark for ambiguous expressions, is undermined by insufficient evidence of its annotation process. While the authors mention that a significant portion of text-image pairs were generated using Qwen-VL-Max, they omit critical details such as the prompting strategies, filtering criteria, and validation mechanisms . The absence of concrete examples (e.g., sample prompts, annotation guidelines) hinders reproducibility and raises concerns about potential biases introduced by LLM-generated texts. Without transparent documentation, it is difficult to assess whether the dataset truly captures real-world referential ambiguity or inadvertently reflects the limitations of the underlying language model .\n\n[1] An efficient and effective transformer decoder-based framework for multi-task visual grounding. In ECCV, 2024.\n[2]Multi-task visual grounding with coarse-tofine consistency constraints. In AAAI, 2025.\n[3]Oneref: Unified one-tower expression grounding and segmentation with mask referring modeling. In NeurIPS, 2024.\n[4]Prompt-driven referring image segmentation with instance contrasting. In CVPR,2024.\n\nQUESTIONS:\nQ1: Could the authors explicitly state the primary scientific concern that SaFiRe aims to address—beyond merely introducing a new framework? Given the experimental limitations (marginal gains over Magnet, and absence of recent SOTA benchmarks like EEVG/C3VG), it is unclear whether the core contribution lies in cognitive modeling or practical performance advancement. \n\nQ2:Could the authors clarify whether the first SFLayer uses the final-layer features from Swin Transformer and BERT, and whether you use LAVT’s cross-modal fusion in the backbone stage?\n\nNote: A thorough response that resolves these concerns and demonstrates the solidity of experimental validations could potentially influence the evaluation score positively.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}