Teaching Diffusion Models to Ground Alpha Matte
Authors: Tianyi Xiang
Weiying Zheng
Yutao Jiang
Tingrui Shen
Hewei Yu
Yangyang Xu
Shengfeng He
Venue: Accepted by TMLR ()

Abstract: The power of visual language models is showcased in visual understanding tasks, where language-guided models achieve impressive flexibility and precision. In this paper, we extend this capability to the challenging domain of image matting by framing it as a soft grounding problem, enabling a single diffusion model to handle diverse objects, textures, and transparencies, all directed by descriptive text prompts. Our method teaches the diffusion model to ground alpha mattes by guiding it through a process of instance-level localization and transparency estimation. First, we introduce an intermediate objective that trains the model to accurately localize semantic components of the matte based on natural language cues, establishing a robust spatial foundation. Building on this, the model progressively refines its transparency estimation abilities, using the learned semantic structure as a prior to enhance the precision of alpha matte predictions. By treating spatial localization and transparency estimation as distinct learning objectives, our approach allows the model to fully leverage the semantic depth of diffusion models, removing the need for rigid visual priors. Extensive experiments highlight our model’s adaptability, precision, and computational efficiency, setting a new benchmark for flexible, text-driven image matting solutions.

Published in Transactions on Machine Learning Research (10/2025)
Teaching Diffusion Models to Ground Alpha Matte
Tianyi Xiang
tiansiing6--c@mmyccoco wwtp:ng...ltta......... edu.
Department of Computer Science City University of Hong Kong
Weiying Zheng
weijgnnzzhergy@ooomeed.b.bbbk
School Computing and Data Science The University of Hong Kong
Yutao Jiang
gm wttp:/ccch.mmmmmmml..omo
School of Computer Science and Engineering South China University of Technology
Tingrui Shen
2082200040113@mmaal.ccco sww..eu....eee..
School of Computer Science and Engineering South China University of Technology
Hewei Yu
hwyu@scuul.c edu.cn
School of Computer Science and Engineering South China University of Technology
Yangyang Xu
zugangyang@at..edu.mm
School of Intelligence Science and Engineering Harbin Instilute of Technology (Shenzhen)
Shengfeng Het
shengfenghe &smu.
School of Computing and Information Systems Singapore Management University
Reviewed on OpenReview: https: // openreview. net/ form? id= 2ge8y
Abstract
The power of visual language models is showcased visual understanding lasks, where language-guided models achieve impressive flexibility and precision. In this paper, we ex- tend this capability to the challenging domain of image matting by framing it soft grounding problem, enabling single diffusion model to handle diverse objects, textures, and transparencies, all directed by descriptive text prompts. Our method teaches the diffusion model to ground alpha mattes by guiding it through a process of instance-level localization and transparency estimation. First, we introduce an intermediate objective that trains the model accurately localize semantic components of the matte based on natural language cues, establishing robust spatial foundation. Building on this, the model progressively refines its transparency estimation abilities, using the learned semantie structure to enhance the precision of alpha matte predictions. By treating spatial localization and transparency estimation as distinct learning objectives, our approach allows the model to fully leverage the semantic depth of diffusion models, removing the need for rigid visual pri- ors. Extensive experiments highlight our model's adaptability, precision, and computational efficiency, setting new benchmark for flexible, text-driven image matting solutions. The code is available https::/ http:///w//ttttcoo/////////////a
prior
(Correpponding authors.

---

Published in Transactions on Machine Learning Research (10/2025)
Portrait Matting
Natural Image Matting (without Trimap)
A WOMAN in gray with Long hair Animal Matting
the transparent/on Left
net like ceiling
Video Instance Matting (without reference mask)
small white rabbit with beards
the male in dark blue shirt
Figure 1: We propose teach a single diffusion model capable of handling various matting tasks using task-specific text guidance (shown below each sample) with keywords marked in bold. By introducing language-driven priors, we unify diverse matting tasks into soft grounding framework. Our novel pipeline for teaching the text-to-image diffusion model achieves state-of-the-art performance on this challenging problem.
1 Introduction
Image matting is a longstanding and foundational task in computer vision, aimed at extracting a foreground object from an image and estimating the transparency of each pixel. Traditionally, this process is modeled by the following equation (Porter & Duff 1984):
I= oF + +1-
(1)
where only the input image 1 is known, while the alpha matte foreground F, and background B col- ors are unknowns. Solving this ill-posed problem has lod researchers to develop varions priors, inchuding
trimaps (Levin et al. 2007 Xu el al. 2017 Yao et al. 2024a), background estimates (Lin et al. 2021
binary masks YYu et al. 2021 Huynh al. 2024), and user-provided inputs (Ye et al. 2024 Li et al. 2024b).
However, these visual-level priors present limitations in many matting scenarios. For instance, trimaps require substantial annotation efforts, while the background and binary masks are unsuitable for dynamic scenes without temporal information. Additionally, interactive inputs are generally limited to static settings and lack generalizability for complex textures natural image matting (see the upper-right example in Fig. 1. Thus, there is a strong need for a unified and user-friendly approach.
Recently, visual-language approaches have achieved significant progress in dense visual prediction tasks (Gavrilyuk al. 2018 Ye et al. 2019 Luo et al. 2020 Wu et al.) 2022p. Notably, text-to-image dif- fusion models like Stable Diffusion (SD) (Rombach et al.
2022 have enabled tasks such open-vocabulary panoptic segmentation (Xu et al., 2023a) and referring image segmentation (Zhao et al. 2023a) by lever- aging capabilities for semantic diflerentiation and cross-modal allention. This significant advancement over previous visual-language models like CLIP [Radford et al. 202I), which primarily focus on image-level similarity and do not offectively capture the pixcl-level correspondences required for tasks like matting. The rich and powerful internal features along with the interpretable cross-atenntion maps naturally encode pixel-level visual- visual-language priors, making SD a promising candidate for matte grounding tasks through
2

---

Published in Transactions on Machine Learning Research (10/2025)
Original Image
SD
CLIP (Visual Branch)
Denoised image from SD
K-means clustering of features from different backbone
cat
standing on a table
Cross Attention Map in SD
Figure 2: Visualization of features from SD (Rombach et. al. 2022 and CLIP (Radford et al. 2021 h (left) and cross-attention maps in SD (right). SD exhibits strong semantic differentiation in its features and cross- modal attention maps, with both inter-correlation and intra-consistency.
natural language guidance, combining the expressive and intuitive nature of language with the strong prior knowledge embodded in the SD model. As visualizod in Fig. [ SD's semantic representations show strong differentiation and consistency both within and across object regions, making especially well suited for the fine-grained demands of advanced alpha matte estimation, which often has to resolve multiple similar objects and complex transparency patterns without relying rigid priors such as trimaps, masks, user interactions.
This raises natural question: can this prior knowledge be applied effectively to alpha matting? Unlike standard visual grounding tasks that typically produce binary masks for object identification, alpha matting requires both precise object localization and fine-grained transparency cstimation, with the alpha value a E [0, 1] in Eq. [1) accurately predicted. We define this unified approach as soft grounding, addressing both instanco-level localization and transparency simultancously for more adaptable and nuanced matting applications.
While SD holds significant potential, directly adapting it for soft grounding tasks introduces challenges due to the simultancous requirements of localization and transparency estimation. Semantic localization relies on high-level semantic understanding, whereas transparency estimation depends on fine-grained, low-level details. Jointly optimizing these sub-tasks within a single model often results in performance trade-offs due to their competing objectives. Previous adaptations of SD for portrait matting (Xu et al.) 2024a. Wang et al. 2024) rely on the assumption of prominent, casily distinguishable foreground object, which simplifies localization. However, this assumption is often invalid in more complex scenarios involving multiple objects or intricate transparency patterns, such as those encountered in natural image matting (Fig. [ which complicate the separation between foreground and background. Therefore, directly applying SD may yield suboptimal results due to the increased difficulty in resolving both localization and transparency.
In this work, we propose a framework to teach SD to ground alpha mattes with any user prompts. Rather than allempting to estimate the alpha matte directly, our approach introduces an intermediate objective that initially guides the model to localize semantic components of the alpha channel. Once this "teacher" model is trained for localization, we introduce a distillation process where "student" model leverages the teacher's learned semantic information to progressively refine transparency estimation. This sequential framework ultimately enables accurate alpha matte prediction with minimal post-processing
This approach offers two primary advantages. First, by avoiding continuous fine-tuning of the teacher model for transparency estimation, we preserve much of its pretrained semantic knowledge, maintaining a clear distinction between localization and transparency tasks. Second, the strong semantic foundation established
3

---

Published in Transactions on Machine Learning Research (10/2025)
by the teacher model enables us to employ more computationally efficient student model, enhancing the practicality of this approach for real-world applications.
Our main contributions are as follows:
We unify various matting tasks into a soft grounding problem, leveraging the Stable Diffusion model's visual-language capabilities.
We introduce distillation framework that teaches SD to ground alpha mattes by disentangling instanco-level localization and transparency estimation, enhancing performance in scenarios involv- ing multiple objects or complex transparency.
This framework enables optimization the student model's structure, improving computational efficiency for practical applications.
Extensive evaluations demonstrate that our method outperforms multiple baselines in soft grounding tasks, achioving competitive speed and generalizing well across different matting categories.
2 Related Work
2.1 Alpha Matte Grounding
Due to the inherent ambiguity in image matting, most existing methods rely heavily on trimaps as prior
guidance (Levin et al., 2007 He et al.] 2011 Chen et al., 2013 Xu et al.] 2017 Hou & Liu 2019 Park
et al. 2022 Wang et al. 2023b Yao et al. 2024a Xu et al., 2023b Hu et al.) 2024). However, generating
accurate trimaps that can distinguish foreground, background, and unknown regions is costly and time-
intensive. Consequently, recent research has focused on exploring alternative priors, including background
information (Lin ct al. 2021 Sengupta et al. 2020). binary masks
Park et al.
2023
Yu et al. 2021 Huynh
2024 Sun et al. et al.
2022
Li et. al.] 2024a Yang et al. 2025). in-contexl priors
Guo et al.
2024 and
interactive inputs such
as points, bounding boxes, and scribbles
Wei et al.
2021 Yang et al.
2022 Yao
ct al.
2024b
Ye et al.
2024 Li al.
2024b: Xia ct al. 2024).
Text-based priors, which offer more flexible and intuitive guidance by dynamically and autonomously iden- tifying matting objects, have also been investigated. For instance, Li et al. (Li et al. 2023a) introduce CLIPMat, which is the first text-based matting method leveraging pre-trained CLIP model (Radford et al. 2021) to integrate visual and textual features for matting. Xu et al. (Xu et al. 2024b) further enhance the feature fusion approach based on Li et al. (2023a). However, since CL.IP was designed for cross-modal similarity across entire images, it struggles with the high semantic precision and detailed requirements of pixel-level malting. In contrast, our method is based on a texl-to-image diffusion model that provides pixel-level visual-language priors and robust semantic differentiation, making it more suited the nuanced demands of image matting.
2.2 Diffusion Models for Image Matting
Diffusion models have shown great potential a variety of applications, including generative tasks (Rahman
et al. 2023 Shi et al.) 2024 Blattmann et al. 2023 Brooks et. al., 2023; Ruiz et al.] 2023 Wang et al.
2023a and dense prediction tasks (Ji ct al.] 2023 Zhao et al.) (2023a Xu et al., 2023a Lee al. 2024 Xu
et al. 2024a Burgert et al. 2023). Some studies have applied the prior knowledge encoded in these models
to image matting tasks.
Guo et al. (Guo et al. p024 leverage the in-context correspondence priors within Stable Diffusion (Rombach et al. 2022), using guidance from a reference image to perform matting of the same object across different scenes. Wang et al. (Wang et al.) 2024) approach matting as a generative task, applying denoising over multiple time steps and fine-tuning the model to predict the alpha matte. However, these methods depend on forecasting alpha mattes directly, limiting their effectiveness to matting a single object and often resulting in inefficiencies due to the high computational cost of diffusion models.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Image Encoder
Soft Grounding in Semantic (Sec. 3.2.1)
Alpho
Soft Grounding in Transparency (Sec. 3.2.2)
Alpha
Image
Text
Fecture Adopter
Morphulagica Operation
Toud exirr blcck
Extrect Semantic Masks
SD U-Net
GE
CLIP Text
Imsge
Feature
Adspter
Fad factor
Teacher U-Net o
"A brown yellow
Saft Semorttc Ground Result
From Soft Grounding to Alpha Matte (Sec. 3.4)
Transparenta Ayn Mining Loss
Semancie Guidance Loss
Original
Sep Cormesioon
Predict A pho
Denal
Wafting
LEARNABLE
Motting
Student U-Net
FROZEN
Loss
Text Fanture Adop ter
Confidance Wap
Algho
CONCATENATION
Soft Grounding Resul
UPSAMPLING
Saft Grounding
Readi
ST Alpha
blcck
Figure 3: Overview of the pipeline of our model, with unimportant operations and blocks omitted for clearer illustration. Motivated by the goal of disentangling two learning objectives, we first adapt the diffusion model to a sub-objective of instance-wise localization of semantic components of the alpha matte (blue part on tep-left). Next, we propose a distillation framework (orange part on the right) to distill the localization information from the teacher model (<%') into the student model (cf) using the Semantic Guidance Loss (Lsg) based on interuel features, while encourage e, to simultaneously explore the transparent-related information via the Transparency Mining Loss (LSTM, LCTM) based attention maps. o learns to predict the coarse alpha matte " and the transparent confidence map R to further enhance a into at full resolution (green part on bottom-lcft).
Alternatively, some methods operate directly in pixel space by diffusing a disturbed trimap (Xu et al. 2023b) or pure noise (Hu et al., 2024) until a clean alpha matte is produced. Li al. (Li al. [024cc] extend this approach to latent space, introducing modified self-attention better model matting context. However, these approaches rely hcavily on trimap conditioning input, limiting their flexibility in practical applications.
3 Method
3.1
Overview
Problem Formulation. Given an input image E xx33 and a foreground text expression T, our goal is to teach the Stable Diffusion (SD) model soft grounding and predict the alpha matte a E [0, x x1 in a single step.
Network Design. Our framework illustrated in Fig. 3 To teach the SD model ground the alpha matte, we disentangle the soft grounding problem into two sub-objectives using an intermediate objective, soft semantic grounding, along with distillation framework. First, the original diffusion model is trained to localize the semantic components of the target alpha matte (see Sec. 3.2.1 Then, asymmetric distillation framework with two tailored objectives guides the model to refine transparency estimation while preserving the localization capability learned in the initial stage (see Sec. 3.2.2p. This approach ensures effective lask disentanglement. Moreover, the distillation framework enables the adoptiio of a computationally efficient model for soft grounding, improving its practicality (see Sec. 3.3). Using the soft grounding results and output features from the well-taught diffusion model, the matting decoder can readily predict the final full-resolution alpha matte (see Sec. 3.4p. We discuss our proposed paradigm below.

---

Published in Transactions on Machine Learning Research (10/2025)
3.2 Soft Grounding Teaching
3.2.1 Soft Grounding in Semantic
We first teach the diffusion model to address an intermediate task: localizing each semantic component of the target alpha matte. This is achieved by fine-tuning the denoising U-Net (ex) starting from the original weights of SD. Specifically, taking the latent code 2 == E() at resolution of H. x W and the corresponding g
linguistic features JT conditional input, ég is fine-tuned to distinguish the semantic regions of alpha matte (the foreground, background, and transparent regions) and to predict classification results for each region. The results are represented $1 E
5 x x3 of I. To achieve this, replace the last latent prediction head in the original ea with a new head comprising two groups of ConyBNReLU layers to predict 87. The linguistic features fr are generated by the frozen CLIP text encoder (Radford et al. 2021] and further refined using a two-layer MLP text adapter, inspired by Gao et al. (2024). We train the model to predict classification probabilities Pı LCE:
[0, 1] [ 4x x3 for each channel in S1 by minimixing the cross-entropy loss
CCE =
Σ & log(pi),
(2)
where sGr represents the GT region representation, obtained by applying morphological operations to the GT alpha, and pf denotes the classification probability for the e-th channel in 87. We denote the fine-tuned co as cg.
3.2.2 Soft Grounding in Transparency
After the convergence of training c,, our next goal is to further explore transparency information based on e until achieving soft grounding. A straightforward approach would be to fine-tune t to predict transparency directly. However, this presents two key challenges. First, fine-tuning oØ for transparency estimation risks disrupting its well-learned semantic knowledge. Second, the computational cost of e is typically high, limiting its practicality for soft grounding. To address both challenges, we propose a novel soft grounding distiilation framework. This framework fully exploits the strong semantic guidance embedded in the intermediate features of the pre-trained eă while enabling the student model (cş) to refine transparency estimation in the corresponding alpha matte. As result, oΦ expected to gain full capabilities for solving the W x1 soft grounding problem. For implementation, eÿ is trained to predict coarse alpha matte at [0, and a transparent confidence map R R x xx1, which indicates regions that may be transparent. These outputs are then used for further upsampling and refining at to a. Next, we introduce two key losses to effectively guide this distillation process.
Strong Semantic Guidance Loss. Inspired by Kim et al. 2022 we introduce a feature-level distillation loss LsG, to ensure that the semantic representation of the student model eg closely aligns with that of the teacher model cf, at an intermediate feature level, formulated as:
EsG =
Σ [lM F - M o (FF))E.
(3)
where Ft denotes the intermediate feature of E at layer (and similarly, F1 for the student model). Here, M is a binary mask with foreground and background regions set to 1 and other regions set to 0. M ensures that semantic supervision dose not interfere with the exploration of transparency details. The symbol c represents the Hadamard product. The projection module T aligns the intermediate features the two models, accounting for differences in their feature spaces due to distinct learning objectives. It consists of 3 learnable convolutional layers (Cv) and an intermediate LeakyReLU layer, defined as:
+ (F) = CV (layxaa(eeayyeeeee
(4)
Transparency Mining Loss.
To further encourage ef to explore transparency details, we aim to extract and mine potential transparency information from o We begin with the self-attention maps of o7 a critical component for modeling intra-object consistency. Since og possesses wear-learned semantic knowledge of alpha, the implicit transparency information used to identify such regions can serve as transparency cues,
6

---

Published in Transactions on Machine Learning Research (10/2025)
guiding ej to further refine transparency details. Given the self-attention map A, SoftMax( 9r-k, derived from feature f. we introduce a self-attention-based transparency mining loss LSTM, to constrain affinities specifically related to the alpha matte. To accomplish this, we propagate the GT alpha, OcT, using the averaged self-attention map, ASA, computed across all self-attention layers at the same resolution. This propagation is performed via. matrix multiplication (AsA @ OCT). We then minimize the difference between the propagated results of the teacher and student models. The rationale is as follows: ASa acts as a transition matrix, while oGr represents the current state. Their matrix multiplication transforms the current state into a resulting state. By constraining this resulting state, we indirectly regulate the a-related affinitics within the self-attention map. This approach enhances the exploration of transparency details without incurring significant computational overhead, as it avoids constraining the entire sell-attention map.
va
We formulate LgTM as follows:
CSTM = -IA acr A @ acTl
where & denotes matrix multiplication (not element-wist multiplication). Note that ogr down-sampled the corresponding resolution using bilinear interpolation to satisfy the requirements for @. We also extract potential transparency cues from the text expression using similar cross-attention map distillation loss LCTM, which directly constrains the cross-attention maps between eğ and eØ formulated as:
LCTM ||ACA AL
(6)
For implementation, all attention maps are grouped by resolution and averaged to compute the loss. The final attention distillation loss is then calculated as the mean of the losses across all resolutions.
Total Objectives. In addition to the two key losses described earlier, we incorporate an L1 loss to train af and a binary cross-entropy (BCE) loss to train R. These are defined as follows:
Lat: =11a" atrlll..
ll
LR - M
log( Mlog(
e (1 M) log( c
(8)
where a'rr represents the down-sampled GT alpha matte, and M := M binary mask that indicates transparent regions. The final objective, Leg, used to train eg, is formulated as:
Lgg = ASTMCSTM + ACTM + AscCsa + Aat + XRCR,
(9)
where the À coefficients are hyperparameters that balance the contribution of each loss term.
3.3 Structural Optimization on E
In our distillation framework, LsG provides strong supervision signals to preserve localization knowledge by directly minimizing discrepancies in the semantic content of internal features. This highlights a key advan- tage of our approach: achieving a balance between maintaining performance and enhancing computational efficiency (see Sec. 3.2.2 To fully exploit this advantage, we apply two primary structural optimization techniques to the vanilla eg (which initially shares the same architecture and parameters as cg)): block pruning and self-attention optimization.
Block Pruning. We prune redundant blocks from the vanilla student model following Kim et al. (0224) resulting a model with fewer parameters. Guided by Lsg, the performance degradation caused by this pruning is effectively mitigated. Additionally, the other loss terms encourage the student model to further assimilate soft grounding knowledge.
Self-Attention Optimization. The self-attention mechanism in Transformer blocks is critical compo- nent for modcling long-range dependencies. However, the matrix multiplication operations in self-attention are compulationally expensive, resulting in an overall time complexity of O(n2d + nd²), which becomes particularly burdensome at higher resolutions, Nevertheless, the soft grounding task has more relaxed accu- racy requirements, as the learned outputs "a and R) provide sufficient flexibility to tolerate errors in the subsequent alpha enhancement process. Inspired by this, we propose that the self-attention operation can be optimized into more computationally efficient form by learning asymmetric sparse correspondences.

---

Published in Transactions on Machine Learning Research (10/2025)
Specifically, we argue that the dense affinity matrix derived from the self-attention map A can be simplified into unidirectional sparse representation. To achieve this, we can learn smaller set of representative feature tokens and use their affinities to approximate the affinities between the original feature tokens and others within a relatively small region. Thus, we introduce a learnable down-sampling operation Ø, which is applied to ʃ when calculating K and (i.e., K$ = WxO(J), V = Wr)(()) while Q remains unchanged. & can be implemented using a single convolution layer with a kernel size and stride of k x k where is chosen from {2'11 < 2 logzhagh and hay denotes the height of the square matrix A₁, theoretically. The quantitative analysis of the reduction in computational cost can be found in the Appendix. Note that LSTM contributes to preserving intra-object cohesion and textural information within the optimized self-attention mechanism.
3.4 From Soft Grounding to Alpha Matte
To up-sample af from a resolution of # W. to at x W, we adopt the detail encoder-decoder structure proposed in Yao et al. (2024a). The encoder takes the concatenation of (I, a²t, R) as input to extract detail features. Before that, " and R are interpolated to match the resolution of I, and R. is processed through a sigmoid function. The dctail features from the encoder's final layer are concatenated with features extracted from E and then passed to decoder to predict the final alpha malte a. For more elficient training, freeze eÿ and train only the parameters of the encoder and decoder. Additionally, single learnable ConvBNReLU block is applied to the output features from eg before concatenation to align the feature spaces. This process is supervised using an L1 loss and a Laplacian loss following Li et al. (2022).
Extend to High-resolution Inference. We implement a simple upsampling module based on sparse convolution (Contributors 2022), integrated with the matting decoder, to enable inference at arbitrary high resolutions (up to 2K). This sparse convolution approach effectively reduces computational cost and memory usage during the upsampling process.
4 Experiment
4.1 Implementation Details
Data Acquisition. The data used to train our model comprises 4 matting datasets (RefMatte (Li et al. 2023a),, P3M10K (Li ct al. 2021a), AM2K (Li ct al. 2022), RMIK (Wang et al. 2023b ), and 1 grounding segmentation dataset (ReICOCO (Kazemzadeh el al., 2014 Considering there are text annotations for P3M10K, AM2K, and RMIK, we adopt BLIP2 (Li et al. 2023b) to generate text annotations for each sample in these datasets by guiding the BLIP2 model to describe the appearance of the object in the imago. For samples from RefMatte and RefcOCO during training, we randomly select one expression for each object if multiple expression annotations exist for the same object. Since there are no matting-level annotations for RefCOCO, we first generate the pseudo trimap according to the mask annotations using morphological operations, then use Yao et al. (2024a) to obtain alpha annotations.
Training Details. All stages of our model's training process adopt a consistent data scheduling strategy. Specifically, we train the model on RefMatte during odd-numbered iterations and on RefCOCO in every even iteration. We also insert a special iteration after every 4 iterations to perform training on P3M10K, AM2K, and RMIK. We randomly select. dataset among the 3 to train our model in this special iteration. We set the kernel size of the morphological operation to 15, and we set (ASTM; λCTM, ÀsG, Agte, Axx) to (10, 0,1.0.1.... 0.5, 10, 1). The timestep input of the SD model is set to 1.0 during both training and inference, which is consistent with previous works (Zhao et al. 2023a Lee et al., 2024 Xu et al., 2024a). This setting can transform the multi-step SD model into a deterministic single-step perception model effectively. Other training settings, including batch size, learning rate, total iterations, and rationales behind setting As, can be found in the A Appendix.
Details about Optimized c. To thoroughly evaluate our proposed framework, we adopt the most extreme pruning scheme proposed in Kim et al.) (2024) to build c,, the tiny setting, which removes blocks in the encoder and blocks in the decoder, and removes the entire middle block in eg. For optimixing self-attention
8

---

Published in Transactions on Machine Learning Research (10/2025)
(SA), directly remove all SA layer at resolution of 64 x 64 following Zhao et al. (202bb) and set the k value for SA layers at 32 32 and 16 x 16 (kaza, k162) as kzza- = -16-= = 2.
4.2 Evaluate Metrics
Following Rhemann al. (2009), we adopt matting metrics for evaluation, including SAD, MSE, Gra- dient Error (GRAD), and Connectivity Error (CONN). These metrics are scaled by 103, 10-3, 3 103, 103, respectively. Lower values indicate better performance across all metrics.
4.3
Comparison on Soft Grounding
Baselines. To comprehensively evaluate our model's performance, we compare it with three types of baselines.
Text-guided matting methods. We select CLIPMat (Li ct al. 2023a) as a fundamental basc- line, which solves the soft grounding problem directly and shares similar settings with ours. Since CLIPMat is currently closed-source, we re-implement and use the same training settings as ours.
Visual grounding with interactive matting methods. We select the representative Ground- ingDINO (GDINO) (Liu ct al., 2024b) the visual grounding method, which producos the bounding box of the foreground object according to the text input. Then select 3 recent interactive mat- ting methods to obtain the alpha given the bounding box, including MatAny Yao et al. 2024b MAM Li et al. 2024b). and SmartMat Ye et al. 2024).
Grounding segmentation with mask-guided matting methods. We also select the latest grounding segmentation method PSALM (Zhang et al.) 2024) and the SD-based referring segmen- tation method ReIVPD (Zhao et al., 2023a) to generate a mask for the foreground object based text input. We then apply two mask- matting methods to derive the alpha matte from the mask, including MaGGIe (Huynh et al. 2024 and MGMat (Yu et al. 2021
For fairness, all basclines with cascaded structure, including the visual grounding part (except PSALM and GDINO) and matting part, are aligned to our training data through fine-tuning. Note that PSALM and GDINO are trained on very largo-scale datascts. We attempted to fine-tune them directly small-scale training dataset, but this yielded poorer results. Consequently, we use their original weights for further comparison.
relatively
Note that among all matting components in the basclines (including MatAny, MAM, SmtrtMat, MGMatting, and MaGGIe), only MaGGle was trained for human matting exclusively, while the others natively support natural image matting. Although MaGGTe is trained for human matting, its network architecture does not incorporate specific optimizations tailored exclusively for human subjects. Furthermore, the authors of MaGGIe demonstrated its strong zero-shot generalization capabilities non-human matting tasks in their paper, indicating that MaGGIe can adapt to natural image matting to some extent.
Benchmarks. We apply two referring natural matting benchmarks, including RefMatte- Test (Li et al. 2023a) and SeMMatt-RWW10 (Li et al.] 2023a), for soft grounding evaluation. Here, the former is a com- position dataset (6,243 instances among 2,500 images) and the latter a real-world dataset (221 instances among 100 images). Every instance these two benchmarks has 4 diffcrent expressions, so we evaluate all baselines and ours using all expressions and report the average result among these expressions. During evaluation, the input resolution for all methods is set to 512 512, and the metrics are also calculated on this resolution. We also report the average inference time per sample in milliscconds, using the same machine with a single RTX 3090.
Quantitative Results. We show the quantitative comparison on soft grounding in Tab. [ We found a sig- nificant performance gap between CLIPMat and ours since the fcature and cross-modal prior within SD model make it casier to solve the soft grounding problem compared with CLIP. Our model also achieves the best performance most of the metrics compared with the cascaded baselines, even when training data is aligned
9

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Comparison on soft grounding task. Our model outperforms all baselines on two referring natural image matting benchmarks (RofMatto-Testsse and RefMatto-RW 100) with faster inference time,
demonstrating the elfectiveness and practicality of our method.
RefMatte- Testset
RefMatte-RWio0
Inference
Methods
SADI
MSE4
GRADD
CONN
SADJ
MSR
GRADD
CONN
Thme. (ms)
Grounding Model
MatAyy
15.52
0.0561
8.41
3.61
19.92
0.0727
9.57
7.71
984.37
GDINO+MAM
15.16
0.0552
8.92
4.59
16.96
0.0026
8.56
10.78
496.30
Interactive Matting
GDINO+ SmartMat
11.09
0.0403
7.91
1.78
16.91
0.0616
9.30
5.54
122.29
MaAGMe
8.71
0.0299
7.63
2.67
9.90
0.0349
8.21
3.75
287.79
Gounding Segmentation Model
PSALM-MGMatt
8.55
0.0301
2.57
9.41
6.83
3.81
272.21
Mask guided Matting
+
0.0308
8.36
3.23
11.41
0.0399
9.64
5.63
220.79
ReVVD+MGMar
8,98
0.0315
7.89
3.33
10.32
0.0371
7,66
5.37
206.21
Soft. Grounding
CLIPMat
26.56
0.1181
14.84
8.15
12
0.1951
21.01
18.97
102.74
Our
3.19
0.0098
3.55
1.69
7.37
0.0264
6.55
5.31
95.14
Image
G'I'
GD+ MatAny
GD+ MAM
GD+ MatrtMat
PS+ MaGGIe
PS+ MGMat
RV+ MaGGIe
RV+ MGMat
CLIPMat
Ours
Figure 4: Qualitative comparison on soft grounding task. The text inputs from top to bottom are: 1) the insect which is darkgray; 2) the white and non-salient and transparent net; 3) a modern-designed glass with a black frame; 4) the red chair made of wood.
among all matting networks. Notably, with all training data aligned, our model still achieves significant per- formance improvement comparing with two strong basclines (RefVPD+MaGGIe and RefVPD+MGMan This evidence highlights that the performance gains primarily stem from our novel soft grounding prob- lem modeling and the distillation framework, demonstrating the importance of our proposed algorithm to the performance gain. Morcover, although adopting PSALM, which based on Large Multi-modal Model (LMM) for grounding segmentation, can obtain lower connectivity error on RW100 benchmark, the other three metrics are still worse than ours and have relatively high inference time. Instead, our framework can teach oootiiize model to achieve the best performance with lower inference time, showing the effectiveness of our proposed framework.
Some qualitative comparisons are shown in Fig. 13 Without properly modeling soft grounding, bascading-based baselines often perform poorly with incorrect semantic and low-quality details as shown in the first two rows. Although some basclines adopt SAM (Kirillov et al. 2023 or LMM {Liu let al. 2024a), they still show sub-optimal performance on soft grounding the real-world (last two rows). More comparison results can be found in the Appendix.
Qualitative Results.
4.4 Comparison on Generalization Ability
Introducing text prior can unify various matting tasks into soft grounding. To evaluate the generalization ability of our model on such fino-grained matting tasks, we select several specialist methods in their own task to compare with our model quantitatively (Tab. 2). All basclines here are directly applied using the officially released weight, and our model is tested without fask-specific tuning. All the metrics are calculated in full
10

---

Published in Transactions on Machine Learning Research (10/2025)
Quantitative comparison on generalization ability across different matting tasks. Top-2 rosults are marked in bold and underlined. Task-specific methods only perform well their own task, while ours can generalize to various matting tasks using text-based guidance without fine-tuning.
Table 2:
AM2K
P3M-P
RefMatto- Test (referring natural)
Method
Type
(animal)
(portrait)
SAD
MSEI
SADL
MSE
SAD1
MSEJ
GFM [Li
2022
animal
12.08
0.0035
347.44
0.2001
239.33
0.1335
P3M- VITAE Ma et
2023
portrait
40.43
0.0204
6.59
0.0015
290.05
0.1616
GenPercept Xu ct al
2024a
portrait
19.04
0.0049
11.02
0.0025
269.50
0.1456
AIM [L 2021bb
natural
28.25
0.0101
45.41
0.0207
336.76
0.1840
Ours
13.81
0.0045
9.53
0,0030
24.19
0.0112
P3M-VITAE (portrait)
GenPercept (portrait)
Image
GT
GFM (animal)
ATM (natural)
Ours
Figure 5: Qualitative comparison on generalization ability across different matting tasks. The text imputs used in ours are (from top to bottom): 1) mother and daughter in field at sunset; 2) a jaguar is silling on top of a rock; 3) the works which are thistle and nom-trunspuuren.
resolution. For all test samples in these benchmarks, we fixed the prompt input of ours as "the foreground person" for P3M-P and "the foreground animal" for AM2K. This ensures that the input conditions for model and the basclines consistent, enabling fair comparison. We found that although the baselines excel in their designated tasks, they often lack generalization and struggle with referring matting across diverse categories. In contrast, our model, leveraging text prior, generalizes effectively to various matting tasks, though it slightly lags behind task-specific experts in their respective domains. Most importantly, our model shows unrivaled performance on referring natural matting tasks, where other baselines falter. We also showcase some qualitative results Fig. 5 We found GFM and P3M lack generalization, while Gen GenPercept and AIM predict over-smoothed alpha. All of them have poor performance on the soft grounding task.
4.5 Ablation Studies
To fully demonstrate the necessity of our key design and modules in our model, we conduct following ablation studies via different training settings and quantify results Tab. 3 using RefMatte-Ri HEEMatte-RW10 as the benchmark.
Problem Disentanglement. First, directly training SD for soft grounding proves highly challenging, resulting in suboptimal performance (row 1) compared to disentangled formulation.
Knowledge Distillation (KD). Next, we explore a two-stage approach where SD is pre-trained for local- ization and then fine-tuned for soft grounding (row 2). This significantly improves performance but remains inferior to our complete KD framework (row 5). These results highlight the importance of preserving se- mantic knowledge and explicitly separating the two sub-objcctives. Furthermore, fine-tuning light eg without KD leads to substantial performance drop (row 3), underscoring the critical role of KD in maintaining balance between efficiency and performance.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: Ablation studies. For instance, (eg SG) means training og on soft grounding task. "0--
denotes the distillation framework. eg is structural optimized to lightweight in row 2'6
Training scttings
Mat. E-D
#
SAD4 MSE) GRAD1 CONN1
SG=Soft Grounding, Local, Localiaiociiiition)
SG)
17.50 0.0645
9.45
8.27
2
(cg
Local.), then fine-tune (c)
SC)
9.75 0.0375
7.89
6.48
3
[<]
Local.), then fine-tune (e)
SG)
13.27 0.0487
9.01
6.73
Local.), (cg
SG)
15.96 0.0534
9.26
Local.), (cg
SG)
7.63 0.0267
6.98
5.35
6
Local.), (eg
SG)
7.37
0.0264
6.55
5.31
Local.), (c) 22 4 (w/o. opti.)
SG)
6.30 0.0223
5.52
4.33
Multi-instance
Low-light
Overlapping
"two women the picture" "
"the man wearing sweater"
"a man in light color shirt"
Figure Robustness Evaluation. Our method demonstrates strong performance across a variety of challenging scenarios.
Other Network Components. We also observe that removing the feature projection module (Φ) confuses ci during distillation, as eØ and eg have distinct learning objectives, leading to degraded performance (row 4), Additionally, integrating a dedicated malling encoder and decoder (Mat. E-D) (row 6, complete framework) further improves the quality of the predicted alpha matte.
Structural Optimization. Lastly, we evaluate the impact of structural optimization. While the vanilla eğ without structural optimization (row achieves higher performance, its inference time is significantly longer (143ms per image under the same setting in Sec. 4.3 Although there is slight performance degradation, our framework effectively teaches the lightweight model to achieve impressive performance with much lower inference time, it is important to note that our optimized model still outperforms most baseline methods, as demonstrated in our evaluations (see Tab. [], validating the effectiveness of our framework in modeling soft grounding. Furthermore, the optimized student model achieves a significant 33.46% reduction in inference time (from 143ms 95.14ms). This reduction particularly valuable in practical applications, such mobile devices, where lower computational requirements enhance user experience. In future work, we aim to explore more advanced optimization strategics to achieve real-time matting while further minimizing performance degradation.
Moreover, with the same number of parameters, comparing with simply inheriting the SD model (row 1), our complete framework (row 7) achieves a significant performance improvement, demonstrating the importance of our proposed algorithm to the performance gain.
More ablation studies on hyperparameters, objectives, and SD) versions are provided in the Appendix.
4.6 Robustness Evaluation and Limitations
Given accurale and well-structured lext prompts, our model demonstrates strong robustnness against various typical disturbances (see Fig. 6 However, like all prompt-based based systems (Zhang et al. 2025 Wu et al. 2025 Qu al. 2025 Wen et al. 2025), susceptible vague or ambiguous prompts, which can degrade performance (see Fig. 7). Future work could explore prompt engineering techniques or adaptive guidance strategies to enhance robustness and flexibility across a broader range of matting scenarios.
12

---

Published in Transactions on Machine Learning Research (10/2025)
Input image
Ours
Input image
Ours
"there is a" (Incomplete)
"yellow Hlwerr"
(Ambiguity)
Figure 7: Some failure cases of our method caused by ambiguous prompts.
DiffMatte (Hu et al 2024
DiffMatte Hu et al 2024
Input image
GT
(Given mannal trimap)
(Given sogmentation-mee trimap)
Ours
Figure 8: Qualitative comparison with pixel-space diffusion matting method DiffMatte. The text inputs of ours are (from top to down): 1) a beautāful young woman with short wany hair; 2) beautiful young woman with long brown hair and blue eyes; 3) a woman with headphones on looking out at the ocean. DiflMatte heavily relies on carefully annotated manual trimaps to produce accurate alpha malles, while our method only needs text prompts and can achieve comparable performance, despite the latent-space diffusion structure may lead to minor dotail loss.
Besides, inheriting biases from pretrained Stable Diffusion models is another inherent limitation of our approach. These biases may impose an upper bound on our model's ability interpret complex prompt content, as this capability is constrained by the pretrained model's understanding of such prompts. For instance, SD models often struggle with concepts requiring precise counting, such prompts like "the eleventh person from the left", leading to difficulties in accurately interpreting such positional information. Additionally, biases may adversely affect performance in handling images with challenging conditions, such as those with highly complex lighting or shadows, potentially limiting the general applicability of our model. These limitations stem from the SD model itself. However, we emphasize that the primary focus of our work is not to perfect the diffusion backbone but to effectively adapt it for novel task. This distinction underscores the unique and significant contributions of our approach in leveraging pretrained models for targeted applications. We also note that future advancements in vision-language models with advanced both prompt and visual understanding capabilities could further enhance progress in soft grounding tasks, and we look forward to such developments. In addition, some structures in SD designed specifically for generation lasks (such as downsampling in VAE) may cause accuracy loss in matting lasks, although our Stage 3 is designed to alleviate this as much as possible. However, compared to pixel-space diffusion matting models (c.g., DiffMatte (Hu al., (024), see Fig 8), which strictly rely on perfect manually annotatod trimaps predict high-quality alpha mattes, our model achieves comparable performance in matting accuracy using only text prompts guidance, demonstrating better practicality and flexibility of approach in real-world
13

---

Published in Transactions on Machine Learning Research (10/2025)
applications. We expect that future work will integrate VAE with lossless image compression technology to more effectively address this problem.
Also, inheriting biases from training data may affect our model. For example, the matting model currently used as a tool for labeling GT alphas in RefCOCO may produce labeling errois in some certain situations, or the overall training data may not. cover certain visual scenes with extreme lighting or environmental conditions. This will affect the generalizability of our model. We look forward to future developments in referencing image matting datasets that are larger, cover more real-world scenes, and have more accurate annotations to support progress the soft grounding problem.
5 Conclusion
In this paper, we unified various matting tasks a soft grounding problem, addressing both instance-level localization and transparency estimation using the visual-language capabilities of Stable Diffusion. Instead of directly training SD, we introduced distillation framework that separates localization and transparency tasks, improving performance in complex scenarios with multiple objects or intricate transparency patterns. This framework also enables a more elficient student model, balancing complexity and accuracy. Extensive evaluations demonstrate our method's effectiveness, with competitive speed and strong generalization across diverse matting categories.
Broader Impact Statement. Our method advances texl-guided image malting but could be misused for image manipulation deepfakes, thus we encourage responsible in creative applications. Besides, since our method leverages pretrained diffusion and vision-language models, it may inherit and amplify biases present in the training data. These biases could manifest as unequal performance across demographic groups or systematic failures in underrepresented categories. Future research should carefully evaluate and document such biases, and explore mitigation strategies such as balanced data curation, prompt robustness analysis, or bias-aware training objectives. Moreover, the performance prompts, and vague or ambiguous prompts can lead to degraded or unintended results. This raises potential concerns about fairness and reliability when the system is deployed in practical scttings. To address this, future work may incorporate adaptive prompting strategies, human-in-the-looe corrections, uncertainty estimation mechanisms to ensure more robust and transparent usage.
our method is inherently sensitive to lext
Acknowledgements. This rescarch is supported by the Guangdong Natural Seience Funds for Distin- guished Young Scholars (Grant 2023B1515020099 Natural Science Foundation of Guangdong Province (2023A 5515099))) Key R&D Project of Guangzhou Science and Technology Plan (2023B01J0002), Na- tional Natural Science Foundation of China (Grant No.: 625021177 the National Research Foundation Sin- gapore under its AI Singapore Programme (AISG Award No: AISG3-GV-2023-01 Foundation Singapore under the AT Singapore Programme (ATSG Award No: ATSCAA-TCC20)3-00--CCKT)) the Singapore Ministry of Education AcRF Tier Grant (Grant No.: MSS25C004), and the Lee Kong Chian Fellowships.
the National Rescarch
References
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiu:2502. 1225602.3390 13923, 2025.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pp. 22563 22563-22575, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, pp. 18392 18402, 2023.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peckaboo: Text to image diffusion models are zero-shot segmentors. In CVPRW, 2023.
Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Kon matting. TRFE TPAMT, 35(9):2175 2188, 2013.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Spconv Contributors. Spconv: Spatially sparse convolution library. hutpp:///ggttuub.ooo//rrvraalll/ spconv 2022.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024.
Zihao Fu, Anthony Man-Cho So, and Nigel Collier. A stability analysis of fine-tuning a pre-trained model.
arXiv preprint arXi::3011008800 2023.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao, Clip-adapter: Better vision-language models with feature adapters. IJCV, 132(():581-595, 2024.
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Coes GM Snock. Actor and action video segmentation from sentence. In CVPR, pp. 5958 5966, 2018.
He Guo, Zixuan Ye, Zhiguo Cao, and Hao Lu. In-context matting. In CVPR, pp. 3711 3720, 2024.
Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. A global sampling method for alpha matting. In CVPR, pp. 2049-2056, 2011.
Qiqi Hou and Feng Liu. Context-aware image matting for simultaneous foreground and alpha estimation. In ICCV, pp. 4130-4139, 2019.
Yihan HU, Yiheng Lin, Wei Wang, Yao Zhao, Yunchao Wei, and Humphrey Shi. Diffusion for natural image matting, In ECCV, pp. 181 199, 2024.
Chuong Huynh, Seoung Wug Oh, Abhinav Shrivastava, and Joon- Joon-Young Lee. Maggie: Masked guided gradual human instance matting. In CVPR, pp. 3870-3879, 2024.
Yuanfeng Ji, Zhe Chen, Euze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In ICCV, pp. 21741 21752, 2023.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In EMNLP, pp. 787-798, 2014.
Bo-Kyeong Kim, Hyoung-Kyu Song, Thibault Castells, and Shinkook Choi. Bk-sdm: A lightweight, fast, and cheap version of stable diffusion. In ECCV, pp. 381 399, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. In ICCV, pp. 4015-4026, 2023.
Hsin- Hsin-Ying Lee, Hung-Ye Yu Tseng, and Ming-Hsuan Hsuan Yang. Exploiting diffusion prior for generalizable dense prediction. In CVPR, pp. 7861-7871, 2024.
Anat Levin, Dani Lischinski, and Yair Weiss. A closed-form solution to natural image matting. JEEE TPAMT, 30(2):228 242, 2007.
Jiachen Li, Roberto Henschel, Vidit Goel, Marianna Ohanyan, Shant Navasardyan, and Humphrey Shi.
Video instance matting. In WACV, pp. 6668 6677, 2024a.
Jiachen Li, Jitesh Jain, and Humphrey Shi. Matting anything. In CVPRW, pp. 1775-1785 2024b.
Jizhizi Li, Sihan Ma, Jing Zhang, and Dacheng Tao. Privacy--reserving portrait matting. In ACM MM, pp. 3501 -5009, 2021a.
Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. IJCAI, pp. 800 806, 2021b.
Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. IJVV, 130:246 266, 2022.
15

---

Published in Transactions on Machine Learning Research (10/2025)
Jizhizi Li, Jing Zhang, and Dacheng Tao. Referring image matting. In CVPR, pp. 22448 12448, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languago-image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742, 2023b.
Xiaodi Li, Zongxin Yang, Ruijic Quan, and Yi Yang. DRIP: Unleashing diffusion priors for joint foreground and alpha prediction in image matting. In NeurIPS, 2024c.
Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Samellmacher-Shiizemnn Real-time high-resolution background matting. In CVPR, pp. 1762-8777 2021.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction luning. NewrIPS, 36, 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, 2024b.
Πly Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Liujuan Cao, Chenglin Wu, Cheng Deng, and Rongrong Ji. Multi- task collaborative network for joint referring expression comprehension and segmentation. In CVPR, pp. 10034 10043, 2020.
Sihan Ma, Jizhizi Li, Jing Zhang, He Zhang, and Dacheng Tao. Rethinking portrait matting with privacy preserving. IJC,, 131(8):2172 2197, 2023.
Marius Mosbach, Maksym Andriushchenko, and Dictrich Klakow. On the stability of fino-tuning bert: Misconceptions, explanations, and strong baselines. arXiv preprinl arxv::2066.04 04884, 2020.
GyuTae Park, SungJoon Son, Jaung Yoo, SeHo Kim, and Nojun Kwak. Matteformor: Transformer- basod image matting via prior-tokens. In CVPR, pp. 11696- 2022.
Kwanyong Park, Sanghyun Woo, Scoung Wug Oh, In So Kweon, and Joon- Young Lee. Mask-guided matting in the wild. In CVPR, pp.
2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint ar Xrv:23307.0193 2023.
Thomas Porter and Tom Duff. Compositing digital images. In SIGGRAPH, pp. 253-259, 1984.
Xiangyan Qu, Gaopeng Gou, Jiamin Zhuang, Jing Yu, Kun Song, Qihao Wang, Yili Li, and Gang Xiong, Proapo: Progrossively automatic prompt optimization for visual classification. In CVPR, pp. 25145-25155, 2025.
Aloc Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8748-8763, 2021.
Tanzila Rahman, Hsin-Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make-a- story: Visual memory conditioned consistent story generation. In CVPR, pp. 2493-2502, 2023.
Christoph Rhemann, Carsten Rother, Jue Wang, Margrit Gelautz, Pushmeet Kohli, and Pamela Rott. A perceptually motivated online benchmark for image matting. In CVPR, pp. 1826-1833, 2009.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-rosolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, pp. 22500- 22510, 2023.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steven M Seitz, and Ira Conelmacher-Shlizeerne Background matting: The world is your green screen. In CVPR, pp. 2291-2300, 2020.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent. YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive point-based based image editing. In CVPR, pp. 9899,8849, 2024.
Yanan Sun, Chi-Keung Tang, and Ying Tai. Human instance matting via mutual guidance and multi- instance refinement. In CVPR, pp. 2647 2656, 2022.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, pp. 12619-1 12629, 2023a.
Yanleng Wang, Lv Tang, Yijie Zhong, and Bo Li. From composited to real-world: Transformer-based natural image matting. IEEE TCSVT, 2023b.
Zhixiang Wang, Baiang Li, Jian Wang, Yu-Lun Liu, Jinwei Gu, Yung- Yu Chuang, and Shin'ichi Satoh. Matting by generation. In SIGGRAPH, pp. 1-11, 2024.
Tianyi Wei, Dongdong Chen, Wenbo Zhou, Jing Liao, Hanqing Zhao, Weiming Zhang, and Nenghai Yu. Improved image matting via real-time user clicks and uncertainty estimation. In CVPR, pp. 15374 15383, 2021.
Zhen Wen, Luoxuan Weng, Yinghao Tang, Runjin Zhang, Yuxin Liu, Bo Pan, Minfeng Zhu, and Wei Chen. Exploring multimodal prompt for visualization authoring with large language models. arXie preprint arXiv:2504.13700, 2025.
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo. Language as queries for referring video object segmentation. In CVPR, pp. 4974 4984, 2022.
Shiyu Wu, Mingzhen Sun, Weining Wang, Yequan Wang, and Jing Liu. Visualprompter: Prompt optimiza- tion with visual feedback for texl-to-inage synthesis. arXiv preprint arXiv:2506.2311 23138 2025.
Ruíhao Xia, Yu Liang, Peng-Tap Tao Jiang, Hao Zhang, Qianru Sun, Yang Tang, Bo Li, and Pan Zhou. Towards natural image matting in the wild via real-scenario prior. arXiv preprint arXiv:2410. 06593, 2024.
Guangkai Xu, Yongtao Ge, Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models. arXie preprint arXiv:2403.0600 2024a.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pp. 2955-2966, 2023a.
Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In CVPR, pp. 2970-2979, 2017.
Yangyang Xu, Shengfeng He, Wenqi Shao, Kwan- Yee K Wong, Yu Qiao, and Ping Luo. Diffusionmat: Alpha matting as sequential refinement learning. arXiv preprint Xin:2311.11553 2023b.
Yong Xu, Xin Yao, Baoling Lin, Yuhui Quan, and Hui Ji. Text-guided portrait image matting. IEFE Transactions on Artificial Intelligence, 2024b.
Peiqing Yang, Shangchen Zhou, Jixin Zhao, Qingyi Tao, and Chen Change Loy. MatAnyone: Stable video matting with consistent memory propagation. In arXiv preprint arX:::2501.1467, 2025.
Stephen DH Yang, Bin Wang, Weijia Li, YiQi Lin, and Conghui He. Unified interactive image matting. arXiv preprint arXim:2205.00 08324, 2022.
Jingfeng Yao, Xinggang Wang, Shusheng Yang, and Baoyuan Wang. Vitmatte: Boosting image matting with pre-trained plain vision transformers. Information Fusion, 103:102091, 2024a.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Jingfeng Yao, Xinggang Wang, Lang Ye, and Wenyu Liu. Matte anything: Interactive natural image matting with segment anything model. Image and Vision Computing, 147:105067, 2024b.
Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In CVPR, pp. 10502 10511, 2019.
Zixuan Ye, Wenze Liu, He Guo, Yujia Liang, Chaoyi Hong, Hao Lu, and Zhiguo Cao. Unifying automatic and interactive matting with pretrained vits. In CVPR, pp. 15585-25594 2024.
Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In CVPR, pp. 1154 1163, 2021.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting few-sample bert fine-tuning, arXiv preprint arXiv:2006. 05987, 2020.
Yuan Zhang, Chun-Kal Mai Fan, Tao Huang, Ming Lu, Sicheng Yu, Junwen Pan, Kuan Cheng, Qi She, and Shanghang Zhang. Autov: Learning to retrieve visual prompt for large vision-language models. arXiv preprint arxi::2006. 16112, 2025.
Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi- modal model. In ECCV, pp. 74-91, 2024.
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unlcashing text-to-image diffusson models for visual perception. In ICCV, pp. 5729 5739, 2023a.
Yang Zhao, Yanwu Xu, Zhisheng Xiao, and Tingbo Hou, Mobilediffusion: Subsecond text-to-image genera- tion on mobile devices. arXiv preprint arXtv:2311.16567, 2023b.
6 Appendix
6.1 Analysis of Self-Attention Optimization
Here, we compare the computational overhead of vanilla self-attention (SA) and the proposed sparse self- attention (Sp.SA) within the framework of our method. Complexity of Vanilla SA. Let n represent the number of input tokens and d the dimension of the token embedding. The computational complexity of single SA operation Le., the total number of addition and multiplication operations (OsA)- can be
caleulated follows:
OSA = 2nd 3 2n'dd+ n²
+3n 2n
Q.K.V
QKK
(Vi)
Sofemax
AV
(10)
= 6nd + 4n² d 4n² -
For instance, given the input feature f, computing Q involves multiplying f E gaxd and Wq E Rdxd resulting in n x d x d addition operations and n x d x d multiplication operations. Hence, the total number for calculating Q is 2nd2. The same applies to the computations of Q K and AV. Additionally, since the normalization (x(vd) -1) and Softmax operations act on an n x n matrix, their complexitics are n² and
(n² 1)+ n 3n - 1, respectively F
exponential
addition
division
Complexity of Optimized SA. As mentioned in Sec. 3.3 in the main paper, introducing the down- sampled convolution with kernel size and stride both equal to k results in KS and ys with a sequence length 22 times shorter than the original K and V. Let j (j k2) be the scale factor of this operation; the corresponding complexity OSp.SA is given as:
*FIOOPs for exponential operation be seen due to common instruction optimization and hardware acceleration rather than using repeating multiplication.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Osp.SA 2nd +
2x j
d x2
2nd +
]
3
Q
R², V with applied
Q.K
(va)-
n
1 1 2n x n
+ 3n
d
jx x xxd
(11)
Softmax
AV8
Convolution operation
4n
4n²
= (3 + ndd +
3
this work, we set kaza 2i = 2 for SA layers at resolutions of 32 x 32 and 16 x 16 respectively, with number of channels dz2= = 640, di 1280. Then, the fraction of complexity reduction is calculated as:
Sp = = 6 =
R3 0.4515
OSa = 32 - 640)
(12)
Osp.SA (n = 162, d 1280, = 2
~ 0.6176
Osa 16 ,,=,,d= d= 1280)
The results show that the proposed Sp.SA can reduce the computational complexity by 54.85% and 38. 28% for the SA layers at 32 x 32 and 16 x 16 resolution, respectively. Given the same j, the complexity reduction is more significant for the SA layer with a longer sequence length, which aligns with the intuition that the computational overhead of SA proportional to the square of the sequence length.
6.2 More Details on Data Curation
6.2.1
Annotation Models Choice
For generating caption annotations, we select a common VLM (BLIP-2 (Li et al. 2023b) to do this. Since captions were generated exclusively for the P3M10K, AM2K, and RMIK datasets, which are trimap-free matting datasets where the most. salient object is unambiguously the foreground. Generating captions to describe the foreground in these datasets is straightforward task for modern VLMs (including BLIP-2). Moreover, the BLIP-2 model was selected for its balance of usability, accuracy, and prevalence in the matting domain. It has been successfully used in prior work (Wang et al. 2024 to generate captions for matting datasets, demonstrating its effectiveness this context.
For generating GT alpha mattes for the ReiCOCO dataset, which contains only segnenntation-leeel annota- tions, we employed ViTMatte (Yao et al.) 2024a) to advance our model's training by producing matting-level annotations. The RefCOCO dataset primarily includes highly opaque objects (e.g., person, animals, tables), where the variance in unknown region of GT mattes is inherently low. Consequently, the choice of matting model has minimal impact on the resulting maltes. ViTMatte was selected for its balance of annotation efficiency and quality.
6.2.2 Quality Control
For generated captions, since modern VLMs have achieved superior performance in generating captions for images, the simplicity of this task ensures that errors, if any, are minimal and do not significantly impact training, the captions only need to capture key foreground characteristics. Consequently, no additional quality control measures were necessary beyond the inherent reliability of these datasets and the robustness of BLIP-2 in this context.
For generated GT alpha mattes, considering certain classes in RelCOCO (e.g., bottles) include both trans- parent and opaque objects, and their associated prompts often lack explicit transparency indicators (e.g., the word "transparent" is rarely used). To ensure annotation accuracy and avoid confusing the model during training, we excluded samples from these ambiguous classes. Additionally, some classes (c.g., bicycles) have segmentation annotations that lack fine-grained texture details (e.g., spokes in bicyele wheels), leading to
19

---

Published in Transactions on Machine Learning Research (10/2025)
low-quality GT alpha mattes. These samples were also removed to maintain the accuracy of the model's learning process and avoid ambignity.
6.2.3 Data Pre-processing
All the images are resized to 512 x 512 for training in all stages. For composition samples from RelMatte Li et al. 2023a), , we use the same augmentation strategy as in Li et al. 2022) to reduce the discrepancy between synthetic and real data.
6.3 More Details on Training
Rationals of Hyperparanter Setting. For the kernel size the morphological operation Kmors we set it 15 balance ovcrostimation (if too large) and undercstimation (if too small) of the transparent region, aligning with common practice in matting methods. For the cross-altention loss LCTM, we use sum reduction to address the presence of many zero matrices in the cross-attention maps (e-g., padding tokens), leading to smaller ÄCTM of 0.1. During training, LSTM and Latr are relatively small, while LsG is larger under mean reduction. To balance these, we set (ASTM; , AsG) to (10, 10, 0.5).
Training Hyperparameters. We show the training parameters in Tab. We adopt Adam (Loshchilov & Hutter 2019) as the optimizer with weight decay as 0.01. All the training work is done on NVIDIA A100 80GB GPU(s). For the scheduler, we adopt LambdaLR scheduler with several specified milestones. When the milestone not reached, we use the lambda function (ax) (1 total iterations
x 00.9 to adjust the learning
rate. Otherwise, the learning rate directly adjusted to the preseted value.
Table 4: Hyperparameters for all training stages.
Initial
Total
Total
Scheduler
Scheduler
GPUs (A100)
Learnable Parameters
Training Time
Training Stage
Learuing Rate
Iserations
Batch Size
Value
Milestone
Soft Grounding
5e-5
50000
32
[0.5, 0.25) [15000, 40000)
861M
18.5h
Semantic
Soft Grounding
5e-5
50000
32
[0.4, 0.25)
110000 35000]
509
20.7h
in Transparency
From Soft Grounding
4e-4
[0.1, 0.05)
[10000, 35000]
2.67M
to Alpha Matte
Clarification on Baselines Fine-tuning. Here provide more clarification regarding the baselines fine- tuning. First, it is well-known that fine-tuning models pretrained on large-scale datasets with small-scale datasets can lead to performance fluctuations (Zhang et al. 2020 Mosbach et al. 2020 Fu et al. 2023). particularly when the training details (c.g., crucial hyperparameters settings and training techniques) during fine-tuning differ significantly from those used in pretraining, or when the model with high complexity.
For Grounding DINO (Liu et al. 2024b), as its training process currently close-source, we could only attempt fine-tuning based on the training details provided in its paper. However, we observed that fine- tuning resulted in inferior performance compared to using the pretrained weights. We attribute this to discrepancies in the training approaches comparing with its undisclosed settings.
For PSALM (Zhang et al. 2024), we attempted fine-tuning based on its publicly available training code but found the performance to be unstable and degraded. We believe this due the high model complexity (PSALM uses an LLM its backbone), which amplifics the challenges of fine-tuning on small-scale datascts.
In contrast, for Stable Diffusion (SD) RRombach et al. 2022). argue that its advantage lies in the superior interpretability of its internal features and attention maps, which exhibit strong semantic differentiation. discussed in Soc. D This property provides SD with unique prior advantage when fine-tuned on downstream tasks supported by small-scale datasets, an advantage not shared by LLM-based methods like PSALM. Furthermore, numerous prior works (Zhao et al.) 2023a Lee et al. 2024 Xu et al.) 2024a Guo et al. 2024 Wang et al. 22024) have provided practical support for fine-tuning SD models on various dense prediction tasks, making this approach more mature. Therefore, we believe that SD offers distinct advantages for the soft, grounding task.
20

---

Published in Transactions on Machine Learning Research (10/2025)
6.4 More about Sparse Up-sampling Module
The design of the sparse up-sampling module is inspired by the progressive refinement decoder proposed in Yu et al. (2021), consisting of two groups of are replaced by sparse convolutional layers (Contributors 2022) in the up-sampling module. The input of each group is the concatenation of the up-interpolated alpha matte and the context feature from the matting decoder (all with a resolution of 512 512), along with the original image at the target resolution. The sparse convolution layer performs calculations only on areas where the alpha matte value is greater than 0.01 and less than 0.99. To train the module, attach it to the pre-trained matting decoder and train the intire network using the same loss function as the matting decoder. The learning rate of the module is set to 2e - 4, and the total number of iterations is set to 10, 000 with batch size of 16.
convolution groups. All the vamilla convolutional layers
6.5 More Robust Analysis
Here, we provide more robust analysis of our model given inaccurate prompt and under complex application scenarios.
6.5.1 Quantitative Analysis on Susceptibility of Noisy/Ambigge Prompt
To quantitatively assess the influence of noisy and ambiguous prompts on our model's performance, we conducted following ablation study via embedding-level disturbance. This study uses the RefMatto-RW100 dataset as the test benchmark. First, for each text prompt in the dataset, we generated two variants version using GPT-4: a noise version and an ambiguous version. Specifically:
Noisy Prompt (Tng): We instructed GPT-4 to remove all nouns and their corresponding adjective descriptions from the original prompt T, thereby producing noisy version that lacks key semantic content and contains only useless function words.
Ambiguous Prompt (Tamh): We instructed GPT-4 to retain only nouns related to object cate- gories, discarding all additional descriptive terms, thus creating an ambiguous prompt with reduced contextual specificity.
To precisely quantify the impact of Tay and Tamb on model performance, we performed measurements the linguistic embedding level. The linguistic features corresponding to the original prompt (fr), noisy prompl (JTny); and ambiguous prompt (Jrams) were extracted. We then injected the features of the noisy and ambiguous prompts into the original prompt's feature in a controlled manner, producing interpolated features IT.. and fT.m.. These interpolated features were used in place of fT for subsequent inference.
The degree of feature injection was controlled by two interpolation parameters, Pny and 9amb which allowed us to systematically evaluate the impact of noisy and ambiguous prompts. Mathematically, the feature interpolation process is defined as:
Proy (1 Bmn)ff + BaySTays Promo = (1 Bamb)ff + Bamb
(13)
Here, Bny, Bamb E [0, 1] control the extent of noisy or ambiguous feature injection, with B 3= corresponding to the original prompt's feature and 8 corresponding to the fully noisy or ambiguous feature.
We tested several values of Bny and Bams and quantified the experimental results, shown in Tab.
Our findings indicate that the model's performance is relatively robust to small perturbations from noisy and ambiguous prompts. For both Bay and 9amb at 0.2, the degradation in performance is modest. At Bay = Bamb = 0.4, the performance decline becomes more noticeable. However, a significant drop in performance observed when Bay or Bamb reaches 0.6 or higher. Notably, noisy prompts tend to have a more severe impact on performance compared to ambignous prompts, particularly at higher B values, evidenced by the larger increases in these metrics.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Ablation Study on Susceptibility of Poissy/mmiguoni Prompt.
Duy
SAD1
MSEJ
GRAD1
CONNI
Rmmb
SAD4
MSET
GRADI
CONNI
7.37
0.0264
6.55
5.31
7.37
0.0264
5.31
8.87
0.0321
7.14
6.23
0.2
8.47
0.0306
6.99
5,89
12.90
0.0474
9.17
7.03
0.4
14.03
0.0517
9.47
6.55
23.29
0.0867
18.11
12.83
0.6
26.15
0.0976
19.86
13.09
0.8
40.29
0.1510
31.03
19.24
0.8
34.55
0.1385
23.96
16.35
(a) Performance with Noisy Prompts (Bny)
(b) Performance with A Ambiguous Prompts (8amb)
6.5.2 Qualitative Analysis on More Complex Scenarios
Although our model is susceptible to noisy and ambignous prompts, it demonstrates strong robustness in more complex scenarios when the prompt accurate. In Fig. g below, we present additional robust, qualitative evaluations that clearly support the model's resilience under more challenging conditions, including multiple objects, instruction-contaiiii prompts, and images with more intricate spatial relationships. Furthermore, while using LLM- based paraphrasing for prompt enhancement can partially mitigate this limitation, we effective solution lies in incorporating prompt engineering into the network architecture
believe that training process, or exploring adaptive guidance strategies to further improve robustness, which is beyond the scope of this paper. While not with in our scope, addressing this challenge presents an interesting research direction that we leave to future work.
"tyzo womenn in the picture"
"Ivo of the girls are wenning jeans"
30
PUWER JD RANKINGS
SEPT. 22
"There are some players in the picture, two of them are wearing orange uniforms. Please extract both of them out in one single matte."
"Three people picture are taking selfie. Try matting them out together one output."
"a person with vine his shoulder
"the man with brown clothes"
Figure 9: More Qualitative Robust Evaluation. Despite some minor prediction artifacts, our model still demonstrates strong robust performance under more complex application scenes.
22

---

Published in Transactions on Machine Learning Research (10/2025)
6.6 More Ablation Studies
Loss Functions. We also explore the effect of each loss adopted in the distillation process in Tab. 0 using RefMatte-RW as the benchmark. Since LsG is the key loss for semantic guidance from e omitting it will lead to a significant performance drop. LSTM and LCTM also evidently improve the final performance, demonstrating the effectiveness of transparency exploration and retention of attention information for the optimized cg- Furthermore, the learned transparent confidence map R via Ln also facilitates the matting E-D in predicting more accurate alpha mattes.
Table 6: Ablation study different losses.
Lgg
ESTM
LCTM
Ln
SAD
MSE
GRAD4
CONNI
14.26
0.0465
8.95
7.09
8,40
0.0297
7.25
6.11
8.13
0.0284
7.06
5.98
7.96
0.0279
6.91
5.80
7.37
0.0264
6.55
5.31
Caption Annotation Model. As discussed in Sec. 6.2.1 our training process of our model is not sensitive to the choice of captioning model. To further verify this, we conduct following ablation study. We replace BLIP-2 with Qwen2.5-VT. (Bai al.) 2025), comparable multi-modal LLM, to generate the caption for the training samples without caption annotation. For consistency, we used the same prompt (* Describe the foreground in the picture, including its clothing, appearance, or behavior.") for both models. The quantitative results (using RelMalte-RW10k as the benchmark) are presented in Tab. 7 The evaluation metrics show very negligible differences between the two models, indicating that our method is robust to the choice of captioning model.
Table 7: Ablation study on caption annotation model.
Models for Caption Annotation
SAD
MSEL
GRADI
CONN1
BLIP-2 (La et al. 2023b
7.37
0.0264
6.55
5.31
Qwen2.5 VT. Bai al. 2025
7.36
0.0261
6.56
5.29
GT Alpha Matte Annotation Model. As discussed in Sec. 6.2.1 to annotate GT alpha matte for RefCOCO dataset, the choice of matting model has minimal impact on the resulting alpha mattes. Other matting models could be substituted with similar outcomes. To validate this, we conducted an ablation study by replacing ViTMatte with DiffMatte (uu et al. 2024) to annotate GT alpha mattes for RefCOCO and retrained stage and stage of our model. We also use RefMatte-RWio as the benchmark for evaluation. The results, shown in Tab. g indicate very negligible performance differences across the evaluation metrics, demonstrating the robustness of our method to the choice of matting model.
Table 8: Ablation study GT alpha matte annotation model.
Model for GT Matte Annotation
SAD,
MSEJ
GRAD,
CONNІ
ViT'atte (Yao et al. 2024a)
7.37
0.0264
6.55
5.31
DiffMatte (Hu et al.
2024
7.37
0.0265
6.54
5.31
Self-Attention Optimization Kernel. We further evaluate the impact of using different kernel sizes k for the down-sampled convolution in the optimization of SA at various resolutions. Our default setting removes SA layers at resolution of 64 x 64 (denoted as kar = -1) and sets ky2a K16 = 2. Therefore, we attempt to increase the kernel size at resolutions of 32 x 32 and 16 16, and also try restoring SA layers at resolution of 64 x 64 with kega = 2 instead of removing them entirely, resulting in three different settings.
In Tab. g we report the quantitative results of these three settings (the last three rows) alongside our default setting (the first row), using RefMatto-RWiOn as the benchmark. The inference time per sample is also evaluated, following the same setup as in Sec. 4.3 of the main paper. We found that using a larger kernel size at resolutions of 32 x 32 and 16 16 can lead to performance degradation, the over-optimized
23

---

Published in Transactions on Machine Learning Research (10/2025)
Table 9: Ablation study on kernel size & in Self-Attention Optimization. Here means remove the SA layers directly.
Inference Time (ms)
Kep
Rgga
116
SAD,
MSEL
GRADI
CONN
2
7.37
0.02644
6.55
5.31
95.14
2
4
8.57
0.0301
7.24
6.05
94.90
4
2
7.65
0.0272
6.70
5.54
94.69
2
7.13
0.0255
6.37
5.19
130.03
Image
Sparse Medule
w/o. Sparse Module
Figure 10: Qualitative ablation comparison of the sparse up-sampling module (SUM). The text inputs for the examples from left to right are: 1) lion yowning with its mouth open; 2) spider web is shown. SUM preserves more high-frequency textures, resulting in clearer alpha matte.
SA layers lose their ability capture global context and intra-object cohesion information. This effect is particularly noticeable at resolution of 16 x 16, where increasing & causes significant performance drop. Moreover, the rumning time is not substantially reduced in either case. We also found that restoring SA layers at resolution of 64 x 64 with h642 = 2 yields only marginal improvements while significantly slowing down the model.
Sparse Up-sampling Module. We conduct the ablation study of the sparse up-sampling module on the RefMatte- dataset in full resolution, which consists of more natural image matting samples. The quantitative result shown in Tab. 10 We found that three metrics (SAD, MSE, GRAD) are improved with the sparse up-sampling module, while the CONN metric is slightly higher. This is mainly because over- smoothed alpha leads to a lower connectivity error value. We then show the qualitative comparison in Fig T0) Despite the higher connectivity error, SUM can produce clearer alpha with finer details, demonstrating the evfectiveness of this module in extending our model lo arbitrary high-resolution inference, with more accurate matting texture preserved compared with direct up-sampling.
Table 10: Ablation study on sparse up-sample module (SUM).
SADI
MSEL
GRAD]
CONN.
Ours (w/o. SUM)
24.87
0.0114
16.43
12.58
Ours (w. SUM)
24.19
0.0112
14.94
12.91
Hyperparameters. We analyze the impact of hyperparameters in the training process using RefMatte- RW100 as the benchmark (sce Tab. 11 We evaluate three alternative settings and find that our default configuration (row 1) consistently outperforms them, demonstrating its effectiveness.
Stable Diffusion Versions. To further assess the generalizability of our framework, apply our method to teach two additional versions of Stable Diffusion (SD v2.0 and SD v2.1) for soft grounding while keeping all other experimental settings unchanged. The evaluation results on RefMatte-RW (see Tab. 12 show that compared to our default setting (SD v1.5), SD v2.x achieves better performance, highlighting the broad applicability of our approach.
Morcover, we also notice that recent SD models are emerged for advanced visual generation, such SDXL. (Podell et. al. 2023) and SD v3.x series (Esse et al.] 2024 SD3.x employs MMDIT, pure
24

---

Published in Transactions on Machine Learning Research (10/2025)
Table 11: Ablation study on hyperparameters.
ASTM
λCTM
AsG
Aatr
AR
Kmor
SADI
MSEI
GRAD]
CONNI
10
0.1
0.5
10
15
7.37
0.0264
6.55
5.31
10
0.1
0.5
10
9.32
0.0354
8,57
6.61
1
0.1
0.5
1
15
7.65
0.0271
6.63
5.32
10
0.1
1
10
7.41
0.0268
6.55
5,29
Table 12: Ablation study on Stable Diffusion (SD) version.
SD version
SAD.
MSEJ
GRAD,
CONN,
SD v1.5
7.37
0.0264
6.55
5.31
v2.0
6.50
0.0232
6.72
4.98
v2.1
6.18
0.0219
6.42
4.70
translormer-bssed model, as the denoising network, and SDXL incorporates more complex multi-scale generation structure. Due to these substantial architectural differences from the UNet architecture (com- prising residual and attention layers), and given that our method is tailored for UNet-based SD models, we acknowledge that our framework cannot yet directly applied to these transformer- based SD models.
Nevertheless, we emphasize the advantages of UNet-basod SD models for downstream tasks supported by relatively small-scale datasets, such our soft grounding problem. UNet-based networks benefit from the inductive bias of convolutional layers, which is particularly suited for small datasets and dense prediction tasks like soft grounding. In contrast, transformer-base long-term modeling for image generation but rely on large-scale training data and may not be as effective for dewnstream lasks with limited data. In future work, we plan to explore adapting transformer-basee SD models to address our soft grounding problem. However, we also highlight that UNet-based SD models possess unique advantages for transfer to dense prediction tasks.
denoising networks (e.g., MMDiT in SD3) excel in
6.7 Towards Video Instance Matting
Leveraging text prior enables to easily extend our model to video instance matting, since the language is also capable of describing a series of frames. We show the qualitative comparison of our model and several strong baselines on video instance matting in Fig. [T] Without any temporal modeling or fine-tuning, our model can still produce high-quality results on vidco sample with given text input. Our model is also robust to multi-instance enviromments and camera viewpoint changes, which demonstrates the powerful practicality of our model in real-world applications. More video instance matting results can also be found in the supplementary video file.
6.8 Generalization on Non-photographic Data
We further evaluated the generalization ability of our model on several non-photographic images, including animation-stylle and oil painting-style samples. Although the objects in these samples tend to have clearer boundaries compared to real-world images, they also exhibit markedly different textual characteristics and lighting effects. Since our training data consisted entirely of photographic images, these samples are consid- ered out-of-distribution (OOD) for our model. Nevertheless, as shown in Fig. [2] guided by textual input, our model still demonstrates strong generalization on these samples, highlighting its robustness and practical applicability in more diverse scenarios.
6.9 Computational Complexity Comparison
We compare the computational cost of our model with baselines with competitive performance in Tab. 13 with number of parameters, GMac, GFlops, and average GPU memory usage as metrics. Our method has both the lowest model size and computational cost among them, demonstrating the efficiency of ours comprehensively.
25

---

Published in Transactions on Machine Learning Research (10/2025)
Input frames
MatAny GDINO+SmartMar PSALM+MaGGIe
ReVVPD+MG
Ours
"the in gray clothes dacing" *
"the woman in pånk dancing"
"the black wearing
"the left zebra" "
Figure 11: Qualitative comparison on video instance matting. The corresponding text inputs are shown in italic below every video clip. The frames are placed following the temporal order in the original video from top to down.
26

---

Published in Transactions on Machine Learning Research (10/2025)
"the girl with dark hair and black clothes"
Input Image
"the girl with pink hair left"
Input Image
"the woman left"
"the male with black clothes on right"
Figure 12: Qualitative results of our model several non-photographic images (animation and oil painting). The corresponding text imputs shown in italic below every sample. Our model shows strong out-of-domain generalizability such data.
Table 13: Computational complexity comparison.
Number of
Average GPU Memory
GMAC&
GFLOPs
parameters (M) 4
Usage (MiB)
PSALM+MGM
1617
1430
2859
23372
R+VPDDMMM
929
959.4
1919
7506
MAGGM
1617
1429
2858
23236
ReVVDD+MaGoO
930
958.9
1918
7670
Ours (w/o. optimization)
861
902.3
1805
7914
Ours
429
769.7
1539
6474
6.10 More Qualitative Results
we show more qualitative comparison results on RefMatto-Test and RefMatte-RW100. Our model outperform other baselines both on synthetic and real-world datasets, demonstrating the superior performance of our model in grounding alpha matte from text descriptions. We also show qualitative results portrait and animal matting (Fig. 14h. The text inputs are generated by BLIP2 (Li et al. 2023b). The results further validate the gencralization ability of our model on different matting tasks.
In Fig. 13
27

---

Published in Transactions on Machine Learning Research (10/2025)
GD+ MatAny
GD+ SmartMat
PS+ MaGGle
PS+ MGMat
RV+ MaGGle
RV+ MGMst
Image
GT
CLiPMat
Oure
MAM
"the hotpink and salient. and transpporent sadnstrral plant
"the thistle and non-salient and transparent mass
"the coral and non-salient and non-transporent maple"
"the Hower which is meroon and non- cen-trensparentt
"the Lavender transparent sprog"
"the linen and non-tranat conn-ttranppaentt dandelion'
"the indevidual wearing black jacket blu jeans"
"the gainsboro and non-salient netting"
"the long haired wearing ovange dress glasses
"it mo0mun <r black I-shun!"
black dog on ils tiptoe"
"the strong blonde woman wearing white slecueless
"a women wears green jacket has brown haärs"
Figure 13: More qualitative comparisons on soft grounding task. The text inputs are shown in italic below. Zoom for better view.
28

---

Published in Transactions on Machine Learning Research (10/2025)
Image
Ground Truth
Ours
Image
Ground
Qurs
standing the beach talking" "
black white cut green eyes"
M
"a cumel with red blankee buck"
THip women standing next each other their turned to
leopard
"a camel with a long nose and long ears"
a woman wearing denim jacket and mask"
"a small white rabbit sitting on top of a rock""
a woman in the spotlight
MAMA
"a camel standing in front of pyramid"
"a towel standing in front shower"
red dress standing the
"a kudu with large horns stonding the wild"
dt mun and woman standing in a forest"
"wwo monkeys are standing together on rock
woman blank fuce stauling in front building"
"a horse with long mame" "
Figure 14: Qualitative results of our model on portrait (left) and animal (right) matting. Zoom in for better view.
29


Reviews:
[{"review_id": "llr2kGmygl", "reviewer": "Reviewer_SfhP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper introduces a new reasonable unified formulation for image matting, i.e., proposing to reformulate diverse matting tasks (portrait, natural image, animal, video, etc.) into a single soft grounding framework, where both instance-level localization and pixel-level transparency estimation are guided by natural language descriptions.\n- The paper proposes a successful practice to finetune a diffusion model, i.e., stable diffusion (SD) into a matting network with specific designs, including a Two-Stage Teacher–Student Distillation Framework and Designs for Disentanglement.\n\nWeaknesses:\n\n- The necessity of tuning a diffusion model, particularly for generation, into a matting network is in doubt. From my understanding, some designs for image generation in stable diffusion models are unfriendly for matting tasks. For example, the downsampling of VAE will lead to information loss and affect the accuracy of matting, which requires high alignment in pixel space. Furthermore, whether the matting network requires such a heavy network is another problem. I do not think matting relies on the generation ability of the model, which may even counteract the accuracy of the pixel-level alignment of the matting results.\n- The training process of tuning the SD network into a matting network is kind of vague. It is unclear for me how the author turns the multi-step SD into a single-step matting network.\n- The motivation of adopting the Two-Stage Teacher–Student Distillation requires more explanation. It is unclear why distillation is a better option compared with directly tuning the SD model.\n- The baseline settings need more details. Specifically, it is unclear whether the compared baselines in the experiments are retrained under the same setting. For example, MatAnyone and Maggie are trained for human matting and may not be suitable for natural images. Moreover, the author may consider comparing with common matting baselines on non-referring commonly used benchmarks. I regard this as a possible option since the proposed method can assign the target for matting, thus it should be able to handle non-referring data as previous methods, e.g., [Lin et al., 2021].\n- Alpha annotations for RefCOCO are generated via morphological operations + another matting model (Yao et al., 2024a), so the ground truth might carry the bias of that model. \n\n[Lin et al., 2021] Real-Time High-Resolution Background Matting", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a unified formulation of diverse image matting tasks as a soft grounding problem, simultaneously addressing instance-level localization and transparency estimation guided by natural language prompts. \n\nThe authors introduce a two-stage teacher–student distillation framework, where a Stable Diffusion model is first adapted for semantic localization, and its knowledge is transferred to a lightweight student model for transparency prediction. The designs include 1) Novel loss designs: including a Semantic Guidance Loss for feature-level alignment and 2) Transparency Mining Losses leveraging attention maps: enabling effective disentanglement of the two objectives. Structural optimizations such as block pruning and sparse self-attention substantially improve inference efficiency, and a sparse upsampling module allows high-resolution inference while preserving fine details.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I am not an expert in image matting, but I know pretty well about the diffusion-based generation. According to the weaknesses discussed before, my main concerns are as follows:\n\n1. While introducing the prompt guidance seems reasonable in Fig. 1, the author should provide more motivations on tuning a diffusion model, particularly for generation, into a matting network. From my understanding, some designs for image generation in stable diffusion models are unfriendly for matting tasks. For example, the downsampling of VAE will lead to information loss and affect the accuracy of matting, which requires high alignment in pixel space. Furthermore, whether the matting network requires such a heavy network is another problem. I do not think matting relies on the generation ability of the model, which may even counteract the accuracy of the pixel-level alignment of the matting results.\n\n2. The author should provide more details on how to tune the stable diffusion (SD) model following Fig. 3, i.e., how to turn the multi-step SD into a single-step matting network.\n\n3. The author should provide more insights on adopting the Two-Stage Teacher–Student Distillation. It is unclear why distillation is a better option compared with directly tuning the SD model.\n\n4. The author should provide more details on the baseline settings. Specifically, it is unclear whether the compared baselines in the experiments are retrained under the same setting. For example, MatAnyone and Maggie are trained for human matting and may not be suitable for natural images. \n\n5. The author may consider comparing with common matting baselines on non-referring commonly used benchmarks. I regard this as a possible option since the proposed method can assign the target for matting, thus it should be able to handle non-referring data as previous methods, e.g., [Lin et al., 2021].\n\n6. The author may explain more on the potential bias of the training data, especially for the alpha annotations for RefCOCO, which are generated via morphological operations + another matting model (Yao et al., 2024a). The ground truth might carry the bias of that model.\n\n7. For image generation like SD, generating accurate content following the prompt is still an unsolved problem. The author may consider discussing the potential problem of the prompt following in this matting task.\n\n8. The paper provides the video results in Fig. 9 of the supplementary material. It seems the result on human matting, e.g., 'the woman in pink dance' is not very good with many incorrect matting details. The author should provide other video matting results such as GD+MatAny.\n\n[Lin et al., 2021] Real-Time High-Resolution Background Matting", "broader_impact_concerns": "The authors should add a Broader Impact Statement for discussion of possible ethical implications, such as the mitigation of dataset bias and prompt sensitivity, training data bias, etc.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3LCOlGa1aR", "reviewer": "Reviewer_AoAt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe proposed approach seems sound that unifies diverse matting tasks under the soft grounding task by integrating localization and transparency in a single model. Also, the proposed teacher-student design seems to work well as it disentangles the localization and transparency learning through leveraging the visual priors of pretrained text-to-image diffusion models. It also enjoys computational efficiency compared to baselines through structure pruning and sparse attention that reduce inference time and supports high-resolution inference efficiently. Lastly, the generalization capability demonstrates the practical usage of the proposed method.\n\n\n### Weaknesses\nOne potential problem would be the dependency towards text prompts, where the models’ performance would vary among the descriptiveness of prompt inputs, and ambiguous text prompts would limit its performance. Also, while the teacher-student framework seems sound, it requires multiple stages in training (i.e., semantic grounding - distillation - matting refinement) with multiple datasets and text annotations, which makes the training complicated. The proposed method would heavily depend on the quality of the pretrained diffusion model, thus the diffusion backbone’s biases would potentially be inherited. The sparse attention enjoys efficient inference, but it seems like there is a slight degradation in performance compared to the unoptimized student model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers image matting problem, which aims to estimate the per-pixel transparency (i.e., alpha matte) to separate foreground and background, by leveraging pre-trained text-to-image diffusion models (e.g., Stable Diffusion). Unlike traditional methods that rely on visual priors such as trimaps or masks, which are often expensive to obtain, there is growing research that uses text-based vision models such as CLIP (i.e., CLIPMat) or Stable Diffusion. However, the author claims that those approaches also lack pixel-level precision or struggles with complex transparency and multi-object cases. To this end, the authors propose an alternative approach to achieve high-quality image matting using diffusion models. In specific, the authors frame matting problem as a soft grounding problem, where it jointly solves instance-level localization and transparency estimation using textual information. They introduce two-stage teacher-student distillation framework, where 1) teacher model is trained by adapting Stable Diffusion for soft semantic grounding (only localization), trained to classify each pixel into foreground, background, or transparent regions using cross-entropy loss with CLIP text embeddings. Then 2), the student model is a lightweight, structure-optimized version distilled from the teacher to perform transparency estimation while preserving semantic localization. Here, they introduce semantic guidance loss that conducts feature-level distillation from teacher, and transparency mining loss that uses self and cross attention maps from teacher to mine transparency cues. Lastly, the matting decoder is attached to refine a coarse alpha matte and a transparency confidence map into full-resolution output. \n\nThrough evaluation on RefMatte-Testset, RefMatte-RW100, AM2K (animal), and P3M-P (portrait), the authors show that the proposed method outperforms CLIPMat and cascaded baselines (e.g., GroundingDINO + matting or PSALM + matting) in both accuracy and speed. Furthermore, it demonstrates zero-shot generalization capability among diverse matting tasks without tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Detailed analysis on prompt dependency.\nSince the method relies on precise, unambiguous text prompts, there should be more robustness analysis rather than few qualitative examples. For example, the authors could add experiments on how noisy or ambiguous prompts affect quantitative performance. Here, I believe the method could be further improved by using prompt enhancement techniques (e.g., using LLM-based paraphrasing). The related question is, if the prompt becomes more detailed, could the model handle more complex scenarios (e.g., multi-object, complex descriptions or complex spatial relationships)?\n\n### Real-world applications.\nThe proposed evaluation is limited to static image benchmarks, but does the method can be applied to dynamic cases (e.g., video matting)? If true, showing video matting experiments would further improve the paper’s contribution. Also, to show out-of-domain generalization, it could be possible to show generalization on non-photographic images such as cartoons or synthetic images. \n\n### Comparisons with baselines.\nWhile PSALM, Grounding DINO + matting could be a firm baseline, they are used with their original weights. Thus, it remains a question why fine-tuning such models degrade the performance, while using Stable Diffusion + proposed method works. Also, showing the parameter counts, FLOPs, or memory usage to contextualize speed vs accuracy trade-offs would strengthen the paper.\n\n### Ablation studies on backbone.\nThe proposed method uses Stable Diffusion v2.0 and v2.1, but it could be interesting if the method applies to different models. For example, how does the proposed method behave when using older models (e.g., SDv1.5) or recent models (e.g., SDXL, SD3, SD3.5)? The scaling behavior of the backbone size would be more beneficial for future research.\n\n### Miscellaneous.\nWhile I believe the paper is clearly written, the presentation could be improved. For instance, Figure 3 is too dense to understand the major detail at glance, and the equations to explain the loss function may overwhelm the readers. Simplifying the diagram for method description and equations would help the readers to understand paper more easily.", "broader_impact_concerns": "The paper could discuss broader impact concerns in their paper. For example, the proposed method could be misused for image manipulation or deepfakes as they provide high-quality matting of natural images. Furthermore, the biases from pretrained model could be inherited to their model, thus analyzing those biases in depth would be helpful.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nt1PkybGRW", "reviewer": "Reviewer_UMCq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Clarity and Motivation: The paper is exceptionally well-written and clearly articulated. The authors provide strong intuition and compelling motivation for their core design choices, especially the teacher-student distillation framework for disentangling localization from transparency estimation. The logic is easy to follow and methodologically sound.\n- Compelling Empirical Performance: The method achieves state-of-the-art results, demonstrating significant improvements over a comprehensive set of baselines on multiple benchmarks, as evidenced by the results in Table 1. The performance gains are not marginal but substantial, establishing a new benchmark for text-driven matting.\n- Effective Visualizations: The submission is supported by high-quality, informative visualizations. For example, Figure 3 offers an excellent and intuitive diagram of the entire pipeline, which greatly aids in understanding the complex interactions between the teacher, student, and various loss components.\n\nWeaknesses\n- Lack of Detail in Data Curation: The methodology for creating the training dataset lacks transparency. The authors state they use BLIP-2 for caption generation and an existing matting model (Yao et al., 2024a) for alpha mask generation. However, this process raises several critical questions that are not addressed:\n  - Quality Control: What measures were implemented to ensure the quality of the automatically generated labels? Noisy or inaccurate captions/mattes could significantly impact training, and the filtering or verification process is not described.\n  - Rationale for Model Choice: The paper does not justify why these specific models were chosen for annotation over other alternatives.\n  - Performance Sensitivity: An analysis of how the final model's performance is affected by the quality of these auto-generated labels is missing. This is a crucial detail for reproducibility and understanding the robustness of the method.\n- Absence of Data-Level Ablation Studies: The paper presents a thorough ablation study on its architectural components (Table 3) but critically omits any ablation on the data itself. This makes it difficult to disentangle the source of the impressive performance gains. The key unanswered question is: How much of the improvement comes from the novel architecture versus the newly curated training data? To isolate the contribution of the proposed model, it would be essential to conduct experiments where a strong baseline method is also trained on the exact same dataset. Without this comparison, the core architectural contribution is confounded with the data's contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The core innovation is a computationally efficient Teacher-Student distillation framework that reframes the task as \"soft grounding.\" The key contributions and new knowledge are:\n- State-of-the-Art Performance with High Efficiency: The paper demonstrates that this approach achieves top-tier results across multiple benchmarks. Crucially, it does so while operating with significantly fewer parameters and faster inference speeds than competing methods. This marks a critical step towards the practical deployment of high-quality matting models.\n- A Unified Framework for Diverse Matting Tasks: The method successfully subsumes various matting sub-tasks (e.g., portrait, animal, referring matting) under a single, versatile text-promptable model. This demonstrates strong generalization and flexibility, moving the field beyond task-specific architectures toward a more universal solution.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To address the concerns raised in the \"Weaknesses\" section, we propose the following adjustments. We believe these changes are essential for substantiating the paper's core contributions and ensuring its scientific rigor.\n\n1. Clarification of the Data Curation and Annotation Pipeline (Critical)\nThe paper's reproducibility and the trustworthiness of its results depend heavily on the dataset. The current description is too high-level. We request that the authors add a dedicated subsection, detailing the data generation process. This section should address the following points:\n- Quality Control Protocol: Please describe the specific steps taken to validate or filter the auto-generated captions from BLIP-2 and alpha mattes from Yao et al. (2024a). Was there a manual review process? Were metrics used to discard low-quality samples? Providing examples of both accepted and rejected auto-generated labels would be highly beneficial.\n- Rationale for Tooling: A justification for choosing BLIP-2 and the specific matting model by Yao et al. (2024a) should be provided. What were their key advantages over other available tools? Please provide the comparison with other annotation models or tools.\n\n2. Data-Level Ablation Study to Isolate Model Contribution (Critical)\nThe current results confound the contribution of the novel architecture with the contribution of the newly curated dataset. To disentangle these two factors, we request the following experiment:\n- Train a Strong Baseline on Your Dataset: Please select a high-performing baseline method from Table 1 (e.g., the previous state-of-the-art) and train it on the exact same training dataset used for your proposed model.\n- Report Comparative Results: Add a row to Table 1 or create a new table that compares your method against this retrained baseline.\nThis experiment is critical because it will provide a fair, apples-to-apples comparison that isolates the architectural improvements. If your model still significantly outperforms the baseline on the same data, it will provide undeniable evidence that your proposed teacher-student framework is the primary driver of the performance gains, thereby solidifying the paper's core contribution. Without this, it remains unclear whether the novelty lies in the model or the data.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)