DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting
Authors: Xiaodi Li
Zongxin Yang
Ruijie Quan
Yi Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Matting
Diffusion Model
Foreground Estimation
Primary area: diffusion_based_models

Abstract: Recovering the foreground color and opacity/alpha matte from a single image (i.e., image matting) is a challenging and ill-posed problem where data priors play a critical role in achieving precise results. Traditional methods generally predict the alpha matte and then extract the foreground through post-processing, often failing to produce high-fidelity foreground color. This failure stems from the models' difficulty in learning robust color predictions from limited matting datasets. To address this, we explore the potential of leveraging vision priors embedded in pre-trained latent diffusion models (LDM) for estimating foreground RGBA values in challenging scenarios and rare objects. We introduce Drip, a novel approach for image matting that harnesses the rich prior knowledge of LDM models. Our method incorporates a switcher and a cross-domain attention mechanism to extend the original LDM for joint prediction of the foreground color and opacity. This setup facilitates mutual information exchange and ensures high consistency across both modalities. To mitigate the inherent reconstruction errors of the LDM's VAE decoder, we propose a latent transparency decoder to align the RGBA prediction with the input image, thereby reducing discrepancies. Comprehensive experimental results demonstrate that our approach achieves state-of-the-art performance in foreground and alpha predictions and shows remarkable generalizability across various benchmarks.

DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting
Xiaodi Li Zongxin Yang", Ruijie Quan 1.2 Yi Yangl.2+
The State Key Lab of Brain-Machine Intelligence. Zhejiang University, Hangzhou, China 2 CCAL, College of Computer Science and Technology, Zhejiang University, Hangzhou, China 3 DBMI, HMS, Harvard University, Boston, USA www.taaw....../OoD/D/ri....
Abstract
Recovering the foreground color and opacity/alpha matte from a single image (i.e., image matting) is a challenging and ill-posed problem where data priors play a critical role in achieving precise results. Due the limited matting datasets, tradi- tional methods usually struggle to produce high-quality estimation. To address this, we explore the potential of leveraging vision priors embedded in pre-trained latent diffusion models (LDM) for estimating foreground RGBA values in challenging scenarios and rare objects. We introduce Drip, novel approach for image matting that harnesses the rich prior knowledge of LDM models. Our method incorporates a switcher and cross-domain attention mechanism to extend the original LDM for joint prediction of the foreground color and opacity. This setup facilitates mutual information exchange and ensures high consistency across both modalities. To mitigate the inherent reconstruction errors of the LDM's VAE decoder, we propose a latent transparency decoder to align the RGBA prediction with the input image, thereby reducing discrepancies. Comprehensive experimental results demonstrate that our approach achieves state-of-the-an art performance foreground and alpha predictions and shows remarkable generalizability across various benchmarks.
1
Introduction
Image matting aims to isolate the foreground object from composited images, Long-standing and fundamental task vision intelligence [1]. It is indispensable for various downstream applications, such as media production, virtual reality, and image/video editing [2, 3J. Mathematically. image
matting begins with solving the inverse problem of the composition equation:
Image; = a F Foreground; + (1-a,) Background,, a E [0. 1].
(1)
where denotes the index of a pixel. Here, all quantities on the right-hand side are unknown, and the prediction of the alpha matte a and foreground color represents an ill-posed problem.
In the past decade, advances deep learning have significantly pushed the boundaries of image matting, rapidly becoming the mainstream direction in this field [2, 4-7]. Despite their impressive performance, two challenges remain unresolved this domain: (i) high-quality foreground color prediction. As illustrated in Fig. most matting methods consist of two stages: namely, alpha prediction with neural networks and foreground isolation via post-processing. These methods typically struggle to generalize and recover high-fidelity foregrounds due to the accumulated errors in alpha prediction and post-processing. (ii) accurate prediction of cemi-transparent objects. When the target be predicted contains large areas of cranptrrnsparrncy (e.g., water glass) or
*Corresponding author: Yi Yang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Compostiion
Compocttio
Prod. Foroground
Error Accumulation
Strong Learrable Prtor
more natural
Matting Network
Post Procssing
LDM-Sased
Trimap
Motting
Trimap
Network
Pred, Alcha
Alpha
Empirial
Pred. Freggroudd
Hyportees
Previous Matting Methods
egg. local smoothing
(a) Previous Methods
(b) Drip (ours)
Figure 1: (a) Matting methods [20, 4-7] commonly predict the alpha matte and then infer the foreground color by post-processing [21], which often relies on empirical assumptions such as local smoothing of the foreground and background, leading to the accumulation of errors. (b) In contrast, our joint prediction approach estimates both the foreground and alpha simultaneously. By leveraging LDM's [8] powerful natural vision prior, our predicted foreground closer natural images.
semi cemi-transparent regions with high-frequence details (e.g.. patterned semi-transpanne cuni-transparent fabric), existing methods still struggle to predict the alpha matte accurately.
High-quality semitransparent matting data is difficult annotate large scale. Therefore, how can we enhance the algorithm's 's generalization capability for semi-transparent objects and achieve high-quality foreground prediction with limited training data? Recently, with the emergence of large pre-trained generative models [8-12], their data priors learned from billions of images (e.g.. LAION [13]) found to be useful for various downstream tasks [14-18]. Back the challenging image matting, we posit that the data priors learned from tons of natural images are also intuitively beneficial. Hence, our key insight is to unleash the data priors from the large pre-trained generative models (LDM) to estimate alpha and foreground simultaneously.
To achieve this, we propose Drip, the unleashing Diffusion priors for joint foreground and alpha prediction method, which follows the diffusion paradigm and jointly generates the foreground and alpha map conditioned on the image and trimap input. Specifically, we wisely design a cross-domain switcher that leverages domain-awand embedding to unify the foreground and alpha generations in a single-diffusion model. This design facilitates mutual information exchange and ensures high consistency between foreground image and alpha. Besides, the pre- pre-trained VAE compresses the image into a compact latent space, significantly reducing training consumption while inevitably missing detailed information. To narrow the errors caused by VAE, we introduce auxiliary latent transparency decoder, which is implemented by inserting the features from early layers in the encoder into the decoder with several learnable zero-conv layers [19]. This latent transparency decoder significantly contributes to high-fidelity foreground image and alpha prediction and also effectively adapts the pre-trained LDM into image matting.
We extensively evaluate the performance of our method through extensive experiments and com- parisons. The results demonstrate that our approach achieves state-of-the of-thert- performance the Composition- 1k test set and exhibits stronger generalizability on other benchmark datasets. Remark- ably, Drip outperforms all the previous methods in the mainstream benchmark, Composition- 1k, where Drip improves the SAD metric of alpha prediction by 3.3% and foreground by 12.1% 1% and MSE metric of alpha by 6.1% and foreground by 28.33%. In summary, the key contributions of this paper are as follows.
To our best knowledge, we introduce the first LDM-driven matting method, Drip, which effectively unleashes the data priors learned from LDM into image matting-
To enable joint prediction of foreground and alpha, we propose a switcher and a cross-domain attention mechanism, facilitating mutual information exchange and ensuring high consistency.
To mitigate the inherent reconstruction errors of the LDM's VAE decoder, propose latent transparency decoder to align the RGBA prediction with the input image.
2
Related Work
Image Matting is aimed to extract the foreground objects from arbitrary natural images [22, 2]. Traditional methods always need the auxiliary user input like trimap [23, 4] and seribble [24, 25]. These methods basically only leverage low-level color structure features, which limits their ability

---

to distinguish foreground details from images. With the success of deep learning, researchers have begun to use deep convolutional neural networks (CNNs) to predict the alpha map an end-to-end fashion [26, 20]. One type takes images and auxiliary trimap or scribble as input and outputs the alpha map [5]. In order to alleviate the demand for trimap, trimap-free methods [27, 6, 28] are proposed to directly predict alpha mattes from the input image, which increases efficiency while sacrificing performance. Although these existing methods achieve impressive results in alpha prediction, accurately predicting foreground and background colors remains an essential yet challenging task for high-quality matting. Tang et al. [29] and Aksoy et al. [30] firstly proposed to address color estimation by sequentially directly predicting the background and foreground colors before alpha prediction. Furthermore, recent method [31] unifies foreground, background, and alpha matte into an end-to-end framework. While, another line of works [32, 28] focuses foreground human extraction and alpha matte prediction. However, these methods are limited by the lack of high-quality labeled data. Meanwhile, the explosion of generative models shows immense potential in providing priors in different tasks. In this work, we explore unleashing diffusion priors within stable diffusion [8] to improve the performance of image matting.
Diffusion Models have emerged as a powerful class of generative models, which learn a reverse denoised process from the Gaussian noise to natural images [33]. In the vanilla DDPMs [33], the sampling process is consuming due to the Markovian property. To speed up the sampling, DDIMs [34] is proposed to provide non- non-Markovian shortcut. Furthermore, LCM [35] just formulate the diffusion process as one-step denoising via an ODE. Besides the speed, a series of works [36, 10, 38] focus increasing the controllability of diffusion models. For instance, Controlnet [19] fine-tunes a Stable Diffusion model with zero convolutions, which proves to be effective in adapting the pre-trained diffusion models to different tasks by adding different conditions.
Diffusion Priors in Visual Perceptive Tasks are prevalent and hot topics. A series of works leverage the diffusion priors in segmentation [17], image enhancement [39], depth estimation [40] and 3D vision [41, 42]. In the context of image matting, Xu et al. [43] propose to formulate alpha prediction as denoised process, and train a condition generation model in DDPMs fashion. However, the vanilla DDPMs have not been scaled-up training due to their expensive computational cost. On the contrary, LDM [8] proposes to compress the features into compact latent space, which obviously reduces the computational cost. And based on it, Stable Diffusion is largely trained on the large-scale dataset [13]. However, LDM generates the image features in the latent space encoded by pretrained VAE. In order to alleviate the domain gap, Marigold [16] finetunes UNet backbone of diffusion models to perform affine invariant monocular depth estimation and exhibit strong generalization capability. Inspired by this, we carefully discuss and propose novel method to unleash the diffusion priors within stable diffusion to improve the performance of image matting while preserving high-fidelity details.
3 Drip
Overview. Drip is an LDM-based matting model designed to predict both foreground and alpha values while ensuring high consistency between these two representations. Given an input image (X) and trimap (Tri) indicating the object to be matted, our goal is to estimate its corresponding Foreground (F)) and alpha (a). Initially, we explore the problem using the diffusion paradigm (see Sec. $3.1). Subsequently, we present our LDM- based matting model (see Sec. $3.2). This model employs a cross-domain switcher to simultancously generate the foreground color and alpha map using single diffusion model. Moreover, through mutual information exchange, the model effectively enhances boundary and texture consistency. To address the challenge of missing caused by VAE compression, the model incorporates an auxiliary latent transparency decoder (see Sec. $3.3). An overview of Drip is provided in Fig. 2.
information
3.1 Problem Formulation
The task of foreground and alpha estimation is to model the mapping f() : (X, + aF, where F E R! x2xWxx x3 represents the foreground and a E # represents the alpha map. The input conditions are an RGB image X E R#XWxx x3 and a trimap Tri E # which consists of three values indicating the foreground, unknown, and background regions, respectively. However, unlike prior works that adopt CNN or transformer as architecture, we employ a diffusion- scheme f()) to model the joint foreground and alpha distribution p( a))).
Diffusion Probabilistic Models [44, 33] define forward Markov chain that progressively transits the sample x drawn from data distribution p(x) into noisy versions 2, (L, = 0y20 + o,E, where
3

---

Predictions
Input and Trimap
Switcher
Latent Pred.
TYMl
Denolsing Net
Concat
21
E
Foreground and Alpha
2
Z
Henation
M(O,1)
Add nose
Concar
Denotsing
training objective
t (~1((T)
Sample Noise
Switcher
(a) Training
(b) Inference
Figure 2: Overview of Drip. (a) During training, the input image X, trimap Tri, ground-truth foreground F, and ground-truth alpha map a are first encoded into latent representations Z Z ZF. and z respectively using the original Stable Diffusion VAE encoder E. After adding noise to 2" and Z all the latents are fed into a U-Net, which generates the output the foreground or alpha domain guided by switcher ($3.2.1). The U-Net is then fine-tuned by optimizing the standard diffusion (3jjective($3.2.33 (b) After executing the T-step denoising schedule, the resulting latents 7f and 7 decoded by transparent latent decoder ($3.3).
NO, /), T is the timestep. arand or are the noisy scheduler terms that control sample quality, In the reverse Markov chain, it learns denoising network e-(-) parameterized by usually structured as U-Net [45] transform Te into It-1 from an initial Gaussian sample zT through iterative denoising.
For the joint foreground and alpha distribution p(F, F. a), given a conditional input image X with its corresponding trimap Tri, the foreground F and the alpha map can be obtained by the generative formulation Markov probabilistic form:
I I
f(x,
(Fr, dr
X,Trri,, Pr, ~ N)O,)).
To enhance computational efficiency and generate higher-resolution images, Stable Diffusion [8] employs the latent diffusion model, where the diffusion steps are performed in the low-dimensional latent space instead of directly operating on the original data. The latent space is formed within the bottleneck of VAE [46], which is trained separately from the denoiser. This design allows latent space compression and facilitates perceptual alignment with the data space.
To translate formulation (Eq. 2) into the latent space, obtain the corresponding latent code for given image using encoder: z = Et), where i E Tri, F, a. It's worth noting that we triplicate the single-channel trimap and alpha map into three channels. Moreover, the denoiser (. subsequently trained in the latent space. To obtain the desired outputs, given latent codes of and Za, the foreground and alpha can be reconstructed using the decoder D: F D(=²)) and á D(z is worth noting that in the Matting task if the reconstructed image is obtained directly from the latent representation of the foreground and alpha without making any modifications to the VAE, significant error can occur.
3.2 LDM-Based Matting Model
We base our model on pretrained text-to- text-to-image LDM (Stable Diffusion v2 [8]), which has learned strong and generalizable image priors from LAION-5B [13]. In order to accept given image and trimap as conditions and simultaneously generate both foreground and alpha outputs, we quadruple the input of the original U-Net and employ a switcher mechanism to expand the capabilities of the original LDM model. Additionally, incorporate cross-domain attention to enhance consistency.

---

Self Attention
Cross- Domain Self Attention
modify
Res Block
Cross Attention
rgb latent
alpha
foreground latent
Figure 3: Demonstration of Cross-Domaie Attention ($3.2.2.. To enhance mutual guidance and ensure contextual consistency, we instead utilize cross-domain self-attention mechanism instead of self-attention to associate the foreground and alpha latent.
3.2.1 Foreground and Alpha Switcher
Previous matting models [20, 4-6] have primarily focused predicting the alpha value and utiliz- ing post-proeessing methods, such as the local smoothness assumption, to estimate the foreground color [21, 47]. However, this two-stage approach often leads to accumulation and suboptimal foreground estimation results, particularly when dealing with transparent objects. To address this limitation, we propose novel approach that leverages LDM-based method predict the fore- ground and alpha values simultaneously. Our method generates more realistic foreground images by incorporating strong natural image prior distribution learned from Stable Diffusion v2 [8].
To incorporate foreground and alpha estimation, one straightforward approach is to finetune two U-Nets separately to model their respective distributions. However, this method introduces additional parameters and fails to capture the inherent connections between foreground and alpha. Motivated by the work [48, 14], we propose novel approach using switcher that enables single stable diffusion model to generate both foreground and alpha values based on indicators. Mathematically, the foreground and alpha values can be obtained follows:
F. f(X,Tri, SF) (rnntt
= J(, Tri, PosEnc(0))
In the above equations, SF and Sa are one-dimensional vectors controlling the foreground and alpha domains, respectively. The switchers are encoded using low-dimensional positional encoding and combined with time embedding within the U-Net architecture.
3.2.2 Cross-Domain Attention
To further facilitate mutual-guided optimization, we introduce modification the self-attention layer in the U-Net architecture, transforming it into a cross doms-n self-attention layer that encourages spatial alignment (refer Fig 3). This operator enhances the geometric consistency between the foreground and alpha channels and accelerates convergence. The cross-domain attention operation, denoted as At)C is defined as follows:
A K,, V.) = Att (W,- h, We (hp @ ha), W. (hp e h,))
(5)
Here, i = F, a. hp and h represent the latent embeddings of the foreground and alpha channels within the transformer blocks, respectively. The symbol @ denotes the concatenation operation. W W., and W,, are the matrices of query, key, and value embeddings, respectively. Finally, Att(-) refers to the softmax attention mechanism.
3.2.3 LDM Loss Function
We adopt annealed multi-resolution noise noises [16] to preserve low-frequency details in the depth and normal maps, as similar values will frequently appear local geometric regions. This deviation proves to be more efficient than a single-scale noise schedule. We perturb the two geometry branches with the same timestep scheduler to decrease the difficulty when learning more modalities. The canonical standard learning objective we utilize is defined as follows:
C Ex Ex (Fr; X, Tri, sp) - ef 11 +ep (au; X,Tri, -
(6)
Here, ef and e are two Gaussian noises independently sampled from annealed multi-scale noise sets for the foreground and alpha, respectively.
5

---

2
E
D
D Dtrans
(XTrri
Foreground or Alpha
Reconstructions
Figure 4: Structure of Transparent Latent Decoder ($3.3). Due to the non-negligible reconstruction loss introduced by the compression of the LDM's VAE, we employ a transparent latent decoder, which takes the output images of the LDM-based matting model and the corresponding latent as inputs, generating results that are more consistent with the details of the composite image.
3.3 Latent Transparency Decoder
The compression of the VAE [46] in LDM introduces a non-negligible reconstruction loss, indicating that æ * (x))))) Previous discriminative tasks using the LDM [8, 12] have primarily focused on tasks such as segmentation and depth estimation. In comparison, matting is task that places more emphasis on capturing fine details, making the errors introduced by the VAE more noticeable.
To address this challenge, draw inspiration from LayerDiifusion [49] and propose novel latent transparency decoder Dirans. This decoder takes the output images of the LDM-based matting model 3.2 and the corresponding latent as inputs generate results that more consistent with the details of the composition image (refer to Fig 4). Mathematically, the expression is given by
(f, (F = Dirans (X, Tri, D(2²",, D(
where 2² and 20 represent the foreground and alpha predictions in the latent space, respectively. Additionally, 'D represents the original VAE.
By incorporating this transparent latent decoder into framework and training it with the matting loss, we aim to improve the fidelity of the predicted foreground and alpha outputs, ensuring that they capture the intricate details present the composition image. This enhancement is particularly crucial for the task of matting, which relies heavily on preserving fine details and boundaries.
4 Experiment
4.1
Experimental Setup
4.1.1 Datasets.
Compasition- dataset [5] is a synthetic dataset consisting of 431 manually labeled foreground images for training and additional 50 foreground images for evaluation. The training set is created by compositing each foreground image with 100 background images sourced from the COCO dataset [S0]. This approach allows for the generation of a sufficient training set despite having a smaller number of unique foreground object images, Similarly, the test set is generated by synthesizing 50 test foreground images using 20 background images from VOC2012 [51]. resulting in total of 1000 test images.
AIM-500 dataset [6] is benchmark for natural image matting that encompasses various object categories. It consists of 500 high-resolution real nature images, each with minimum short side length of 1080 pixels. Unlike other natural matte datasets [52-54] that often limited specific classes of human and animal images, AIM-500 offers a more diverse range of objects. Evaluating the performance on the AIM benchmark helps assess the model's ability to generalize well to natural images rather than solely fitting the distribution of synthetic images. Therefore, the evaluations conducted on both the Composition- Ik and AIM benchmarks complement each other, providing comprehensive understanding of matte models in real-world scenarios.
4.1.2 Implementation Details.
We employed Diffusers [55] with Stable Diffusion v2 [8] as the backbone for implementing our Drip. Text conditioning was disabled by providing empty text input. To accommodate the two additional conditions, the weights of the first layer of the UNet were copied three times as initialization. After training the LDM-based matting network, the obtained output and latent results were utilized as inputs for training the latent transparent decoder.

---

Image
FBA &
FBA
FBA &P
Ours d
Ours F
Ours á
Figure 5:
Qualitative Result Foreground. The visual results compared with FBA [31] on AIM- 500[6]. Please zoom in for the best view.
Table 1:
Comparison of our Drip with -fate-of-the-Art (SOTAs) on the synthetic dataset Composition
1k [5] and the natural dataset AIM-500 [6], focusing on four metrics for alpha prediction results. ($4.2).
Composition- lk
AIM-500
Method
Publication
SAD
MSE
Grad
Conn
SAD
MSE
Grad
Conn
DIM
[5]
CVPR 17
50.4
14.0
31.0
50.8
49.3
14.7
29.3
47.1
IndexNet
[56]
ICCV*19
45.8
13,0
25.9
43.7
FBA
[31]
ArXiv'20
25.8
5.2
10.6
20.8
HATT
[27]
CVPR 20
44.0
7.0
29.3
46.4
479.2
270.0
238.6
474.0
Alpha
AIM
[6]
IJCAT'21
43.9
16.1
33.1
43.2
GFM [32]
IJCV'22
52.7
21.3
46.1
52.9
MFormer
[57]
CVPR'22
23.8
4.0
8.7
18.9
ViTMatte
IF'23
21.5
3.3
7.2
16.2
DiffMat
[43]
ArXiv'24
22.8
4.0
6.8
18.4
Ours
20.8
17.8
17.3
1.5
14.7
To enhance the diversity of the dataset, we performed various data augmentation techniques during the training of the 2D image set. These included random horizontal flipping, cropping, and photometric distortion. Besides, in order to enhance the foreground, we followed LayerDiffusion [49] to fill the foreground image. Pixels with Alpha values equal zero in the foreground have no impact on the appearance of the alpha-blendes blended image when assigning any color. Nevertheless, since neural networks tend to produce high-frest nigh-frequency patterns surrounding image edges, avoiding unnecessary edges in the RGB channels prevents potential artifacts. To achieve this, we applied Gaussian blurring regions of the foreground image where the Alpha value was strictly equal zero.
During training, the DDPM noise scheduler [58] with 1000 diffusion steps was applied. At inference time, the DPM solver scheduler [59]was employed, and only 10 steps were sampled. The model was trained for 30,000 steps with a total batch size of 96, using an image size of 512 x 512 exclusively on the Composition- 1k dataset [5]. The entire training procedure typically took approximately days when executed on cluster consisting of 4 Nvidia Tesla A100-80GB GPUs. For optimization, the Adam optimizer was used with a learning rate of -0
4.1.3 Evaluation Metric.
We employ standard evaluation metrics to assess the quality of alpha predictions. Specifically, we report the Sum of Absolute Differences (SAD), Mean Square Error (MSE), Gradient loss (Grad), and Connectivity loss (Conn). A lower value for these metrics indicates higher quality alpha matte.
Additionally, we follow the evaluation methodology of FBA matting [31] evaluate the quality of foreground predictions. Since colors other than the foreground object region are not used, we only consider the region where the ground truth alpha, denoted attt, is located and it weighting factor. We apply the SAD and MSE to evaluate the agt predictions. A lower value for these metrics indicates better quality foreground prediction.

---

Compasition
GT
Trimap
DIM
VitMatte
DiffMatte
Ours
Figure 6: Qualitative Result of Alpha. The visual results compared with previous SOTA methods on AIM-500[6]. Please zoom for the best view.
4.1.4 Baselines.
For alpha estimation, we consider several state-of-the-ee methods as baselines, including both trimap-based approaches and automated methods. Among the automated methods, HATT [27] AIM [6], and GFM [32] are included. On the other hand, the trimap-based methods consist of DIM [5], IndexNet [56], FBA [31], MFormer [57], and ViT'Mttt [7]. These trimap-based methods are all deep barning-basee. with the backbone architecture gradually transitioning from convolutional networks to transformer-bat based networks. Furthermore, with the development of diffusion, the contemporary method DiffMat [43] is also based on diffusion. However, it important to note that DiftMat [43] does not operate in the latent space, nor does it leverage the priori of the natural image distribution learned from diffusion.
For foreground estimation, we compare our proposed method with a limited set of baselines, as many matting algorithms primarily focus on alpha estimation rather than foreground estimation. In our comparison, we include Global-Matting [60] and KNN-Mattin [61] as non-deep learning methods. We consider ContextAware- Matting [62] and FBAMatting [31] for deep learning-bast based methods. These approaches leverage deep neural networks to estimate the foreground and have demonstrated promising results in previous studies.
Quantitative Result Qualitative Result
Table Comparison with SOTAs($4.2) 2).
The results are shown Table for alpha. The re- sults indicate that our method outperforms others by a large margin and achieves state-of-the-an (SOTA)
Method
Publication
Composition lk
SAD
MSE
Global
[60]
CVPR'I II
220.39
36.3
Foreground
KNN
[61]
TPAMI' 13
281.9
36.3
performance. On the Composition- dataset, our method improves the SAD metric by 0.7 (+3.3%)
CA
[62]
ICCV'19
61.72
3.24
LBS
[29]
CVPR'19
49.7
8.6
and the MSE metric by 0.2 (+6.1%) 1%) compared the ViTMatte [7] method. The performance improve-
FBA
[3]]
ArXiv 20
38.8
6.0
Ours
34.1
ments are even more noticeable on the AIM-500 dataset, where our method improves the SAD by
26.6 and the MSE by 14.6 compared to the AIM [6] approach. For the foreground metrics, as shown
in Table 2, our method demonstrates significant improvements compared the FBA-Matting [31]. Specifically, we improve the SAD metric by 4.7 (+12.1%) and the MSE metric by 1.7 (+28.33%)
8

---

Table 3: A set of ablative experiments about our proposed modules on the AIM-500 [6].(S4.3)
à
aF
àf
Switcher
CTAut
af
SAD
MSE
SAD
MSE
SAD
MSE
Decoder
SAD
MSE
SAD
MSE
18.1
1.7
21.1
3,8
23.7
5.2
21.3
2.1
25.3
4.8
17.8
1.5
21.3
4.1
26.7
5,9
17.3
1.5
20.6
3.7
17.3
1.5
20.6
3.7
23.3
4.9
(a) Joint prediction of foreground and alpha
(b) Transparent latent decoder
Additionally, visualize some qualitative results in comparison with other baselines. As demon- strated in Figure Fig. 5, our method produces foreground predictions that are more closely aligned with natural images, and the resulting RGBA outputs detail. As shown in Fig. 6, for natural datasets, our results are significantly closer to the true values, demonstrating the strong generalization ability of our method.
consistent with the original image in
4.3 Ablation Study
We conduct ablation studies on AIM-500 [6] to investigate the contributions of the proposed joint prediction of foreground and alpha and the transparent latent decoder.
Joint Prediction of Foreground and Alpha We first investigate the effect of the proposed cross- domain attention mechanism on the joint foreground and alpha estimation(cf. $3.2). As shown in Ta- ble 3a, when the cross-domain attention module is removed, we observe a decrease in the predic- tion accuracies of both the foreground and alpha representations. Importantly, the accuracy of the final RGBA composition, obtained by multiplying the predicted foreground and alpha, decreases more significantly. This suggests that the information in- teraction between the two modalities, facilitated by cross-domain attention, increases the consistency of spatial information between the foreground and al- pha representations. This finding verifies that cross- domain self-attention effectively correlate the two representations, enabling them to benefit from each other's contextual cues mutually.
Composition
w/o Dranns
Full Model
Figure 7: Ablation Result. When the trans- parent latent decoder is not used, the generated output exhibits significant differences in low- level details with the original input.
For the switcher, we also notice a clear reduction in performance, suggesting that giving additional embedding information rather than simply distinguish- ing between foreground and alpha with channel order is helpful in relation to the neural network's ability to distinguish and utilize information from the two modalities.
Transparent Latent Decoder Next, we ablate the effect of the (era $3.3). As reported in Table 3b, using the original VAE decoder without Dtrans leads to a degradation in the prediction performance both alpha and foreground. In addition, the qualitative results on composition- Ik also clearly show that the low-level details correspond better after applying Derans, mitigating the errors induced by VAE compression.
Conclusion and Future Work
This work addresses two key challenges in image matting: high-quality foreground prediction and accurate trmi-transparent object alpha estimation. To overcome these, we propose Drip method, which leverages data priors from large pre-trained generative models to jointly predict the foreground and alpha. Drip utilizes a switcher and a cross domain attention for consistent foreground-alh alpha generation and a latent transparency decoder to enhance fidelity. Extensive experiments demonstrate Drip achieves sota performance Composition- 1k and stronger generalization on other benchmarks.
While the Drip method demonstrates strong performance in image matting tasks, several key limita- tions warrant consideration: i) Model Complexity and Deployment. The incorporation of latent diffusion models (L.Mss) substantially increases the architectural complexity of the approach. This added complexity may impact deployment and inference efficiency, particularly in real-time or

---

resource-conseee constrainsded environments where computational overhead is critical. ii) Inherited Biases from Generative Priors. Drip's methodology relies on the extensive priors captured within pre- trained LDMss. Consequently, Drip inherits any biases present in the original generative model. These biases could adversely affect the method's performance on certain types of images or domains, such as those with highly complex lighting and shadows, or those containing numerous small texture details, thereby limiting its general applicability.
Acknowledgements. This work was supported by the National Natural Science Foundation of China (U2336212) and the Fundamental Research Funds for the Central Universities (No. 226-2022- 00051).
References
[1] Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. Frontiers of Information Technology Electronic Engineering, 2024.
[2] Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep image matting: comprehensive survey, arXiv preprint 240402230 2023.
[3] Zongxin Yang, Jiaxu Miao, Yunchao Wei, Wenguan Wang, Xiaohan Wang, and Yi Yang. Scalable video object segmentation with identification mechanism. TPAMI, 2024.
[4] Jian Sun, Jiaya Jia, Chi- Chi-Keung Tang, and Heung-Youn Yeung Shum. Poisson matting, ACM Trans. on Graphics (TOG), 23(3):315-2211 2004.
[5] Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[6] Jizhizi Li, Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. arXiv preprint arXiv:2107.07235, 2021.
[7] Jingfeng Yao, Xinggang Wang, Shusheng Yang, and Baoyuan Wang. Vitmatte: Boosting image matting with pretrained plain vision transformers, 2023.
[8] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[9] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, lyy Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[10] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding, Advances in Neural Information Processing Systems (NewrIPS), 2022.
[11] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2Xiv:2204.06125,, 1(2):3, 2022.
[12] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[13] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[14] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. arXiv preprint 1211 2024.
10

---

[15] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang. and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified as a video agent). In ICMI., 2024.
[16] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger. Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-basee image generators for monocular depth estimation. arXiv preprini urXiv:2312.02/// 2023.
[17] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In IEEE Confer- ence Computer Vision and Pattern Recognition (CVPR), 2023.
[18) Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. IEEE International Conference on Computer Vision (ICCV), 2023.
[19] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[20] Donghyeon Cho, Yu-Wing Tai, and Inso Kweon. Natural image matting using deep convolu- tional neural networks. In European Conference on Computer Vision (ECCV), 2016.
[21] Thomas Germer, Tobias Uelwer, Stefan Conrad, and Stefan Harmeling. Fast multi-level foreground estimation. IEEE International Conference on Pattern Recognition (ICPR), 2021.
[22] Anat Levin, Dani Lischinski, and Yair Weiss. closed-form solution to natural image matting. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 30(2):228-22 242 2008.
[23] Jue Wang and Michael F Cohen. Optimized color sampling for robust matting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2007.
[24] Anat Levin, Dani Lischinski, and Yair Weiss. closed-form solution to natural image matting. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 30(2):228-24 2007.
[25] Xue Bai and Guillermo Sapiro. A geodesic framework for fast interactive image and video segmentation and matting. In IEEE International Conference on Computer Vision (ICCV), 2007.
[26] Xiaoyong Shen, Xin Tao, Hongyun Gao, Chao Zhou, and Jiaya Jia. Deep automatic portrait matting. European Conference on Computer Vision (ECCV), 2016.
[27] Yu Qiao, Yuhao Liu, Xin Yang, Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei. Attention-guided hierarchical structure aggregation for image matting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
(28] Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan, and Rynson WH Lau, Modnet: Real-time trimap-free portrait matting via objective decomposition. In AAAI Conference on Artificial Intelligence, 2022.
[29] Jingwei Tang, Yagiz Aksoy, Cengiz Öztireli, Markus H. Gross, and Tunç Ozan Aydin. Learning- based sampling for natural image matting. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3055-3063 Computer Vision Foundation IEEE, 2019.
[30] Yagiz Aksoy, Tunç Ozan Aydin, and Marc Pollefeys. Designing effective inter-pixel information flow for natural image matting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 228-2362222222222 IEEE Computer Society, 2017.
[31] Marco Forte and François Pitié. f. b, alpha matting. arXiv preprint arXiv:2003.07711 2020.
[32] Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. International Journal of Computer Vision (IJCV), 130(2):246-266, 2022.
[33] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
11

---

[34] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500, 2020.
[35] Simian Luo, Yigin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:23/0.04378, 2023.
[36] Dewei Zhou, You Li, Fan Ma, Zongxin Yang, and Yang. Miga++: Advanced multi-instance generation controller for image synthesis. arXiv preprint arXiv:2407.02322, 2024.
[37] Xiaolong Shen, Jianxin Ma, Chang Zhou, and Zongxin Yang. Controllable 3d face generation with conditional style code diffusion. In AAAI, volume 38, pages 481 2024.
[38] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-imane -mae diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[39] D Zhou, Z. Yang, and Y Yang. Pyramid diffusion models for low-light image enhancement. arxiv 2023. IJCAI, 2023.
[40] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In IEEE International Conference on Computer Vision (ICCV), 2023.
[41] Zechuan Zhang, Zongxin Yang, and Yi Yang. Sifu: Side-view conditioned implicit function for real-world usable clothed human reconstruction. In CVPR, pages 9936-9947, 2024.
[42] Yuanyou Xu, Zongxin Yang, and Yi Yang. Seeavatar: Photorealistic text-to-3d avatar generation with constrained geometry and appearance. arXiv preprint arXiv:2312.08888 2023.
[43] Yangyang Xu, Shengfeng He, Wenqi Shao, Kwan-Ye Yee K Wong, Yu Qiao, and Ping Luo. Diffu- sionmat: Alpha matting as sequential refinement learning. arXiv preprint arXiv:2311.13533 2023.
[44] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In The International Conference Machine Learning (ICML), 2015.
[45] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted interven- tion (MICCAI), 2015.
[46] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv:1312.61 14, 2013.
arXiv preprint
[47] Marco Forte. Approximate fast foreground colour estimation. In IEEE International Conference on Pattern Recognition (ICPR), 2021.
[48] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, al. Wonder3d: Single image to 3d using cross- diffusion. arXiv preprint arXiv:2310.15008 2023.
[49] Lvmin Zhang and Maneesh Agrawala. Transparent image layer diffusion using latent trans- parency, 2024.
[50] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[51] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International Journal of Computer Vision (IJCV), 88:303-338, 2010.
12

---

[52] Jinlin Liu, Yuan Yao, Wendi Hou, Miaomiao Cui. Xuansong Xie, Changshui Zhang, and Xian- sheng Hua. Boosting semantic human matting with coarse annotations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[53] Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang. Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[54] Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian L Curless, Steven M Seitz, and Ira Kemelmacher-Shizzemann Real-time high-resolution background matting. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[55] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-an diffusion models. btops://gihhe con/huggingface/ diffusers, 2022.
[56] Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu. Indices matter: Learning index for deep image matting. In IEEE International Conference on Computer Vision (ICCV), 2019.
[57] Jiaojiao Li, Yihong Leng, Rui Song, Wei Liu, Yunsong Li, and Qian Du Mformer: Taming masked transformer for unsupervised spectral reconstruction. IEEE Transactions Geoscience and Remote Sensing, 61:1-12, 2023.
[58] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020.
[59] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models, 2023.
[60] Kaiming He, Christoph Rhemann, Carsten Rother, Xiaoou Tang, and Jian Sun. A global sampling method for alpha matting. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2011.
[61] Qifeng Chen, Dingzeyu Li, and Chi-Keung Tang. Knn matting. IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 350))2175--2888 2013.
[62] Qiqi Hou and Feng Liu. Context-aware image matting for simultaneous foreground and alpha estimation. In IEEE International Conference on Computer Vision (ICCV), 2019.
[63] Shanchuan Lin, Linjie Yang, Imran Saleemi, and Soumyadip Sengupta. Robust high- regouuesoottion video matting with temporal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
13

---

A
Appendix
This appendix contains additional details for the Neurips 2024 submission, titled "DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction Image Matting". The appendix is organized as follows:
AA.1 offers more implementation details of the multi-level annealed noise.
S A.2 offers more implementation details of the augmentation to pad the foreground.
A.33 provides more information about the transparent latent decoder.
$A.4 provides ablation results on the number of timesteps for the AIM [6] dataset.
A.1 Annealed Multi-Resolution Noise
In order to make Idm handle the details better, we adopt the annealed multi-resolution noise used in [16, 14] .In crafting the standard multi-resan musti-ressoution noise, array of Gaussian noise images is initially sampled to construct a pyramid with gradient of resolutions, which are then merged using upscaling, weighted averaging, and normalization methods. Each level i of this pyramid assigned weight following the formula 8", where 8 is a decimal fraction between O and 1, signifying the extent of the influence exerted by noise reduced resolutions. To better align the resultant noise with the Gaussian distribution outlined the foundational DDPM framework, the weights of the upper levels i are modified in accordance with diffusion schedule. Specifically, at each time step t, the i-th level is endowed with weight calculated as (s /T)), with 7' representing the cumulative number of diffusion steps. As result, the weight allocated levels with diminished resolution progressively reduced as the schedule nears its noise-free terminal point.
Algorithm 1 annealed pyramid
1: b, w, h t r.shape (Get the shape the input tensor)
2: Worl
hon t- (Save the original dimensions)
3: noise t geenenoisee pen__oise_likee (b * Woii hon) (Create a noise tensor)
4: --0
5: repeat
6:
rand() 2 2 (Generate random scale factor)
7: w < max (1, [Won (''))
8:
h t max(1, [han/ (r (Compute the current feature map size)
9:
somp_noise_ao & gen gen_noise____eeeeeeee _oiose_iieeeebbbbbeeeeeeeee * * to * h) (Generate a temporary noise tensor)
for j - t b c Wort hori do
11:
Lids t j%Wori
Vid ]
13:
new x < Dide (Won/ [w)]
new y < Videe & (hori h)]
15:
new new_idx + new y + new
16:
noiseli] < noiselj] + temp_ temp_noise nowseileww_idss (0inesteps//00000) * discount" end for
18:
i ++i+1
until 10 or (w S 1 and h S 1) (If already reached minimum resolution, break out)
20:
result +- noise noise.std (Normalize the noise tensor)
return result
A.2
Padded Foreground
In order to enhance the foreground effect and avoid potential artifacts, we apply a fill technique [49] to the foreground image. Pixels in the foreground with an alpha value equal to zero, regardless of their color, do not affect the appearance of the final alpha-blended image. However, neural networks tend to produce high-frequency patterns at the edges of the image, avoiding unwanted edges in the RGB channel helps prevent potential artifacts. To address this issue, we apply Gaussian blurring to regions of the foreground image where the alpha value is strictly equal to zero.
14

---

22.9
21.4
SAD
19.9
SAD
18.4
16.9
1
4
10
25
50
100
# Inference Steps (log scale)
Figure 8: Impact of Denoising Steps Performance. The performance of method improves
the number of denoising steps increases, with diminishing marginal gains as the number of timesteps becomes larger.
Algorithm 2 padded fg(alpha, paddee_fi((alpha fg)
h, < shape(alpha) [Get the height and width of the input image)
2: mask < gen (alk(al mask where alpha are and alpha > are 0}
1-00 0
4: while 64 do filtered_iman iimgemaaee-- < gaussian_blur(ff..
blur(fg. (13, 13),0) (apply gaussian blur to foreground)
6: j -+0
while j < h x w do
8:
mask = then
[gli] + filtered_imagee (update foreground if mask indicates transparent region)
10:
end if
j j ++
12: end while
i i i+1
14: end while
return fg (Return the updated foreground image)
A.3 Detail of Transparent Latent Decoder
The core component of our approach is the transparent latent decoder, which implemented as U-Net architecture. To train this module, we randomly sample the step number between 1 and 10 to get the output of latent matting network, which then use as input. We utilize a loss function commonly used in matting algorithms to optimize the model. Specifically, we employ the combined matting loss, which includes separate components such as the l loss [7, 63] la loss [63], laplacian loss [62, 63], and gradient penalty loss [7] for both alpha and foreground. The objective function
becomes:
Lmat
Li Lt, + Liop Lggrad
(8)
A.4 Ablation Results on Timesteps
We conducted ablation studies on the number of diffusion timesteps used in our method on the AIM [6] dataset. As shown in Figure 8, increasing the number of timesteps generally improves the performance, though the gains diminish as the number of timesteps becomes larger.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately summarize the key claims, contribu- tions, and scope of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the conclusion section we discussed the limitation of our method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those
discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: the implementation part the experiment section we elaborate on the details needed for method reproduction.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided not.
If the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification:
Codes will be released after the paper is accepted. Data are available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/ for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: lelguuddee/oeeeaaiiiiieeeee
//nips..
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: the implementation part of the experiment section we elaborate on the details needed for method reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
All experiments are conduct on fixed random seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main
claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It OK to report --sggma error bars, but one should state The authors should preferably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the implementation part of the experiment section we elaborate on the details needed for method reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http///eurrppp.ppe wttpiipp..pppubbiee
Answer: [Yes]
Justification:
The research in the paper complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the reference section we accurately cite the datasets, papers and codebases used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswil thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [No]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
21


Reviews:
[{"review_id": "M5YamSwSPi", "reviewer": "Reviewer_YFFw", "summary": "The authors present the clear and straightforward method to improve image matting performance using an LDM-based model. The model is a conditioned LDM which can predict both the Alpha mask and the foreground, basically doing RGB-A prediction from an image and a trimap. To further adapt to the data domain, the authors also finetune a specialized decoder which is proven to be helpful. The overall design of the model and training method is intuitive and effective with sufficient evaluation. The training data is limited while can be scaled up by more data collection.", "questions": "- The transparent latent decoder is helpful. However, will it be more useful to also train an transparent auto-encoder by finetuning the existing antoencoder? When training the decoder, did the authors prepare and precompute the data from a trained diffusion model?\n- How to speed up the process and could we reduce the steps of sampling?\n- Any proposed methods to scale up the training data for a more robust model training?\n- Will Trimap be necessary? It will be great if the model can directly do foreground object RGB-A extraction without inputting the Trimap.", "limitations": "- Trimap is needed for the model to do RGB-A extraction. Some dropping of the conditions may be helpful.\n- Data is quite limited and training longer time may degrade the performance.\n- Cross-domain self-attention is not fully evaluated.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear presentation of the methods with a straightforward implementation. The results show obvious improvement over existing discriminative models. \n- The cross-domain attention and finetuned decoder are necessary and interesting to explore.", "weaknesses": "- The training dataset still looks quite small. The model may not have good generalization capability to more unseen data in the wild. Also the batch size and training time is limited. \n- When the authors mentioned 'prior', is it referring to the pre-trained LDM model? Did the authors use like LoRA to finetune the model on the smaller dataset? Will the performance become worse as the training goes longer?\n- There seems no evidence showing the advantages of the cross-domain self-attention. Could the authors provide more details?\n- It seems the switcher is just conditioned embedding for choosing outputs. Will we be able to predict both at the same time without inputing the switcher values?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Training Dataset Size and Model Generalization**\n\nThank you for raising concerns about our dataset size and model generalization. Our training dataset is substantial, comprising 431 foreground images and a background library of 82,783 images. Each foreground image is paired with 100 different backgrounds (L206-L209), **resulting in at least 43,100 training samples**. Additionally, **we applied data augmentation techniques** such as random horizontal flipping, cropping, and photometric distortion, further increasing the effective training dataset size (L227-L229). Thus, our dataset is quite extensive.\n\nFor generalization, our model **trained on the Composition-1k dataset was evaluated on the independent AIM-500 real-world dataset. The results showed that our model generalizes well to unseen data (L283-284)**.\n\nWe fully acknowledge the importance of dataset size for generalization. However, matting is a dense annotation task that is extremely challenging to label. This is precisely **why we incorporated the strong prior knowledge from the latent diffusion model trained on LIAON**, which significantly aids in improving generalization capabilities.\n\n**W2: Clarification on Prior Knowledge and Fine-Tuning with LoRA**\n\nWhen we refer to 'prior' in our paper, **it indeed pertains to the pre-trained Latent Diffusion Model (L147-148)**. Regarding your question about using LoRA to fine-tune the model on a smaller dataset, we have experimented with this approach. Our findings indicate that fine-tuning with LoRA **resulted in poor performance and difficulty in convergence**. Consequently, there was no issue of overfitting or performance degradation over extended training periods. We hope this clarification addresses your concerns.\n\n**W3: Evidence for Cross-Domain Self-Attention Advantages**\n\nIn our paper, we **conducted an ablation study (L288-305) to evaluate this component**. The quantitative results demonstrate that cross-domain self-attention improves both alpha prediction and foreground color prediction accuracy. This improvement is attributed to the **high correlation between alpha and foreground color**, as both describe information about the foreground object. Cross-domain attention enhances the interaction and consistency of this information, leading to better overall performance.\n\n**W4: Switcher**\n\nThe switcher **is indeed an embedding designed to help the network distinguish between the two outputs**: foreground color and alpha value. Both outputs are **predicted simultaneously**. Our **ablation study (L306-309)** demonstrates that providing this additional embedding information, rather than merely differentiating between foreground and alpha by channel order, enhances the neural network’s ability to distinguish and effectively utilize information from the two modalities.\n\n**Q1: Transparent Latent Decoder**\n\nWe appreciate the suggestion to fine-tune the VAE. However, solely fine-tuning the VAE is not ideal for several reasons:\n\n1. **Consistency with Pre-trained Weights:**\nTo leverage the pre-trained weights in the LDM, especially in the U-Net, it is crucial to maintain the consistency of the latent distribution output by the VAE encoder. Over-finetuning the VAE encoder could disrupt this consistency, negatively impacting performance.\n2. **Compression:**\nThe VAE inherently introduces a non-negligible reconstruction loss due to its compression nature, which is problematic for high-resolution tasks like matting that require low-level detail. This limitation, as mentioned in the paper (L189-190, L310-314), necessitates the use of a transparent latent decoder to meet the high precision demands of matting tasks.\n\nOur current approach, combined with the transparent latent decoder, offers a balanced solution that addresses the high-resolution requirements of matting tasks while leveraging the strengths of pre-trained LDM components.\n\n**Q2: Steps of Sampling**\n\nIn the appendix of our paper, **we have included results and analyses regarding the impact of denoising steps on performance (L507-510)**. The **specific values are shown in the table below**. \n\n|Steps | SAD |\n| --- | --- |\n| 1 | 22.8 |\n| 5 | 17.8 |\n| 10 | 17.3 |\n| 20 | 17.2 |\n\nOur findings indicate that **increasing the number of timesteps generally enhances performance, although the improvement diminishes as the number of timesteps increases**. Additionally, our experiments show that setting the sampling steps to at least 5 generally yields satisfactory results.\n\n**Q3: Scaling Up Training Data**\n\nThank you for your comments and for inquiring about potential methods to scale up the training data for more robust model training. **We have plans for future research that involve using methods such as layerdiffusion to generate additional data**. This generated data will then be screened **with a human-in-the-loop approach to ensure quality**. In our current study, **to maintain a fair comparison with other methods, we used the same training dataset**.\n\n**Q4: Necessity of Trimap**\n\nThank you for your comment regarding the use of trimap in our model. We understand the desire for a model that can directly perform RGB-A extraction without requiring a trimap. However, we have found that **the use of a trimap is still necessary for achieving accurate results, especially when multiple foreground objects are present in an image**.\n\nTo address this concern, we conducted **a qualitative experiment where the trimap was set to an unknown region across the entire image**. This situation introduces ambiguity about which foreground object to extract, **as illustrated in Figure D of the rebuttal PDF**. For example, with a trimap indicating unknown regions, the model faces difficulty in deciding whether to extract the rabbit or the flower, leading to potential confusion and reduced accuracy.\n\nAdditionally, **to ensure fair comparison with previous methods, we have maintained consistent settings and trimaps across our experiments(L251-257)**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paU3R8xM8X", "reviewer": "Reviewer_L9RY", "summary": "The paper introduces Drip, an approach to image matting that leverages vision priors from pre-trained latent diffusion models (LDM). Drip incorporates a switcher and cross-domain attention mechanism for joint prediction of foreground color and opacity, ensuring high consistency. A latent transparency decoder mitigates reconstruction errors. Experiments demonstrate good performance and generalizability across benchmarks.", "questions": "NIL", "limitations": "The limitation is not well discussed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The experiment is good and quite comprehensive", "weaknesses": "Like many other papers on the application using the DNN. The paper present an NN architecure for image matting. The building blocks are all standard one, and the design does not really reflect the uniqueness of the problem from other image editing. Overall, I am not convinced that such a paper really benefit the progress of image matting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Contribution to Image Matting**\n\nOur primary contributions and insights are not focused on the neural network architecture or block design. Instead, we concentrate on exploring how to leverage the priors learned by well-scaled image generation models to address the ill-posed problem of image matting. Here are the unique aspects of our work:\n\n1. **Joint Estimation of Foreground and Background**: In previous works, foreground extraction required complex post-processing. Our experiments demonstrate that leveraging the priors from generative models facilitates foreground image prediction. Furthermore, the alpha map and foreground have a close domain relationship, and joint estimation benefits both.\n2. **Addressing Domain Gaps with Generative Models**: We tackle a significant technical challenge by adapting image generation models for alpha and foreground estimation. This involves bridging the severe domain gap between the RGB images encoded and decoded by VAEs and the RGB-A images required for matting.\n\nWe believe that our approach of integrating generative model priors into the matting process brings a novel perspective and advances the field of image matting. Thanks for your valuable feedback.\n\n**W2: Inadequate Discussion of Model Limitations**\n\nThank you for your feedback. We acknowledge that the discussion of limitations is crucial. **In the original paper, we have addressed the limitations of our model in the \"Conclusion and Future Work\" section (L322-331).** We have outlined several aspects where the model might face challenges or limitations.\n\n**If there are specific areas or additional limitations you believe were not adequately covered, we would greatly appreciate your insights.** We are open to discussing and incorporating any further considerations you might suggest during the discussion phase and are willing to make necessary revisions to address these points comprehensively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GaQhMF6WFg", "reviewer": "Reviewer_8UU8", "summary": "This paper introduces DRIP, a novel image matting method that leverages pre-trained LDMs to jointly predict foreground color and alpha mattes. By integrating a cross-domain attention mechanism and a latent transparency decoder, DRIP addresses the limitations of traditional methods, achieving significant performance improvements on synthetic and natural datasets. The key contributions include enhanced prediction consistency, high-fidelity results, and setting new benchmarks in image matting accuracy.", "questions": "See weakness.", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper presents a pioneering approach by utilizing pre-trained LDMs for image matting, incorporating cutting-edge methodologies such as the latent transparency decoder. This method markedly enhances performance in image matting, establishing new SOTAs and overcoming shortcomings (e.g., details in semitransparent regions) of existing SOTA methods.\n\n1. Robust empirical evidence supporting the proposed method is provided through comprehensive ablation studies.\n\n2. The manuscript is eloquently composed and well-organized. The logical progression is sound and accessible to a wide readership.\n\n3. The innovative application of pre-trained LDMs is a successful paradigm for future endeavors in additional computer vision applications.", "weaknesses": "1. The computational complexity is unclear. Their high computational demands may impede practical application, particularly in real-time or resource-constrained settings.\n\n2.  The evaluation is predominantly conducted on synthetic datasets and a limited number of natural image benchmarks. More extensive testing on diverse real-world scenarios would enhance the demonstration of the model’s robustness and generalizability.\n\n3.  The approach depends on pre-trained LDMs, which may propagate existing biases from the training data. Can the matting model perform effectively on objects that are not well-represented by the original SD model, such as rare objects?\n\n4.  This article develops its model based on the Stable Diffusion model. It raises the question of whether it retains the textual input feature of the SD model. If so, how is image captioning handled? If not, could this omission lead to a degradation in the performance of cross-attention in SD?\n\n5. During inference, is classifier-free guidance utilized? If so, what impact does it have on the matting performance?\n\n6. Has there been any consideration for fine-tuning based on layer diffusion?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Computational Complexity**\n\nThank you for your valuable comments regarding the computational complexity of our model. We understand the importance of addressing computational demands, particularly for real-time or resource-constrained applications. **In the limitation section of our paper, we discussed the issues related to model complexity and deployment, specifically noting that the use of latent diffusion models substantially increases the architectural complexity of our approach (L323-L326)**. **Below, we provide detailed information** about the computational complexity:\n\n| Image Resolution | Memory Usage (MB) |\n| --- | --- |\n| 512 x 512 | 4297 |\n| 1024 x 1024 | 7057 |\n\nThe table above shows the computation time and memory usage for processing images of various resolutions on a standard GPU (NVIDIA RTX A6000) using float16 precision. While the latent diffusion models increase the computational load, the performance remains within acceptable limits for many practical applications.\n\n**W2: Real World Scenarios** \n\nThank you for raising this important point. In the original paper, we demonstrated our model’s generalizability by **training on synthetic datasets and evaluating on unseen real-world datasets, where it achieved outstanding results (L283-284)**. To further validate our model’s robustness, we have **supplemented our evaluation with tests on the VideoMatte240K dataset** [1], also using the model trained on the Composition-1k dataset. The quantitative results are as follows:\n\n| Method | MAD | MSE | dtSSD |\n| --- | --- | --- | --- |\n| MODNet | 9.41 | 4.30 | 2.23 |\n| Our Model | 5.32 | 2.21 | 1.98 |\n\nAdditionally, **the qualitative results are provided in Figure C of the rebuttal PDF**. These results show that our model performs well on unseen video test data, further demonstrating its generalization capability.\n\n[1] Real-time high-resolution background matting. CVPR’21\n\n**W3: Biases From Pretrained Model**\n\nThank you for raising this important concern. As mentioned in the limitations section of our paper, **our approach indeed relies on pre-trained Latent Diffusion Models (LDMs), which may carry biases from their training data (L326-331)**. Regarding **rare objects that are not well-represented by the original SD model, this is an area that requires further research, and currently, there is no suitable dataset for evaluation**. However, it is undeniable that the training dataset used for LDMs, such as LAION-5B, is extremely large. This extensive dataset is a significant reason why our method demonstrates better generalization capabilities compared to other approaches (L266-284).\n\n**W4: Textual Input Feature**\n\nThank you for raising this question. Our approach indeed builds on the Stable Diffusion (SD) model (L147-149), but we have customized it to focus specifically on the image matting task, **intentionally omitting the textual input feature of the SD model**. This decision was made to **simplify the model's architecture and optimize it for the specific requirements of image matting**.\n\nBy doing so, we ensure that the cross-attention layers fully capture the intricate details and dependencies in the visual input without being influenced by text-based guidance. Our state-of-the-art (SOTA) results across multiple datasets (L266-284) demonstrate that omitting the textual input does not lead to a degradation in performance.\n\n**W5: Classifier Free Guidance**\n\nIn our paper, **we clarified that our approach uses an empty text input (L222-223)**. Classifier-free guidance is typically employed to enhance the ability of a model to follow textual descriptions by combining the predictions from a given text prompt with those from an empty text prompt. However, in our method, the text input is deliberately left empty, which means that CFG is not utilized.\n\n**W6: Fine-Tuning Based on Layer Diffusion**\n\nWe appreciate the suggestion to consider fine-tuning based on layer diffusion. Our current approach leverages pre-trained latent diffusion models (LDMs) for robust performance (L7-9). However, we recognize the potential benefits of layer diffusion fine-tuning. **We are exploring various fine-tuning strategies to improve our model's robustness and accuracy and plan to incorporate these techniques in future iterations.** Thank you for your valuable feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plAuaCduh2", "reviewer": "Reviewer_TjXY", "summary": "The paper describes a new alpha matting model, which builds upon Stable Diffusion v2, adding various blocks including a switcher for training for alpha and foreground color prediction using the same model, cross-domain attention and an alpha decoder. Unlike the usual emphasis on optimizing models for solely alpha prediction, the authors also tune their model to also accurately predict foreground colors. Their results are competitive with the state-of-the-art alpha matting models in alpha estimation.", "questions": "* How does your model handle different trimap shapes? Could you comment on the robustness w.r.t. the trimap accuracy?\n* Did you test your model on video matting? Would you expect the alpha/foreground color predictions to be temporally coherent (given coherent trimaps of course)? \n* Would your model potentially benefit from end-to-end training where the encoder weights are not frozen?", "limitations": "Some discussion on limitations is present in the final section. If space permits, I'd encourage further discussion of failure cases.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The emphasis on estimating foreground colors is spot on. In fact alpha mattes by themselves are of little practical use in the absence of foreground color estimates.\n* Alphas and foreground colors are predicted efficiently through a single model thanks to a foreground-alpha switcher. This is more efficient than training an additional model for estimating foreground colors, and as Tab 3. suggests seems to work slightly better than directly predicting alphas and foreground colors using a single model without a switcher.  \n* The alpha mattes predicted by the proposed model is competitive with the state-of-the-art.", "weaknesses": "* The major weakness of this work is the absence of a proper baseline for the foreground color estimation. This weakness is further amplified by the authors' emphasis on foreground color estimation throughout the paper. For instance, Tang et al. Learning-based Sampling for Natural Image Matting. CVPR'19. sequentially predicts background and foreground colors before alpha prediction. Another alpha matting method by Aksoy et. al. Information-Flow Matting. 2017 has an explicit part for accurate color prediction. In addition to such methods that explicitly predict foreground/background colors, existing alpha matting methods can be trivially be extended to additionally predict foreground and background colors. Given all that, the evaluation presented in Table 3 is insufficient to prove the claims on accurate foreground color prediction.  \n* Overall the technical novelty of the paper is limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Absence of Baselines for Foreground Color Estimation**\n\nThank you very much for highlighting the need for a proper baseline in foreground color estimation. We agree that this aspect warrants further discussion and comparison.\n\n1. **Related Work Discussion**\nIn the related work section, **we initially focused on methods that simultaneously predict alpha and foreground RGB, as well as those that output alpha followed by post-processing to obtain foreground RGB (L79-83)**. However, we acknowledge that the two works mentioned by the reviewer that adopt a sequential approach where foreground colors are predicted before alpha. This is indeed a significant method, and **we will include a detailed discussion of these works in the revised related work section**.\n2. **Evaluation with Baselines**\nTo address the reviewer's concern regarding the absence of a proper baseline, **we have added an evaluation of the Learning-based Sampling (LBS) method as a foreground estimation technique**. The dataset and metrics for foreground estimation are consistent with those in the paper (L206-212, L245-249). **The qualitative results are presented in Figure.A of the rebuttal PDF, and the quantitative results are provided in the table below.** The evaluation shows that our method, when leveraging the powerful priors from latent diffusion models (LDM), significantly outperforms other methods, achieving state-of-the-art (SOTA) performance in foreground color estimation.\n    | Method | SAD | MSE |\n    | --- | --- | --- |\n    | LBS | 49.7 | 8.6 |\n    | Ours | 34.1 | 4.3 |\n\n**W2: Technical Novelty**\n\nThank you for your valuable feedback. Our approach addresses significant challenges in the ill-posed problem of image matting by incorporating prior knowledge through latent diffusion models. Here are our main technical contributions (L64-69):\n\n1. **Transforming Generative Models for Matting**: We modified LDMs, which are generative models, into discriminative models for matting by modeling the task as conditional generation, making this the first LDM-driven matting method.\n2. **Joint Prediction of Foreground and Alpha**: To bridge the gap between LDM's RGB output and our RGBA output, we introduced a switcher and cross-domain attention mechanism. These facilitate mutual information exchange, ensuring high consistency and accurate joint predictions.\n3. **Mitigating VAE Reconstruction Errors**: We proposed a latent transparency decoder to address the VAE's inherent reconstruction error, aligning RGBA predictions with the input image to preserve necessary details for high-quality matting.\n\n**Q1: Different Trimap Shapes**\n\nWe acknowledge the importance of trimap accuracy and have ensured our model's robustness. During training, **we use data augmentation (L227-234), varying the size and number of dilation and erosion kernels to enhance robustness to different trimap shapes**. For the Composition-1k and AIM-500 datasets, **predefined trimaps** facilitate method comparisons.\n\nTo validate our model's robustness, **we conducted additional experiments with different trimap shapes**, adding two new trimaps with medium and large shapes in the Composition-1k dataset, using 10 and 20 iterations with kernel sizes of 50 for dilation and erosion.\n\nThe **qualitative results of these tests are shown in Figure B of the rebuttal PDF**. As expected, the larger trimap includes more unknown regions, increasing the prediction challenge. However, our model still performs well. The **quantitative results in the table below** show that although alpha prediction accuracy decreases with larger unknown regions, our model maintains high precision, demonstrating its robustness.\n\n| Trimap Shape | SAD | MSE |\n| --- | --- | --- |\n| Large | 21.6 | 3.2 |\n| Medium | 21.2 | 3.2 |\n| Small | 20.8 | 3.1 |\n\n**Q2: Video Matting**\n\nWe **supplemented our evaluation with tests on the VideoMatte240K dataset** [1] using the model trained on Composition-1k. For each frame, we generated a trimap with erosion and dilation (kernel size 10, iterated five times) and conducted tests at a resolution of 512×288. Evaluation metrics included MSE(L242-244) and dtSSD for temporal coherence.\n\nThe **quantitative results are presented below**. We included results from a human video matting method, MODNet[2]. Since our model uses trimaps for each frame, our results generally outperform those of human video matting methods.\n\n| Method | MAD | MSE | dtSSD |\n| --- | --- | --- | --- |\n| MODNet | 9.41 | 4.30 | 2.23 |\n| Our Model | 5.32 | 2.21 | 1.98 |\n\nThe **qualitative results, shown in Figure C of the rebuttal PDF**, indicate that our model maintains good performance on unseen video test data. This also demonstrates the generalization capability of our model.\n\n[1] Real-time high-resolution background matting. CVPR’21\n\n[2] Modnet: Real-time trimap-free portrait matting via objective decomposition. AAAI’22\n\n**Q3: End-to-end Training**\n\nThank you for the insightful suggestion. End-to-end training with unfrozen encoder weights presents several challenges. Firstly, **optimizing the latent space while treating it as an optimization target can destabilize the training process**. Additionally, training both the VAE encoder and decoder simultaneously **increases memory usage significantly**.\n\nThis approach would also **require balancing three types of loss functions**: latent space constraints, pixel space constraints, and latent space regularization, **necessitating an extensive hyperparameter search** and increasing computational costs exponentially.\n\nWhile end-to-end training could offer benefits, **the costs and complexities currently outweigh those of our two-stage approach**. Implementing this during the rebuttal period is impractical. However, we appreciate the suggestion and acknowledge its potential for future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Summary of Revisions:**

To all reviewers,

We would like to express our sincere gratitude for your valuable efforts. We have meticulously reviewed all the feedback provided and made the necessary revisions to our paper. Below is a summary of the major changes incorporated into the final version. Additionally, the qualitative results are included in the PDF file below. In response to the concerns and comments raised by the reviewers, we have carefully addressed each point in the following point-to-point response.

**The major changes are as follows:**

1. Added a table to clarify the relationship between sampling steps and accuracy, which complements Figure 8 in the appendix, in response to Reviewer YFFw’s comments.
2. Included a visualization figure to illustrate the necessity of the Trimap, as requested by Reviewer YFFw.
3. Reported detailed computational complexity values to address the concerns raised by Reviewer 8UU8.
4. Conducted additional experiments on the video matting dataset, incorporating the feedback from Reviewers 8UU8 and TjXY.
5. Enhanced the related work section on foreground color estimation, following the recommendations of Reviewer TjXY.
6. Introduced a new baseline for foreground color estimation using the Composition-1k dataset, as suggested by Reviewer TjXY.
7. Performed an experiment on alpha prediction with different Trimap shapes to further demonstrate the robustness of our method, based on Reviewer TjXY’s comments.

We eagerly look forward to further discussions. Thank you for your thoughtful consideration.

Meta-review: The paper introduces DRIP, a novel approach for image matting that leverages latent diffusion models (LDMs) to jointly predict foreground color and alpha matte. The method incorporates a switcher and cross-domain attention mechanism to improve prediction consistency and mitigate reconstruction errors. The approach is validated on several benchmarks, showing strong performance.
We are glad to accept the paper, despite slightly divergent reviews. The paper offers valuable contributions to the field of image matting, particularly in the joint prediction of foreground and alpha. However, concerns were raised about the computational complexity, the generalization to real-world scenarios, and the robustness of the method. The authors have addressed most of these concerns in the rebuttal, leading to a cautious recommendation for acceptance. 
We encourage the authors to improve their camera-ready version based on all the reviews.
Decision: Accept (poster)