DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting

Authors: xiaodi  li
zongxin  yang
ruijie  quan
yi  yang
alpha pred pred
  froeground
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:48:31

Abstract: Recovering the foreground color and opacity/alpha matte from a single image (i.e., image matting) is a challenging and ill-posed problem where data priors play a critical role in achieving precise results. Traditional methods generally predict the alpha matte and then extract the foreground through post-processing, often failing to produce high-fidelity foreground color. This failure stems from the models' difficulty in learning robust color predictions from limited matting datasets. To address this, we explore the potential of leveraging vision priors embedded in pre-trained latent diffusion models (LDM) for estimating foreground RGBA values in challenging scenarios and rare objects. We introduce Drip, a novel approach for image matting that harnesses the rich prior knowledge of LDM models. Our method incorporates a switcher and a cross-domain attention mechanism to extend the original LDM for joint prediction of the foreground color and opacity. This setup facilitates mutual information exchange and ensures high consistency across both modalities. To mitigate the inherent reconstruction errors of the LDM's VAE decoder, we propose a latent transparency decoder to align the RGBA prediction with the input image, thereby reducing discrepancies. Comprehensive experimental results demonstrate that our approach achieves state-of-the-art performance in foreground and alpha predictions and shows remarkable generalizability across various benchmarks.

Main paper: 
 introduction: 
 Image matting aims to isolate the foreground object from composited images, a long-standing and fundamental task in vision intelligence [1] . It is indispensable for various downstream applications, such as media production, virtual reality, and image/video editing [2, 3] . Mathematically, image matting begins with solving the inverse problem of the composition equation:EQUATIONwhere i denotes the index of a pixel. Here, all quantities on the right-hand side are unknown, and the prediction of the alpha matte α and foreground color represents an ill-posed problem.In the past decade, advances in deep learning have significantly pushed the boundaries of image matting, rapidly becoming the mainstream direction in this field [2, [4] [5] [6] [7] . Despite their impressive performance, two challenges remain unresolved in this domain: (i) high-quality foreground color prediction. As illustrated in Fig. 1 , most matting methods consist of two stages: namely, alpha prediction with neural networks and foreground isolation via post-processing. These methods typically struggle to generalize and recover high-fidelity foregrounds due to the accumulated errors in alpha prediction and post-processing. (ii) accurate prediction of semi-transparent objects.When the target to be predicted contains large areas of semi-transparency (e.g., a water glass) or [20, [4] [5] [6] [7] commonly predict the alpha matte and then infer the foreground color by post-processing [21] , which often relies on empirical assumptions such as local smoothing of the foreground and background, leading to the accumulation of errors. (b) In contrast, our joint prediction approach estimates both the foreground and alpha simultaneously. By leveraging LDM's [8] powerful natural vision prior, our predicted foreground is closer to natural images. semi-transparent regions with high-frequency details (e.g., patterned semi-transparent fabric), existing methods still struggle to predict the alpha matte accurately.High-quality semitransparent matting data is difficult to annotate on a large scale. Therefore, how can we enhance the algorithm's generalization capability for semi-transparent objects and achieve high-quality foreground prediction with limited training data? Recently, with the emergence of large pre-trained generative models [8] [9] [10] [11] [12] , their data priors learned from billions of images (e.g., LAION [13] ) are found to be useful for various downstream tasks [14] [15] [16] [17] [18] . Back to the challenging image matting, we posit that the data priors learned from tons of natural images are also intuitively beneficial. Hence, our key insight is to unleash the data priors from the large pre-trained generative models (LDM) to estimate alpha and foreground simultaneously.To achieve this, we propose Drip, the unleashing Diffusion priors for joint foreground and alpha prediction method, which follows the diffusion paradigm and jointly generates the foreground and alpha map conditioned on the image and trimap input. Specifically, we wisely design a cross-domain switcher that leverages domain-aware embedding to unify the foreground and alpha generations in a single-diffusion model. This design facilitates mutual information exchange and ensures high consistency between foreground image and alpha. Besides, the pre-trained VAE compresses the image into a compact latent space, significantly reducing training consumption while inevitably missing detailed information. To narrow the errors caused by VAE, we introduce an auxiliary latent transparency decoder, which is implemented by inserting the features from early layers in the encoder into the decoder with several learnable zero-conv layers [19] . This latent transparency decoder significantly contributes to high-fidelity foreground image and alpha prediction and also effectively adapts the pre-trained LDM into image matting.We extensively evaluate the performance of our method through extensive experiments and comparisons. The results demonstrate that our approach achieves state-of-the-art performance on the Composition-1k test set and exhibits stronger generalizability on other benchmark datasets. Remarkably, Drip outperforms all the previous methods in the mainstream benchmark, Composition-1k, where Drip improves the SAD metric of alpha prediction by 3.3% and foreground by 12.1% and MSE metric of alpha by 6.1% and foreground by 28.33%. In summary, the key contributions of this paper are as follows.• To our best knowledge, we introduce the first LDM-driven matting method, Drip, which effectively unleashes the data priors learned from LDM into image matting. • To enable joint prediction of foreground and alpha, we propose a switcher and a cross-domain attention mechanism, facilitating mutual information exchange and ensuring high consistency. • To mitigate the inherent reconstruction errors of the LDM's VAE decoder, we propose a latent transparency decoder to align the RGBA prediction with the input image. 
related work: 
 Image Matting is aimed to extract the foreground objects from arbitrary natural images [22, 2] . Traditional methods always need the auxiliary user input like trimap [23, 4] and scribble [24, 25] . These methods basically only leverage low-level color or structure features, which limits their ability to distinguish foreground details from images. With the success of deep learning, researchers have begun to use deep convolutional neural networks (CNNs) to predict the alpha map in an end-to-end fashion [26, 20] . One type takes images and auxiliary trimap or scribble as input and outputs the alpha map [5] . In order to alleviate the demand for trimap, trimap-free methods [27, 6, 28] are proposed to directly predict alpha mattes from the input image, which increases efficiency while sacrificing performance. Although these existing methods achieve impressive results in alpha prediction, accurately predicting foreground and background colors remains an essential yet challenging task for high-quality matting. Tang et al. [29] and Aksoy et al. [30] firstly proposed to address color estimation by sequentially or directly predicting the background and foreground colors before alpha prediction. Furthermore, recent method [31] unifies foreground, background, and alpha matte into an end-to-end framework. While, another line of works [32, 28] focuses on foreground human extraction and alpha matte prediction. However, these methods are limited by the lack of high-quality labeled data. Meanwhile, the explosion of generative models shows immense potential in providing priors in different tasks. In this work, we explore unleashing diffusion priors within stable diffusion [8] to improve the performance of image matting.Diffusion Models have emerged as a powerful class of generative models, which learn a reverse denoised process from the Gaussian noise to natural images [33] . In the vanilla DDPMs [33] , the sampling process is time-consuming due to the Markovian property. To speed up the sampling, DDIMs [34] is proposed to provide a non-Markovian shortcut. Furthermore, LCM [35] just formulate the diffusion process as one-step denoising via an ODE. Besides the speed, a series of works [36, 10, 37, 38] focus on increasing the controllability of diffusion models. For instance, Controlnet [19] fine-tunes a Stable Diffusion model with zero convolutions, which proves to be effective in adapting the pre-trained diffusion models to different tasks by adding different conditions.Diffusion Priors in Visual Perceptive Tasks are prevalent and hot topics. A series of works leverage the diffusion priors in segmentation [17] , image enhancement [39] , depth estimation [40] and 3D vision [41, 42] . In the context of image matting, Xu et al. [43] propose to formulate alpha prediction as a denoised process, and train a condition generation model in DDPMs fashion. However, the vanilla DDPMs have not been scaled-up training due to their expensive computational cost. On the contrary, LDM [8] proposes to compress the features into a compact latent space, which obviously reduces the computational cost. And based on it, Stable Diffusion is largely trained on the large-scale dataset [13] .However, LDM generates the image features in the latent space encoded by pretrained VAE. In order to alleviate the domain gap, Marigold [16] finetunes UNet backbone of diffusion models to perform affine invariant monocular depth estimation and exhibit strong generalization capability. Inspired by this, we carefully discuss and propose a novel method to unleash the diffusion priors within stable diffusion to improve the performance of image matting while preserving high-fidelity details.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In the conclusion section we discussed the limitation of our method. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
drip: 
 Overview. Drip is an LDM-based matting model designed to predict both foreground and alpha values while ensuring high consistency between these two representations. Given an input image (X) and a trimap (T ri) indicating the object to be matted, our goal is to estimate its corresponding Foreground (F ) and alpha (α). Initially, we explore the problem using the diffusion paradigm (see Sec. §3.1). Subsequently, we present our LDM-based matting model (see Sec. §3.2). This model employs a cross-domain switcher to simultaneously generate the foreground color and alpha map using a single diffusion model. Moreover, through mutual information exchange, the model effectively enhances boundary and texture consistency. To address the challenge of missing high-frequency information caused by VAE compression, the model incorporates an auxiliary latent transparency decoder (see Sec. §3. 3 ). An overview of Drip is provided in Fig. 2 .The task of foreground and alpha estimation is to model the mapping f (•) : (X, T ri) → (F, α), where F ∈ R H×W ×3 represents the foreground and α ∈ R H×W represents the alpha map. The input conditions are an RGB image X ∈ R H×W ×3 and a trimap T ri ∈ R H×W , which consists of three values indicating the foreground, unknown, and background regions, respectively. However, unlike prior works that adopt CNN or transformer as architecture, we employ a diffusion-based scheme f (•) to model the joint foreground and alpha distribution p(F, α).Diffusion Probabilistic Models [44, 33] define a forward Markov chain that progressively transits the sample x drawn from data distribution p(x) into noisy versions ϵ ∼ N (0, I), T is the timestep, α t and σ t are the noisy scheduler terms that control sample quality. In the reverse Markov chain, it learns a denoising network ϵ θ (•) parameterized by ϵ usually structured as U-Net [45] to transform x t into x t-1 from an initial Gaussian sample x T through iterative denoising.x t ∈ (1, T )|x t = α t x 0 + σ t ϵ, where Switcher Sample Noise � ~�(�) � Tri � � � � � � � � � � � � � � � �� � � � � � � � Denoising Net � � � � � � 2 � � � � � � � � � � � � training objective Switcher �(0, � ) t Denoising Net t � � � � � � � �� � � � � � -1 � � � -1 � � 0 � � 0 � (X,� Tri � � � � � � � � � � � � � � � �� � � � � � � � Denoising Net � � � � � � 2 � � � � � � � � � � � � training objective Switcher �(0, � ) t Denoising Net t � � � � � � � �� � � � � � -1 � � � -1 � � 0 � � 0 � (X,For the joint foreground and alpha distribution p(F, α), given a conditional input image X with its corresponding trimap T ri, the foreground F and the alpha map α can be obtained by the generative formulation in Markov probabilistic form:EQUATIONTo enhance computational efficiency and generate higher-resolution images, Stable Diffusion [8] employs the latent diffusion model, where the diffusion steps are performed in the low-dimensional latent space instead of directly operating on the original data. The latent space is formed within the bottleneck of VAE [46] , which is trained separately from the denoiser. This design allows latent space compression and facilitates perceptual alignment with the data space.To translate our formulation (Eq. 2) into the latent space, we obtain the corresponding latent code for a given image using an encoder:z (i) = E(i), where i ∈ X, T ri, F, α. It's worth noting that we triplicate the single-channel trimap and alpha map into three channels. Moreover, the denoiser ϵ θ (•) is subsequently trained in the latent space. To obtain the desired outputs, given latent codes z F and z α , the foreground and alpha can be reconstructed using the decoder D:F = D(z F ) and α = D(z α ).It is worth noting that in the Matting task if the reconstructed image is obtained directly from the latent representation of the foreground and alpha without making any modifications to the VAE, a significant error can occur.We base our model on a pretrained text-to-image LDM (Stable Diffusion v2 [8] ), which has learned strong and generalizable image priors from LAION-5B [13] . In order to accept a given image and trimap as conditions and simultaneously generate both foreground and alpha outputs, we quadruple the input of the original U-Net and employ a switcher mechanism to expand the capabilities of the original LDM model. Additionally, we incorporate cross-domain attention to enhance consistency. 2). To enhance mutual guidance and ensure contextual consistency, we instead utilize a cross-domain self-attention mechanism instead of self-attention to associate the foreground and alpha latent.Previous matting models [20, [4] [5] [6] have primarily focused on predicting the alpha value and utilizing post-processing methods, such as the local smoothness assumption, to estimate the foreground color [21, 47] . However, this two-stage approach often leads to error accumulation and suboptimal foreground estimation results, particularly when dealing with transparent objects. To address this limitation, we propose a novel approach that leverages an LDM-based method to predict the foreground and alpha values simultaneously. Our method generates more realistic foreground images by incorporating a strong natural image prior distribution learned from Stable Diffusion v2 [8] .To incorporate foreground and alpha estimation, one straightforward approach is to finetune two U-Nets separately to model their respective distributions. However, this method introduces additional parameters and fails to capture the inherent connections between foreground and alpha. Motivated by the work [48, 14] , we propose a novel approach using a switcher that enables a single stable diffusion model to generate both foreground and alpha values based on indicators. Mathematically, the foreground and alpha values can be obtained as follows:EQUATIONEQUATIONIn the above equations, s F and s α are one-dimensional vectors controlling the foreground and alpha domains, respectively. The switchers are encoded using low-dimensional positional encoding and combined with time embedding within the U-Net architecture.To further facilitate mutual-guided optimization, we introduce a modification to the self-attention layer in the U-Net architecture, transforming it into a cross-domain self-attention layer that encourages spatial alignment (refer to Fig 3) . This operator enhances the geometric consistency between the foreground and alpha channels and accelerates convergence. The cross-domain attention operation, denoted as AttCD(•), is defined as follows:EQUATIONHere, i = F, α, h F and h α represent the latent embeddings of the foreground and alpha channels within the transformer blocks, respectively. The symbol ⊕ denotes the concatenation operation. W q , W k , and W v are the matrices of query, key, and value embeddings, respectively. Finally, Att(•) refers to the softmax attention mechanism.We adopt annealed multi-resolution noise noises [16] to preserve low-frequency details in the depth and normal maps, as similar values will frequently appear in local geometric regions. This deviation proves to be more efficient than a single-scale noise schedule. We perturb the two geometry branches with the same timestep scheduler to decrease the difficulty when learning more modalities. The canonical standard learning objective we utilize is defined as follows:EQUATIONHere, ϵ F t and ϵ α t are two Gaussian noises independently sampled from annealed multi-scale noise sets for the foreground and alpha, respectively. � � � � � � 2 � � � � � � � � � � � Switcher t � � � � � � -1 � Self AttentionThe compression of the VAE [46] in LDM introduces a non-negligible reconstruction loss, indicating that x ̸ = D(E(x)). Previous discriminative tasks using the LDM [8, 12] have primarily focused on tasks such as segmentation and depth estimation. In comparison, matting is a task that places more emphasis on capturing fine details, making the errors introduced by the VAE more noticeable.To address this challenge, we draw inspiration from LayerDiffusion [49] and propose a novel latent transparency decoder D trans . This decoder takes the output images of the LDM-based matting model 3.2 and the corresponding latent as inputs to generate results that are more consistent with the details of the composition image (refer to Fig 4) . Mathematically, the expression is given byEQUATIONwhere ẑF and ẑα represent the foreground and alpha predictions in the latent space, respectively. Additionally, D represents the original VAE.By incorporating this transparent latent decoder into our framework and training it with the matting loss, we aim to improve the fidelity of the predicted foreground and alpha outputs, ensuring that they capture the intricate details present in the composition image. This enhancement is particularly crucial for the task of matting, which relies heavily on preserving fine details and boundaries. [50] . This approach allows for the generation of a sufficient training set despite having a smaller number of unique foreground object images. Similarly, the test set is generated by synthesizing 50 test foreground images using 20 background images from VOC2012 [51] , resulting in a total of 1000 test images.AIM-500 dataset [6] is a benchmark for natural image matting that encompasses various object categories. It consists of 500 high-resolution real nature images, each with a minimum short side length of 1080 pixels. Unlike other natural matte datasets [52] [53] [54] that are often limited to specific classes of human and animal images, AIM-500 offers a more diverse range of objects. Evaluating the performance on the AIM benchmark helps assess the model's ability to generalize well to natural images rather than solely fitting the distribution of synthetic images. Therefore, the evaluations conducted on both the Composition-1k and AIM benchmarks complement each other, providing a comprehensive understanding of matte models in real-world scenarios.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
implementation details.: 
 We employed Diffusers [55] with Stable Diffusion v2 [8] as the backbone for implementing our Drip.Text conditioning was disabled by providing empty text input. To accommodate the two additional conditions, the weights of the first layer of the UNet were copied three times as initialization. After training the LDM-based matting network, the obtained output and latent results were utilized as inputs for training the latent transparent decoder.Ours � FBA � Ours � FBA �� ImageOurs �� FBA � Figure 5 : Qualitative Result of Foreground. The visual results compared with FBA [31] on AIM-500 [6] . Please zoom in for the best view.Table 1 : Comparison of our Drip with State-of-the-Arts (SOTAs) on the synthetic dataset Composition-1k [5] and the natural dataset AIM-500 [6] , focusing on four metrics for alpha prediction results.( §4.2). To enhance the diversity of the dataset, we performed various data augmentation techniques during the training of the 2D image set. These included random horizontal flipping, cropping, and photometric distortion. Besides, in order to enhance the foreground, we followed LayerDiffusion [49] to fill the foreground image. Pixels with Alpha values equal to zero in the foreground have no impact on the appearance of the alpha-blended image when assigning any color. Nevertheless, since neural networks tend to produce high-frequency patterns surrounding image edges, avoiding unnecessary edges in the RGB channels prevents potential artifacts. To achieve this, we applied Gaussian blurring to regions of the foreground image where the Alpha value was strictly equal to zero.During training, the DDPM noise scheduler [58] with 1000 diffusion steps was applied. At inference time, the DPM solver scheduler [59] was employed, and only 10 steps were sampled. The model was trained for 30,000 steps with a total batch size of 96, using an image size of 512 × 512 exclusively on the Composition-1k dataset [5] . The entire training procedure typically took approximately 2 days when executed on a cluster consisting of 4 Nvidia Tesla A100-80GB GPUs. For optimization, the Adam optimizer was used with a learning rate of 1 • 10 -5 .We employ standard evaluation metrics to assess the quality of alpha predictions. Specifically, we report the Sum of Absolute Differences (SAD), Mean Square Error (MSE), Gradient loss (Grad), and Connectivity loss (Conn). A lower value for these metrics indicates a higher quality alpha matte.Additionally, we follow the evaluation methodology of FBA matting [31] to evaluate the quality of foreground predictions. Since colors other than the foreground object region are not used, we only consider the region where the ground truth alpha, denoted as α gt , is located and use it as a weighting factor. We apply the SAD and MSE to evaluate the α gt F predictions. A lower value for these metrics indicates a better quality foreground prediction. For alpha estimation, we consider several state-of-the-art methods as baselines, including both trimap-based approaches and automated methods. Among the automated methods, HATT [27] , AIM [6] , and GFM [32] are included. On the other hand, the trimap-based methods consist of DIM [5] , IndexNet [56] , FBA [31] , MFormer [57] , and ViTMatte [7] . These trimap-based methods are all deep learning-based, with the backbone architecture gradually transitioning from convolutional networks to transformer-based networks. Furthermore, with the development of diffusion, the contemporary method DiffMat [43] is also based on diffusion. However, it is important to note that DiffMat [43] does not operate in the latent space, nor does it leverage the priori of the natural image distribution learned from diffusion.Composition GT Trimap DIM VitMatte Ours DiffMatteFor foreground estimation, we compare our proposed method with a limited set of baselines, as many matting algorithms primarily focus on alpha estimation rather than foreground estimation. In our comparison, we include Global-Matting [60] and KNN-Matting [61] as non-deep learning methods. We consider ContextAware-Matting [62] and FBAMatting [31] for deep learning-based methods. These approaches leverage deep neural networks to estimate the foreground and have demonstrated promising results in previous studies. The results are shown in Table 1 for alpha. The results indicate that our method outperforms others by a large margin and achieves state-of-the-art (SOTA) performance. On the Composition-1k dataset, our method improves the SAD metric by 0.7 (+3.3%) and the MSE metric by 0.2 (+6.1%) compared to the ViTMatte [7] method. The performance improvements are even more noticeable on the AIM-500 dataset, where our method improves the SAD by 26.6 and the MSE by 14.6 compared to the AIM [6] approach. For the foreground metrics, as shown in Table 2 , our method demonstrates significant improvements compared to the FBA-Matting [31] . Specifically, we improve the SAD metric by 4.7 (+12.1%) and the MSE metric by 1.7 (+28.33%). We conduct ablation studies on AIM-500 [6] to investigate the contributions of the proposed joint prediction of foreground and alpha and the transparent latent decoder.switcher sample noise �~�(�) � Tri � � � � � � � � � � � � � � � �� � � � � � � denoising net � � � � � � 2 � � � � � � � � � � � � training objective switcher �(0, �) t denoising net t � � � � � � � �� � � � 0 � � 0 � (X,Tri )Composition GT w/o Full ModelFigure 7 : Ablation Result. When the transparent latent decoder is not used, the generated output exhibits significant differences in lowlevel details with the original input.We first investigate the effect of the proposed crossdomain attention mechanism on the joint foreground and alpha estimation(cf. §3.2). As shown in Table 3a, when the cross-domain attention module is removed, we observe a decrease in the prediction accuracies of both the foreground and alpha representations. Importantly, the accuracy of the final RGBA composition, obtained by multiplying the predicted foreground and alpha, decreases more significantly. This suggests that the information interaction between the two modalities, facilitated by cross-domain attention, increases the consistency of spatial information between the foreground and alpha representations. This finding verifies that crossdomain self-attention can effectively correlate the two representations, enabling them to benefit from each other's contextual cues mutually.For the switcher, we also notice a clear reduction in performance, suggesting that giving additional embedding information rather than simply distinguishing between foreground and alpha with channel order is helpful in relation to the neural network's ability to distinguish and utilize information from the two modalities.Transparent Latent Decoder Next, we ablate the effect of the D trans (cf. §3.3). As reported in Table 3b , using the original VAE decoder without Dtrans leads to a degradation in the prediction performance of both alpha and foreground. In addition, the qualitative results on composition-1k also clearly show that the low-level details correspond better after applying D trans , mitigating the errors induced by VAE compression.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: In the implementation part of the experiment section we elaborate on the details needed for method reproduction. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion and future work: 
 This work addresses two key challenges in image matting: high-quality foreground prediction and accurate semi-transparent object alpha estimation. To overcome these, we propose Drip method, which leverages data priors from large pre-trained generative models to jointly predict the foreground and alpha. Drip utilizes a switcher and a cross domain attention for consistent foreground-alpha generation and a latent transparency decoder to enhance fidelity. Extensive experiments demonstrate Drip achieves sota performance on Composition-1k and stronger generalization on other benchmarks.While the Drip method demonstrates strong performance in image matting tasks, several key limitations warrant consideration: i) Model Complexity and Deployment. The incorporation of latent diffusion models (LDMs) substantially increases the architectural complexity of the approach. This added complexity may impact deployment and inference efficiency, particularly in real-time or resource-constrained environments where computational overhead is critical. ii) Inherited Biases from Generative Priors. Drip's methodology relies on the extensive priors captured within pretrained LDMs. Consequently, Drip inherits any biases present in the original generative model. These biases could adversely affect the method's performance on certain types of images or domains, such as those with highly complex lighting and shadows, or those containing numerous small texture details, thereby limiting its general applicability.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: Codes will be released after the paper is accepted. Data are available.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: In the implementation part of the experiment section we elaborate on the details needed for method reproduction.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: All experiments are conduct on fixed random seed.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: In the implementation part of the experiment section we elaborate on the details needed for method reproduction.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: In the reference section we accurately cite the datasets, papers and codebases used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: This appendix contains additional details for the Neurips 2024 submission, titled "DRIP: Unleashing Diffusion Priors for Joint Foreground and Alpha Prediction in Image Matting". The appendix is organized as follows:• §A.1 offers more implementation details of the multi-level annealed noise.• §A.2 offers more implementation details of the augmentation to pad the foreground.• §A.3 provides more information about the transparent latent decoder.• §A.4 provides ablation results on the number of timesteps for the AIM [6] dataset.In order to make ldm handle the details better, we adopt the annealed multi-resolution noise used in [16, 14] .In crafting the standard multi-resolution noise, an array of Gaussian noise images is initially sampled to construct a pyramid with a gradient of resolutions, which are then merged using upscaling, weighted averaging, and normalization methods. Each level i of this pyramid is assigned a weight following the formula s i , where s is a decimal fraction between 0 and 1, signifying the extent of the influence exerted by noise at reduced resolutions. To better align the resultant noise with the Gaussian distribution outlined in the foundational DDPM framework, the weights of the upper levels i > 0 are modified in accordance with a diffusion schedule. Specifically, at each time step t, the i-th level is endowed with a weight calculated as (s i /T ) t , with T representing the cumulative number of diffusion steps. As a result, the weight allocated to levels with diminished resolution is progressively reduced as the schedule nears its noise-free terminal point.Algorithm 1 annealed_pyramid_noise(x,timesteps,discount) w ← max(1, ⌊w ori /r i ⌋)8:h ← max(1, ⌊h ori /r i ⌋) {Compute the current feature map size} x idx ← j%w ori 12:y idx ← ⌊j/w ori ⌋ 13:new_x ← ⌊x idx * (w ori /w)⌋ i ← i + 1 19: until i ≥ 10 or (w ≤ 1 and h ≤ 1) {If already reached minimum resolution, break out} 20: result ← noise/noise.std {Normalize the noise tensor} 21: return resultIn order to enhance the foreground effect and avoid potential artifacts, we apply a fill technique [49] to the foreground image. Pixels in the foreground with an alpha value equal to zero, regardless of their color, do not affect the appearance of the final alpha-blended image. However, neural networks tend to produce high-frequency patterns at the edges of the image, so avoiding unwanted edges in the RGB channel helps prevent potential artifacts. To address this issue, we apply Gaussian blurring to regions of the foreground image where the alpha value is strictly equal to zero. h, w ← shape(alpha) {Get the height and width of the input image} 2: mask ← gen_mask(alpha) {a mask where alpha = 0 are 1 and alpha > 0 are 0} i ← 0 4: while i < 64 do filtered_image ← gaussian_blur(f g, (13, 13), 0) {apply gaussian blur to foreground} 6:j ← 0 while j < h × w do 8: if mask[j] == 1 then f g[j] ← filtered_image[j] {update foreground if mask indicates transparent region} 10: end if j ← j + 1 12:end while i ← i + 1 14: end while return f g {Return the updated foreground image}The core component of our approach is the transparent latent decoder, which is implemented as a U-Net architecture. To train this module, we randomly sample the step number between 1 and 10 to get the output of latent matting network, which we then use as input. We utilize a loss function commonly used in matting algorithms to optimize the model. Specifically, we employ the combined matting loss, which includes separate components such as the l 1 loss [7, 63] , l 2 loss [63] , laplacian loss [62, 63] , and gradient penalty loss [7] for both alpha and foreground. The objective function becomes:EQUATIONWe conducted ablation studies on the number of diffusion timesteps used in our method on the AIM [6] dataset. As shown in Figure 8 , increasing the number of timesteps generally improves the performance, though the gains diminish as the number of timesteps becomes larger.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The abstract and introduction accurately summarize the key claims, contributions, and scope of the paper. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research in the paper complies with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of the work performed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper introduces DRIP, a novel approach for image matting that leverages latent diffusion models (LDMs) to jointly predict foreground color and alpha matte. The method incorporates a switcher and cross-domain attention mechanism to improve prediction consistency and mitigate reconstruction errors. The approach is validated on several benchmarks, showing strong performance.
We are glad to accept the paper, despite slightly divergent reviews. The paper offers valuable contributions to the field of image matting, particularly in the joint prediction of foreground and alpha. However, concerns were raised about the computational complexity, the generalization to real-world scenarios, and the robustness of the method. The authors have addressed most of these concerns in the rebuttal, leading to a cautious recommendation for acceptance. 
We encourage the authors to improve their camera-ready version based on all the reviews.

Reviews: review: paper_summary: paper_summary: The authors present the clear and straightforward method to improve image matting performance using an LDM-based model. The model is a conditioned LDM which can predict both the Alpha mask and the foreground, basically doing RGB-A prediction from an image and a trimap. To further adapt to the data domain, the authors also finetune a specialized decoder which is proven to be helpful. The overall design of the model and training method is intuitive and effective with sufficient evaluation. The training data is limited while can be scaled up by more data collection.
strength_weakness: strength_weakness: - The training dataset still looks quite small. The model may not have good generalization capability to more unseen data in the wild. Also the batch size and training time is limited. 
- When the authors mentioned 'prior', is it referring to the pre-trained LDM model? Did the authors use like LoRA to finetune the model on the smaller dataset? Will the performance become worse as the training goes longer?
- There seems no evidence showing the advantages of the cross-domain self-attention. Could the authors provide more details?
- It seems the switcher is just conditioned embedding for choosing outputs. Will we be able to predict both at the same time without inputing the switcher values?
questions: questions: - The transparent latent decoder is helpful. However, will it be more useful to also train an transparent auto-encoder by finetuning the existing antoencoder? When training the decoder, did the authors prepare and precompute the data from a trained diffusion model?
- How to speed up the process and could we reduce the steps of sampling?
- Any proposed methods to scale up the training data for a more robust model training?
- Will Trimap be necessary? It will be great if the model can directly do foreground object RGB-A extraction without inputting the Trimap.
limitations: limitations: - Trimap is needed for the model to do RGB-A extraction. Some dropping of the conditions may be helpful.
- Data is quite limited and training longer time may degrade the performance.
- Cross-domain self-attention is not fully evaluated.
paper_summary: paper_summary: The paper introduces Drip, an approach to image matting that leverages vision priors from pre-trained latent diffusion models (LDM). Drip incorporates a switcher and cross-domain attention mechanism for joint prediction of foreground color and opacity, ensuring high consistency. A latent transparency decoder mitigates reconstruction errors. Experiments demonstrate good performance and generalizability across benchmarks.
strength_weakness: strength_weakness: Like many other papers on the application using the DNN. The paper present an NN architecure for image matting. The building blocks are all standard one, and the design does not really reflect the uniqueness of the problem from other image editing. Overall, I am not convinced that such a paper really benefit the progress of image matting.
questions: questions: NIL
limitations: limitations: The limitation is not well discussed.
paper_summary: paper_summary: This paper introduces DRIP, a novel image matting method that leverages pre-trained LDMs to jointly predict foreground color and alpha mattes. By integrating a cross-domain attention mechanism and a latent transparency decoder, DRIP addresses the limitations of traditional methods, achieving significant performance improvements on synthetic and natural datasets. The key contributions include enhanced prediction consistency, high-fidelity results, and setting new benchmarks in image matting accuracy.
strength_weakness: strength_weakness: 1. The computational complexity is unclear. Their high computational demands may impede practical application, particularly in real-time or resource-constrained settings.

2.  The evaluation is predominantly conducted on synthetic datasets and a limited number of natural image benchmarks. More extensive testing on diverse real-world scenarios would enhance the demonstration of the model’s robustness and generalizability.

3.  The approach depends on pre-trained LDMs, which may propagate existing biases from the training data. Can the matting model perform effectively on objects that are not well-represented by the original SD model, such as rare objects?

4.  This article develops its model based on the Stable Diffusion model. It raises the question of whether it retains the textual input feature of the SD model. If so, how is image captioning handled? If not, could this omission lead to a degradation in the performance of cross-attention in SD?

5. During inference, is classifier-free guidance utilized? If so, what impact does it have on the matting performance?

6. Has there been any consideration for fine-tuning based on layer diffusion?
questions: questions: See weakness.
limitations: limitations: Yes
paper_summary: paper_summary: The paper describes a new alpha matting model, which builds upon Stable Diffusion v2, adding various blocks including a switcher for training for alpha and foreground color prediction using the same model, cross-domain attention and an alpha decoder. Unlike the usual emphasis on optimizing models for solely alpha prediction, the authors also tune their model to also accurately predict foreground colors. Their results are competitive with the state-of-the-art alpha matting models in alpha estimation.
strength_weakness: strength_weakness: * The major weakness of this work is the absence of a proper baseline for the foreground color estimation. This weakness is further amplified by the authors' emphasis on foreground color estimation throughout the paper. For instance, Tang et al. Learning-based Sampling for Natural Image Matting. CVPR'19. sequentially predicts background and foreground colors before alpha prediction. Another alpha matting method by Aksoy et. al. Information-Flow Matting. 2017 has an explicit part for accurate color prediction. In addition to such methods that explicitly predict foreground/background colors, existing alpha matting methods can be trivially be extended to additionally predict foreground and background colors. Given all that, the evaluation presented in Table 3 is insufficient to prove the claims on accurate foreground color prediction.  
* Overall the technical novelty of the paper is limited.
questions: questions: * How does your model handle different trimap shapes? Could you comment on the robustness w.r.t. the trimap accuracy?
* Did you test your model on video matting? Would you expect the alpha/foreground color predictions to be temporally coherent (given coherent trimaps of course)? 
* Would your model potentially benefit from end-to-end training where the encoder weights are not frozen?
limitations: limitations: Some discussion on limitations is present in the final section. If space permits, I'd encourage further discussion of failure cases.

Comments: title: Thanks for the authors' response
comment: Dear Reviewer,

We are delighted to hear that our responses have resolved all of your concerns! If you have any further inquiries or doubts, please don’t hesitate to inform us. We are determined to address any remaining issues and respond promptly!

Thanks again for your thorough review and reconsideration of our rebuttal!
Dear Reviewer YFFw,

We sincerely appreciate your time and effort in reviewing our submission and providing valuable comments. We have provided a detailed response to each concern you raised and hope they have adequately addressed your concerns. As the author-reviewer discussion phase is coming to an end **(Aug 13 11:59pm AoE)**, we would like to confirm whether our response has effectively addressed your questions. If you have any other questions or concerns, please do not hesitate to contact us.

Best regards,

Authors
Dear Reviewer,

Thank you for your valuable feedback on our manuscript. We appreciate your comments and have carefully considered the points you raised. As you suggested, we will include a comparison to the Aksoy et al. '17 method in the revised manuscript. This will allow us to more comprehensively demonstrate the superior performance of our approach in the FG color estimation task. We believe that this additional comparison, along with the existing evaluation against the Learning Based Sampling method and other baselines presented in the original paper, will significantly strengthen the evidence for the advantages of our proposed method.

We are grateful for your support and guidance throughout the review process. If you have any other feedback or suggestions, please feel free to share them. Thank you again for your time and consideration.
Dear reviewers, the authors are eagerly awaiting your response. The author-reviewer discussion closes on Aug 13 at 11:59 pm AoE. Thanks!
Dear reviewers, as we approach the final two days, please take a moment to review the author's responses and join the discussion. Thank you!
Thank you for the clarifications. The experiment against the Learning Based Sampling method presented in W1 partly addresses one of my main concerns with this submission. I'm accordingly increasing my initial rating. It would be a lot more convincing to also include Aksoy et al. '17 as a baseline in the final manuscript, since their work specialize in accurate FG color estimation.
Thank you for your detailed rebuttal and for addressing my concerns thoroughly. The additional experiments and insights were very helpful in clarifying the approach. The method of leveraging pre-trained LDMs for image matting is innovative and well-presented. The clarity and organization of the manuscript are notable, and I appreciate the thoroughness of this work. I stand by my recommendation to accept this paper.
Dear reviewers, a gentle reminder of the participation of the reviewer-author discussion. Thank you!
Dear reviewers, do the authors' responses answer your questions or address your concerns? Thanks.