MP-Mat: A 3D-and-Instance-Aware Human Matting and Editing Framework with Multiplane Representation
Authors: Siyi Jiao
Wenzheng Zeng
Yerong Li
Huayu Zhang
Changxin Gao
Nong Sang
Mike Zheng Shou
Venue: ICLR 2025 Poster (2025)
Keywords: image matting
image editing
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Human instance matting aims to estimate an alpha matte for each human instance in an image, which is challenging as it easily fails in complex cases requiring disentangling mingled pixels belonging to multiple instances along hairy and thin boundary structures. In this work, we address this by introducing MP-Mat, a novel 3D-and-instance-aware matting framework with multiplane representation, where the multiplane concept is designed from two different perspectives: scene geometry level and instance level. Specifically, we first build feature-level multiplane representations to split the scene into multiple planes based on depth differences. This approach makes the scene representation 3D-aware, and can serve as an effective clue for splitting instances in different 3D positions, thereby improving interpretability and boundary handling ability especially in occlusion areas. Then, we introduce another multiplane representation that splits the scene in an instance-level perspective, and represents each instance with both matte and color. We also treat background as a special instance, which is often overlooked by existing methods. Such an instance-level representation facilitates both foreground and background content awareness, and is useful for other down-stream tasks like image editing. Once built, the representation can be reused to realize controllable instance-level image editing with high efficiency. Extensive experiments validate the clear advantage of MP-Mat in matting task. We also demonstrate its superiority in image editing tasks, an area under-explored by existing matting-focused methods, where our approach under zero-shot inference even outperforms trained specialized image editing techniques by large margins. Code is open-sourced at https://github.com/JiaoSiyi/MPMat.git.

Published as a conference paper at ICLR 2025
MP-MAT: A 3D-AND-INSTANCE-AWAAR HUMAN MATTING AND EDITING FRAMEWORK WITH
MULTIPLANE REPRESENTATION
Siyi Jiao Wenzheng Zeng2*1 Yerong Li
Huayu Zhang*
Changxin Gao
Nong Sang Mike Zheng Shou
Key Laboratory of Image Processing and Intelligent Control, School of Artificial Intelligence and Automation, Huazhong University of Science and Technology. China SSho Lab, National University of Singapore
{jiaosiyi7, mike. cheng.shoop. gmail 1oaii..oo com venzhengzeng@u. nus. nus.edu .dd fyrlee, hyzhang, egao, msang)ehus edu.cr
ABSTRACT
Human instance matting aims estimate alpha matte for each human instance in an image, which is challenging as easily fails in complex cases requiring disentangling mingled pixels belonging to multiple instances along hairy and thin boundary structures. this work, we address this by introducing MP-Mat, a novel 3DD-and-instance-aaal matring framework with multiplane representation, where the multiplane concept designed from two different perspectives: scene geometry level and instance level. Specifically, we first build feature-level multi- plane representations to split the scene into multiple planes based on depth differ- ences. This approach makes the scene representation 3D-aware, and can serve an effective clue for splitting instances in different 3D positions, thereby improv- ing interpretability and boundary handling ability especially occlusion areas. Then, we introduce another multiplane representation that splits the scene in an instance-level perspective, and represents each instance with both matte and color. We also treat background special instance, which is often overlooked by isting methods. Such an instance-level representation facilitates both foreground and background content awareness, and is useful for other down-stream tasks like image editing. Once built, the representation can be reused to realize controllable instance-level image editing with high efficiency. Extensive experiments validate the clear advantage of MP-Mat in matting task. We also demonstrate its superi- ority in image editing tasks, an area under-explored by existing matting-focused methods, where our approach under zero-shot inference even outperforms trained specialized image editing techniques by large margins. Code is open-sourced http:///ggtuub.oomma/aiii///////
INTRODOCTION
Human matting one of the foundation tasks computer vision that can widely serve for applica- tions such as image editing, image compositing, and film post-production (Zhu et al., 2017; Chen et al., 2018; Sengupta et al., 2020; Lin et al., 2021; 2023). Despite the development of effective algorithms, most methods focus on human matting under single-instance scenarios, which cannot fully align with real-world applications, where multiple instances could exist in complex scene. In recent years, InstMatte (Sun al., 2022) formally introduced the multi-instance matting fomulation, which separates the image into combination of multiple instance layers and background layer:
Σ a;C +
1
'Equal contribution, 'Proeect lead, *Corresponding author.

---

Published as a conference paper at ICLR 2025
Instance M Matting
MP-Mat
Instance Editing
Instance removal ( (x) & dragging (+)
Imput
DragonDifftsion
MP-Mat (Ours)
cclusion reordering (r)
GLANOUR
GLANOUR 23
Maggle
MP-Mat (Ours)
Input
DragonDiffusion
MP-Mat (Ours)
Figure 1: The proposed MP-Mat can perform well in both instance matting and editing tasks, out- performing existing state-of-ther specialist models designed for individual tasks. Distinguished areas are highlighted with bounding boxes, where MP-Mat preserves finer details and better retains regions that should remain semantically unchanged.
where a; [0, 1] denotes the opacity (alpha matte) of the i-th foreground, whose value is the ultimate task goal. The task is actually ill-defined problem since the foreground color C. back- ground color and the alpha value a; are left unknown. Compared with single-instance assumption, multi-instance matting poses additional challenges. Specifically, the algorithm should be instance- aware (i.e., can localize and distinguish different human instances), and also needs to preserve com- plex and fine instance edges. Maintaining the integrity of each instance without blurring the edges particularly challenging in cases where instances are in contact or occluded (Ke et al., 2021).
A motivation of this work is to establish layered representation to facilitate instance matting in complex scenarios, where we split the scene into different layers based on 2 different perspectíves: depth-oriented and instance-oriented. Our design of such layered concept is partially inspired by the idea of Multiplane Image (MPI) (Tucker & Snavely, 2020), a learnable 3D representation that proposed for novel view synthesis. MPI aims to learn 3D scene representation with multiple RGBA planes from source image, which can then be used to synthesize different novel views of the scene. The learned multiple RGBA planes can split the scene based on the depth difference. We argue that such a depth-oriented plane-splitting idea could be potentially helpful for matting in multi-instance scenarios. Specifically, different instance may lie in different depth positions, so they can be effectively distinguished by grouping them into different planes that represent distinct depth level, which could be potentially helpful for handling occlusions under complex scenes. However, argue that naively using MPI for instance matting not feasible, it is build on low pixel-level based on depth differences for only pixel-level novel view synthesis and without instance-awareness.
Based the aforementioned analysis, propose MP-Mat, 3DD-and-istaanc-awaae matting framework that is built on meticulously designed multiplane representations. Specifically, our for- mulation mainly consists of parts: scene geometry-level multiplane representation (SG-MP) and instance-level multiplane representation (Inst-MP). The SG-MP is built to split the scene into multi- ple planes based on the depth differences, making the scene representation 3D-aware, and can serve as an effective clue for splitting instances in different 3D positions. Different from the existing MPI representations, the proposed SG-MP is built on feature-level, rather than the low pixel-level. The benefits lie in 2 aspects: (1) Compared with low-level RGBA, building MP features on high-level deep features can contain more semantic information to better represent the scene geometry as well as fine-grained texture context, which can be useful for the subsequent instance-level analysis based on it; (2) When optimized together with subsequent instance-leved Inst-MP for instance-level per- ception tasks (e.g., instance localization and matting), the SG-MP feature will also receive relevant gradient, making the plane division and scene representation become more instance-wwar
Besides the built SG-MP, also introduce instance-level multiplane representation (Inst-MP) that splits the scene in an instance-level perspective, and represent each instance with both matte and color (as shown in Fig. 2). Our formulation has several benefits: (1) Different from most mat-
2

---

Published as a conference paper at ICLR 2025
ting methods that only predict alpha matte, our proposed representation additionally estimates the color of each foreground, enabling better foreground content awareness, resulting in a higher mat- ting accuracy; (2) Besides foreground instances, Inst-MP also explicitly models the background as a special instance. Such formulation is different from existing works that only focus on foreground matte modeling, and enables better handling of the boundary of instance and background, especially when occlusion occurs; (3) The proposed instance-level multiplane representation obeys the inte- gration property for image rendering (i.e.. the integral of the representation equals the whole RGB image), and thus can be easily used for downstream tasks like image editing where instance-level manipulation can be directly processed on separate feature planes. Once built, the representation can be reused to realize controllable instance-level image editing with very high efficiency.
We conduct extensive experiments to demonstrate the superiority of the proposed framework. Specifically, for instance matting, our MP-Mat outperforms existing methods by large margins (i.e., at least 2.76 SAD in HIM-100K dataset (Liu et al., 2024) and 4.16 SAD in SMPMat dataset (Jiao et al., 2024)). Besides, we also take a further step to explore the capacity of MP-Mat on image editing task that is actually under-explored by existing matting-focuse method can also perform well on this potential downstream task and even outperforms existing spe- cialized image editing techniques by large margins and with high efficiency, even under zero-shot inference. A qualitative comparison is shown in Fig. 1, where distinguished regions are highlighted with bounding boxes. For both types of tasks, MP-Mat can actually preserve finer details and better retains regions that should remain semantically unchanged. We hope our work can inspire future research in related fields, including but not limited to image matting and image editing.
methods. We found that our
RELATED WORK
2.1 INSTANCE MATTING
Elder matting works (Xu et al., 2017; Tan et al., 2018; Lu et al., 2019; Zhang et al., 2019; Qiao et al., 2020: Li & Lu, 2020; Sun et al., 2021: Yu et al., 2021b; Ke al., 2022; Li et al., 2022; Ma al., 2023) assume single-instanee condition without multi-instance awareness, which remains a gap with many real-world scenarios. Human instance matting a recently emerged task that differs from the traditional one, as it requires simultaneously localizing multiple instances and distinguishing their mattes in an instance-level manner. Such a task poses more challenges as it easily fails in complex cases requiring disentangling mingled pixels belonging to multiple instances along hairy and thin boundary structures. Mainstream methods (Hu & Clark, 2019; Sun et al., 2022; Huynh et al., 2024) for this task typically rely on instance-level masks (He et al., 2017; Wang et al., 2020) as input and gradually refine them to predict the matte. While these methods provide a certain level of instance-awareness, it is relatively weak as they largely depend on pre-set instance segmentation module. Although a recent work (Liu et al., 2024) achieves independent instance-aware capability, its performance still falls short of the SoTA methods in the mainstream setting, primarily due the lack of external guidance. Different from these works, we build layered representations to emphasize 3D and full instance awareness. Specifically, the build SG-MP decomposes the scene based on depth variation, thereby improving interpretability and boundary handling ability especially in occlusion areas. Our Inst-MP, besides representing the matte of foreground instances, also explicitly models the background and color of each instance, resulting better context awareness for both foreground and background. Inst-MP also processes good properties for downstream image editing tasks that under-explored by existing matting-focused focused methods.
2.2 INSTANCE CONTROLLABLE IMAGE EDITING
The task aims to impose instance-level editing (e.g., instance removal, dragging) on the image in a harmonious way. Recently, diffusion models (Yildirim al. (2023); Shi et al. (2024); Ekin et al. (2024); Sheynin et al. (2024); Yang et al. (2024b); Ren et al. (2024)) have brought significant break- throughs in editing tasks. Despite the effectiveness, they generally need more denoising steps for high-quality generation, which is usually more time costly. Existing methods are also of weak instance-aware capabilities and cannot fully guarantee that theoretically unchanged regions remain unaffected, resulting in poor performance sic property of our proposed instance-level multiplane representation (Inst-MP) can also enable an easier and more direct way to achieve instance-level editing, where accurate instance-level feature manipulation can be done on separate feature planes without affecting other regions, thereby enhanc- ing the consistency of the edited image. Experiments show that our approach, even under zero-shot
instance-level editing tasks. In this work, the intrin-
3

---

Published as a conference paper at ICLR 2025
Scene Geometry-leeve el
wwweeeeeeelllalaae Multiplane Regresestation (SG-MP) Extraction
Instance-L. M Wultiplane Representation (last-M1 Extraction
Instance Q Queries (0.
{ ]
Scene Encoder
Transformer Decoder
Refinement Moduule
Digh Kad
PGN
(M
SG-MP
Inst-MP
Algha Matte Embeeddinge
Selt: Atteatioa
NILP,
Seltmax
(0
Color Embeddings
0
Sell- Attention
Figure 2: The overall framework of the proposed MP-Mat. @ indicates concatenation operation.
inference, significantly outperforms trained specialized image editing techniques. Another distin- guished advantage is that once Inst-MP is built, subsequent editing will take negligible time.
2.3 3D SCENE REPRESENTATION
3D scene representation has been widely used for tasks like 3D reconstruction (Mescheder et al., 2019; Wu al., 2024) and view synthesis (Kong al., 2024; Luiten et al., 2024). In this work, one of motivations is to make instance matting become more 3D-aware, Our design is partially inspired by the idea of Multiplane Image (MPI) (Tucker & Snavely, 2020; Zhou et al., 2018), 3), learnable 3D representation that is proposed for novel view synthesis. MPI aims to learn multiple planes that decompose the scene according to depth variations, which also share similar spirit with some other relevant concepts like layered depth images (LDI) (Shade al., 1998). We argue that such depth- oriented plane-splitting idea could be potentially helpful for matting in multi-instance scenarios. However, naively using MPI for instance matting not feasible, as is build on low pixel-level based on depth differences for only pixel-level novel view synthesis and without instance awareness. On the contrary, our formulation differs to within aspects: (1) MPI split the planes based on depth variations, while we design the multiplane representation from 2 different perspectives (scene geometry level and instance level) to be better aware of both depth and instance-level information; (2) MPI is built on low pixel-level, while our scene geometry level representation is built on high- level deep features, which is of better capacity for understanding scene geometry as well as fine- grained texture context. It also enables flexible end-to-end training with the subsequent instance- level task to enrich its eettanc--awarenee ability; (3) MPI generally used for low pixel-level tasks like novel view synthesis (Tucker & Snavely, 2020) and bokeh rendering (Peng et al., 2022; Rao, 2023) that lack of instance awareness, while our goal is to solve instance-level perception problems, with different formulation from original MPI.
METHOD
As shown in Fig. 2, our MP-Mat framework mainly consists of 2 parts: scene geometry-level mul- tiplane representation and instance-level multiplane representation. In the following, we will first introduce MP-Mat for instance matting detail. Then, will also describe how our method can be applied for instance-controllabl image editing tasks.
3.1 SCENE GEOMETRY-LEVE MULTIPLANE REPRESENTATION (SG-MP)
We first build multiplane representations to split the scene into multiple planes based on depth dif- ferences. This approach makes the scene representation 3D-aware, and can serve an effective clue for splitting instances in different 3D positions, thereby alleviating occlusion effects. To be specific, introduce plane-distinetive masks to represent plane information, which is obtained by plane generation network. They will be concated with the deep feature encoded from RGBD input, to form the so-called scene geoeetry-level multiplane representation.
4

---

Published as a conference paper at ICLR 2025
3.1.1 PLANE GENERATION NETWORK (PGN)
This network aims generate the plane-distinctive masks based on the input single view RGBD image, for plane information representation. To be more specific, we first define N planes that aim to split the scene based on distinction in depth and instance-level information, which we represent them masks. The PGN will gradually generate N plane-distinctive (uniformly sampled according to the scene depth map) and the actual RGBD image inputs, where the final refined masks will simuitaneously possess scene depth-distinetive and instance-aware characteristics. Note that the input depth map here can be easily estimated using off-the-shelf depth estimators, and the actual cost is comparable pre-instance mask generation in existing mainstream mask-guided instance matting methods (Sun et al., 2022; Huynh et al., 2024).
using what we call plane-distineto
masks {M based on set of predefined initial depths {d N
As illustrated in Fig. we first shared lightweight CNN & extract global feature for plane i at the initial depth d;:
Ed
Then we apply the self-attention operation to {f1) to obtain {f())=
(f) Self-Attention (U))
(3)
The intuition here is to adjust the corresponding depth information of each plane at the feature level by exchanging the geometry and appearance information among {f1) The adjusted feature S; is then decoded to the adjusted depth d; using a shared MLP Dj:
d: Da(s)
(4)
After plane depth adjustment, we then use an interpreter to generate the plane-distinctive masks based on the adjusted plane depths and the original RGBD inputs:
{ = Softmax((((I,,dd))
(5)
where H is UNet-like architecture, and the spatial resolution of M, aligns with the extracted deep feature from the scene encoder. The mask delicately allocates every visible pixel from the source viewpoint to each plane. We concat the -aate-distinetive masks and the deep scene feature to form the so-called scene geometry-level multiplane representation.
3.2 INSTANCE-LEVEL MULTIPLANE REPRESENTATION (INST-MP)
Given the obtained scene geometry-level multiplane representation, subsequently use it to derive instance-level multiplane representation {(c, a:) {=) where serves the plane index, with i corresponding to the background plane, and ranging from to S representing instance-level plane for each distinct human instances within the image.
Actually, the goal of instance matting task (i.e., instance-level matte {at);-) sub-set of our instance-level multiplane representation. Our formulation has several benefits: (1) Different from most matting methods that only predict alpha matte, our proposed representation additionally es- timates the color of each foreground, enabling better foreground content awareness; (2) Different from existing works that only focus on foreground matte modeling, our representation also explic- itly model the background by regarding it as special foreground instance. Such formulation enables better handling of the boundary of instance and background, especially when occlusion occurs; (3) The proposed instance-level multiplane representation obeys the integration property for image dering: I S Gay, Based on this property, we can reuse the built representation realize controllable instance-level image editing with very high efficiency (detailed in Section 4). Next, we will introduce how to obtain the instance-level multiplane representation in detail.
3.2.1 INSTANCE QUERY
Here we use learnable queries {Q (K > S S
1) to capture instance-level features, where S denotes the actual instance amount within image and also regard background special
5

---

Published as a conference paper at ICLR 2025
instance, resulting in the total number as S 1. 1. The queries will collect the corresponding instance- level features from the obtained scene geometry-leve multiplane representation, through a standard multi-layer transformer decoder Zou et al. (2023) that consists of multi-head self-attention and cross-attention with fully-connected feed-forward networks (FFNs). Then, we use different pre- diction heads (composed of MI.P) map the query feature Qi into distinct embeddings: opacity embeddings O E Eº that gauge transparency, and color embeddings Of E Re that capture hue information, where C' is the feature dimension. We also derive an uncertainty map O" E R H xw (H x W corresponds the spatial resolution of input image) estimate the uncertainty of the predictions, which will be used for subsequent feature refinement.
3.2.2 REFINEMENT MODULE
We design refinement module that utilizes the uncertainty map {Or K to enhance {0:),- and K
{OTh) J==1. and finally output the instance-level multiplane representation. Specifically, given the acquired uncertainty map, the top T% (i.e., 10% our implementation) of pixels with the highest uncertainty values selected, yielding filtered guidance masks { for refinement. The mask R2 is subsequently concatenated with 0Tn and Of to indicate the region of high uncertainty for each instance-level representation. Then, self-attention is adopted among different instance-level features to refine (reconsider) the representations of each instance:
K (0
===1 Self-Attetio R ,R)
(6)
{0
K ==
Self-Attetion( ( (0,, R2 )
Finally, we predicts the alpha matte a; and the color c; based on O." and 0f:
MattePredictor (O!") {ar j i=1 (a K ColorPredictor (0f)
(7)
where both MattePredictor and ColorPredictor are two-layer ML.s that decode the alpha/color em- beddings to predict the final alpha matte/color.
3.2.3 MODEL TRAINING
The whole MP-Mat framework is trained in an end-to-end manner. We employ multi-task loss,
L = Dretect X2h
(8)
Matting = Laipha + Llap + Lcomp
where Detect is the loss for instance detection, including localization and classification loss. The detection predictions are derived from instance-level matte results (the bounding box that can tightly cover the pixel with m > 0). We then perform bipartite matching (Carion et al., 2020) between the instance-level detection predictions and GTs to obtain the correspondence between predictions and GTs, and then calculate the standard matting loss La which encompasses alpha loss, pyramid Laplacian loss and composition loss following standard formulation (Sun et al., 2022).
4 MP-MAT FOR INSTANCE CONTROLLABLE IMAGE EDITING
Due to the good properties of Inst-MP mentioned in Sec. 3.2, MP-Mat can achieve instance-level image editing direct, accurate, and nearly free manner. The overarching idea is that instance- level editing can be achieved by directly processing on separate instance-level planes within Inst-MP. Once the Inst-MP built, subsequent image editing will only take negligible time, and the editing process is training-frees free. Here will describe how it works for different sub-tasks in detail.
Instance removal aims to delete specified foreground instance from the image without affecting its overall harmony. This problem can be treated reassigning the alpha matte of the removed instance j to its backward instances (including background). For target instance j. We first define 22 as the set of pixel (x.y) where (x, y) 0. For every pixel in 22,, we assign its alpha the instance plane I with (2, y) > 0 and is closest behind (can be estimated by the average depth of the plane):
ax(2r, v) ore (2. v) 0rg (x,

---

Published as a conference paper at ICLR 2025
Then, the edited image can be represented as:
I'= 1=0,
(10)
Occlusion reordering aims to switch the occlusion relationship between the source instance and target instance (e.g., changing from instance p occluding q to q occluding p). We first consider simplified case where no additional instances are positioned between p and (indicated by depth position) for demonstration purposes. In this case, the task can be done by swapping the alpha values of the two instances within their intersection regions:
df (2, v) = ap(2, y).
(11)
(2, v) = ag (z, 8),
(12)
where (x.y) refers to the pixel position that satisfy (c. 0) A (cp(z,y) > 0). The edited image can be then synthesized with the updated a;:
S
1'=
Gaa; + Gooz + cg/...
(13)
For general cases where other instances may exist between p and q. we first sort the plane based on their depth from the closest to the fastest to the camera plane, resulting in sorted plane index set 2, {.... p. p .... q- I, 9, ..... Then, we perform the aforementioned alpha swap between each pair of adjacent planes iteratively until the desired order is achieved (i.e., f.... 9, P- 1, 2.....
1, .).....). We provide more detailed description and pseudo code Appendix
Instance dragging aims to drag target instance to a new desired position, which can be divided into two categories: drag across images and drag within one image. The drag across image task can be divided into three steps: (1) Feed the reference image I into MP-Mat to get its Inst-MP, and separate the plane t (ce, 0x)) that corresponds to the target instance. (2) Crop (ct, ar) to form (ct, a{) by extracting the rectangular region that tightly covers the pixels with positive alpha value. (3) Set up a new plane (Cnews anew) with zero initialization (i.c., all pixel values are 0), and add the cropped (c,, az)) to it at the desired position that additional transformations, such as rescaling and rotation, can also be applied to (c,, oe) before adding. (4) Add the resulting new plane to the target image I:
the target image, resulting in (chow d Note
I' cee new + (1 dnnew)
(14)
For dragging within one image, it can be regarded as a combination of instance removal and dragging across images when the target image remains the same as source.
EXPERIMENTS
5.1 TASK, DATASET, AND METRICS
Instance matting, We first validate MP-Mat on the multi-instance matting task, where we conduct experiments on 2 datasets: the real image subset of the HIM-100K dataset (Liu et al., 2024), and the SMPMat dataset, recent synthetic matting dataset. We evaluate different models using two major quantitative metrics: sum of absolute differences (SAD) and mean square error (MSE, we report the 102 scaled value). Lower values for these metries indicate better alpha matte results. We also report other widely used metrics in Appendix G.
Instance editing. This one of the potential downstream applications for matting, but existing focused research has not explored their actual usability and performance on related tasks. To fill this gap, we conduct extensive experiments on 3 sub-tasks with compelling application value: (I) Instance removal that aims to delete specified foreground instances from the image without affecting its overall harmony: (2) Occlusion reordering that aims to modify the occlusion relationships among foreground instance in controlled and harmonious manner; (3) Instance dragging that aims to
move any instance to the desired position harmoniously. We conduct experiments on the GQA-
inpaint dataset (Yildirim al., 2023) for instance removal task and use Mean LI Loss, Mean L.2
7

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison of instance matting.
Table Effects of SG-MP and
Inst-MP in global perspective.
Dataset
Method
HIM-10OK
SMPMat
SG-MP Inst-MP
SAD
MSE
SAD
MSE
SAD
MSE
35.85
0.87
Instance-agnostic
33.78
0.86
FBA(+Mask RCNN)
38.25
0.95
42.36
1.12
29.46
0.60
FBA(+SOLO)
38.18
0.94
41.23
0.96
26.75
0.49
FBA(+EVA)
37.76
0.91
39.82
0.95
MG(+Mask RCNN)
40.51
0.97
41.58
1.06
MG(4SOLO)
39.26
0.95
40.79
0.95
Table
Components effects
MG(+EVA)
38.19
0.94
39.73
0.95
within SG-MP extraction.
Instance-aware
InstMatte
37.34
0.93
40.19
0.96
Depth
PGN
SAD
MSE
E2E-HIM
32.22
0.84
38.55
0.93
33.78
0.86
Maggie
29.48
0.78
37.41
0.91
31.82
0.74
MP-Mat (Ours)
26.75
0.49
33.25
0.81
26.75
0.49
Table 4: Component effect in Inst-MP extraction.
Background Estimation
Color Estimation
Refinement
SAD
MSE
29.46
0.60
28.53
0.55
27.79
0.52
26.75
0.49
Input
EVA+MG
InstMatte
EZE-HIM
Maggie
MP-Mat.
Figure 3: Qualitative comparisons. Distinguished areas are highlighted with bounding boxes.
Loss, and PSNR metrics for evaluation, where lower values for Mean L2 Loss and Mean L11 Loss are preferable, while higher PSNR values indicate better quality. For occlusion reordering task, due the lack of data with ground truth, we construct a synthetic dataset ORHuman that can theoretically ensure the correctness of the derived ground truth (see Appendix C). For instance dragging, we only give qualitative comparisons due to the lack of evaluation benchmarks.
5.2 INSTANCE MATTING
Compared methods. Our method is compared with the methods designed for this tasks, including InstMatte (Sun et al., 2022), E2E-HIM (Liu et al., 2024) and Maggie (Huynh al., 2024). We also compared with composite approaches following Liu et al. (2024). Specifically, we tailor the mask-guided matting model MG (Yu et al., 2021a) and FBA (Forte & Pitié, 2020) with off- fffftff-shelf instance segmentation models Fang et al. (2023)) to adapt it to the multi-instance matting task.
He et al. (2017), SOLO Wang et al. (2020), and EVA
Main results. The main performance comparison on HIM-100K and SMPMat dataset shown in Tab. 1. It can be observed that the proposed MP-Mat outperforms existing methods by large margins
8

---

Published as a conference paper at ICLR 2025
on both datasets (i.e., least 2.76 SAD in HIM-1008 dataset and 4.16 SAD in SMPMat dataset), demonstrating the superiority of the proposed method.
Qualitative analysis. We also give some qualitative results in Fig. 1 and Fig. 3. It can be observed that our method is superior in (1) distinguishing fine-grained boundaries such as hair regions, and (2) with better instance-aware ability, especially under occlusion areas caused by human interactions. More results can be found in Appendix L..
5.3 ABLATION STUDIES
Here we conduct ablation studies on the HIM-100K dataset to analyze the effectiveness of the pro- posed components. We first validate the individual effects of the proposed multiplane representations (i.e., SG-MP and Inst-MP) from a global perspective. Then, we go deeper to analyze the effect of each component within the proposed SG-MP and Inst-MP, demonstrating their effects while also revealing insights for further research. Other studies can be found in Appendix.
Effects of the proposed multiplane representations. Here we analyze the individual effect of the proposed scene geometry-level multiplane representation (SG-MP) and instance-level multiplane representation (Inst-MP) from a global perspective. From Tab. 2, it can been seen that: (1) Without SG-MP, the performance drops drastically (SAD from 26. 75 to 33.78). This validates the importance of explicit 3D scene representation for multi-instance matting task. Our designed SG-MP makes the scene representation 31D-aware, and can serve as an effective clue for splitting instances in different 3D positions, thereby alleviating occlusion effects; (2) Inst-MP can further boost SAD by a large margin (i.e., 2.71), verifying its effectiveness. Another benefits of such design is for its flexibility and high efficiency on down stream tasks like instance-level image editing, as discussed in Sec. 5.4.
Component effects within SG-MP extraction. From Tab. 3, it can be summarized that: (1) Depth information is useful, as it can bring auxiliary 3D information; (2) Only sending depth map as input is not sufficient enough. With our proposed Plane Generation Network (PGN) for multiplane representation at feature level, the performance further boosts by 5.07 in SAD, which is much larger than the performance gain solely from depth input (1.96 in SAD). This essentially demonstrates the effectiveness of our explicit multiplane feature representation extraction design.
Component effects within Inst-MP extraction. From Tab. 4, it can be summarized that: (1) Be- sides focusing on foreground instances, adding explicit background estimation is beneficial for mat- ting accuracy, it enables better handling of the boundary of instance and background; (2) Besides matte estimation, adding color estimation for each instance can enable better content awareness and thus boost the performance; (3) The proposed uncertainty guided refinement can further facilitate performance, as it can adjust ambiguity at fine-grained boundaries.
5.4 INSTANCE EDITING
For all tasks mentioned in Sec. 5.1, we compare our method with SOTA image editing-focused methods (Yildirim et al., 2023; Shi et al., 2024). We finetune the image dliting-focused methods on the target dataset for higher performance, while our MP-Mat adopts a zero-shot inference manner.
Instance removal. From Tab. 5, we can observe that: (1) MP-Mat significantly outperforms existing editing- based methods and with high efficiency (also refer to Fig. 4 (a) for qualitative comparison). (2) When combined with an off-the-shelf background inpainting model (Yu et al., 2018), the per- formance of MP-Mat can be further boosted by large margins, we attribute this to the inadequate background modeling capacity of our matting- based methods that also partially caused by lack of training data for such themes. (3) The editing time cost within the same image becomes negligible once Inst-MP is constructed MP-Mat, further verifying the superiority of our design.
Occlusion reordering, From Tab. 6, we can also observe the significant superiority of MP-Mat in both effectiveness and efficiency (i.e., more than 10% advantage on Mean LI Loss, 8.5db on PSNR, and 68% on speed), From Fig. 4 (b), we can observe that when editing target semantics, exiting SOTA methods will also unintentionally alter content that should remain unchanged, such as human faces. In contrast, our method better preserves these elements. The reason is that we conduct editing on individual instance-level planes, where the appearance (color) information remains unchanged, and the only modification is the alpha value (opacity) for new image rendering (users can also inject

---

Published as a conference paper at ICLR 2025
Table 5: Quantitative comparison on instance removal. BI refers an off-the-shelf background
inpainting method used to enhance the inpainting of the corresponding region. Bold text indicates the best performance, and underlined text represents the second-best performance.
Editing Method
Mean L1 Loss
Mean L2 Loss
PSNR (t)
Time per editing (s)
Inst-inpaint
12.69%
2.58%
23.09db
0.1982
Dragon Diffusion
12.
2.49%
22. 27.1
0.2139
Matting Method
Mcan L1 Loss (C)
Mean L1 Loss (4)
PSNR (1)
Time per image (8)
Time per editing (s)
Ours
9.37%
1.96%
23.58db
0.1117
0.0003
Ours (w/ BI)
3.73 %
0.84 %
25.79db
0.1117
0.0169
Table 6: Quantitative comparison occlusion reordering.
Method Inst-inpaint Dragon Diffusion
Mean LI Loss (4)
Mcan L2 Loss ())
PSNR (1)
Speed (s)
12.65%
3.42%
21.2db
0.2547
13.85%
3.66%
19.82db
0.2849
Ours
3.26
0.79 %
28.32db
0.1739
Input
Inst- inst-inpaint
DrapoaDiffusion MP-Mat
MP-Mat++l IBI
Input image
DragonDiffusion
MP-Mat
(a) Instance removal (O)
(c) Instance dragging within image (+)
Ingut
Inst-ingaiot
DragonDiffusion
M.-Mat
Target Image
Reference imtaage
DragonDlffssion
MP-Mat
(b) oclusion reardering (Fn)
(d) Instance dragging images
(7)
Figure 4: Qualitative comparisons for editing tasks. Yellow boxes highlight the distinguished areas.
some customized transformation like rescaling and rotation, especially for dragging task). This actually provides better guarantee of not affecting the semantics of areas that should not be altered, further demonstrating the superiority of our approach and highlighting its potential in editing tasks.
Instance dragging. Here we only gave qualitative comparison due to the lack of benchmark datasets. As Fig. (c) and (d), MP-Mat can preserve finer details such as hair and watch af- ter dragging, which highlights another advantage of our approach. Overall, the results highlight the potential of our matting- focused methods for image editing tasks.
CONCLUSION, LIMITATION, AND BROADER IMPACT
matting framework that is built on meticulously designed multiplane representations. We design layered representations from perspectives: the scene decomposition based on depth differencee, and the instance-level multiplane representation (Inst-MP), which focuses on instance-level modeling. These representations excel handling occlu- sion effects and are better aware of both foreground and background content, leading to significant performance improvement for instance matting. Additionally, our design demonstrates strong po- tential for instance-level image editing, a relatively underexplored area in existing mauting-focusse methods. Remarkably, our approach, even under zero-shot inference, outperforms specialized image editing techniques by large margins and with high efficiency.
Conclusion. In this work, propose MP-Mat,
geometry-leve, multiplane representation (SG-MP), which emphasizes
Limitation. Despite the effectiveness, our work mainly focuses on human instances rather than general categories. This is partially due to the lack of data for multi-instance matting for other categories, and we leave this to future works.
Broader impact. Besides matting and editing, our layered representation with depth cues may enable extension on other tasks, like 3D photography Shih et al. (2020); Tucker & Snavely (2020), inpainting Shih et al. (2020), distortion correction Wang al. (2024), etc, where the fine-grain grained boundary matting and background modeling ability of MP-Mat may bring additional benefits to enhance those relevant tasks.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENT
Mike Shou is only supported by the Singapore Ministry of Education Academic Research Fund Tier 1 FY2023 Reimagine Research Scheme.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-211221 229. Springer, 2020.
Quan Chen, Tiezheng Ge, Yanyu Xu, Zhiqiang Zhang. Xinxin Yang, and Kun Gai. Semantic human matting. In Proceedings of the 26th ACM international conference Multimedia, pp. 618-626, 2018.
Yigit Ekin, Ahmet Burak Yildirim, Erdem Eren Caglar, Aykut Erdem, Erkut Erdem, and Aysegul Dundar. Clipaway: Harmonizing focused embeddings for removing objects via diffusion models. arXiv preprint arXiv:2406.09366 2024. 3
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 19358 19369, 2023. 17
Marco Forte and François Pitié. f. b, alpha matting. arXiv preprint arXiv:2003.07711 2020. 20
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017. 8
Qiqi Hou and Feng Liu. Context-aware image matting for simultaneous foreground and alpha esti- mation. pp. 4130-4139, 2019. 20
Guanqing Hu and James Clark. Instance segmentation based semantic matting for compositing applications. In 2019 16th Conference on Computer and Robot Vision (CRV), pp. 135-142. IEEE, 2019. 3
Chuong Huynh, Scoung Wug Oh, Abhinav Shrivastava, and Joon- nounYoung Lee. Maggie: Masked guided gradual human instance matting. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 3870-3879, 2024. 3, 5, 8, 15,
Siyi Jiao, Wenzheng Zeng, Changxin Gao, and Nong Sang. Dfimat: Decoupled flexible interactive matting in multi-person scenarios. In Proceedings of the Asian Conference on Computer Vision, 2024. 3
Lei Ke, Yu-Wing Tai, and Chi-Keung Tang. Deep occlusion aware instance segmentation with overlapping bilayers. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 4019 4028, 2021. 2
Zhanghan Ke, Jiayu Sun, Kaican Li, Qiong Yan. and Rynson WH Lau. Modnet: Real-time trimap- free portrait matting via objective decomposition. volume 36, pp. 1140-1147, 202,, 3
Xin Kong, Shikun Liu, Xiaoyang Lyu, Marwan Taher, Xiaojuan Qi, and Andrew J Davison. Escher- net: A generative model for scalable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9503-9513, 2024. 4
Jizhizi Li, Sihan Ma, Jing Zhang, and Dacheng Tao. Privacy portrait matting. pp. 3501- 3509, 2021a. 21
Jizhizi Li. Jing Zhang, and Dacheng Tao. Deep automatic natural image matting. arXiv preprint arXiv:2107.07235, 2021b.
Jizhizi Li, Jing Zhang, Stephen J Maybank, and Dacheng Tao. Bridging composite and real: towards end-to-end deep image matting. 130(2):246-266 2022. 3, 17, 21
11

---

Published as a conference paper at ICLR 2025
Yaoyi Li and Hongtao Lu. Natural image matting via guided contextual attention. pp. 11450-11457 2020. 3
Chung-Chine Ching Lin, Jiang Wang, Kun Luo, Kevin Lin, Linjie Li, Lijuan Wang, and Zicheng Liu. Adaptive human matting for dynamic videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10229-10238, 2023.
Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian I. Curless, Steven M Seitz, and Ira Semmelmacher-Shiizemmne Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8762-8771, 2021.
Qinglin Liu, Shengping Zhang, Quanling Meng, Bineng Zhong, Peiqiang Liu, and Hongxun Yao. End-to-end human instance matting. IEEE Transactions on Circuits and Systems for Video Tech- nology, (4(4)):663-264) 2024. doi: 10. 1211109/CSTTT223333444444
3, 7, 8, 21,
Hao Lu, Yutong Dai, Chunhua Shen, and Songcen Xu. Indices matter: Learning to index for deep image matting. pp. 3266-3275, 2019.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pp. 800-8000000000 809. IEEE, 2024. 4
Sihan Ma, Jizhizi Li, Jing Zhang. He Zhang, and Dacheng Tao. Rethinking portrait matting with privacy preserving. pp. 1-26, 2023.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEF/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019. 4
Juewen Peng. Jianming Zhang, Xianrui Luo, Hao Lu, Ke Xian, and Zhiguo Cao. Mpib: An mpi- based bokeh rendering framework for realistic partial occlusion effects. In The Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Yu Qiao, Yuhao Liu, Xin Yang. Dongsheng Zhou, Mingliang Xu, Qiang Zhang, and Xiaopeng Wei. Attention-guided hierarchical structure aggregation for image matting. pp. 13676-13685, 2020. 3,
Zhefan Rao. Single portrait image matting and bokeh effect synthesis via multiplane images. Mas- ter's thesis, HKUST, MonthOfPublication 2023. 4
Jiawei Ren, Mengmeng Xu, Jui-Chieh Wu, Ziwei Liu, Tao Xiang, and Antoine Toisoul. Move anything with layered scene diffusion. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6380-6388 6389, 2024.
Soumyadip Sengupta, Vivek Jayaram, Brian Curless, Steven M Seitz, and Ira Kemelmacher- Shlizerman. Background matting: The world is your green screen. pp. 2291-2300, 2020.
Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In Pro- ceedings of the 25th annual conference on Computer graphics and interactive techniques, pp. 231-242, 1998. 4
Shelly Sheymin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. Pro- ceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp. 8871- 8879, 2024.3 3
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent YF Tan, and Song Bai. Dragdiffusion: Harnessing diffusion models for interactive based image edit- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9839-8849) 2024. 3, 9, 17
Meng-Li Li Shih, Shih- Yang Su, Johannes Kopf, and Jia-Bin Huang. 3d photography using context- aware layered depth inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8028-8038, 2020. 10
12

---

Published as a conference paper at ICLR 2025
Yanan Sun, Chi-Keung Tang. and Yu-Wing Tai. Semantic image matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11120-11129, 2021.
Yanan Sun, Chi- Chi-Keung Tang, and Yu-Win Wing Tai. Human instance matting via mutual guidance and 2022. 1, 5, 6, 8, 15, 17, 19, 20, 21
multi-instance refinement. pp.
Fuwen Tan, Crispin Bernier, Benjamin Cohen, Vicente Ordonez, and Connelly Barnes. Where and who? automatic semantic- aware person composition. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1519-1528. IEEE, 2018.
Jingwei Tang, Yagiz Aksoy, Cengiz Oztireli, Markus Gross, and Tune Ozan Aydin. Learning-baser sampling for natural image matting- pp. 3055-3063, 2019. 20
Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In Proceed- ings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 551-560, 2020. 2, 4, 10
Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. Solov2: Dynamic and fast instance segmentation. Advances in Neural information processing systems, 33:17721-17752, 2020. 8
Zhixiang Wang, Yu-Lun Liu, Jia-Bin Huang, Shin ichi Satoh, Sizhuo Ma, Gurunandan Krishnan, and Jian Wang. Disco: Portrait distortion correction with perspective-aware 3d gans. International Journal of Computer Vision, 132(11): 1)::111)47--588 2024. 10
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao. Daniel Watson, Pratul Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffuuson priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21551-21561, 2024. 4
Ning Xu, Brian Price, Scott Cohen, and Thomas Huang. Deep image matting, pp. 2000-297999 2017.
3, 17, 20
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406. 09414, 2024a. 17
Zhen Yang. Ganggui Ding, Wen Wang. Hao Chen, Bohan Zhuang, and Chunhua Shen. Object-aware inversion and reassembly for image editing. In The Twelfth International Conference on Learning Representations, 2024b. URL https: wttp/pppeeeveeeeeeeeeeeeeeeeeeee n forum aaat/forumidddpce/ 2id=dpcxxiM1cccmmmmmmmmm 3
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint: Instructing to remove objects with diffusion models. arXiv preprint 23W::23004.02246, 2023. 3, 7. 17
Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of vir- tual normal for depth prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5684-5693, 2019. 17
Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023. 15, 16, 17
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5505-5514, 2018. 9
Qihang Yu, Jianming Zhang, He Zhang, Yilin Wang, Zhe Lin, Ning Xu, Yutong Bai, and Alan Yuille. Mask guided matting via progressive refinement network. pp. 1154-1163, 2021a. 8, 21
Zijian Yu, Xuhui Li, Huijuan Huang, Wen Zheng, and Li Chen. Cascade image matting with de- formable graph refinement. In Proceedings of the IEEE/CVF International Conference on Com- pufer Vision, pp. 7167-7176, 2021b.
13

---

Published as a conference paper at ICLR 2025
Yunke Zhang, Lixue Gong, Lubin Fan, Peiran Ren, Qixing Huang, Hujun Bao, and Weiwei Xu. A late fusion cnn for digital matting. pp. 1469-7478, 2019. 3
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely, Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint /805. 2018.
Bingke Zhu, Yingying Chen, Jinqiao Wang, Si Liu, Bo Zhang, and Ming Tang. Fast deep matting for portrait animation on mobile phone. In Proceedings of the 25th ACM international conference on Multimedia, pp. 297-305, 2017. 1
Junhao Zhuang. Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. In European Conference on Computer Vision, pp. 195-211. Springer, 2024. 17
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jac Lee. Seg- ment everything everywhere all at once, arXiv preprint arXiv:2304.06711 067118 2023. 6
14

---

Published as a conference paper at ICLR 2025
APPENDIX
We provide supplementary material for a deeper understanding and more analysis related to the main paper, arranged follows:
1. More detailed illustration of MP-Mat for occlusion reordering (Appendix A)
2. Implementation details (Appendix B)
Details about the constructed ORHuman Dataset (Appendix C)
4. Model's robustness across different depth estimators (Appendix D)
More ablation studies (Appendix E)
6. Details the metric calculation instance matting (Appendix F)
More results under other metries (Appendix G)
8. Additional evaluation on foreground estimation (Appendix H)
Separate evaluation on occlusion and occlusion-free cases (Appendix I)
10. Experiments on instance-agnostic dataset (Appendix J)
Generalization capability beyond human category (Appendix K)
12. More qualitative results (Appendix L)
A MORE DETAILED ILLUSTRATION OF MP-MAT FOR OCCLUSION REORDERING
In the manuscript, we primarily convey the main editing process of MP-Mat through textual de- scriptions with part of core equations, due to space limitations. Here, we provide a more detailed illustration those processes, including additional equations and pseudo code, especially for occlu- sion reordering.
A.1 PREREQUISITE
For image that needs to be edited, we first feed it into our MP-Mat to acquire the Inst-MP repre- sentation:
R= ((a = = MPMMl(11, D)
where + is the prediction instance number (including one for background), and D is the corre- sponding depth map of I. Then, we also estimate the depth of each instance-level plane by averaging the pixels with positive alpha values:
R, d.) = { (,,,),)d)))) dd.,ddd Aug (d())
where Aug means average operation and f2, is the pixel set containing the positions of the foreground area of plane i:
Ω {xx v)} as (x, y)
Based on the acquired plane depth d,, sort the Inst-MP R in ascending order of depth, where Ro is the closest instance plane to the camera, and Rs represents the background with its depth considered infinity. The resulting sorted will be used for the subsequent editing.
Note that the utilized depth information here can be easily estimated using off- the-shelf depth esti- mators (e.g., (Yin et al., 2023)), and the actual cost is comparable to pre-instance mask generation in existing mainstream mask-guided instance matting methods (Sun et al., 2022; Huynh et al., 2024).
A.2 OCCLUSION REORDERING
Definition: Switch the occlusion relationship between the source instance and target instance. Here take: changing from instance p occluding y to occluding p for example.
15

---

Published as a conference paper at ICLR 2025
Here focus the general cases where other instances may exist between p and q. The readers can also refer to the manuscript where a simplified case is also provided for easier understanding.
The initial instance index list of R. can be represented as: {0, 1, 2, .... P. 1, .......... 1, 9, S}
where R has already been sorted based on depth information as illustrated in See. A.I. Then, we perform alpha swap between planes finally achieve updated index order:
10,1,2,.... 4 l,... 9 1, II,,,..,.)) S}
The alpha swap operation can not be done by naively swapping the alpha value between plane and plane q, since the effect from other planes positioned between them will also affect the final rendering. Thus, decompose this task into iteratively swapping between each pair of adjacent planes until the desired order is achieved. The pscudo-code can be found at Algorithm
Algorithm Algorithm of Occlusion Reordering
Inputs: Sorted Inst-MP (R), instances that require reordering (p,g)
# Outputs: new rendered image (I')
Function Swap ( (R, indeæ11 indear2)
# Iddentffy the intersection area of two instance.
!2 {xx () [[indez]]] (x] 3) >> 0 A [[index2[[<<[[,.]]] 0}
for (x, y) E f2 do
# Alpha swapping between the intersection area. Riinder1] (x, 3) ++ R[inder2][ ('') y)) end
return
Function Rendering (R)
='=0 for inder t 0 to len R) do 1'= R[inder] a'] R[indez]]'e ['c]
end
return I'
Function Reordering p)
# Iteratively swap plane p the target position g.
for index f p + to do
R Swap (R index) end
# Intermediate plane order: {0, 1, 2..,, 1,..,, 1,2,,,...,
# Iteratively swap plane the target position.
for index t q to + 1 do
R Swap RR g. inder) end
# Final plane order: 10 2, 9.P 1,.., I,p, S}
S}
1' Rendering (R)
= Reordering p.
B IMPLEMENTATION DETAILS
B. GENERAL IMPLEMENTATION DETAILS
For both training and testing, we obtain the input depth map through an off-the-shelf monocular depth estimator (Yin et al., 2023). We train MP-Mat 4 NVIDIA RTX 2080Ti GPUs with total batch size 4 (1 per GPU). The training employs the SGD optimizer with a momentum of 0.9 and weight decay of 0.0005 for 50,000 iterations. The learning rate is initialized at 0.01 and is adjusted 0.9
by multiplying with set to 5. In the loss function, the hyperparameters Åι 1 and X2 5.
iter
For the uncertainty map generation, the hyperparameter K
Hter
16

---

Published as a conference paper at ICLR 2025
B.2 MORE ILLUSTRATION ON THE LOSS CALCULATION
Here, we provide details regarding the classification loss part of Dpetect mentioned in the manuscript. Specifically, this loss is designed to constrain the classification of queries during each network forward pass (i.e., determining whether a query corresponds to a real human instance or a redundant prediction labeled as the "non-object" class). Before calculating the loss, each query is passed through binary classification head to obtain the predicted 2-class probabilities.
B.3 PSEUDO LABEL GENERATION IN OCCLUDED AREA
One objective of MP-Mat is to predict foreground color information alongside alpha prediction. For instance editing tasks (i.e., instance removal, occlusion reordering, and instance dragging), aware- ness of occluded regions is also crucial, as inpainting capabilities are fundamentally required. To this end, we incorporate pseudo-grousd truth RGB content for occluded regions as direct supervision, enabling the model to learn the color information behind occlusions.
We use PowerPaint Zhuang al. (2024), inpainting method, generate such pseudo-labels. Specifically, for each target human instance, we first identify the surrounding instances that may have occlusion relationships with it. Since we do not know which part has occlusion, we treat the closest 4 instances (based on the bounding box center, if they exist) as the surrounding instances. We then merge the ground-truth masks of these surrounding instances into a single mask map that indicates the instances we want to remove (aims to uncover the occluded areas, including the part that may potentially belong to the target instance). We feed such mask along with the original im- age into the inpainting model, which generates a new image where the surrounding instances are removed, and the occluded areas are inpainted. We then use the acquired new image to generate the new pseudo matting labels (alpha) using an off-the-shelf method Fang al. (2023). The foreground color label is defined as the RGB pixel value where its corresponding alpha value>0.. Note that we only modify the ground truth for the changed areas compared to the original image (i.e., the previ- ously occluded region that is now uncovered), ensuring that the original grouding-truth information remains unchanged. The entire process can be completed automatically without data filtering.
C ORHUMAN DATASET
For the occlusion reordering task, due to the lack of data with ground truth, we construct a synthetic dataset ORHuman that can theoretically ensure the correctness of the derived ground truth. The data synthetic pipeline are as follows:
(1) Collect set of instances (A cali E 21, NN))) along with corresponding high-resolution background image B. Here, we collect instances from image matting datasets (Qiao al., 2020; Xu et al., 2017; Li al., 2021b), and high-resolution background images without salient objects from the BG-20K dataset (Li et al., 2022).
(2) Randomly generate the overlay order for this set of instances.
(3) Following the synthesis scheme in Sun et al. (2022), overlay the instances onto the background
image in the generated order to obtain the original image Io.
(4) Randomly select instances p and q whose occlusion relationships need to be swapped, and ex-
change their overlay order during the image synthesis process.
(5) Generate the corresponding mnstance-swaped image I, (i.e., the desired ground-truth) according to the new overlay order after the swap.
Based on this schedule, can obtain diverse data pairs (.. p) + {I,} that can be used for both training and evaluation. Some examples are shown in Fig. 5. In total, the constructed ORHuman dataset includes 24,000 training samples, 8,000 validation samples, and 8,000 test samples. Note that in our work, during comparison, our training set is used to fine-tune the existing editing-base methods (Yildirim et al., 2023; Shi al., 2024) for higher performance, while our MP-Mat has not been trained on it and adopts a zero-shot inference on the evaluation set.
D MODEL'S S ROBUSTNESS ACROSS DIFFERENT DEPTH ESTIMATORS
Here we further provide experiments to demonstrate the actual robustness of our method and offer an analysis of the reasons behind it. As in Tab. 7, our method is robust to various depth estimators Yin et al. (2023); Yang et al. (2024a); Yin al. (2019) (spanning various scales and even including
17

---

Published as a conference paper at ICLR 2025
before reorder
after reorder
before reorder
after reorder
Figure 5:
Examples of the sample pairs within the proposed ORHuman dataset, where we show the pairs of images before and after occlusion reordering (indicated by yellow arrows).
relatively older one published more than 5 years ago, with significantly lower performance than the recent ones). We argue that such robustness is ensured by the design of MP-Mat itself, which is in aspects: (1) The plane generation network (PGN) in MP-Mat works in an adaptive manner that gradually refines the plane depth based not only input depth but also RGB scene information, more evidence can be found in Tab. 2 in the Manuscript, where the performance gain of PGN is much larger than the introduction of depth map itself; (2) we build SG-MP feature level, which contains not only plane information but also high-level semantic context from the deep feature of RGB image: (3) The precise instance-leve perception is ultimately guaranteed by the design of Inst- MP; (4) During optimization, the SG-MP feature will also receive relevant gradient from Inst-MP, making the plane division and scene representation become more instance-aware.
Also, another non-negligible factor is that depth estimation algorithms have already reached a rela- tively mature stage, even with lightweight ones, and we believe that the future development of depth estimators can further ensure the accuracy of MP-Mat.
Method
Depth Estimator
Depth Esumation on NYU
Instance Mating on HIM-I10K
RMSEJ
Params]
Speed]
SADJ
MSEJ
VNL (ICCV'19)
0,364
2.7M
30ms
27.24
0.51
Metric3d, used our (CVPR'23)
0,266
198M
183ms
0.49
MPMat
Depth anything V2-VIT-B (Neurips') 24)
0.241
335M
213ms
26.68
0.49
Mean
26.88
0.50
Std
0.22
0.01
Table 7: Performance under different depth estimators.
E MORE ABLATION STUDIES
E. PLANE NUMBER N IN SG-MP
We determine the value based on parameter search with an initial intuition. Specifically, we believe the number of planes should equal to or larger than the number people in the scene (including one background). Based such intuition, we used the average number of instances in our training set (6.95 + 1) as the starting point, and then performed a parameter search. From Tab. 8, it can be observed that: (1) Performance remains relatively stable across different plane numbers; (2) The optimal value is 12, which approximately equals the mean instance number plus one standard deviation (6.95 + 3.48 11.43). Note that roughly set N 10 in our method; (3) We also report the computational overhead SG-MP under different plane numbers. can be observed that the additional cost introduced by more planes is negligible. E.2 UNCERTAINTY THRESHOLD T
The value is decided by greedy search. Here we also conduct experiment show the performance under different threshold values and provide more insights. The result is listed in Tab. 9. We can observe that: (1) When T > 0, the performance improves consistently, which demonstrates the effectiveness the proposed incertainty-guided refinement; (2) The performance improvements relatively stable with different T values. The optimal value is 10%. Here we also provide
18

---

Published as a conference paper at ICLR 2025
Num planes
SAD
MSE
Parameters (MB)
Memory usage (GB)
8
27.49
0.51
17.52
0.239
9
26.96
0.51
17.32
0.246
10
26.75
0.49
17.52
0.253
11
26.73
0.50
17.52
0,260
12
26.47
0.48
17.52
0.267
13
26.82
0,49
17.52
0.274
27.28
0.50
17.52
0.281
Table 8: Ablation studies on plane number on the HIM-100K dataset.
some intuitive explanations: We argue that the challenging regions are mainly boundaries among instances, which often exhibit higher uncertainty. However, the ratio of such regions is not very large because boundaries only occupy a small portion of the total pixels of an instance.
T (%)
SAD
MSE
27.79
0.52
27.24
0.51
10
26.75
0.49
15
26.82
0.49
Table 9: Ablation studies on uncertainty threshold T on HIM-100K dataset.
E.3 THE SCALING BEHAVIOR WITH THE NUMBER OF INSTANCES
Here we evaluate MP-Mat based on various instance numbers on HIM-100K, where we report MAD (normalized SAD across pixels because SAD is sensitive to pixel numbers) and MSE. From Tab. 10, it can be seen that (1) When instance number>> 1, noticeable performance degradation occurs. We argue this is due to the intrinsic challenge of multi-instance scenarios (e.g.. the need to distinguish different instances and handle occlusions). (2) The performance is relatively stable when instance
number> 1, demonstrating the potential working stability of MP-Mat across different scenarios.
Num instances
MAD
MSE
3.47
0.45
5.66
0.48
3
5.93
0.50
5.32
0.49
5,81
0.49
5.45
0.48
Table 10: The scaling behavior with the number of instances on HIM-100K dataset.
E.4 QUERY NUMBER K
From Tab. 11, it can be observed that the matting performance increases as the number of queries grows, although the improvement becomes relatively smaller the number increases. We set 25 queries in our work.
F DETAILS ON THE METRIC CALCULATION ON INSTANCE MATTING
Here we will illustrate how we calculate SAD and MSE for instance matting task. Besides matting accuracy, the metric calculation also considers the instance awareness aspect. Specifically, in multi- instance cases, we need to determine the correspondence between each predicted instance-level result and the ground truth instance. Metrics are then computed within each matched pair. This process actually shares similar spirits to the IMQ metric calculation InstMatte Sun et al. (2022).
More specifically, first match GT instances with predictions based IoU (each GT match at most one prediction, with higher IoU given priority). For unmatched GT instances (indicating missed detection happens), the corresponding matched prediction is set to all-zero pixel map.
19

---

Published as a conference paper at ICLR 2025
Num_queries
Num queries
SAD
MSE
15
27.94
0.52
20
27.55
0.51
25
26.75
0.49
30
26.36
0,49
40
26.08
0.48
Table 11: Ablation studies on the query number HIM-100K dataset.
Similarly, for unmatched predictions (indicating false positive), the corresponding matched GT is set to an all-zero pixel map. After the matching process, we can compute SAD or MSE on each GT- prediction pair. In this way, issues such as missed detections and false positives are also reflected in the performance evaluation, in addition to the reflection of instance localization and matting accuracy within matched true positive predictions.
G MORE RESULTS UNDER OTHER METRICS
For instance matting task, we provide more metrics here, including Grad, Conn, and IMQ series pro- posed in InstMatte Sun et (2022). From Tab. 12, can be observed that our method outperforms existing SOTA methods across all metrics consistently, further demonstrating its superiority.
Method
SADI
MSE
Grad]
Conn
IMQmad
IMQmse
IMQgradt
IMQconn
InstMatte
37.34
0.93
18.47
37.59
56.84
73.39
56.52
58.77
E2E-HIM
32.22
0.84
14.51
34.82
32,97
57.43
33.24
33.95
Maggie
29.48
0.78
16.93
30.16
60.36
78.81
61.70
62.19
MP-Mat (Ours)
26.75
0.49
14.26
27.83
63.75
81.49
64.26
65.44
Table 12: Quantitative result under other metrics.
H ADDITIONAL EVALUATION ON FOREGROUND ESTIMATION
we conduct experiments on the Composition- dataset Xu et al. (2017) whose goal is foreground estimation evaluation. We compare MP-Mat with existing matting methods that have foreground estimation capability (note that all of the existing multi-instand instance matting methods do not have such capability). The result is shown in Tab. 13, which demonstrates the superiority of our method in foreground estimation.
Method
SAD]
MSE]
SampleNet Tang et al. (2019)
95.36
9.26
FBA Matting Forte & Pitié (2020)
74.85
5.03
Context Aware Matting Hou & Liu (2019)
61.72
3.24
MP-Mat (Ours)
57.44
2.95
Table 13: Quantitative results on foreground estimation on the HIM-100K dataset.
I SEPARATE EVALUATION ON OCCLUSION AND OCCLUSION-FREE CASES
Here, we propose new metrics to evaluate performance on occluded cases. Specifically, since occlu- sion is very common and not well-suited for evaluation at the image level, we adopt a more general- ized approach by grouping occlusion and occlusion-free cases pixel-level manner. Particularly, all predicted pixels are categorized into two groups based the ground truth (GT): occluded pixels and non-occluded pixels. For a given image, if the GT shows that least two instances have an alpha value greater than 0 at specific pixel, that pixel is grouped into the occluded set; otherwise, will be grouped into the non-occluded set. We then calculate the SAD metric separately within these sets, resulting in SAD-O for occluded pixels and SAD-NO for non-occluded pixels. The performance is shown in Tab. 14. We observe that our method demonstrates a more obvious advan- tage in more challenging occluded regions, highlighting the superiority and potential of MP-Mat. (Note that SAD is sensitive to the number of pixels, so the value of SAD-O tends to be lower than that SAD-NO due the relatively smaller amount of occluded pixels.)
20

---

Published as a conference paper at ICLR 2025
Method
SAD-O
SAD-NO 18.48
SAD
InstMatte
18.86
37.34
E2E-HIM
10.40
21.82
32.22
Maggie
12.65
16.83
29.48
MP-Mat
9.14
17.61
26.75
Table 14: Evaluation on occlusion and occlusion-fii cases.
J EXPERIMENTS ON INSTANCE-AGNOSTIN DATASET
We conduct experiments the P3M dataset Li al. (2021a). From Tab. 15, it can be seen that MP-Mat still outperforms cxisting arts by a noticeable margin. It indeed verifies the superiority of the proposed method.
Datasets
Method
P3M-500-P
P3M-500-NP
SAD
MSE
SAD
MSE
InstMatte
12.49
0.0048
16.71
0.0057
E2E-HIM
7.96
0.0024
9.25
0.0030
Maggie
8.94
0.0030
11.39
0.0037
MP-Mat (Ours)
6.88
0.0022
8.37
0.0028
Table 15: Quantitative results on P3M dataset.
K GENERALIZATION CAPABILITY BEYOND HUMAN CATEGORY
Theoretically, our method can be generalized instances of other categories (i.e., not limited to human). Some reasons we mainly focus on human instances in our manuscript are: (1) There is lack of suitable datasets, as no natural multi-instance matting datasets focus on categories beyond humans. (2) Human instance matting itself holds significant application value and can be easily extended to other categories (as long as with sufficient data support). Existing multi-instance matting works also focus human instances, and we follow this setting by starting with humans.
Here we further conduct experiments to show the generalization ability of MP-Mat beyond human instance. Specifically, we conduct experiment AM-2K dataset Li et al. (2022), which contains multiple (i.e., 20) instance categories (e.g., cat, dog, camel) but under single-instance scenarios. As in Tab. 16, our method still outperforms other instance matting counterparts by large margins, demonstrating its strong generalization ability across different categories. We believe that the devel- opment of high-quality multi-instance datasets will further advance the relevant research.
AM-2K
Method
SADJ
MSE
InstMatte
14.25
00053
E2E-HIM
12.39
0.0041
Maggie
13.83
0.0050
MP-Mat (Ours)
10.57
0.0029
Table 16: Quantitative results AM-2k dataset.
L MORE QUALITATIVE RESULTS
Here we provide more visual comparisons in addition to the examples given by Fig. and Fig. 3 in the manuscript. From rows 1, 2, 4, and 5 in Fig. 6, it can be observed that our MP-Mat preserves finer details (e.g., hair, hands, and other interactive boundaries between instances) better than existing approaches (Yu et al., 202 Sun et al., 2022; Liu et al., 2024; Huynh et al., 2024), and in some cases, even surpasses the human-labelee ground truth (Rows and 5). Besides, our method demonstrates strong instance awareness. For example, out of 5 existing methods Sun et al. (2022);
21

---

Published as a conference paper at ICLR 2025
Liu et al. (2024); Huynh al. (2024) fail to detect instances assign incorrect pixel associations in rows 3 and 5, while our method robustly distinguishes instances even at a small scale (Row 3) and with challenging boundaries (Row 5). In summary, the proposed MP-Mat excels in instance localization, instance differentiation, and fine detail perception.
During the rebuttal period, further provide more visualization results, including the visualization of the learned multi-pla multi-planes and more qualitative examples of more complex hairstyles.
Input.
EVA+MG
InstMatte
E2E-HIM
Maggie
MP-Mal
GT
Figure 6: Additional visual comparison on instance matting.
Imageo
MP-Mat Mat
Figure 7: Qualitative examples of more complex hairstyles.
Qualitative examples of more complex hairstyles. As in Fig 7, can be observed that MP-Mat can well handle challenging cases with complex hair hairstyles, further demonstrating its effectiveness.
Visualization of the learned multi-planes. From Fig. 8 (a), it can be observed that the learned plane-distinctive masks in SG-MP reasonable, they correctly divide the scene based on depth
22

---

Published as a conference paper at ICLR 2025
Image
Plane 1
Plane 2
Plane
Plane 4
Plane 5
Plane 6
Plane
Plane 8
Plane 9
Plane 10
(a) Visualization of the leamed daaneddittnnttie masks in SG-MP.
alpha
color
alpha
color
Instance 1
Instance 2
alpha
color
alpha
color
Instance
Instance
alpha
color
alpha
color
Instance
Instance (background)
(b) Visualization of the learned Inst-MP representation.
Figure 8: Visualizations of the learned representations SG-MP and Inst-MP.
variance. Moreover, as shown in Fig. 8 (b), our Inst-MP precisely extracts each instance's alpha values and foreground information (color), providing a solid and efficient foundation for subsequent image editing tasks.
23


Reviews:
[{"review_id": "W88JmU9ajF", "reviewer": "Reviewer_dZDg", "summary": "The authors presented a MPI combined with instance matting method to represent image, and shows its generalization capability for instance-aware image editing. The paper is mainly focusing on human-background representation. The architecture is a transformer-based network which consists of plan generation network and refinement module. Strong supervision makes it generate both the color and instance masks all together, to enable the model to do zero-shot image editing easily. The idea is intuitive and interesting.", "questions": "- Does the training dataset contain de-occluded instance mask and RGB contents? Why the model can learn the de-occluded contents without special supervision? Is it because the dataset is synthetic?\n- Is it possible to show each layer in the MPI after learning? It will be interesting to visualize each layer in images. \n- How can the model be generalized to object matting in longer term? Is it possible to extend it to a diffusion-based framework for better generation quality in the future?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of combining MPI with instance matting is intuitive and effective.\n- The zero-shot performance on instance editing task is convincing.", "weaknesses": "- The paper is more about human matting and editing, so better to indicate that in the title to avoid confusion and overclaiming. \n- The generation quality of occluded areas, including occluded people or complicated background, can be not good enough, since the model is a transformer-based framework, but not trained in a diffusion-based way. \n- The model relies on the off-the-shelf depth estimator to obtain accurate depth information as 3D geometry. So its performance is highly relaying on the depth map accuracy. It is somehow a constraints which may limit the model performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fPJzEwR2CV", "reviewer": "Reviewer_dGBZ", "summary": "The paper introduces a  3D- and -instance-aware matting framework with multiplane representations. The multiplane concept is designed from two different perspectives: scene geometry level and instance level. The approach builds feature-level multiplane representations which split the scene into multiple planes based on depth differences. This is followed by further splitting of the planes on an instance-level, at which point each instance is associated with an alpha value and a color value.\nThe detailed splitting into a complex multiplane representation permits other operations, such as instance-level image editing.", "questions": "See the weaknesses section for questions to be answered in the author responses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The multiplane representation is novel, and the two-phase splitting by depth then by instance makes a lot of sense (in terms of different ways that a multi-plane representation could be built).\nThe paper extends the MPI technique of Tucker and Snavely (2020) by using deep features rather than raw pixel values.\nThe detailed splitting into a complex multiplane representation permits other operations, such as instance-level image editing. However, the representation presumably is rich enough to enable other image editing pipeline operations, such as creation of stereo pairs (for 3D movies), or doing video editing. This could be mentioned in the paper and the title of the paper could perhaps be changed to include the editing aspect, since this is a definite contribution of the approach, and somewhat compensates for the computation costs of building the multiplane representations.\n\nExperiments show the method works well compared to other approaches, and the ablations effectively show the influence of the various parts of the method.", "weaknesses": "An obvious cost of the proposed approach is that it requires depth information as input (e.g. an RGBD image). The paper states \"The input depth map here can be easily estimated using off-the-shelf depth estimators, and the actual cost is comparable to pre-instance mask generation in existing mainstream mask-guided instance matting methods.\" No support is given for these claims in the paper, however. How do errors in the depth map affect performance? (i.e. how sensitive is the approach to depth map errors?)\nNo information is given in the experiment section how the value of N (number of depth planes) was set. There should be an ablation or hyperparameter search to determine an appropriate value.\n\nThe paper states \"i ranging from 1 to S representing instance-level plane for each distinct human instances within the image\", but does not say where these human instances come from. Are these detected by some external semantic (to identify humans) instance segmentation algorithms? If so, which ones were used? In the experiment section all that was mentioned is that certain datasets were used, and my assumption is that the ground truth instance labels were used as input. But some guidelines should be given on how the instances (and depth maps) would be generated when presented with unseen data which were not in the training datasets.\nAlso, no information or experiments are given as to the effect of errors in the instance segmentation on the performance of the approach.\n\nIn Table 1, both headings are labeled \"instance agnostic\". One of the these (the lower one) should be instance dependant.\n\nA minor point, but the proposed method is not limited to human matting, so it is unclear why the authors chose to focus the paper on human matting. Nothing in the approach is specific to human instances. Any other type of instance (e.g. horses) could just as well be used, and would give the method more generality and applicability. There needs to be specific information which is unique to human instances for the approach to really be called human instance matting. Otherwise, it is just instance matting.\nAs an example of this point consider the paper by Tan et al (WACV 2018 - which should be referenced as an early work in this area) states \"To simplify the task, we restrict our problem by focusing on human instance composition, because human segments exhibit strong correlations with their background and because of the availability of large annotated data. \" This motivates the approach based on the focus on human instances, which the current paper under review does not do at all.\nTan et al. \"Where and who? automatic semantic-aware person composition.\" 2018 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 2018.\nThe conclusion does shed some light on this, stating \"our work mainly focuses on human instances. This is partially due to the lack of data for multi-instance matting for other categories.\", but this does not mean that the approach is necessarily a human instance matting technique. It could be used for non-human matting if such training data was used.\n\nIn section 2.1 the paper writes \"Previous image matting (...) is conducted under the single-instance assumption without multi-instance awareness...\". This is not true. There are previous works that do multi-instance matting. \nFor example, the introduction of the paper writes: \"InstMatte (Sun et al., 2022) first introduced the fomulation of multi-instance matting...\". But even this statement is not true. Sun et al 2022 was not the first paper to formulate multi-instance matting. For example, the paper:\nHu and Clark. \"Instance segmentation based semantic matting for compositing applications.\" 2019 16th Conference on Computer and Robot Vision (CRV). IEEE, 2019 uses instance segmentation to delineate foreground objects to be matted, and uses the simple iterative approach of matting these one at a time, with the other instances temporarily being considered as background.\n\nThere should be references to the early papers on semantic matting, as the more recent ones build on the ideas from these. Besides the two papers mentioned earlier, the following paper should be mentioned:\nSun,  Tang, and Tai. \"Semantic image matting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nThere are two references for the same paper in the (Qinglin Liu, Shengping Zhang, Quanling Meng, Bineng Zhong, Peiqiang Liu, and Hongxun Yao. End-to-end human instance matting. IEEE Transactions on Circuits and Systems for Video Technology, 2023.) One should be deleted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v02Va5YSDs", "reviewer": "Reviewer_ibAt", "summary": "This paper introduces a novel approach to instance matting by incorporating depth maps and multiplane image (MPI) representations with a tailored neural architecture. This design can handle occlusions, demonstrated through evaluations on two human instance matting benchmarks. Additionally, the authors evaluate their matting method for image editing and show advantages over diffusion-based methods.", "questions": "- The proposed method may rely heavily on the quality of the depth estimation algorithm. How might the model's performance vary with different monocular depth estimators?\n- How to set up the query during inference?\n- Additionally, it's unclear how the method does inpainting for tasks like instance removal (e.g., removing a person as shown in Figure 4(a); without the BI module, this proposed method still can in paint). Similarly, it would be beneficial to elaborate on handling occluded person inpainting (e.g., the person in Figure 4(b)).\n- Since there is an advantage in dealing with occlusions, separating the evaluation benchmark into two parts (occlusion-based or occlusion-free) and discussing how the model performs on different categories. \n- Based on the above, adding the instance-agostic benchmark (such as PPM) would help show the capability to process details. The samples in the instance matting benchmark are too small to see details.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Integrating depth maps and MPI representations into instance matting is novel and useful for handling occlusions. This approach is conceptually sound and well-motivated.\n- The method shows high quantitative performance on two benchmarks and also demonstrates effectiveness in image editing tasks, showing the potential application scope of the proposed framework.", "weaknesses": "- Only SAD and MSE metrics are provided, whereas other widely used metrics, such as Grad and Conn, are absent.\n- Since the paper emphasizes the significance of foreground estimation, additional metrics evaluating foreground estimation would strengthen the evaluation. \n- While quantitative results are promising, there are too few visual examples to fully assess the advantages. And most examples involve relatively simple cases (e.g., individuals with simple hairstyles). Including more challenging examples, such as individuals with complex hairstyles, would provide a more comprehensive evaluation. \n- The paper's visual results are also relatively small and make it hard to access the matting quality; larger figures are recommended for better clarity.\n- Line 519 claims stronger theoretical guarantees for content preservation. However, this seems to primarily benefit from the regression model, while the compared methods utilize diffusion-based generative models. Therefore, I didn't realize there was any theoretical guarantee for this.\n- Typos: In Line 389, \"Instance-agnostic\" should be \"Instance-aware.\"\n- Missing references: (1) a highly related thesis on using MPI for image matting: Single portrait image matting and bokeh effect synthesis via multiplane images, Master Thesis, 2023, HKUST. (2) Occasion-aware segmentation.\n- There will be some confusion mentioned in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f1OYJbBsXj", "reviewer": "Reviewer_Yuvj", "summary": "This paper presents MP-MAT, a framework using dual multiplane representations (Scene Geometry-level and Instance-level) for human instance matting. The approach achieves state-of-the-art results in both matting (2.76+ SAD improvement) and image editing tasks, with an uncertainty-guided refinement module enhancing performance in ambiguous regions. Notably, MP-MAT's zero-shot performance in editing outperforms specialized methods while maintaining high efficiency.", "questions": "- What drove your choice for the number of planes in SG-MP? Is performance sensitive to this parameter?\n- Could you provide the complete architecture of your plane generation network?\n- How did you determine the optimal uncertainty threshold T%?\n- What are the minimum GPU memory requirements for storing dual representations?\n- Have you tested the method on non-human objects?\n- Can you provide quantitative results on instance dragging tasks?\n- How does performance degrade with imperfect depth maps?\n- What's the scaling behavior with the number of instances?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Introduces a dual multiplane framework that combines geometric depth understanding with instance-level processing, adapting concepts from view synthesis to matting. The treatment of background as a special instance and uncertainty-guided refinement for ambiguous regions represent significant technical advances.\n- Demonstrates comprehensive validation through multiple datasets, achieving substantial improvements (2.76+ SAD on HIM-100K) and efficient runtime. The thorough ablation studies and zero-shot performance on editing tasks surpassing specialized methods provide evidence of the method's effectiveness.\n- Presents a well-structured technical solution with clear diagrams and effective visualizations. \n- The transparent discussion of limitations and systematic presentation of components enhance understanding and reproducibility.", "weaknesses": "- The method is tested exclusively on human instances, lacking validation on other object categories or across different datasets. \n- The heavy reliance on external depth estimation creates potential error propagation issues. The paper lacks theoretical analysis of when/why the multiplane approach might fail and doesn't explore how the number of planes affects the performance/efficiency tradeoff.\n- Key ablation studies regarding plane number sensitivity and computational overhead from dual representation are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GxeDwVouqU", "reviewer": "Reviewer_Vmaq", "summary": "This paper presents a new instance matting method that enhances 3D scene representation by constructing feature-level multiplane representations. It also introduces multiplane representation to segment the scene at the instance level. Additionally, foreground prediction refinement is incorporated to supervise and assist the network in learning instance-specific features.", "questions": "The paper uses SAD and MSE as evaluation metrics, but it’s important to clarify how these metrics are calculated.  I think the evaluation for multiple instances may differ from traditional single-instance matting. Additionally, given that InstMatt employs the IMQ metric, it would be helpful to explain why this metric was not used in the current evaluation.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces depth information into the instance matting task, which is technically sound in its approach through the use of multilayer representations.\n2. The instance multiplane representation offers a new approach distinct from existing methods like E2E-HIM’s instance guidance and Maggie’s IDEmbedding.\n3. The method demonstrates strong performance across two datasets, highlighting its effectiveness.", "weaknesses": "1. The innovation in foreground prediction has been discussed in other works such as Context Aware Matting, FBAMatting, and SampleNet, making it less novel.  In FBAMatting, the network predicts the alpha matte, foreground, and background simultaneously, which is similar to the design of the two decoders in this paper, but with the additional prediction of the background. This reduces the novelty of the proposed design.\n2. The method description is not entirely clear; for instance, the shape of O_i^u seems inconsistent between lines 273 and 279.  It is recommended to clarify the shape of O_i^u and specify whether it involves a k-dimension.\n3. The framework appears closely related to E2E-HIM, particularly with the use of bipartite matching during training, but certain aspects, such as the number of queries S, are not clearly explained. It is recommended that the comparison with E2E-HIM be enhanced by clarifying whether there are differences in the bipartite matching during training and in parameters such as the number of queries. Additionally, an ablation study on the hyperparameters would be beneficial.\n4. There seem to be issues with the references, with some entries being repeated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)