Zero-shot Image Editing with Reference Imitation
Authors: Xi Chen
Yutong Feng
Mengting Chen
Yiyang Wang
Shilong Zhang
Yu Liu
Yujun Shen
Hengshuang Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Editing
Image Customization
Diffusion Model
Primary area: diffusion_based_models

Abstract: Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e.g., some relative pictures come across online), without having to cope with the fit between the reference and the source. Such a design requires the system to automatically figure out what to expect from the reference to perform the editing. For this purpose, we propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images in a self-supervised manner. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives. We also construct a benchmark to facilitate further research.

Zero-shot Image Editing with Reference Imitation
Xi Chen'
Yutong Feng Yu Liu
Mengting Chen
Yiyang Wang'
Shilong Zhang'
Yujun Shen
Hengshuang Zhao i-
'The University of Hong Kong
Alibaba Group
Ant Group
Sounce
Source
BOOS
reforec
eeferaace
Roult
BOOST
Roult
Figure 1: Diverse editing results produced by MimicBrush where users only need to specify the to-edit regions in the source image (i.e., white masks) and provide in-the-wild reference image illustrating how the regions are expected after editing. Our model automatically captures the semantic correspondence between them, and accomplishes the editing with feedforward network execution.
Abstract
Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, help Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e.g., some relative pictures come across online), without having to cope with the fit between the reference and the source. Such design requires the system automatically figure what expect from the reference to perform the editing. For this purpose, propose a generative training framework, dubbed Mimi cBrush, which randomly selects two frames from video clip, masks some regions of one frame, and learns to recover the masked regions using the information from other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images
exercise their creativity more conveniently.
self-supervised manner. We experimentally show the effectiveness of our method under various test cases well its superiority over existing alternatives. We also construct a benchmark to facilitate further research.
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ref. Iuage
Text Prscupe
Ret. Musicllin
Ref. Image
Source Image
Model
Sounce Imagee
Model
Image
Model
Source Mask
Source Mask
Souvee Minic
(x) Impainting
(b) Composition
[c] Ours
Figure 2: Conceptual comparisons for different pipelines. To edit a local region, besides taking the source image and souree mask (indicates the to-edit region), inpainting models use text prompts to guide the generation. Image composition methods take a reference image along with a mask/box to crop out the specific reference region. Differently, our pipeline simply takes reference image, the reference regions are automatically discovered by the model itself.
1 Introduction
Image editing enables various applications for creating novel content, e.g., adding new object(s), modifying attributes, or translating image styles. Recently, powered by the large-scale pre-trained text-to-image diffusion models [33, 30, 36], the range of capacity for editing models [18, 3, 7, 2, 43, 15, 26, 12] also expands significantly.
The advanced editing methods could satisfy large variety of user requirements for modifying either a full image [2, 18] or its local regions [43, 26, 15, 7, 18, 3]. However, it is still challenging for existing editing models to fit the requirements of complicated practical scenarios. For instance, as shown in Fig. 1, it is required to modify the sole of a shoe by referring to another one, or to paste a specified pattern
given mug. Such kind of editing is important for real applications like product design, character creation, and special effects, etc.
For this kind of local editing, existing works take the source image with binary mask as input. As shown in Fig. 2 (a), inpainting [43, 52] methods re-generate the masked region following text instructions. However, it is not feasible describe the desired outcomes only with texts. For example, in Fig. 1, the design of shoes or the colors of hair is hard to describe accurately in text. Composition methods [7, 38, 37, 53] take reference image as input, along with mask/box representing the reference area, as shown in Fig. 2 (b). They could insert an "individual object" from the reference image into the source image but struggle to deal with local components (like shoe soles and human hair) or local patterns (like logos and texture). These methods require to carefully extract the reference area from the image. Nevertheless, local components are inherently intertwined with the context and are hard to understand when isolated from the whole object. Besides, their [7, 38, 53] training process requires the mask pairs to indicate the same object in different states (e.g., two video frames). Object-level mask pairs are feasible to obtain, but it is difficult to get the part-level pairs at scale.
To deal with the aforementioned requirements, we propose a novel pipeline of editing, termed imitative editing. As illustrated in Fig. 2 (c), given a source image with a masked area for editing, requires only the reference image without masks. Then, imitative editing targets to fill in the masked area by automatically finding and imitating the corresponding part in the reference image. Such pipeline formulates more convenient interactions, without strictly separating the reference components from the whole image. Besides, it reaches harmonious blending referring to the relation between the reference region and its surroundings (e.g.. the sole and vape of the shoe).
To achieve imitative editing, we design a framework called Mimi chushh with dual diffusion U- Nets tackle the source and reference images. More specifically, we train it in a self-supervised manner, where we take two frames from a video to simulate the source and reference images. As the video frames contain both semantic correspondence and visual variations, Mimi cBrush learns to discover the reference region automatically and repaint it into the source image with a natural combination to its surroundings. In Mimi cBrush, we send the masked source image into an imitative U-Net and the reference image into a reference U-Net, respectively. Then the attention keys and values of the reference U-Net are injected into the imitative U-Net, which assists in completing the masked regions. As demonstrated in Fig. 1, MimicBrushi overcomes the variations between the source and reference images in different poses, lightings, and even categories. The generated region highly preserves the details of the visual concepts in the reference image, and harmoniously interacts with the backgrounds. For a more comprehensive evaluation the proposed method, we also construct high-quality benchmark of imitative editing. The benchmark includes two main tasks, i.e. part composition and texture transfer. Each task covers several sub-tracks inspired by practical applications, e.g., fashion and product design.
2

---

Reference Image
Augment
Reference U-Net
Image Latest AAIAN
lecae DUWN
Video Flainss
Mack 0 1.331000
Laten
Concal K,
Source Image
Masking
Augment
Imitative U-Net
Depth Estimation
Projector
Figure 3: The training process of Mimi cBrush. First, we randomly sample two frames from a video sequence as the reference and source image. The source image are then masked and exerted with data augmentation. Afterward, we feed the noisy image latent, mask, background latent, and depth latent of the source image into the imitative U-Net. The reference image is also augmented and sent to the reference U-Net. The dual U-Nets are trained to recover the masked area of source image. The attention keys and values of reference U-Net are concatenated with the imitative U-Net to assist synthesis of the masked regions.
2 Related Work
Image inpainting. Traditional image inpainting methods [21, 22, 51, 50] only leverage the background information complete the masked regions. Powered by the text-to-image diffusion models, recent works [43, 30, 2, 15, 26] leverage text prompts to guide the generation content of the editing regions. These works fully leverage the flexibility of prompts to generate diversified content. A potential limitation is that only using text prompts could not fully express the user's intention for some specific demands.
Image customization. To generate the image for the given subject with high fidelity, customization methods optimize new "word" or use LoRAs As to learn specific concepts. However, most of the customization methods [34, 23, 24, 11, 1, 13] tackles the full object. Besides, they require 3-5 exemplar images and rely subject-specifie fine-tuning which last half hour. Among them, RealFill [41] and CLiC [35] could customize the local region in context. However, RealFill requires 3-5 images for the same scene, and the finetuned model could only complete local regions of the trained scene. CLiC [35] could customize the local patterns to edit different objects, but it only demonstrates cnter-ya generalizations and still requires subject-specifie fine-tuming
Image composition. This topic explores inserting given object into a specific location of the background with harmonious blending. The early works involved a long pipeline of image segmentation, pasting, and harmonization [39, 9, 4, 14, 10, 5]. Diffusion-based methods propose end- to-end solutions and support the pose variations of the reference object. Paint-by maint-b--exmplle [47] and ObjectStitch [37] use a CLIP image encoder to extract the representation the object. AnyDoor [7] uses DINOv2 [27] encoders and collects training samples from videos. Later works [53, 54, 28, 44] add camera parameters or text prompts to increase the controllability. However, they mainly focus on inserting the full-object. Part-level composition poses higher demands for modeling the interaction between the editing region and the surrounding context.
Method
3.1 Overall Pipeline
The overall framework of Mimi cBrush is demonstrated in Fig. 3. To realize imitative editing, we design architecture with dual diffusion models and train it in self-supervised manner.
Video data contains naturally consistent content, and also shows visual variations such as different postures of the same dog. Thus, randomly pick two frames from video clip the training

---

samples of Mimi cBrush. One frame serves as the source image, where we mask out some of its regions. Another frame serves as the reference image, assisting the model to recover of the masked source image. Throughout this way, Mimi mimihBruh learns to locate the corresponding visual information (e.g. the dog's face), and repaint it into the masked area in the source image. To ensure the repainted part is harmoniously blended into the source image, Mimi cBrush also learns to transfer the visual content into the same posture, lighting, and perspective. It is noteworthy that such a training process is built on raw video clips without text or tracking annotations, and can be easily scaled up with abundant videos.
cimush leverages dual branch of U-Nets, i... imitative and reference U-Net, taking the source and reference images as input, respectively. The two U-Nets share their keys and values in the attention layers, and are tamed to complete the masked source image by seeking the indications from the reference image. We also exert data augmentation on both images to increase the variation between source and reference image. At the same time, a depth map is extracted from the unmasked source image and then added to the imitative U-Net as an optional condition. In this way, during inference, users could decide whether enable the depth map of source image to preserve the shape of the objects in the original source image.
3.2 Model Structure
Our framework majorly consists of an imitative U-Net, reference U-Net, and depth model. In this section, we elaborate on the detailed designs of these components.
Imitative U-Net. The imitative U-Net is initialized with stable diffusion- 1.5 [33] inpainting model. It takes a tensor with 13 channels as the input. The image latent (4 channels) takes charge of the diffusion procedure from an initial noise the output latent code step by step. We also concatenate a binary mask (1 channel) to indicate the generation regions and a background latent (4 channels) of the masked source image. In addition, we project the depth map into 4-channel depth latent to provide shape information. The original U-Net also takes the CLIP [31] text embedding as input via cross-attention. In this work, we replace it with the CLIP image embedding extract from the reference image. Following previous works [7, 49], we add a trainable projection layer after the image embedding. We do not include this part in Fig. for the simplicity of illustration. During training, all the parameters of the imitative U-Net and the CLIP projection layer are optimizable.
Reference U-Net. Recently, bunch of works [56, 17, 46, 6, 58, 45] prove the effectiveness of leveraging an additional U-Net to extract the -he-ggained features from the reference image. For this part, we do not claim novelty and apply similar design termed reference U-Net. It is initialized from a standard stable diffusion- 1.5 [33]. It takes the 4-channel latent of the reference image to extract multi-level features. Following [46], we inject the reference features into the imitative U-Net in the middle and upper stages by concatenating its keys and values with the imitative U-Net as the following equation. In this way, the imitative U-Net could leverage the content from the reference image to complete the masked regions of the source image.
Attention = softmax( Qi . cat K,, K,)
) -cat(VV V1, VT))
Vil
Depth model. We leverage Depth Anything [48] to predict the depth maps of the unmasked source image a shape control, which enables Mimuhh to conduct texture transfer. We freeze the depth model and add a trainable projector, which projects the predicted depth map (3-channel) to the depth latent (4-channel). During training, set a probability of 0.5 to drop the input of the depth model as all-zero maps. Thus, the users could take the shape control as an optional choice during inference.
3.3 Training Strategy
To fully unleash the cross-image imitation ability of Mimi cBrush, we further propose some strategies to mine more suitable training samples. Considering that our goal is to conduct robust imitative editing even cross categories, the philosophy of collecting training data could be summarized as twofold: First, we should guarantee that the correspondence relation exists between the source and reference images. Second, we expect large variations between the source image and the reference image, which is essential for the robustness of finding the visual correspondence.

---

Part Composition
Scane
Mat
Rri heap
Foxt Proege
Menic
SSIM PANK
CuiPles cal
Inter-ID Inittaion
Wio GT
Juner-ID Reconstraction
HY GIT
Texture Transfer
Saaee Trug
Sonve Mark
Soused Image
Dato
Scaze Mast
Toxt Phonge
Matic
FANE TPIPS
Inter-1D Imitation
GT
Ianer-ID Reconstruction
Figure 4: Sample illustration for our benchmark. covers the task of part composition (first row) and texture transfer (second row). Fach task includes Inter-ID) and imner-II) track. The annotated data and evaluation metrics for each track are illustrated beside the exemplar images.
Data selection. During training, we sample two frames from the same video. Following [8], use SSIM [42] as an indicator of the similarity between video frames. We discard the frame pairs with too-big or too-small similarities to guarantee that the selected image pair contains both semantic correspondence and visual variations.
Data augmentation. To increase the variation between the source image and the reference image, exert strong data augmentations. Besides applying the aggressive color jitter, rotation, resizing, and flipping, we also implement random projection transformation simulate the stronger deformation.
Masking strategy. A simple baseline is to divide the source image into x N grid and randomly mask each grid. However, find this purely random masking tends to cause large portion of easy cases. For example, as the background (e.g., the grassland, the sky) occupies large areas with repeated content/textures, learning complete these regions does not require the model to seek guidance from the reference image. To find more discriminative regions, we apply SIFT [25] matching between the source and reference images and get series of matching points. Although the matching results not perfect, they are sufficient to assist us in constructing better training samples. Specifically, we increase the masking possibility of the grids with matched feature points.
Considering that collecting high-quality images much easier than videos, we also construct pseudo frames by applying augmentations on the static images and leveraging the object segmentation results for masking the source image. The segmentation masks also improve the robustness Mimi cBrush to support masks in more arbitrary shapes.
In general, Mimi cBrush does not rely on the heavy annotations of the training data. It fully benefits from the consistency and variation of video data, and also leverages image data to expand the diversity, which makes the training pipeline more scalable.
3.4 Evaluation Benchmark
Imitative editing is novel task, we construct our own benchmark to systematically evaluate the performance. As shown in Fig. 4, we divide the applications into two tasks: part composition and texture transfer, and we set the inter-II and inner-II track for each task.
Part composition estimates the functions of discovering the semantic correspondence between the source and reference image and compositing the local parts. The inter-ID track aims to composite the local parts from different instances even different categories. We collect data from the following topics: fashion, animal, product, and scenario. For each topic, we manually collect 30 samples from Pexels [29] thus 120 samples in total, where each sample contain the source and reference image pair. We manually draw the source mask to define the composition requirement. As the generated results do not have ground truth, we annotate the reference regions and write text prompts for the expected result. Thus, we could follow DreamBooth [34] to calculate the DINO [27] and CLIP [31] image similarities between the generated region and the annotated reference region. In addition, we also report the CLIP text similarity between the edited image and the text prompts.
5

---

Source Mask
Reference
Firefly Pronne
pbE Box
AnyDoor wi Mask
Ours
Figure Qualitative comparisons. Noticing that other methods require additional inputs. Firefly [32] takes the detailed prompts descriptions. Besides, we mark the specific reference regions with boxes and masks for Paint-by-Exampa [47] and AnyDoor [7]. Even though, MimicBrush still demonstrates prominent advantages for both fidelity and harmony.
We also set an inner-ID track, where we collect 30 image pairs from DreamBooth(344] manually mask the discriminative regions of the source image, and use reference images to complete them. The reference would be an image containing the same instance in different scenarios. Thus, the unmasked source image could serve as the ground truth to compute SSIM [42]. PSNR [16], and LPIPS [57].
Texture transfer requires strietly maintaining the shape of the source objects and only transferring the texture/pattern of the reference image. For this task, we enable the depth map as an additional condition. Different from the part composition that seeks the semantic correspondence, in this task we mask the full objects thus the model could only discover correspondence between the textures (reference) and the shape (source). We also formulate inter-ID and inner-ID tracks. The former involves 30 samples with large deformations from Pexels [29], like transferring the leopard texture cap in Fig. 4. The latter contains additional 30 examples from the DreamBooth [34] dataset. We follow the same data formats and evaluation metrics as part composition.
Experiments
4.1 Implementation Details
Hyperparameters. In this work, all experiments are conducted with the resolution of 512 512. For the images with different aspect ratios, we first pad the images square and then resize them to 512 x 512. During training, we use the Adam [19] optimizer and set the learning rate as le-5. Experiments are conducted with total batch size of 64 on 8x A100 GPUs. For the masking strategy of the source image, we randomly choose the grid number N N from 3 to 10. We set 75% chances to drop the grid with SIFT-matched features and set 50% chances for other regions. We add the reference U-Net as classifier-free guidance and drop it during training with the probability of 10%. During inference, the guidance scale is 5 as default.
Training data. We collect 100 k high-resolution videos from open- sourced websites like Pexels [29]. To further expand the diversity of training samples, we use the SAM [20] dataset that contains 10 million images and billion object masks. We construct pseudo frames by applying strong data augmentations on the static images from SAM and leverage the object segmentation results for masking the source image. During training, the sampling portions of the video and SAM data are 70% versus 30% as default.

---

Table I: Quantitative comparisons for part composition on our constructed benchmark. The left part of the table reports the evaluation results on the inner-ID track. The right part estimates the inter-ID track. Mimi cBrush demonstrates superior performance for each track with the most simplified interaction form. "T" or "T" denotes the image or text similarity.
inner-ID PSNR (t)
inter-ID
SSIM (t)
LPIPS (U)
DINO-I (t)
CLIP-1 (t)
CLIP-T (f)
PoE [47] w/o Box
0.51
15.17
0.49
41.44
81.00
29.45
PoE [47] w/ Box
0.51
16.09
0.48
42.70
81.10
29.30
AnyDoor [7] w/o Mask
0.42
12.73
0.56
43.41
78.56
28.45
AnyDoor [7] w/ Mask
0.44
14.09
0.50
61,30
86.08
29.39
MinicBrush
0.70
17.54
0.28
56.48
84,30
30,08
Table 2:
User study results. We let annotators rank the results of different methods from the best to the worst from three aspects: fidelity, harmony, and quality. We report both the number of the best picks and the average rank for comprehensive comparison.
Fidelity Beat
Fidelity Bank
Hamony
Hamony Nowk
Quality
Quality Rank
PbE [47] w/ Box
10.8%
2.64
29.2%
2.57
15.8%
2.59
AnyDoor Door [7] w/o Mask
2.8%
2.46
3.0%
2.35
4.2%
2.45
Any Door [7] w/ Mask
30.6%
2.77
22.6%
2.72
29.4%
2.71
MimicBruah,
55.8%
2.11
45.2%
2.34
50.6%
2.23
4.2 Comparisons with Other Works
In this section, we compare Mimi cBrush with other methods that could realize similar functions. Noticing that imitative editing is novel task, no existing methods could perfectly align input formats. Thus, we allow additional inputs for other methods. For example, we give additional masks or boxes for AnyDoor [7] and Shint-by-Examppe [47] indicate the reference regions. We also pick the state-of-the-art inpainting tool Firefly [32] and feed it with detailed text descriptions.
Qualitative comparison. We visualize the qualitative results in Fig. 5. Although Firefly [32] accurately follows the instructions and generates high-quality results, is hard for the text prompt to capture the details of the desired outputs, especially for logos patterns like tattoos. Paint- by-example [47] requires a cropped reference image in which the reference regions are centered. However, even if provide this kind of input, as this model only uses single token to represent the reference, it cannot guarantee the fidelity between the generated region and the reference region
We carefully annotate the masks of the reference region and fed them to AnyDoor [7]. It demonstrates stronger abilities for identity preservation but fails to synthesize harmonious blending. We analyze that there are two main reasons: first, some local parts could not be well understood when cropped out of the context. Second, most the training samples of AnyDoor [7] are full objects. It requires paired mask annotation for the same instance in different video frames for training. The paired masks are feasible to collect for full objects but hardly feasible for local parts. However, Mimi cBrush gets around this problem by leveraging the model to learn the correspondence itself in the full context instead of using the paired masks. In this way, Mimi cBrush shows significant superiorities compared with previous methods for completing arbitrary parts with full reference image.
Quantitative comparison. We also report the results on our constructed benchmark for part composition in Tab. 1. For the inner-ID track with ground truth, MimicBrush shows the dominant performance even though we give additional conditions for other works. For inter-ID imitation, is more challenging to discover the correspondent reference region. MimicBrush could still get competitive performance compared with AnyDoor [7]. We should note that the reference mask is given to Any Door. Therefore, it could forcedly locate the reference regions thus taking some advantages for the evaluation. However, as demonstrated in Fig. 5, struggles to generate harmonious blending and preserve the fine details.
User study. Considering the metrics could not fully reflect human preferences to some extent, we also organized a user study. We let 10 annotators rank the generation results of different models on our benchmark (introduced in Sec. 3.4). We evaluate each sample from three aspects: fidelity, harmony, and quality. Fidelity considers the ability to preserve the identity of the reference region. Harmony estimates whether the generated regions could naturally blend with the background. Quality regards whether the generated regions contain fine details in good quality. Those three aspects evaluated independently. Results are listed in Tab. 2, MimicBruse carns significantly more preferences.
7

---

Ref Source
CLIP
DINOV2
U-Net
Ref & Source
CLII
DINOv2
U-Net
Figure 6: Ablation study for reference feature extractors. CLIP and DINOv2 encoders could also achieve imitative editing but lag behind the U-Net in preserving the fine details.
Table 3: Ablation study for different reference feature extractors. U-Net demonstrates consistent
advantages across different evaluation tracks and metrics compared with CLIP and DINOv2.
Compositon
Testure Trander
SSIM (*)
PSNK (1)
LPIPS [1] [IINO-] (1) CLIP4 (1)
CLIP-T (1)
SSIM (1)
PSNK (1)
LPMPS (1)
DINO-1 (1)
CLIP4 (1)
CLIPT (?)
CLIP D.66
16.78
45.00
30.05
0.75
16.78
37.56
DINOv2 Enooder
0.67
16.50
0.32
48
62:40
29.84
0.74
17.27
0.27
46.34
78.00
30.61
Table 4: Ablation study for training strategies. In the first block, we verify the importance of training data and augmentation. the second block, we explore different strategies for masking the source image. The performance of our full pipeline is given at the bottom.
Part Comportton
Textine Tranfar
SSIM
PSNR (1)
DINO-C(1)
CL)P-I CLIP-T
PSNR (1)
DINO-I CLIP-1
Image Data Only
0.67
14,95
0.33
39.68
79.90
29.12
0.70
15.10
0.31
41.30
77.80
30.72
Aug.
0.68
0.30
50.55
53.20
29.51
0.74
0.26
31.23
Single Bottu
0.66
15.97
0.32
47.41
$2.10
28:77
0.72
16.24
0.31
48.52
75.10
29.30
Gidaas
0.68
17.10
0.30
49.17
83.94
20
0.74
17.35
0.27
50.09
79.50
31.05
Mask Gidaao
0.68
16.97
0.30
50.09
$2.55
29.84
0.73
17.25
0.27
48.58
81.00
30.35
Gidara
0.67
16.11
0.30
$3.00
20.75
0.71
0.28
52.46
81.75
30.02
SinicSruch
0.70
17.54
0.28
$6.48
$4.30
30.05
0.75
17.73
0.26
49.33
79.44
31.75
4.3 Ablation Studies
In this section, we conduct extensive ablations to verify the effectiveness of different components. Referece feature extractor. MimicBrushi leverages dual U-Net structure to model extractor the features from the source and reference image respectively. Some previous works [40, 55] prove that the pre-trained diffusion models contain strong prior to capture semantic correspondence. We explore whether an asymmetric structure could still learn the semantic correspondence under our self-supervised training pipeline. We replace the reference U-Net with the CLIP/DINOv2 image encoder and inject the 16 16 patch tokens within the cross-attention layers. Visual comparisons provided in Fig. 6, CLIP and DINOv2 also successfully locate the reference region, but U-Net illustrates clear superiorities for preserving the fine details. We also conduct quantitative results in Tab. 3, where conclude that CLIP and DINOv2 also reach competitive performance. However, as U-Net gives multi-level representations with higher resolutions, and the feature space is naturally aligned with the initiative U-Net, gives better results when serving a reference feature extractor.
Training strategies. In Tab. 4, We first verify the effectiveness of the "video-based" training pipeline. When using the statistics images only, the performance for each task drops significantly. It shows that object deformation or variation videos is vital for realizing imitative editing. Afterward, we remove the strong color jitter, resizing, and projection transformation. We observe clear degradation in part composition. specifically for the inter-ID track. verifies the importance augmentation for robust semantic correspondence matching.
In the second block, we explore the different masking strategies the source image and report the performance using a single box and different gid ratios. As introduced in Sec. 3.3, these purely random masking strategies could cause large number of low-quality training samples. contrast, we leverage SIFT matching to enhance the masking and reach better performances (bottom row).

---

N
Source Mask
Reference
Result
Source + Mask
Reference
Result
Figure 7: Diverse applications supported by MimicBrush. Our methods could be applied conveniently for product design, accessories wearing, editing the scene images, and refining the flawed generation results of other methods. MimicBrush is able to edit multiple regions in pass.
4.4 Qualitative Analysis
In this section, we give more visual examples and discuss the potential applications. As demonstrated in Fig. Mimi cBrush could deal with images from various topics and domains. The first row illustrates the application for product design. The second row shows some examples of jewelry- hard extract,
wearing. It should be noticed that the segmentation masks for the necklace but cBrush gets rid of the segmentation step and directly transfers the necklace from the reference image the source image. the third row, show that Mimic cBrush could also deal with backgrounds also nature effects, proving its strong generalization ability.
The last row illustrates practical application that could leverage Mimi cBrush to refine the generation results other works. the left example, we improve the fidelity for the image generated by AnyDoor [7]. In the right example,
prst-processing
mark multiple to-edit regions in the source image generated by Cones-2 [24] and provide a concatenated reference image containing both objects. We observe that Mimi cBrush could refine all the to-edit regions in single pass.
Conclusion
We present novel form of image editing with simple interactions, called imitative editing. In our setting, users are only required to mark the editing region on the source image and provide reference image that contains the desired visual elements. Mimi cBrush automatically finds the corresponding reference region to complete the source image. To achieve imitative editing, we take full advantage of the consistency and variation of videos and design a self-supervised training pipeline that uses one frame to complete the masked regions of another frame. Mimi cBrush demonstrates impressive performances various editing tasks and supports a wide range applications. facilitate future explorations, we construct benchmark comprehensively evaluate imitative editing. This work is expected to bring new inspiration for the community to explore more advanced techniques for image generation and editing.

---

Limitations & potential impacts. MimicBrush demonstrates robust performance. However, it could fail locate the reference region when the region too small or multiple candidates exist in the reference image. In this case, users should crop the reference image to zoom in on the desired regions. Mimi cBrush could deal with wide range of images, thus making it possible to produce some content with negative impacts. Therefore, we would add censors to filter out the harmful content when we release the code and demo.
Acknowledgement. This work is supported by the National Natural Science Founda- tion of China (No.62201484), HKU Startup Fund, and HKU Seed Fund for Basic Re- search.
References
O. Avrahami, K. Aberman, O. Fried, D. Cohen-Or, and D. Lischinski. Break-a-scene: Extracting multiple concepts from a single image. SIGGRAPH Asia, 2023.
[2] T. Brooks, A. Holynski, and A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023. 2,
[3] M. Cao, X. Wang. Z. Qi, Y. Shan, X. Qie, and Y. Zheng. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In ICCV, 2023.
[4] B.-C. Chen and A. Kae. Toward realistic image compositing with adversarial learning. In CVPR, 2019. 3
[5] H. Chen, Z. Gu, Y. Li. J. Lan, C. Meng, W. Wang, and H. Li. Hierarchical dynamic image harmonization. In ACMMM, 2022.
[6] M. Chen, X. Chen, Z. Zhai, C. Ju. X. Hong. Lan, and S. Xiao. Wear-any-way: Manipulable virtual try-on via sparse correspondence alignment. ECCV, 2024.
[7] Chen, L. Huang. Y. Liu, Shen, Zhao, and Zhao. Anydoor: Zero-shot object-level image customization. In CVPR, 2024, 2, 3, 4, 6, 7,9,
[8] X. Chen, Z. Liu, M. Chen, Y. Feng. Y. Liu, Y. Shen, and H. Zhao. Livephoto: Real image animation with text-guided motion control. In ECCV, 2024. 5
[9] W. Cong, J. Zhang, L. Niu, L. Liu, Z. Ling, W. Li, and L. Zhang. Dovenet: Deep image harmonization via domain verification. In CVPR, 2020. 3
[10] W. Cong, X. Tao, L. Niu, J. Liang. X. Gao, Q Sun, and L. Zhang, High-rcsolution image harmonization via collaborative dual transformations, In CVPR, 2022.
[11] R. Gal, Y. Alaluf, Y. Atzmon, Patashmik, A. H. Bermano, G. Chechik, and D. Cohen-Or. An image worth one word: Personalizing text-to-image generation using textual inversion. ICLR, 2023.
[12] Gu, Y. Wang. N. Zhao, W. Xiong, Q. Liu, Z. Zhang, Zhang, J. Zhang, Jung, and E. Wang. Swapanything: Enabling arbitrary object swapping in personalized image editing. 20Xiv:204405717,, 2024.
[13] Y. Gu, X. Wang, J. Z. Wu, Y. Shi. Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In NeurIPS, 2023.
[14] Z. Guo, H. Zheng, Y. Jiang, Z. Gu, and B. Zheng. Intrinsic image harmonization. In CVPR, 2021. 3
[15] A. Hertz, R. Mokady, Tenenbaum. K. Aberman, Pritch, and Cohen-or. Prompt-to-promee image editing with cross-attention control. In ICLR, 2023. 2,
[16] A. Hore and D. Ziou. Image quality metrics: Psnr vs. ssim. In ICPR, 2010. 6
[17] L. Hu, X. Gao, P. Zhang, K. Sun, B. Zhang, and L. Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. CVPR, 2024. 4
[18] B. Kawar, S. Zada, O. Lang, O. Tov, H. Chang. T. Dekel, I. Mosseri, and M. Irani. Imagic: Text- based real image editing with diffusion models. In CVPR, 2023.
[19] D. P. Kingma and J. Ba. Adam: method for stochastic optimization. arXiv.1412.69800 2014.
10

---

[20] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitchead, A. C. Berg.
W.-Y. Lo, et al. Segment anything. In ICCV, 2023.
[21] W. Li, Z. Lin, K. Zhou, L. Qi, Y. Wang, and J. Jia. Mat: Mask-aware transformer for large hole image inpainting. In CVPR, 2022.
[22] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang. A. Tao, and B. Catanzaro, Image inpainting for irregular holes using partial convolutions, In ECCV, 2018. 3
[23] Z. Liu, R. Feng. K. Zhu, Y. Zhang, K. Zheng. Y. Liu, D. Zhao, J. Zhou, and Y. Cao. Cones: Concept neurons in diffusion models for customized generation. In JCML, 2023.
[24] Z. Liu, Y. Zhang, Y. Shen, K. Zheng, K. Zhu, R. Feng, Y. Liu, D. Zhao, J. Zhou, and Y. Cao. Cones 2: Customizable image synthesis with multiple subjects. In NeurIPS, 2023. 3,9
[25] D. G. Lowe. Object recognition from local scale-invariant features. In ICCV, 1999.
[26] R. Mokady, A. Hertz, K. Aberman, Y. Pritch, and D. Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In CVPR, 2023. 2.3
[27] M. Oquab, T. Dareet, T. Moutakanni, H. Vo, M. Szafranice, V. Khalidov, Fernandez, D. Haziza, F. Massa,
A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. TMLR, 2024. 3,5
[28] Y. Pan, C. Mao, Jiang, Z. Han, and Zhang, Locate, assign, refine: Taming customized image inpainting with text-subject guidance. arXiv:2403. 19534, 2024. 3
[29] Pexels. The best free stock photos, royalty free images & videos shared by creators. https://wwy pexels. com, 2024. 5,6
[30] D. Podell, English, K. Lacey, A. Blattmann, T. Dackhorn J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv:2307.01952 2023. 2,
[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, ct al. Learning transferable visual models from natural language supervision. In ICML, 2021. 4, 5
[32] A. reseachers. Adobe firefly: Free generative ai for creatives. https://firefly. com/ generate/inpatt 2023. 7,
[33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 2,
[34] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023, 3, 5,
[35] M. Safaee, A. Mikaeili, O. Patashnik, D. Cohen Or, and A. Mahdavi-Amiri Clic: Concept learning in context. In CVPR, 2024. 3
[36] C. Saharia, W. Chan, S. Saxena. L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes.
B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurlPS, 2022. 2
[37] Y. Song, 7. Zhang, 7. Lin, S. Cohen, B. Price, J. Zhang, Y. Kim, and D. Aliaga. Objectstitch: Object compositing with diffusion model. In CVPR, 2023. 2,
[38] Y. Song, Z. Zhang, Z. Lin, S. Cohen, B. Price, Zhang, Y. Kim, H. Zhang, Xiong, and D. Aliaga. Imprint: Generative object compositing by learning prreetipproserving representation. In CVPR, 2024.
[39] K. Sunkavalli, M. K. Johnson. W. Matusik, and H. Plister. Multi-scale image harmonization. SIGGRAPH. 2010. 3
[40] L. Tang, M. Jia, Q. Wang, C. P. Phoo, and Hariharan. Emergent correspondence from image diffusion. In NeurlPS, 2023. 8
[41] L. Tang, N. Ruiz, Q. Chu, Y. Li, A. Holynski, D. E. Jacobs, B. Hariharan, Pritch, N. Wadhwa, K. Aberman, et al. Realfill: Reference-driven generation for authentic image completion. In SIGGRAPH, 2024. 3
[42] Z. Wang, A. C. Bovik, H. R. Sheikh, and E, P. Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 6,
11

---

[43] S. Xie. Z. Zhang, Z. Lin, T. Hinz. and K. Zhang, Smartbrush: Text and shape guided object inpainting with diffusion model. In CVPR, 2023. 2,3
[44] S. Xie, Y. Zhao, 7. Xiao, K. C. Chan, I. Y. Xu, K. Zhang, and T. Hou. Dreaminpainter: Text-guided subject-driven image inpainting with diffusion models. arXiv:2312.03771, 2023. 3
[45] Z. Xu, M. Chen. Z. Wang, L. Xing, Z. Zhai, N. Sang, J. Lan, S. Xiao, and C. Gao. Tunnel try-on: Excavating spatial-temporal tunnels for high-quality virtual try-on in videos. In ACMMM, 2024. 4
[46] 7. Xu, J. Zhang, J. H. Liew, H. Yan, J.-W. Liu, C. Zhang, J. Feng, and M. 7. Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In CVPR, 2024. 4
[47] B. Yang, S. Gu, B. Zhang, T. Zhang, X. Chen, X. Sun, D. Chen, and F. Wen. Paint by example: Exemplar- based image editing with diffusion models. In CVPR, 2023. 6, 7,
[48] L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024. 4
[49] H. Ye, J. Zhang, S. Liu, X. Han, and W. Yang. Ip-adanter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv:2308.0677 2023. 4
[50] J. Yu, Z. Lin, Yang, X. Shen, X. Lu, and T. S. Huang. Generative image inpainting with contextual atlention. In CVPR, 2018. 3
[51] J. Yu, Lin, J. Yang, X. Shen, X. Lu, and T. Huang. Free-form image inpainting with gated convolution. In ICCV, 2019.
[52] T. Yu, R. Feng, R. Feng. Liu, X. Jin, W. Zeng, and Z. Chen. Inpaint anything: Segment anything meets image inpainting. arXiv:2304. 06790, 2023. 2
[53] Z. Yuan, M. Cao, X. Wang. Z. Qi, C. Yuan, and Y. Shan. Customnet: Zero-shot object customization with variable-viewpoints text-to-image diffusion models. arXiv:2310.191 197844 2023. 2,3
[54] B. Zhang, Y. Duan, J. Lan, Y. Hong, H. Zhu, W. Wang, and L. Niu. Controlcom: Controllable image composition using diffusion model, arXiv:2308. 10040, 2023. 3
[55] J. Zhang, C. Herrmann, J. Hur, L. P. Cabrera, V. Jampani, D. Sun, and M.-H. Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. In NewrIPS, 2023.
[56] L. Zhang. Reference-only controlnet. https:// con//iiauilllsddweeeeeeeee www.:::illddddwwwwd-oeeeeeddeeee controoiicntttoneee// discussions/ 1236, 2023. 4
[57] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018. 6
[58] S. Zhang, L. Huang, X. Chen, Y. Zhang, Z.F. Wu, Y. Feng, W. Wang. Y. Shen, Y. Liu, and P. Luo. Flashface: Human image personalization with high fidelity identity preservation. 4rXiv:2403. 1700, 2024.
12

---

Appendix
In the Appendix, we begin by providing further descriptions of our constructed benchmark in Appendix A, along with the proposed metries. Subsequently, we delve into the details of each step of the user study in Appendix B. Following that, we offer additional analysis and discussion on the training strategies in Appendix C. Finally, Appendix D, we include more visual demonstrations and qualitative analysis for various applications.
A Benchmark Details
In this section, we provide detailed descriptions of our constructed benchmark and the proposed metrics. We include examples of part composition and texture transfer, and visualize the annotation for each track in Fig. A1. For the inter-ID tracks, we present a source image, source mask, reference image. reference mask, and a text prompt to specify the expected results. The reference mask and text prompts are used solely for evaluation, where we calculate DINO and CLIP image similarities, as well CLIP text similarity. Specifically, to compute the DINO and CLIP image similarities, first remove the background of both the reference image and the generated results using the reference masks and source masks. Then, we calculate minimum bounding box for the two masks, cropping both the generated and reference regions accordingly. The DINO and CLIP similarities are calculated between these two cropped regions with clean backgrounds.
For inner-ID tracks, since the source image can be considered as the ground truth, we omit annotating prompts for CLIP test scores. Instead, we compute the SSIM, PSNR, and LPIPS metrics. Similar to the cropping operation used before calculating the DINO and CLIP image similarities, we also utilize the masks to obtain the bounding box around the generated region. This box then employed crop the corresponding regions from both the generated result and the source image for computing the SSIM, PSNR, and LPIPS. Although the reference masks are not utilized for model input or metric computation, we still annotate the masks for each sample. This enables evaluation for other methods like Paint- saunt-y--xaann by-Example [47] and AnyDoor [7].
Inter-ID
h
A little boy is wearing #red searf#
Part Composition
Source Image
Source Mask
Ref. Image
Ref. Mask
Text Prompt
Generated Result
Imee--i
Source Image
Source Mask
Ref. Image
Ref. Mask
Generated Result
toy car with the texture of #a gray sweaterjt.
Inter-ID
Texture Transfer
Source Image
Source Mask
Ref. Image
Ref. Mask
Text Prompt
Generated Result
Inner ID
Source Image
Source Mask
Ref. Image
Ref. Mask
Generated Result
Figure A1: Visual demonstrations for the samples in our constructed benchmark. We provide one example for each track to illustrate the detailed data formats.
13

---

Source
Reference
Image Data
Weak Aug.
Naive Masking
Ours
Figure A2: Qualitative ablations for different training strategies. We present the results for various training strategies, including training with only image data, applying weak augmentations, and utilizing naive masking strategies. For comparison, we also provide the generation results of our full-version model.
B
User Study Details
As outlined in the main paper, we conducted a user study involving 10 annotators to assess the generation results of various methods. We devised source mask, reference image, and reference mask on the left side to clarify the expected editing outcomes. On the right side, the user interface displays the four results predicted by four different methods, each marked with an index. Annotators were tasked with answering four single-choice questions regarding the best, second-best, third-best, and fourth-best results index. To enhance the reliability of the user study results, we implemented several measures:
interface that presents the source image,
First, provided comprehensive documentation and rating examples for each task. Annotators were initially tasked with rating a small subset of data. Once their annotations passed our examination, they continued with the remaining samples.
Second, prevent annotators from distinguishing the results of different methods based on additional information, we standardized the image size of each result and shuffled the order of the four results randomly for each sample. We recorded the shuffled indexes maintain consistency.
Third, as human evaluation was required from three different aspects (fidelity, harmony, quality), these aspects were evaluated as three separate annotation tasks ensure independence and accuracy.
C Training Strategy Analysis
In this section, we provide additional analysis and qualitative ablation results for the training strategies. We follow the experimental settings introduced Tab. of the main paper. depicted in Fig. A2, when only utilizing statistical images for training, the first two rows indicate that the model is still
14

---

capable of learning semantic correspondence. However, it struggles to adapt to pose and lighting variations, resulting in rigid blending. This underscores the importance of incorporating visual variations from video data for our method to conduct robust and generalized imitative editing. We also showcase generation results using weak data augmentation and naive masking strategy (i.e., masking the random grid with 50% probability). Both approaches exhibit clear performance degradation compared to our full version of MimicBrush, which employs strong data augmentation and matching-guided masking strategy.
D More Visualization Results
This section presents a comprehensive array of examples highlighting the exceptional capabilities of methodology in diverse contexts.
Fig. A3 showcases results from inner-ID reconstruction where a specified region on the initial image is repainted using another image of the same instance as reference. These outcomes illustrate our method's precision accurately identifying the target region and retaining intricate details with high fidelity.
In Fig. A4, further examples of part composition are provided. The first row demonstrates seamless editing process, which facilitates the combination of multiple visual elements in a cohesive manner. The next two rows exhibit variations across assorted topics and domains, showcasing our method's robust adaptability.
In Fig. A5, we present instances of texture transfer applications. Here, utilize the depth map to maintain the original structure of the source image while artfully transferring texture from the reference image. This technique facilitates myriad of imaginative designs, allowing creators infuse elements with new aesthetic qualities while respecting the underlying form. Such versatility only provides tool for artists expand their creative palette but also offers practical solutions for industries requiring rapid visualization of texture variations without altering the product's shape. Consequently, our method stands as a testament to the convergence of creativity and technical innovation, opening new avenues for design exploration.
15

---

00
Source Mask
Reference
Result
Source + Mask
Reference
Result
Figure A3: Demonstrations for inner-ID reconstruction. In this scenario, we repaint masked local area of the source image using reference from the same instance. For the final two rows, we implement shape control utilizing the depth map for enhanced precision.
Source Mask
Reference
Result
Source Mask
Reference
Result
Figure A4: Demonstrations for part composition. The first row illustrates a case of continuous editing, allowing for step-by-step composition of multiple visual elements. The subsequent rows provide examples across various themes such as animals, fashion, and product design.
16

---

Reference
Source Mask
Source + Mask
Reference
Result
Source + Mask
Reference
Result
Figure A5: Demonstrations for texture transfer. When applying the shape control, Mimi cBrush could strictly follow the original shape and transfers the novel texture from the reference images.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract outlines the overall content and our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we include the discussion the section of conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We give sufficient details and hyper-parameters for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [No]
Justification: According to the regulations of the company, we would release the code and benchmark after internal check.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
public/guides/ https//oooSSuuisiinnnnee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiis
httpps.ccppubiic///ecc S ssi wwwiinsssoooiiiiiioo sesionnyocy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We report the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The papers on our topic do not provide an error bar, instead we run the experiments multiple times to report the average scores.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: //tear://///ips////eeeee ct/ sec/public//eeeeee ctw.ihhesccellllecccc ines?
Answer: [Yes]
Justification: Yes, we follow the code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss it in the section of the conclusion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our methods are not so risky, and we would add censor when we release the demo.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, checked the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not have assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Do not involve human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Do not apply.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "sJbR9TTtL2", "reviewer": "Reviewer_iASj", "summary": "The paper proposes a new form of image editing, termed imitative editing. In this scenario, the user can edit the local area of the source image with a similar area to the reference image. This requires the system to automatically figure out what to expect from the reference to perform the editing. To achieve this goal, the author first designed a new generative network, dubbed MimicBrush. Then, the paper also constructs the benchmark from the video clip. The paper demonstrates both qualitative and quantitative results to validate the effectiveness.", "questions": "Please refer to the weakness part.\n\nMy main concerns are about the depth of discussion and motivation of the proposed network and task, and the writing and formulation that are not serious enough to reach the criterion of Neurips.", "limitations": "The author already addresses the limitations of the work.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed imitative editing is reasonable, novel, and inspiring. As explained in the paper, such an editing form can facilitate other real-world applications in fashion and product design.\n\n\n2. The paper contributes a valuable benchmark for the proposed task, including diverse topics and examples, which can benefit later research. \n\n\n3. The demonstrated examples show desired and good qualitative and quantitative editing results.", "weaknesses": "# Motivation & Novelty\n**Unclear motivation and advantages**: The core learning algorithm of the paper is an imitative U-Net, a reference U-Net, and a depth model. However, the advantages and motivations of such designs are not clearly discussed and validated. For example, some other networks such as Controlnet and IP-adapter, can provide conditional information from the reference image. Why is injecting features from the imitative U-Net better on the proposed imitative editing task? \n\n**Unclear novelties in terms of learning compared with existing methods**: The proposed imitative task is new in that the system can edit part of the object whereas the latest methods mainly insert full objects. However, the novelty and difference seem to only lie in the training data (full object v.s. part object). The learning paradigm seems to be the same. Can these existing works do well in the proposed imitative editing task if trained with the collected part-level data? The essential difference **in terms of learning and formulation** between part-level editing and full-object editing is not addressed. My understanding is that they are essentially the same learning paradigm but in different concrete forms.\n\n\n# Formulation & Writing\n**Unclear notations**: \n1. The paper proposes a new task and a new framework. However, the problem and algorithm are not rigorously formulated. For example, what are the input, output, and masks **mathematically**? \n\n2. What is the training objective (loss function) in the pipeline? What is the loss used in the depth model?\n\n3. For concatenation and injection of K, V from the reference U-Net, which layers are injected? Are cross-attention and self-attention processed in the same way?\n \n\n# Experiments & Evidence\n**Limited editing types**: The demonstrated editing types are limited. \n1. For example, for the same source image, can the user edit different parts of the object with different reference images?\n2. Does the proposed method support general image editing such as changing objects (cat to dog), or shape (round cake to square cake)?\n\n**Unclear motivation on the SAM data**: The paper mentions using the SAM data in training except for the Video clips. How does the SAM data help the learning in this task? I think after the strong augmentation, the spatial position of different objects does not change. So, how does this help to learn the correspondence between the source image and reference image?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Motivation & Novelty**\n\nWe disagree. There exist misunderstandings about our motivation and novelty. The \"structure of dual U-Net\" and the \"data difference\" are not our contributions. Please refer to **A. Core contributions and novelty** in the global rebuttal to see our motivation.\n\n``1. Motivation for the dual U-Nets.``\n\nWe explain the issues of Reference U-Net:\n\n- **First, Reference U-Net is a common practice, we do not claim contribution.** As mentioned in line 121, Reference U-Net is widely used in many tasks that require a reference image [56, 17, 46, 6, 58, 45]. These works have already proved the advantages of this structure over previous works like IP-adapter or controlnet. We use Reference U-Net solely as our feature injection module and do not claim it as our novelty, so we provide citations and do not focus on this part to discuss. \n\n- **Second, we have compared different strategies in Tab.3 and Fig.6.** \"CLIP\" represents the strategy of IP-Adapter, and \"DINOv2\" represents the strategy of AnyDoor. We observed that the Reference U-Net preserves details better, as it provides feature maps with higher resolution. \n\n``2. Novelties in terms of learning.``\n\nThe core difference is not the data, but the \"mask-free\" task formulation and the training strategy for \"correspondence learning\". \n\nPlease refer to **B. Motivation of getting rid of the reference mask** in the global rebuttal block to see the differences between the previous object-level insertion methods.\n\n---\n\n**Formulation & Writing**\n\n``1. The input, output, and masks mathematically``\n\nOur task formulation is clearly demonstrated in Fig. 1, 2, 5, 6, 7. The input and outputs are straightforward: the model takes a source image $I_{source}^{h \\times w \\times 3}$, a binary source mask (indicating the region to edit) $M_{source}^{h \\times w \\times 1}$ , and a reference image $I_{ref}^{h \\times w \\times 3}$, and then predicts the edited image after iterative denoising.  The input and output could be formulated as follows:\n\n$  I_{result}^{h \\times w \\times 3} = f(I_{source}^{h \\times w \\times 3}, I_{ref}^{h \\times w \\times 3},  M_{source}^{h \\times w \\times 1} ) $\n\n``2. What is the training objective in the pipeline? What is the loss used in the depth model?``\n\nWe use the default loss function of diffusion models, which predicts the added noise at each step. The depth model is frozen during training (without loss). We did not emphasize these aspects as we did not make any modifications from the stable diffusion baseline. \n\nAs suggested, we include some basic formulations of diffusion models. \n\nFirst, the ground truth target image( unmasked source image, with augmentation ) and the reference image, are encoded into latent space via VAE encoders. We note the target latent and reference latent as $\\mathbf{z}^{tar}_0$, and $\\mathbf{z}^{ref}_0$ at time step 0. During training, we randomly sample a timestep $t$ and add Gaussian noises $ \\epsilon $ on the target latent:\n\n$ \\mathbf{z}^{tar}_t = \\sqrt{\\bar{\\alpha}_t} \\mathbf{z}^{tar}_0 + \\sqrt{1 - \\bar{\\alpha}_t} \\boldsymbol{\\epsilon}\\ $\n\nWe note the U-Nets as $ \\boldsymbol{\\epsilon}_{w} $, which takes the target latent(imitative U-Net), the reference latent(Reference U-Net), the CLIP image embedding of the reference image(noted as $c^{ref}$), and timestep $t$ as inputs to predict the added noise with MSE loss：\n$ \\mathbf{L}_t = \\left| \\boldsymbol{\\epsilon}_w(\\mathbf{z}^{tar}_t, \\mathbf{z}_0^{ref}, c^{ref}, t) - \\epsilon \\right|^2_2\\  $\n\nWe will include these details in the revised version.\n\n\n``3. For concatenation and injection of K, V from the reference U-Net, which layers are injected? Are cross-attention and self-attention processed in the same way?``\n\nAs discussed in \"Motivation and Novelty,\" reference U-Net is a common practice [56, 17, 46, 6, 58, 45]. We do not claim contribution so we provide citations for the implementation details. Our implementation follows previous work[45] that injects K,V into self-attention layers of unet decoder layers. We will add these details in the revision.\n\n---\n\n**Experiments & Evidence**\n\n``1. For the same source image, can the user edit different parts of the object with different reference images?``\n\nYes, there are two ways to achieve this: \n\n1) concatenate multiple reference images like Fig.7 (the bottom-down example), where we concatenate the two reference images and mark multiple regions to be edited. The model can handle this case in a single pass.\n\n2) edit different parts progressively in multiple turns, like Fig. A4 in the appendix (the first example of Hinton), we show how \"cat ears,\" \"scarf,\" and \"glasses\" are added from different reference images to the source image.\n**We also add more examples as required in Fig.R1 in the rebuttal PDF file.**\n\n\n``2. Does the proposed method support general image editing such as changing objects (cat to dog), or shape (round cake to square cake)?``\n\nImage editing is a broad topic, and our goal is not to tackle all editing tasks. In this paper, we propose a novel form of imitative editing that encompasses \"part composition,\" \"texture transfer,\" and \"post-refinement\" for general categories. This already represents a wide scope, and no previous methods have achieved this combination.\n\nExamples for changing objects are added in Fig.R2(first row) in the rebuttal PDF file.\nBesides, although we could not support all the editing tasks, we could act as their post-refinement to enhance the performance. As in the last row of Fig.7, we could refine the artifacts of AnyDoor(which conducts object changing) and Cones-2 (which could change the shape and attributes of customized objects).  More examples of post-refinement are given in Fig.R2 in the rebuttal PDF. Our method could enhance the performance for a wide range of editing tasks only if they require a reference image. \n\n``How does the SAM data help the learning in this task?`` \n\nPlease refer to **C. Importance of SAM Data** in the global rebuttal block.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VjGFTMd8ee", "reviewer": "Reviewer_KHnJ", "summary": "The paper introduces a novel approach called imitative editing aimed at enhancing user creativity in image editing tasks. Traditional image editing often involves matching references to the source image, which can be challenging. In contrast, imitative editing allows users to directly draw inspiration from in-the-wild references without needing to precisely align them with the source image. This paper introduced a new approach called MimicBrush, a generative training framework that leverages video frame pairs to learn semantic correspondence and recover masked regions in images.", "questions": "please see above weaknesses. Overall I think this paper is trying to address a new and interesting editing scenario, which will make image editing more convenient.", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "--This paper proposed a novel task, Imitative editing, which is a new editing scenario that enables users to leverage diverse reference images without the need for exact matches, promoting more intuitive and less restrictive editing interactions. \n--The training strategy is self-supervised by utilizing two frames coming from the same video is intuitive. To ensure effective training, the paper further introduced data selection that select pairs without too big or too small variations, data augmentation to increase the task difficulty, and masking strategy to mask patches similar to reference in order to force the model to take guidance from reference for reconstruction.\n--This paper proposed a test benchmark containing part composition and texture transfer to evaluate the performance of imitative editing, which is good.", "weaknesses": "--Regarding part composition and texture transfer, if I understand correctly, the proposed framework appears capable of simultaneously addressing both tasks. However, based on the training procedure, I am curious about how the model distinguishes between referencing parts and textures. This differentiation does not seem explicitly addressed in the training process, potentially leading to the model misinterpreting user intentions during the inference stage.\n\n--Regarding ablation experiment, this paper present the training strategies about training data, augmentation and masking method. It would be better to also ablate the necessity of image training data. And for all the ablations, it would better to have qualitative comparisons to illustrate how such training strategy affect the visual quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment and constructive suggestions, we will follow you suggestions to polish our paper for the revision.\n\n``How the model distinguishes between referencing parts and textures.``\n\nDuring inference, users have the option to enable \"depth control.\" The depth map serves as a strong condition for maintaining the shape of the original content. When enabled, the model preserves the original shape, thus forced to conduct texture transfer. Without the depth map, the model has no constraints to retain the original shape, allowing it to change the shape thus performing part composition. During training, we randomly drop the depth map to ensure the model is compatible with both tasks.\n\n\n``Ablate the necessity of image training data``\n\nThank you for your constructive suggestions. We have found that image data is also important for our methods. Please refer to **C. Importance of SAM Data** in the global rebuttal block for details.\n\n\n``Qualitative comparisons for the ablation study``\n\nWe agree that qualitative comparisons can help readers better understand our methods. In fact, we have already included qualitative ablations for different training strategies in Fig.A2 of the appendix. We will add more examples in the revision. We also add visualized ablation studies for the training data in the rebuttal PDF Fig.R3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZ5BDmDAKn", "reviewer": "Reviewer_8FSf", "summary": "To achieve more precise imitative editing, the paper proposed a training framework, called MimicBrush. It randomly selects two frames from a video clip, masks some regions of one frame, and learn to recover the masked regions using the information from the other frame. Also, it constructs a benchmark, consisting two main tasks, i.e., part composition and texture transfer, to evaluate the performance comprehensively.", "questions": "1. How is rank calculated in table 2?\n2. Regarding line #195, does it mean that when the randomly selected grid number is 4, it would drop 4x75%=3 for SIFT-matched features and 4x50%=2 for other regions? So, in total, it would drop 5 grids?\n3. What does the gid ratio mean in line #261?\n4. The paper structure can be improved. Sec. 4.4 and Sec. 4.2 contain the “Qualitative comparison”. I suggest having only one Qualitative comparison section and separating it into subsections: “comparing with others” and “diverse applications without comparing with others”.\n5. The mask representation is hard to identify, e.g., the right case in the first row or the left case in the third row in Fig. 7. I suggest to change a clearer representation.", "limitations": "The authors have shared the limitations and potential societal impact.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The setting for part-level image editing is a novel idea. I believe it will be very helpful for designers to leverage imitative editing in their use cases.\n\n- The paper provides comprehensive experiments to prove its effectiveness.\n\n- The paper is well-written and easy-to-follow.", "weaknesses": "The definition of inter vs. inner is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment and constructive suggestions. We will follow your recommendations to polish our paper for the revision.\n\n`` W1. The definition of inter vs. inner is unclear.`` \n\n**Inter-instance** refers to cases where the source and reference images are of different instances, such as compositing the ear of a \"cat\" onto the head of a \"man\". This track corresponds to the most interesting and fantastic applications. However, since there is no ground truth for the composited target (e.g., the given man with the given cat's ear), we evaluate this track using metrics like DINO/CLIP scores (lines 174-176).\n\n**Inner-instance** refers to cases where the source and reference images are of the same instance. This track is relevant to post-refinement applications. For example, we might use AnyDoor or DreamBooth to generate a new image of a given bag and find that the generated logos or textures differ from the reference image. Our method can use the reference image to repaint the flawed region, even if they are not in the same pose. In our benchmark, we use real image pairs of the same instance as the source and reference images, manually masking some regions of the source image. In this scenario, the unmasked source image itself serves as the ground truth, allowing us to calculate SSIM, PSNR, and LPIPS.\n\nWe will make it clearer in the revision.\n\n\n\n``Q1. How is rank calculated in table 2?``\n\nWe directly average the rank numbers for all test examples, which allows us to compare different methods effectively.\n\n\n``Q2. Regarding line #195, does it mean that when the randomly selected grid number is 4, it would drop 4x75%=3 for SIFT-matched features and 4x50%=2 for other regions? So, in total, it would drop 5 grids?``\n\nAs mentioned in L195, “we randomly choose the grid number N×N from 3 to 10”, thus if we select grid number is 4, we have 4 x 4 = 16 patches.  We first split them into SIFT-matched patches (e.g., 4 patches) and no-matched patches (e.g., 12 patches).  Then, we drop  4 * 0.75 = 3 matched ones and 12 * 0.5 = 6 non-matched ones. We will make it clearer, thank you for your suggestions.\n\n``Q3. What does the gid ratio mean in line #261?``\n\nThe grid ratio here refers to the ratios of masked grid patches, which correspond to the results in Tab.4. We report the results for masking 25%, 50%, and 75% of grid patches. We will clarify this in the text. Thank you for your suggestions.\n\n``Q4 & Q5. The paper structure and mask representation.``\n\nThank you for your suggestions. We will reorganize the paragraphs and use more effective ways to present the masks, such as using different colors or providing additional binary masks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eg91YaB02n", "reviewer": "Reviewer_q758", "summary": "The paper proposes a method to modify specific parts of the image with the content from the reference image and consistently with the original content. It combines the technique of dual diffusion to replace the key and value features, which is used in MasaCtrl, of the source image with the reference image. The method is trained in a self-supervised manner, in which some parts of the image are masked, and the model is trained to recover them.", "questions": "What if the source image is a complicated image that contains some objects and the desired modified object is not a salient object?", "limitations": "The paper is not so novel; it reuses components from existing techniques. The proposed method is not actually zero-shot as it requires training.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand.\n- The end-to-end pipeline doesn’t need fine-tuning for each image.\n- The proposed method is friendly for users that only give the source image, reference image, and mask, and then the model automatically gets the correct content from the reference image and aligns for the original part that is specific by mask.\n- Experimental results are impressive with various categories and metrics.", "weaknesses": "- The proposed method is not actually zero-shot as it requires training. It requires expensive resources for the training model.\n- The method presented in this paper lacks novelty. It uses the old technique (i.e., replacing the key, value feature from MasaCtrl [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, ICCV 2023]) and combines with an old training strategy that recovers the region that is specific by mask.\n- The source and reference images must be the same scale as the object, and the source image must contain only one or a salient object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "``W1. The proposed method is not actually zero-shot as it requires training. It requires expensive resources for the training model.``\n\n**This comment is erroneous**. “Zero-shot” is different from “training-free.” Zero-shot means that we train the model on some examples, and the model’s ability can generalize to unseen examples without case-by-case fine-tuning or optimization. Hence, our setting is exactly \"zero-shot.\"\n\nAdditionally, our model is not training-expensive. Our final model is trained on 8 GPUs, and we have verified that our results can be reproduced with 2 or 4 GPUs. Besides, we do not require any additional human annotations for training our model.\n\n\n``W2. The novelty and techniques of our methods.`` \n\nWe disagree. There exist misunderstandings about our motivation and novelty. \n\nFirst, the dual u-net structure similar to MasaCtrl is not our contribution, it is a common practice for injecting the features of reference images, which is widely used in recent papers [56, 17, 46, 6, 58, 45]. Besides, we tackle totally different tasks with MasaCtrl.\n\nSecond, the core training strategy is not the MAE-like mask completion, but \"correspondence learning\" between two images.  We fully utilize the consistency and variation of video frames, randomly mask one video frame, and use another \"full video frame\" for completion.\n\nPlease refer to **A. Core contributions and novelty.** in the global rebuttal block to see our contribution and novelty.\n\n\n``W3. The source and reference images must be the same scale as the object, and the source image must contain only one or a salient object.``\n\nIt is not true.\n\nFirst, we support source and reference objects of different scales. For example, in Fig. 5, the second row features Hinton and Tyson, and in Fig. A4 in the appendix, the first and second rows illustrate the scarf and ears. These examples all have quite different scales Additionally, as mentioned in L145-147, we add strong augmentations like resizing during training, making our model robust to scale variance.\n\nSecond, the source image can contain multiple objects. As shown in the bottom-right example of Fig. 7, our model performs well in one pass. In this paper, we primarily illustrate images with a single salient object for clear visualization. However, our method can support complex source images, and we have added more examples in Fig. R1 in the rebuttal PDF, where the source images contain **complicated scenes and large scale-variations**. We will include these examples in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable suggestions. We emphasize the motivation and novelty in the global rebuttal and  respond to each reviewer individually. Additionally, we include more figures and tables in the attached PDF file.

---

### A. Core contributions and novelty.

We would like to reaffirm our core contributions below.

- **Novel form for image editing.** Unlike the commonly studied task of "object insertion", our proposed "imitative editing" offers a new editing form, which does *not* require users to specify the region of interest from the reference image. Such a design **sufficiently simplifies the editing process**, because users only need to brush once (*i.e.*, the source image) instead of twice (*i.e.*, both the source image and the reference image), and hence clearly distinguishes our approach from existing methods. It is also noteworthy that getting rid of the reference mask is essential for *part-level editing*, making our algorithm more **flexible** in practice.
- **Novel training strategy.** To accomplish "imitative editing", it requires the model to **automatically** analyze the semantic correspondence between the to-edit region of the source image and the "full" reference image. For such a purpose, our approach benefits more from the training strategy but not the data. Concretely, we propose to leverage the temporal consistency and variation across video frames, and urge the model to recover the masked region of one video frame by *only* using the information from another frame *without* specifying the corresponding region. To our knowledge, our proposed training pipeline offers a brand new way of using video data for diffusion-based editing.
- **Benchmark for the novel task.** We construct a diversified benchmark that contains multiple tracks and covers a large variety of scenarios. Besides, we come up with various metrics to thoroughly evaluate the compare different methods. Our benchmark could be beneficial for further explorations for imitative editing.

---

### B. Motivation of getting rid of the reference mask.

As discussed above, this work presents a novel editing form, which allows users to *not* specify the region of interest from the reference image. Here, we would like to highlight **the necessity and the practical significance** of such a design.

- It is challenging for users to segment some local regions like tattoos or thin necklaces, even with intelligent tools.
- Local regions like tattoos are inherently intertwined with the context and difficult to understand when isolated.
- During training, methods like AnyDoor require image pairs with the same instance in two images with masks. While it's easy to collect object-level pairs through tracking or matching, gathering high-quality mask pairs for general local parts is of high cost. By removing the reference mask, we can independently mask one video frame and use another frame (without the corresponding mask) to train the model.

---

### C. Importance of SAM Data

SAM data contains the masks, we randomly choose a mask and then make two different crop (with different box sizes and locations) around the mask to make an image pair. Besides, we add strong augmentations to guarantee the variations. In this case, the masked regions are contained in both the source and reference images, but in different locations and sizes. 

We find the SAM data important for the training procedure in the following aspects:

- Increased Diversity: Image data helps expand the diversity of categories and scenarios. It is challenging to collect category-balanced, high-quality videos at scale, but it is relatively easy to use open-source image datasets like SAM.
- Mask Compatibility: Image datasets like SAM contain large-scale masks. Using these real masks makes the model more compatible with user-drawn masks of arbitrary shapes.

**We add qualitative ablation results in the rebuttal PDF (Fig. R3) and quantitative ablations in the rebuttal PDF (Tab. R1).**  The results show that SAM data and video data are both important.

Meta-review: The paper presents a new form of image editing called imitative editing. The review comments are mixed. Overall, the reviewers appreciate the formulation of the novel task and the proposed evaluation for this task. However, they have different opinions on the technical contributions. Specifically, the dual U-Net architecture and KV replacement are existing techniques. As the authors stated, architectural design is not the main contribution. The interesting bit about this work lies in the training strategy that uses sampled frames from a video and training the model without correspondence information. The authors' rebuttal did clarify these points and provided additional examples of small objects. 

After the rebuttal, the reviewers' ratings remain diverged. The AC agrees that the paper's contribution is not in architecture design and data, but in the training strategy for this new task. As the new task is appropriately motivated and the evaluation is solid, the AC believes that this work would lead to more exploration and improvement on the new proposed image editing formulation. The AC thus recommends to accept.
Decision: Accept (poster)