Zero-shot Image Editing with Reference Imitation

Authors: xi  chen
yutong  feng
mengting  chen
yiyang  wang
shilong  zhang
yu  liu
yujun  shen
hengshuang  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 05:22:44

Abstract: Image editing serves as a practical yet challenging task considering the diverse demands from users, where one of the hardest parts is to precisely describe how the edited image should look like. In this work, we present a new form of editing, termed imitative editing, to help users exercise their creativity more conveniently. Concretely, to edit an image region of interest, users are free to directly draw inspiration from some in-the-wild references (e.g., some relative pictures come across online), without having to cope with the fit between the reference and the source. Such a design requires the system to automatically figure out what to expect from the reference to perform the editing. For this purpose, we propose a generative training framework, dubbed MimicBrush, which randomly selects two frames from a video clip, masks some regions of one frame, and learns to recover the masked regions using the information from the other frame. That way, our model, developed from a diffusion prior, is able to capture the semantic correspondence between separate images in a self-supervised manner. We experimentally show the effectiveness of our method under various test cases as well as its superiority over existing alternatives. We also construct a benchmark to facilitate further research.

Main paper: 
 introduction: 
 Image editing enables various applications for creating novel content, e.g., adding new object(s), modifying attributes, or translating image styles. Recently, powered by the large-scale pre-trained text-to-image diffusion models [33, 30, 36] , the range of capacity for editing models [18, 3, 7, 2, 43, 15, 26, 12 ] also expands significantly.The advanced editing methods could satisfy a large variety of user requirements for modifying either a full image [2, 18] or its local regions [43, 26, 15, 7, 18, 3] . However, it is still challenging for existing editing models to fit the requirements of complicated practical scenarios. For instance, as shown in Fig. 1 , it is required to modify the sole of a shoe by referring to another one, or to paste a specified pattern to a given mug. Such kind of editing is important for real applications like product design, character creation, and special effects, etc.For this kind of local editing, existing works take the source image with a binary mask as input.As shown in Fig. 2 (a), inpainting [43, 52] methods re-generate the masked region following text instructions. However, it is not feasible to describe the desired outcomes only with texts. For example, in Fig. 1 , the design of shoes or the colors of hair is hard to describe accurately in text. Composition methods [7, 38, 37, 53 ] take a reference image as input, along with a mask/box representing the reference area, as shown in Fig. 2 (b). They could insert an "individual object" from the reference image into the source image but struggle to deal with local components (like shoe soles and human hair) or local patterns (like logos and texture). These methods require to carefully extract the reference area from the image. Nevertheless, local components are inherently intertwined with the context and are hard to understand when isolated from the whole object. Besides, their [7, 38, 53] training process requires the mask pairs to indicate the same object in different states (e.g., two video frames).Object-level mask pairs are feasible to obtain, but it is difficult to get the part-level pairs at scale.To deal with the aforementioned requirements, we propose a novel pipeline of editing, termed imitative editing. As illustrated in Fig. 2 (c), given a source image with a masked area for editing, it requires only the reference image without masks. Then, imitative editing targets to fill in the masked area by automatically finding and imitating the corresponding part in the reference image. Such a pipeline formulates more convenient interactions, without strictly separating the reference components from the whole image. Besides, it reaches a harmonious blending referring to the relation between the reference region and its surroundings (e.g., the sole and vape of the shoe).To achieve imitative editing, we design a framework called MimicBrush, with dual diffusion U-Nets to tackle the source and reference images. More specifically, we train it in a self-supervised manner, where we take two frames from a video to simulate the source and reference images. As the video frames contain both semantic correspondence and visual variations, MimicBrush learns to discover the reference region automatically and repaint it into the source image with a natural combination to its surroundings. In MimicBrush, we send the masked source image into an imitative U-Net and the reference image into a reference U-Net, respectively. Then the attention keys and values of the reference U-Net are injected into the imitative U-Net, which assists in completing the masked regions. As demonstrated in Fig. 1 , MimicBrush overcomes the variations between the source and reference images in different poses, lightings, and even categories. The generated region highly preserves the details of the visual concepts in the reference image, and harmoniously interacts with the backgrounds. For a more comprehensive evaluation of the proposed method, we also construct a high-quality benchmark of imitative editing. The benchmark includes two main tasks, i.e., part composition and texture transfer. Each task covers several sub-tracks inspired by practical applications, e.g., fashion and product design.  
related work: 
 Image inpainting. Traditional image inpainting methods [21, 22, 51, 50] only leverage the background information to complete the masked regions. Powered by the text-to-image diffusion models, recent works [43, 30, 2, 15, 26] leverage text prompts to guide the generation of content of the editing regions. These works fully leverage the flexibility of prompts to generate diversified content. A potential limitation is that only using text prompts could not fully express the user's intention for some specific demands.Image customization. To generate the image for the given subject with high fidelity, customization methods optimize a new "word" or use LoRAs to learn specific concepts. However, most of the customization methods [34, 23, 24, 11, 1, 13] tackles the full object. Besides, they require 3-5 exemplar images and rely on subject-specific fine-tuning which last half an hour. Among them, RealFill [41] and CLiC [35] could customize the local region in context. However, RealFill requires 3-5 images for the same scene, and the finetuned model could only complete local regions of the trained scene. CLiC [35] could customize the local patterns to edit different objects, but it only demonstrates inner-category generalizations and still requires subject-specific fine-tuning.Image composition. This topic explores inserting a given object into a specific location of the background with harmonious blending. The early works involved a long pipeline of image segmentation, pasting, and harmonization [39, 9, 4, 14, 10, 5] . Diffusion-based methods propose endto-end solutions and support the pose variations of the reference object. Paint-by-example [47] and ObjectStitch [37] use a CLIP image encoder to extract the representation of the object. AnyDoor [7] uses DINOv2 [27] encoders and collects training samples from videos. Later works [53, 54, 28, 44] add camera parameters or text prompts to increase the controllability. However, they mainly focus on inserting the full-object. Part-level composition poses higher demands for modeling the interaction between the editing region and the surrounding context.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: we include the discussion in the section of conclusion.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overall pipeline: 
 The overall framework of MimicBrush is demonstrated in Fig. 3 . To realize imitative editing, we design an architecture with dual diffusion models and train it in a self-supervised manner.Video data contains naturally consistent content, and also shows visual variations such as different postures of the same dog. Thus, we randomly pick two frames from a video clip as the training samples of MimicBrush. One frame serves as the source image, where we mask out some of its regions. Another frame serves as the reference image, assisting the model to recover of the masked source image. Throughout this way, MimicBrush learns to locate the corresponding visual information (e.g., the dog's face), and repaint it into the masked area in the source image. To ensure the repainted part is harmoniously blended into the source image, MimicBrush also learns to transfer the visual content into the same posture, lighting, and perspective. It is noteworthy that such a training process is built on raw video clips without text or tracking annotations, and can be easily scaled up with abundant videos.MimicBrush leverages a dual branch of U-Nets, i.e., imitative and reference U-Net, taking the source and reference images as input, respectively. The two U-Nets share their keys and values in the attention layers, and are tamed to complete the masked source image by seeking the indications from the reference image. We also exert data augmentation on both images to increase the variation between source and reference image. At the same time, a depth map is extracted from the unmasked source image and then added to the imitative U-Net as an optional condition. In this way, during inference, users could decide whether to enable the depth map of source image to preserve the shape of the objects in the original source image.Our framework majorly consists of an imitative U-Net, a reference U-Net, and a depth model. In this section, we elaborate on the detailed designs of these components.Imitative U-Net. The imitative U-Net is initialized with a stable diffusion-1.5 [33] inpainting model. It takes a tensor with 13 channels as the input. The image latent (4 channels) takes charge of the diffusion procedure from an initial noise to the output latent code step by step. We also concatenate a binary mask (1 channel) to indicate the generation regions and a background latent (4 channels) of the masked source image. In addition, we project the depth map into a 4-channel depth latent to provide shape information. The original U-Net also takes the CLIP [31] text embedding as input via cross-attention. In this work, we replace it with the CLIP image embedding extract from the reference image. Following previous works [7, 49] , we add a trainable projection layer after the image embedding. We do not include this part in Fig. 3 for the simplicity of illustration. During training, all the parameters of the imitative U-Net and the CLIP projection layer are optimizable.Reference U-Net. Recently, a bunch of works [56, 17, 46, 6, 58, 45] prove the effectiveness of leveraging an additional U-Net to extract the fine-grained features from the reference image. For this part, we do not claim novelty and apply a similar design termed reference U-Net. It is initialized from a standard stable diffusion-1.5 [33] . It takes the 4-channel latent of the reference image to extract multi-level features. Following [46] , we inject the reference features into the imitative U-Net in the middle and upper stages by concatenating its keys and values with the imitative U-Net as the following equation. In this way, the imitative U-Net could leverage the content from the reference image to complete the masked regions of the source image.EQUATIONDepth model. We leverage Depth Anything [48] to predict the depth maps of the unmasked source image as a shape control, which enables MimicBrush to conduct texture transfer. We freeze the depth model and add a trainable projector, which projects the predicted depth map (3-channel) to the depth latent (4-channel). During training, we set a probability of 0.5 to drop the input of the depth model as all-zero maps. Thus, the users could take the shape control as an optional choice during inference.To fully unleash the cross-image imitation ability of MimicBrush, we further propose some strategies to mine more suitable training samples. Considering that our goal is to conduct robust imitative editing even cross categories, the philosophy of collecting training data could be summarized as twofold: First, we should guarantee that the correspondence relation exists between the source and reference images. Second, we expect large variations between the source image and the reference image, which is essential for the robustness of finding the visual correspondence. Data selection. During training, we sample two frames from the same video. Following [8] , we use SSIM [42] as an indicator of the similarity between video frames. We discard the frame pairs with too-big or too-small similarities to guarantee that the selected image pair contains both semantic correspondence and visual variations.Data augmentation. To increase the variation between the source image and the reference image, we exert strong data augmentations. Besides applying the aggressive color jitter, rotation, resizing, and flipping, we also implement random projection transformation to simulate the stronger deformation.Masking strategy. A simple baseline is to divide the source image into N × N grid and randomly mask each grid. However, we find this purely random masking tends to cause a large portion of easy cases. For example, as the background (e.g., the grassland, the sky) occupies large areas with repeated content/textures, learning to complete these regions does not require the model to seek guidance from the reference image. To find more discriminative regions, we apply SIFT [25] matching between the source and reference images and get a series of matching points. Although the matching results are not perfect, they are sufficient to assist us in constructing better training samples. Specifically, we increase the masking possibility of the grids with matched feature points.Considering that collecting high-quality images is much easier than videos, we also construct pseudo frames by applying augmentations on the static images and leveraging the object segmentation results for masking the source image. The segmentation masks also improve the robustness of MimicBrush to support masks in more arbitrary shapes.In general, MimicBrush does not rely on the heavy annotations of the training data. It fully benefits from the consistency and variation of video data, and also leverages image data to expand the diversity, which makes the training pipeline more scalable.Imitative editing is a novel task, we construct our own benchmark to systematically evaluate the performance. As shown in Fig. 4 , we divide the applications into two tasks: part composition and texture transfer, and we set the inter-ID and inner-ID track for each task.Part composition estimates the functions of discovering the semantic correspondence between the source and reference image and compositing the local parts. The inter-ID track aims to composite the local parts from different instances or even different categories. We collect data from the following topics: fashion, animal, product, and scenario. For each topic, we manually collect 30 samples from Pexels [29] thus 120 samples in total, where each sample contain the source and reference image pair. We manually draw the source mask to define the composition requirement. As the generated results do not have ground truth, we annotate the reference regions and write text prompts for the expected result. Thus, we could follow DreamBooth [34] to calculate the DINO [27] and CLIP [31] image similarities between the generated region and the annotated reference region. In addition, we also report the CLIP text similarity between the edited image and the text prompts. Firefly [32] takes the detailed prompts descriptions. Besides, we mark the specific reference regions with boxes and masks for Paint-by-Example [47] and AnyDoor [7] . Even though, MimicBrush still demonstrates prominent advantages for both fidelity and harmony.We also set an inner-ID track, where we collect 30 image pairs from DreamBooth [34] , manually mask the discriminative regions of the source image, and use reference images to complete them. The reference would be an image containing the same instance in different scenarios. Thus, the unmasked source image could serve as the ground truth to compute SSIM [42] , PSNR [16] , and LPIPS [57] .Texture transfer requires strictly maintaining the shape of the source objects and only transferring the texture/pattern of the reference image. For this task, we enable the depth map as an additional condition. Different from the part composition that seeks the semantic correspondence, in this task we mask the full objects thus the model could only discover correspondence between the textures (reference) and the shape (source). We also formulate inter-ID and inner-ID tracks. The former involves 30 samples with large deformations from Pexels [29] , like transferring the leopard texture on a cap in Fig. 4 . The latter contains an additional 30 examples from the DreamBooth [34] dataset. We follow the same data formats and evaluation metrics as part composition.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer Justification: Yes, we follow the code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
implementation details: 
 Hyperparameters. In this work, all experiments are conducted with the resolution of 512 × 512.For the images with different aspect ratios, we first pad the images as a square and then resize them to 512 × 512. During training, we use the Adam [19] optimizer and set the learning rate as 1e-5.Experiments are conducted with a total batch size of 64 on 8× A100 GPUs. For the masking strategy of the source image, we randomly choose the grid number N × N from 3 to 10. We set 75% chances to drop the grid with SIFT-matched features and set 50% chances for other regions. We add the reference U-Net as classifier-free guidance and drop it during training with the probability of 10%.During inference, the guidance scale is 5 as default.Training data. We collect 100 k high-resolution videos from open-sourced websites like Pexels [29] .To further expand the diversity of training samples, we use the SAM [20] dataset that contains 10 million images and 1 billion object masks. We construct pseudo frames by applying strong data augmentations on the static images from SAM and leverage the object segmentation results for masking the source image. During training, the sampling portions of the video and SAM data are 70% versus 30% as default. In this section, we compare MimicBrush with other methods that could realize similar functions.Noticing that imitative editing is a novel task, no existing methods could perfectly align our input formats. Thus, we allow additional inputs for other methods. For example, we give additional masks or boxes for AnyDoor [7] and Paint-by-Example [47] to indicate the reference regions. We also pick the state-of-the-art inpainting tool Firefly [32] and feed it with detailed text descriptions.Qualitative comparison. We visualize the qualitative results in Fig. 5 . Although Firefly [32] accurately follows the instructions and generates high-quality results, it is hard for the text prompt to capture the details of the desired outputs, especially for logos or patterns like tattoos. Paintby-example [47] requires a cropped reference image in which the reference regions are centered. However, even if we provide this kind of input, as this model only uses a single token to represent the reference, it cannot guarantee the fidelity between the generated region and the reference regionWe carefully annotate the masks of the reference region and fed them to AnyDoor [7] . It demonstrates stronger abilities for identity preservation but fails to synthesize harmonious blending. We analyze that there are two main reasons: first, some local parts could not be well understood when cropped out of the context. Second, most of the training samples of AnyDoor [7] are full objects. It requires paired mask annotation for the same instance in different video frames for training. The paired masks are feasible to collect for full objects but hardly feasible for local parts. However, MimicBrush gets around this problem by leveraging the model to learn the correspondence itself in the full context instead of using the paired masks. In this way, MimicBrush shows significant superiorities compared with previous methods for completing arbitrary parts with a full reference image.Quantitative comparison. We also report the results on our constructed benchmark for part composition in Tab. 1. For the inner-ID track with ground truth, MimicBrush shows the dominant performance even though we give additional conditions for other works. For inter-ID imitation, it is more challenging to discover the correspondent reference region. MimicBrush could still get competitive performance compared with AnyDoor [7] . We should note that the reference mask is given to AnyDoor. Therefore, it could forcedly locate the reference regions thus taking some advantages for the evaluation. However, as demonstrated in Fig. 5 , it struggles to generate harmonious blending and preserve the fine details.User study. Considering the metrics could not fully reflect human preferences to some extent, we also organized a user study. We let 10 annotators rank the generation results of different models on our benchmark (introduced in Sec. 3.4). We evaluate each sample from three aspects: fidelity, harmony, and quality. Fidelity considers the ability to preserve the identity of the reference region. Harmony estimates whether the generated regions could naturally blend with the background. Quality regards whether the generated regions contain fine details in good quality. Those three aspects are evaluated independently. Results are listed in Tab. 2, MimicBrush earns significantly more preferences. (↑) PSNR (↑) LPIPS (↓) DINO-I (↑) CLIP-I (↑) CLIP-T (↑) SSIM (↑) PSNR (↑) LPIPS (↓) DINO-I (↑) CLIP-I (↑) CLIP-T (↑) • ImageIn this section, we conduct extensive ablations to verify the effectiveness of different components.Referece feature extractor. MimicBrush leverages a dual U-Net structure to model extractor the features from the source and reference image respectively. Some previous works [40, 55] prove that the pre-trained diffusion models contain strong prior to capture semantic correspondence. We explore whether an asymmetric structure could still learn the semantic correspondence under our self-supervised training pipeline. We replace the reference U-Net with the CLIP/DINOv2 image encoder and inject the 16 × 16 patch tokens within the cross-attention layers. Visual comparisons are provided in Fig. 6 , CLIP and DINOv2 also successfully locate the reference region, but U-Net illustrates clear superiorities for preserving the fine details. We also conduct quantitative results in Tab. 3, where we conclude that CLIP and DINOv2 also reach competitive performance. However, as U-Net gives multi-level representations with higher resolutions, and the feature space is naturally aligned with the initiative U-Net, it gives better results when serving as a reference feature extractor.Training strategies. In Tab. 4, We first verify the effectiveness of the "video-based" training pipeline. When using the statistics images only, the performance for each task drops significantly. It shows that object deformation or variation in videos is vital for realizing imitative editing. Afterward, we remove the strong color jitter, resizing, and projection transformation. We observe a clear degradation in part composition, specifically for the inter-ID track. It verifies the importance of augmentation for robust semantic correspondence matching.In the second block, we explore the different masking strategies for the source image and report the performance using a single box and different gid ratios. As introduced in Sec. 3.3, these purely random masking strategies could cause a large number of low-quality training samples. In contrast, we leverage SIFT matching to enhance the masking and reach better performances (bottom row).Figure 7 : Diverse applications supported by MimicBrush. Our methods could be applied conveniently for product design, accessories wearing, editing the scene images, and refining the flawed generation results of other methods. MimicBrush is able to edit multiple regions in one pass.In this section, we give more visual examples and discuss the potential applications. As demonstrated in Fig. 7 , MimicBrush could deal with images from various topics and domains. The first row illustrates the application for product design. The second row shows some examples of jewelrywearing. It should be noticed that the segmentation masks for the necklace are hard to extract, but MimicBrush gets rid of the segmentation step and directly transfers the necklace from the reference image to the source image. In the third row, we show that MimicBrush could also deal with backgrounds also nature effects, proving its strong generalization ability.The last row illustrates a practical application that we could leverage MimicBrush as a post-processing to refine the generation results of other works. In the left example, we improve the fidelity for the image generated by AnyDoor [7] . In the right example, we mark multiple to-edit regions in the source image generated by Cones-2 [24] and provide a concatenated reference image containing both objects. We observe that MimicBrush could refine all the to-edit regions in a single pass. 
conclusion: 
 We present a novel form of image editing with simple interactions, called imitative editing. In our setting, users are only required to mark the editing region on the source image and provide a reference image that contains the desired visual elements. MimicBrush automatically finds the corresponding reference region to complete the source image. To achieve imitative editing, we take full advantage of the consistency and variation of videos and design a self-supervised training pipeline that uses one frame to complete the masked regions of another frame. MimicBrush demonstrates impressive performances for various editing tasks and supports a wide range of applications. To facilitate future explorations, we construct a benchmark to comprehensively evaluate imitative editing. This work is expected to bring new inspiration for the community to explore more advanced techniques for image generation and editing.Limitations & potential impacts. MimicBrush demonstrates robust performance. However, it could fail to locate the reference region when the region is too small or multiple candidates exist in the reference image. In this case, users should crop the reference image to zoom in on the desired regions. MimicBrush could deal with a wide range of images, thus making it possible to produce some content with negative impacts. Therefore, we would add censors to filter out the harmful content when we release the code and demo. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We discuss it in the section of the conclusion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: Our methods are not so risky, and we would add censor when we release the demo.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, we checked the license.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
model: 
 To edit a local region, besides taking the source image and source mask (indicates the to-edit region), inpainting models use text prompts to guide the generation. Image composition methods take a reference image along with a mask/box to crop out the specific reference region. Differently, our pipeline simply takes a reference image, the reference regions are automatically discovered by the model itself. 
 
 Appendix: In this section, we provide detailed descriptions of our constructed benchmark and the proposed metrics. We include examples of part composition and texture transfer, and we visualize the annotation for each track in Fig. A1 . For the inter-ID tracks, we present a source image, source mask, reference image, reference mask, and a text prompt to specify the expected results. The reference mask and text prompts are used solely for evaluation, where we calculate DINO and CLIP image similarities, as well as CLIP text similarity. Specifically, to compute the DINO and CLIP image similarities, we first remove the background of both the reference image and the generated results using the reference masks and source masks. Then, we calculate a minimum bounding box for the two masks, cropping both the generated and reference regions accordingly. The DINO and CLIP similarities are calculated between these two cropped regions with clean backgrounds.For inner-ID tracks, since the source image can be considered as the ground truth, we omit annotating prompts for CLIP test scores. Instead, we compute the SSIM, PSNR, and LPIPS metrics. Similar to the cropping operation used before calculating the DINO and CLIP image similarities, we also utilize the masks to obtain the bounding box around the generated region. This box is then employed to crop the corresponding regions from both the generated result and the source image for computing the SSIM, PSNR, and LPIPS. Although the reference masks are not utilized for model input or metric computation, we still annotate the masks for each sample. This enables evaluation for other methods like Paint-by-Example [47] and AnyDoor [7] .A little boy is wearing a #red scarf#. As outlined in the main paper, we conducted a user study involving 10 annotators to assess the generation results of various methods. We devised a user interface that presents the source image, source mask, reference image, and reference mask on the left side to clarify the expected editing outcomes. On the right side, the user interface displays the four results predicted by four different methods, each marked with an index. Annotators were tasked with answering four single-choice questions regarding the best, second-best, third-best, and fourth-best results index. To enhance the reliability of the user study results, we implemented several measures:First, we provided comprehensive documentation and rating examples for each task. Annotators were initially tasked with rating a small subset of data. Once their annotations passed our examination, they continued with the remaining samples.Second, to prevent annotators from distinguishing the results of different methods based on additional information, we standardized the image size of each result and shuffled the order of the four results randomly for each sample. We recorded the shuffled indexes to maintain consistency.Third, as human evaluation was required from three different aspects (fidelity, harmony, quality), these aspects were evaluated as three separate annotation tasks to ensure independence and accuracy.In this section, we provide additional analysis and qualitative ablation results for the training strategies. We follow the experimental settings introduced in Tab. 4 of the main paper. As depicted in Fig. A2 , when only utilizing statistical images for training, the first two rows indicate that the model is still capable of learning semantic correspondence. However, it struggles to adapt to pose and lighting variations, resulting in rigid blending. This underscores the importance of incorporating visual variations from video data for our method to conduct robust and generalized imitative editing. We also showcase generation results using weak data augmentation and a naive masking strategy (i.e., masking the random grid with 50% probability). Both approaches exhibit clear performance degradation compared to our full version of MimicBrush, which employs strong data augmentation and a matching-guided masking strategy.This section presents a comprehensive array of examples highlighting the exceptional capabilities of our methodology in diverse contexts.Fig. A3 showcases results from inner-ID reconstruction where a specified region on the initial image is repainted using another image of the same instance as a reference. These outcomes illustrate our method's precision in accurately identifying the target region and retaining intricate details with high fidelity.In Fig. A4 , further examples of part composition are provided. The first row demonstrates a seamless editing process, which facilitates the combination of multiple visual elements in a cohesive manner. The next two rows exhibit variations across assorted topics and domains, showcasing our method's robust adaptability.In Fig. A5 , we present instances of texture transfer applications. Here, we utilize the depth map to maintain the original structure of the source image while artfully transferring texture from the reference image. This technique facilitates a myriad of imaginative designs, allowing creators to infuse elements with new aesthetic qualities while respecting the underlying form. Such versatility not only provides a tool for artists to expand their creative palette but also offers practical solutions for industries requiring rapid visualization of texture variations without altering the product's shape. Consequently, our method stands as a testament to the convergence of creativity and technical innovation, opening new avenues for design exploration. Justification: The abstract outlines the overall content and our contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper presents a new form of image editing called imitative editing. The review comments are mixed. Overall, the reviewers appreciate the formulation of the novel task and the proposed evaluation for this task. However, they have different opinions on the technical contributions. Specifically, the dual U-Net architecture and KV replacement are existing techniques. As the authors stated, architectural design is not the main contribution. The interesting bit about this work lies in the training strategy that uses sampled frames from a video and training the model without correspondence information. The authors' rebuttal did clarify these points and provided additional examples of small objects. 

After the rebuttal, the reviewers' ratings remain diverged. The AC agrees that the paper's contribution is not in architecture design and data, but in the training strategy for this new task. As the new task is appropriately motivated and the evaluation is solid, the AC believes that this work would lead to more exploration and improvement on the new proposed image editing formulation. The AC thus recommends to accept.

Reviews: review: paper_summary: paper_summary: The paper proposes a new form of image editing, termed imitative editing. In this scenario, the user can edit the local area of the source image with a similar area to the reference image. This requires the system to automatically figure out what to expect from the reference to perform the editing. To achieve this goal, the author first designed a new generative network, dubbed MimicBrush. Then, the paper also constructs the benchmark from the video clip. The paper demonstrates both qualitative and quantitative results to validate the effectiveness.
strength_weakness: strength_weakness: # Motivation & Novelty
**Unclear motivation and advantages**: The core learning algorithm of the paper is an imitative U-Net, a reference U-Net, and a depth model. However, the advantages and motivations of such designs are not clearly discussed and validated. For example, some other networks such as Controlnet and IP-adapter, can provide conditional information from the reference image. Why is injecting features from the imitative U-Net better on the proposed imitative editing task? 

**Unclear novelties in terms of learning compared with existing methods**: The proposed imitative task is new in that the system can edit part of the object whereas the latest methods mainly insert full objects. However, the novelty and difference seem to only lie in the training data (full object v.s. part object). The learning paradigm seems to be the same. Can these existing works do well in the proposed imitative editing task if trained with the collected part-level data? The essential difference **in terms of learning and formulation** between part-level editing and full-object editing is not addressed. My understanding is that they are essentially the same learning paradigm but in different concrete forms.


# Formulation & Writing
**Unclear notations**: 
1. The paper proposes a new task and a new framework. However, the problem and algorithm are not rigorously formulated. For example, what are the input, output, and masks **mathematically**? 

2. What is the training objective (loss function) in the pipeline? What is the loss used in the depth model?

3. For concatenation and injection of K, V from the reference U-Net, which layers are injected? Are cross-attention and self-attention processed in the same way?
 

# Experiments & Evidence
**Limited editing types**: The demonstrated editing types are limited. 
1. For example, for the same source image, can the user edit different parts of the object with different reference images?
2. Does the proposed method support general image editing such as changing objects (cat to dog), or shape (round cake to square cake)?

**Unclear motivation on the SAM data**: The paper mentions using the SAM data in training except for the Video clips. How does the SAM data help the learning in this task? I think after the strong augmentation, the spatial position of different objects does not change. So, how does this help to learn the correspondence between the source image and reference image?
questions: questions: Please refer to the weakness part.

My main concerns are about the depth of discussion and motivation of the proposed network and task, and the writing and formulation that are not serious enough to reach the criterion of Neurips.
limitations: limitations: The author already addresses the limitations of the work.
paper_summary: paper_summary: The paper introduces a novel approach called imitative editing aimed at enhancing user creativity in image editing tasks. Traditional image editing often involves matching references to the source image, which can be challenging. In contrast, imitative editing allows users to directly draw inspiration from in-the-wild references without needing to precisely align them with the source image. This paper introduced a new approach called MimicBrush, a generative training framework that leverages video frame pairs to learn semantic correspondence and recover masked regions in images.
strength_weakness: strength_weakness: --Regarding part composition and texture transfer, if I understand correctly, the proposed framework appears capable of simultaneously addressing both tasks. However, based on the training procedure, I am curious about how the model distinguishes between referencing parts and textures. This differentiation does not seem explicitly addressed in the training process, potentially leading to the model misinterpreting user intentions during the inference stage.

--Regarding ablation experiment, this paper present the training strategies about training data, augmentation and masking method. It would be better to also ablate the necessity of image training data. And for all the ablations, it would better to have qualitative comparisons to illustrate how such training strategy affect the visual quality.
questions: questions: please see above weaknesses. Overall I think this paper is trying to address a new and interesting editing scenario, which will make image editing more convenient.
limitations: limitations: The authors have adequately addressed the limitations.
paper_summary: paper_summary: To achieve more precise imitative editing, the paper proposed a training framework, called MimicBrush. It randomly selects two frames from a video clip, masks some regions of one frame, and learn to recover the masked regions using the information from the other frame. Also, it constructs a benchmark, consisting two main tasks, i.e., part composition and texture transfer, to evaluate the performance comprehensively.
strength_weakness: strength_weakness: The definition of inter vs. inner is unclear.
questions: questions: 1. How is rank calculated in table 2?
2. Regarding line #195, does it mean that when the randomly selected grid number is 4, it would drop 4x75%=3 for SIFT-matched features and 4x50%=2 for other regions? So, in total, it would drop 5 grids?
3. What does the gid ratio mean in line #261?
4. The paper structure can be improved. Sec. 4.4 and Sec. 4.2 contain the “Qualitative comparison”. I suggest having only one Qualitative comparison section and separating it into subsections: “comparing with others” and “diverse applications without comparing with others”.
5. The mask representation is hard to identify, e.g., the right case in the first row or the left case in the third row in Fig. 7. I suggest to change a clearer representation.
limitations: limitations: The authors have shared the limitations and potential societal impact.
paper_summary: paper_summary: The paper proposes a method to modify specific parts of the image with the content from the reference image and consistently with the original content. It combines the technique of dual diffusion to replace the key and value features, which is used in MasaCtrl, of the source image with the reference image. The method is trained in a self-supervised manner, in which some parts of the image are masked, and the model is trained to recover them.
strength_weakness: strength_weakness: - The proposed method is not actually zero-shot as it requires training. It requires expensive resources for the training model.
- The method presented in this paper lacks novelty. It uses the old technique (i.e., replacing the key, value feature from MasaCtrl [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing, ICCV 2023]) and combines with an old training strategy that recovers the region that is specific by mask.
- The source and reference images must be the same scale as the object, and the source image must contain only one or a salient object.
questions: questions: What if the source image is a complicated image that contains some objects and the desired modified object is not a salient object?
limitations: limitations: The paper is not so novel; it reuses components from existing techniques. The proposed method is not actually zero-shot as it requires training.

Comments: title: Response to the rebuttal
comment: Dear Reviewer, 

Thank you for your response. However, we **disagree** with your comments, and **it seems that our rebuttal is ignored**. This comment does not involve the new information that we claimed in the rebutall.

``I acknowledged the authors' insights on "the Reference U-Net preserves details better, as it provides feature maps with higher resolution" in rebuttal. However, this argument is generally beneficial for all image tasks and does not sufficiently address how it facilitates the proposed task.``

We have responded in the rebuttal that:  

"We have compared different strategies in Tab.3 and Fig.6. "CLIP" represents the strategy of IP-Adapter, and "DINOv2" represents the strategy of AnyDoor. We observed that the Reference U-Net preserves details better, as it provides feature maps with higher resolution."

In this case, we conducted detailed ablation studies to explore the structure and gave both qualitative and quantitative analyses in the paper.  Considering this structure is **not our contribution**, we believe this experiment is already sufficient, and the advantage of this structure is exactly verified "in the proposed task".  However, 
 our claims around this experiment seem to be ignored,  so we disagree with this comment.

`` So, the paper provides me with an impression of using existing (or sort of new) techniques and strategies with new data to build an application`` 

We have repeatedly claimed in the rebuttal that **the dual U-Net is not our contribution or novelty** and carefully explained **the major differences are beyond the data**.  However, these explanations seem to be ignored, and the new comments still focus on **the novelty of the Reference U-Net and training data**. It seems that the misunderstanding still strongly exists. 

To make our contribution clearer, we would add a small paragraph at the end of the introduction section to summarize our contributions and novelties. In addition, we would add an explanation paragraph in Section 3.2 to claim the dual U-Net is not the contribution of this paper in the revised version.


We appreciate your comments and please carefully read our responses and the initial rebuttal to see if these concerns have been addressed. We would be happy to get your suggestions to polish our paper. Thank you.
Dear authors,

I have read the rebuttal and other reviewer's comments. Thanks for the efforts in the experiments and clarification!

My comments mainly proceed from the novelties and contributions in learning, i.e., theoretical understanding and derivation, OR the validated design of approaches and insights. 

Thus, the main components (Dual-Unet, KV replacement) are existing techniques (as also stated in RWq758). I acknowledged the authors' insights on "the Reference U-Net preserves details better, as it provides feature maps with higher resolution" in rebuttal. However, this argument is generally beneficial for all image tasks and does not sufficiently address how it facilitates the proposed task. So, the paper provides me with an impression of using existing (or sort of new) techniques and strategies with new data to build an application. Since this is not a theory paper, I think the novelties in the design of approaches can be limited though I acknowledged the interesting editing effects added in the rebuttal.

Besides, considering the overall formulation of the current version of the paper, I choose to maintain my score.
Dear Reviewer, 

Thank you for your acknowledgment. 

``the way your contribution is presented in the paper confused me``

Thank you for pointing it out. To make our contribution clearer, we would add a small paragraph at the end of the introduction section to summarize our contributions and novelties. In addition, we would add an explanation paragraph in Section 3.2 to claim the dual U-Net is not the contribution of this paper in the revised version. 

`` It is only novel in the context of user experiments.``

First, image editing is an application-oriented task, and we formulate a more convenient editing form without the reference mask. We think "this novelty in user experience" is already quite important, and could make great contributions for the image editing community. 

Besides, this "novelty for users" could not be simply archived by applying existing techniques, it also requires the support of our novel training strategies.  Our video-based training strategy is specifically designed for learning semantic correspondence without reference masks, which is important for realizing our "imitative editing".

``p/s: I haven't received your examples of small objects.``

We have already sent the link to ACs,  maybe they have not read the message. As the discussion period is going to end we will not be able to make further responses. Please remember to ask ACs for the results if they forgot to send you the link.

By the way, we believe that the examples in the second row of the rebuttal PDF like "candles" and "socks" are already small. Considering that the model receives a "source mask" to indicate the to-edit region, the difficulties for the model would be similar even for more complicated scenarios. 

In addition, we appreciate your constructive discussions and we would make our paper clearer according to your suggestions.
Dear Authors,

Thank you for your further explanations.
My concerns are mostly addressed, but it is still not so convincing.
I can only increase my rating to Borderline Accept.

I already know that the dual U-Net design is not your novelty/contribution. One thing I want to point out is that the way your contribution is presented in the paper confused me. I don't think your method is actually novel; it is only novel in the context of user experiments.

p/s: I haven't received your examples of small objects.
Dear Reviewer, 

We have added examples for even smaller objects and given the link to ACs, you may receive this link from ACs later. Here we describe the examples:

- A camera shot filled with large numbers of fruits, where some fruits are partially obscured by others. We edited one of the fruits that was partially obscured.

- A complex rural scene with more than ten people and ten dogs or sheep, where there is significant occlusion between the people and animals. We replaced one of the heavily obscured dogs with a tiger.

- A windowsill filled with plants (more than 10), with some pots partially obscuring others. We edited one of the pots according to the provided reference image.

Please note that editing very small areas is only a corner case with limited practical usage in real-world applications and is unrelated to the main focus of our paper. However, we have still provided the requested samples. We hope these examples fully address your concerns. If you have any new questions, feel free to reach out to us at any time.
Dear Reviewer iASj,

As the deadline for the discussion phase approaches, we wanted to kindly remind you to read our rebuttal. We highly value your feedback and are eager to incorporate your suggestions to improve our work.

If there are any aspects that require further clarification or additional information, we would be more than happy to engage in continued discussion and provide any necessary details.
Thank you for your response. I believe that the contribution and novelty are still misunderstood.

`` it seems that you are proposing a novel architecture (Imitative U-Net)``

We claim multiple times that **the dual U-Net design is not our novelty/contribution, instead, it is a common practice**. We do not claim contribution. As mentioned in line 121, this structure is widely used in many tasks that require a reference image [56, 17, 46, 6, 58, 45]. These works verified the advantages of "dual U-Net" over previous works like IP-adapter or controlnet. We leverage this type of design solely as our feature injection module and do not claim it as our novelty.

`` However, I still maintain that the overall method may not be entirely novel. ``

We are the **only work** that could realize local region editing with a reference image ***even without providing the reference mask***, for this motivation, please refer to "B. Motivation of getting rid of the reference mask" in the global rebuttal.
In addition, we are the **only work** that trains the model using one frame to complete another frame to learn the correspondence. As **the whole task and training strategy** are completely novel, we believe the novelty is enough. Again, we would like to reaffirm our core novelty below.

**Novel form for image editing.** Unlike the commonly studied task of "object insertion", our proposed "imitative editing" offers a new editing form, which does not require users to specify the region of interest from the reference image. Such a design sufficiently simplifies the editing process, because users only need to brush once (i.e., the source image) instead of twice (i.e., both the source image and the reference image), and hence clearly distinguishes our approach from existing methods. It is also noteworthy that getting rid of the reference mask is essential for part-level editing, making our algorithm more flexible in practice.

**Novel training strategy.** To accomplish "imitative editing", it requires the model to automatically analyze the semantic correspondence between the to-edit region of the source image and the "full" reference image. For such a purpose, our approach benefits more from the training strategy but not the data. Concretely, we propose to leverage the temporal consistency and variation across video frames, and urge the model to recover the masked region of one video frame by only using the information from another frame without specifying the corresponding region. To our knowledge, our proposed training pipeline offers a brand new way of using video data for diffusion-based editing.

**Benchmark for the novel task**. We construct a diversified benchmark that contains multiple tracks and covers a large variety of scenarios. Besides, we come up with various metrics to thoroughly evaluate the compare different methods. Our benchmark could be beneficial for further explorations for imitative editing.

``Could you test a more complex example that includes editing smaller objects within the image?``

Please notice that, in the second row of Fig.R1 of the rebuttal PDF, we have added examples for editing "the candle", "the hat", and even "the socks of a full-body human".  **We have already shown these small things that occupy less than 5% pixels of the source images**. 

We kindly remind you that, as the "to-edit" region could be specified by users with a "source mask". In this way, **there would not be any challenges** in distinguishing the "to-edit" region even if the environment is complicated with distracters or this region is small. 

According to the rebuttal policy, the PDF files could not be updated. We would contact ACs to see if we could give you more examples.


**Please check if this response addresses your concerns, if you have further questions, we are happy to discuss them.**
Dear Authors,

Thank you for your detailed rebuttal.

First, I would like to acknowledge that I agree with your definition of the zero-shot method.

Second, while I appreciate the detailed explanation of your contributions and their novelty, I believe that the novelty is primarily focused on user experiments related to image editing tasks. From your presentation in the paper, it seems that you are proposing a novel architecture (Imitative U-Net). However, I still maintain that the overall method may not be entirely novel.

Finally, the examples provided in Fig. R1 do not fully convince me. Could you test a more complex example that includes editing smaller objects within the image? For instance, an image with a large number of overlapping objects or objects that do not interact with each other would be more compelling.  

In conclusion, I would like to keep my rating as 'BorderLine Reject.’
Thanks for the detailed response. The rebuttal addressed my concerns.
Dear Reviewer q758,

As the deadline for the discussion phase approaches, we wanted to kindly remind you to read our rebuttal. We highly value your feedback and are eager to incorporate your suggestions to improve our work.

If there are any aspects that require further clarification or additional information, we would be more than happy to engage in continued discussion and provide any necessary details.
Dear Reviewer,

We kindly remind you about our submitted rebuttal. We understand that your time is valuable, and we greatly appreciate the effort you put into reviewing our work. If you have any questions or require further clarification on any points, please don't hesitate to reach out. We look forward to your feedback and hope for a favorable consideration.
Thank you very much for your recognition of our work. We greatly value the feedback you provided, and we will follow the discussion to make this paper clearer and better.
Dear Reviewer,

We would like to gently remind you of our submitted rebuttal. We understand the demands on your time and sincerely appreciate the effort you invest in reviewing our work. If there are any questions or if you need further clarification on any points, please feel free to reach out. We look forward to your feedback and hope for a positive outcome.

Thank you for your time and consideration.
Thank the author for providing further information and contributing great work to the image editing community! I have read other reviewers' comments and the rebuttal information. I would like to remain my rating as 'Accept.’
Dear Reviewer,

We kindly remind you about our submitted rebuttal. We understand that your time is valuable, and we greatly appreciate the effort you put into reviewing our work. If you have any questions or require further clarification on any points, please don't hesitate to reach out. We look forward to your feedback and hope for a favorable consideration.