MaGIC: Multi-modality Guided Image Completion
Authors: Hao Wang
Yongsheng Yu
Tiejian Luo
Heng Fan
Libo Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Multi-modality
Image Completion
Diffusion Model
Primary area: generative models

Abstract: Vanilla image completion approaches exhibit sensitivity to large missing regions, attributed to the limited availability of reference information for plausible generation. To mitigate this, existing methods incorporate the extra cue as guidance for image completion. Despite improvements, these approaches are often restricted to employing a *single modality* (e.g., *segmentation* or *sketch* maps), which lacks scalability in leveraging multi-modality for more plausible completion.
In this paper, we propose a novel, simple yet effective method for **M**ulti-mod**a**l **G**uided **I**mage **C**ompletion, dubbed **MaGIC**, which not only supports a wide range of single modality as the guidance (e.g., *text*, *canny edge*, *sketch*, *segmentation*, *depth*, and *pose*), but also adapts to arbitrarily customized combinations of these modalities (i.e., *arbitrary multi-modality*) for image completion.
For building MaGIC, we first introduce a modality-specific conditional U-Net (MCU-Net) that injects single-modal signal into a U-Net denoiser for single-modal guided image completion. Then, we devise a consistent modality blending (CMB) method to leverage modality signals encoded in multiple learned MCU-Nets through gradient guidance in latent space. Our CMB is *training-free*, thereby avoiding the cumbersome joint re-training of different modalities, which is the secret of MaGIC to achieve exceptional flexibility in accommodating new modalities for completion.
Experiments show the superiority of MaGIC over state-of-the-art methods and its generalization to various completion tasks.

Published as a conference paper at ICLR 2024
MAGIC: MULTI-MODALITY GUIDED IMAGE COMPLETION
Hao Wang 1.2-
Yongsheng Yu Tiejian Luo'
Heng Fan*
Libo Zhang
'ccooo of Computer Science and Technology, University of Chinese Academy of Sciences Institute of Software, Chinese Academy of Sciences
'Department of Computer Science, University of Rochester
*Department of Computer Science and Engineering, University of North Texas wanghao 1 844mails. ucas wcaasaccccco cn; yongsheng yu@rochester edu; tjluoêucas. ac. heng. fan@unt. edu; libo@iscas.ac.oo
ABSTRACT
Vanilla image completion approaches exhibit sensitivity to large missing regions, attributed to the limited availability of reference information for plausible genera- tion. To mitigate this, existing methods incorporate the extra image completion. Despite improvements, these approaches are often restricted employing a single modality (e.g., segmentation sketch maps), which lacks scalability in leveraging multi-modality for more plausible completion. In this paper, we propose novel, simple yet effective method for Multi-modal Guided Image Completion, dubbed MaGIC, which not only supports wide range of single modality the guidance (e.g., text, canny edge, sketch, segmentation, depth, and pose), but also adapts to arbitrarily customized combination of these modal- ities (i.e., arbitrary multi-modalitt) for image completion. For building MaGIC, we first introduce modality-speciff conditional U-Net (MCU-Net) that injects single-modal signals into U-Net denoiser for single-modal guided image comple- tion. Then, we devise consistent modality blending (CMB) method to leverage modality signals encoded in multiple learned MCU-Nets through gradient guidance in latent space. Our CMB is training-free thereby avoiding the cumbersome joint re-training of different modalities, which is the secret of MaGIC to achieve excep- tional flexibility in accommodating show the superiority of MaGIC over state-of-the-ar methods and its generalization various completion tasks.
guidance for
modalities for completion. Experiments
Trompl "Show mosurfain: the hanutiful lake."
Large-scar scale image completion
?
Input images
Tbe
Expple
Serch (Seribbe)
Canyy Edy
Pronppé "a thomls up buy and smmllin 22 TT""
Real- woal-world image editing
Figure I: Illustaation of our MaGIC for image completion tasks including outpainting (first row) and real user-input editing (second row) under multi-modalin guidance.
'Hao Wang and Yongsheng Yu contributed equally to this work.
*Corresponding author: I Libo Zhang. This work was supported by Youth Innovation Promotion Association,
CAS(2020111).

---

Published as a conference paper at ICLR 2024
1 INTRODUCTION
Lugmayr et al., (022). involving the concealment of a portion of an image and prompting a model to imaginatively restore it, has long been a subject of extensive research with many applications, such as object removal (Suvorov et al.
Image completion (Criminisi et al. 2003 Li ct al., 2022;
2022 Criminisi et al., 2003), image compositing (Levin et al. 2004), photo restoration (Wan et al. 2020). etc. Typical image completion approaches (Li et al., 2022: Suvorov et al. 20022) are prone to struggle with complex or large masking regions due to inadequate reference information. This limitation causes ambiguity to the completion model over restoration elimination and leads to noticeable artifacts in completed images, degrading the quality.
An intuitive solution to overcome the above limitation is to incorporate user-input (Horita et al. 2022 Yu et al. 2019. Zheng et al. 2022a) or prediction- based (Nazeri et al. 2019
Yu et al. 2022 2022
Guo al. 2021, Dong a (022) guidance, e.g., text (Avrahami al. 2023 Xie et al.
Nichol et al. 2022 Wang et al. (023)). edge (Nazeri et al.) 2019 Guo et al. 2021 Yu et al. 2022).
or segmentation (Yu al. 2022; Liao al. 2020 Zheng et al. 2022b), into image completion.
However, these approaches are limited to performing image completion under only single-modality
guidance, which inflexible in employing the multi-modality, especially more than two modalities simultaneously. for plausible generation and leads to limited application scenarios.
Recently, denoising diffusion probabilistic model (Ho et al. 2020) has been widely employed and demonstrated superior performances in exxt-to-image synthesis (Rombach et al.
2022
Gu et al.
2022. Ramesh et al.] 2022) and text-driven image manipulation fields (Kim et al. 2022] Avrahami
et al. 2022, Kawar et 2023). In addition to lext, many approaches (Bansal et al. [023] Yu et al.
2023 Chen et al., 2023 Avrahami et al., 2022) have explored the integration of extra guidance
modality, such segmentation, sketch, pose, and even position of generated object, into diffusion models in a training-free way. These methods involve designing energy loss associated with the input guidance and guiding its gradient on the latent codes during inference, yet they tend to fail to maintain fine-grained structural information, resulting in insufficient control over the generated results. Meanwhile, several training- ceiiing--rrrrrddde approaches (Mou et al.. 2023 Zhang & Agrawala 2023) have further enhanced the control of input modality over diffusion models by introducing an auxiliary conditional network encode modality and directly add the encoded features to the intermediate features of frozen diffusion models. These methods bring in fresh insights and pave the way for incorporating guidance signals into image completion. Nevertheless, simply transferring these ideas to multi-modalit image completion is not trivial as the introduction of each new modality necessitates the joint training of all auxiliary conditional networks. How to effectively integrate multi-modality guidance for image completion in a scalable and flexible manner remains an open problem.
In this paper, propose MaGIC, a novel, simple yet effective framework for Multi-modality Guided Image Completion, especially when there are more than two modalities at the same time. MaGIC is designed to be scalable and flexible, allowing it to merge various modalities, including but not limited to text, canny edge, sketch, segmentation, depth, and pose, in an arbitrary combination as guidance for image completion (see Fig. F and Fig. 2. To build MaGIC, there are two core ingredients, including a modality-specifh conditional U-Net (MCU-Net) and a consistent modality blending (CMB) method, performed in two stages.
Specifically, the proposed MCU-Net, composed of a standard U-Net denoiser from the pre-trained stable diffusion (Rombach et al. 2022) and a simple encoding network, which injects single modality guidance signal into the U-Net denoiser to attain single-modal guided completion. The MCU-Net will be individually finetuned under each single modality, in the first stage. Then, to achieve multi- modtitmo guidance, the CMB algorithm is proposed in the second stage to flexibly aggregate guidance signals from any combination of previously learned MCU-Nets. The CMB leverages guidance loss to gradually narrow the distances between the intermediate features from the original pretrained U-Net denoiser and multiple MCU-Nets during the denoising sample stage, which ensures that the former features do not deviate too much from the original feature distribution during multi-modality guidance. Compared with the naive approach of achieving multi-modaliity guided completion by jointly re-training unified model, CMB is training -free and allows for the flexible addition or removal of guidance modalities, avoiding cumbersome re- re-training and preserving the feature distribution of the original U-Net denoiser.
2

---

Published as a conference paper at ICLR 2024
pandaa lighting with Coodiny
lore
Zago supermon
Aleiie glacie Iachgmond Solfa gloure the lett Wak prrlly day the right
Multi-modaliiy guidunce
Ground truth
Output
Figure 2: Illustration of our MaGIC for real user-input editing task using various combination of multi multi-modality guidance.
To verify the proposed MaGIC, conduct extensive experiments various tasks including image inpainting, outpainting, and real user-input editing, using the COCO (Lin et al. 2014), Places2 (Zhou et al. 2018), and in-the-wild data. Our results demonstrate the superiority of MaGIC over image completion and controllable generation baselines in terms of image quality. In addition, we find that, surprisingly, the CMB of MaGIC is also well applicable for multi-modality guided image generation, showing its generality and potential for generative tasks. Fig. 3 illlustrates the architecture of our approach.
In summary, our contributions are four-fold: (i) we propose a novel approach of MaGIC for flexible and scalable multi-modality guided image completion. To the best of our knowledge, MaGIC the first to widely support arbitrary multi-modality guided image completion; (ii) we present simple yet effective MCU-Net to effectively and adaptively inject a modality as guidance for image completion; (iii) we introduce novel CMB algorithm that combines arbitrary multiple modalities for image completion without need for additional training and (iv) using MaGIC, we achieve performance superior to that of other state-of-the-ee approaches.
RELATED WORK
Auxiliary-based image completion. The moxiliary-based image completion methods aim to enhance the structure and texture of completed images by incorporating predicted human-provided prior information. Early approaches primarily focus on using single modality (e.g., edge (Nazeri et al.
2019; Guo et al. 2021] Dong al. 2022 Zheng al. 2022a) or segmentation (Zheng et 2022b]
Liao et al. 2020 as the auxiliary guidance for imge completion. Recently, inspired by the superior- performing diffusion models (Ho al., 2020 Dhariwal & Nichol 2021 Rombach et al. 2022).
text-based auxiliary solutions have been proposed for image completion (Wang et al. 2023 Avrahami
ct al. 2023, Nichol et al. 2022 Avrahami et al. 2022). providing more user-friendly image editing applications.
However, prompt text alone not sufficient. Because the above methods are constrained by the training requirements of auxiliary guidance, it is difficult to flexibly add more types of modalities as guidance for completion. Our MaGIC can incorporate arbitrary combination of multiple modalities for more plausible completion results (see Fig. [ of single-modality conditional networks, and allows for plug-and-pla integration into the conditional image completion process without the need for additional cumbersome joint re-training.
again). is versatile, requiring only the optimization
3

---

Published as a conference paper at ICLR 2024
Controllable image generation with diffusion models. Diffusion models (Ho et al. 2020 Dhariwal & Nichol, 202I Rombach et al. 2022; Song et al. 2023) have drawn extensive attention in image
generation owing to their remarkable results and stable training. These methods can be broadly categorized into vuin-required and train-free approaches. The former achieves powerful generation control by training large-scale data fine-tuning a conditional control sub-network on pre-trained diffusion models (e.g., (Rombach et al., 2022)). Recent research (Zhang & Agrawala [023] Mou ct al, 2023) has introduced various modalities (e.g., keypose point maps, sketch maps, etc) for generation. However, it fails to simultaneously use multi- modality as guidance. Differently, train-free
solutions YYu et al.) 2023] Chen et al., [023 Bansal al. [023 Jeong et al., 2023) leverage the
multi-step nature of diffusion models, explicitly introducing guidance signals during the iterative
denoising process and achieving style (Jeong al. (023), layout (Chen et al. 2023; Bansal et al.
2023), face identity (Bansal et al., 2023] Yu et al. 2023). segmentation map (Bansal et al. 2023
Yu et al. 2023) guidance without fine-tuning. Yet, they strugele to leverage fine-grained structural guidance (e.g., canny edge) as conditions, potentially resulting in degraded guidance (Yu et al. 2023).
Our MaGIC is inspired by the above image generation approaches, but different two aspects. First, MaGIC achieves multi-modality guidance without joint re- re-training while improving the effectiveness of fine tne-grained structure guidance. In addition, MaGIC goes beyond controllable generation and can be applied to guided completion and real-world editing tasks.
3 MAGIC: MULTI-MODALITY GUIDED IMAGE COMPLETION
Masked images Xm X O are obtained by corrupting images x with binary masks m E (0, Hx where x E R #xwx3 are original RGB images with width W and height II. Given a known region Xm = x m), the goal of image completion is to learn function p(xmlAm)) that completes the missing mask area with visually realistic and structurally coherent content. To mitigate the inherent ambiguity of completion model, the direction of restoration climination is controlled through the auxiliary guidance C. In the following sections, we start by outlining necessary diffusion steps in 3.1 for formulating our method, then elaborate MaGIC, addressing auxiliary guidance via proposed MCU-Net in 3.2 and multi-modality integration by our CMB algorithm in 3.3
3.1 PRELIMINARIES
Diffusion models. Denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) are generative models that learn the true distribution p(x)) by iteratively denoising a randomly sampled noise image Xy. In each denoising step, U-Net model is trained to predict the noise based on the objective function,
(x,, 0) = min(En,.e-Noonlll (g(x))))).
where X; = v0,xo + VI E represents the intermediate noised image obtained after applying noise I times to the clean image xp, and a, = IT',, (1 - B,)) is series of fixed hyperparameters based on the variance schedule B,, s E [1,7]] 77. The model can be further generalized to conditional generation
DDariwal & Nichol 2021 Ho & Salimans 2021 with predicted noise becoming cg(x, f, C).
Stable diffusion. We consider stable diffusion (SD) inpainting model (Rombach et al. 2022) as the main backbone in the subsequent method sections. Instead of beginning with isotropic Gaussian noise samples in pixel space, the SD model first maps clean images to their corresponding latent space Z through E(.). Here, E(-) is an autoencoder with a left inverse D, ensuring x D o E(x). Owing to the lower inference overhead of U-Net in the latent space, SD has emerged as important class of recent image generators based on diffusion (Rombach et al., 2022] Saharia et al. 2022 Zhang & Agrawala 2023 Avrahami et al., 2023). Specifically, the initial latent codes of iterative denoising process employ random žσ Z E R xxa where signifies s-fold reduction in spatial dimensions. The mask and encoding masked image serve as conditions for the U-Net, modifying the objective function in Eq. to
WGr, 1, my, Xm,, 0) = min(Bauee- c,(2,, My.xx)))l Xma )11))
(2)
where my E "x"x*xx denotes the s-fold nearest- neighbo downsampling of the input mask m, and Xm_ = E(xx) indicates embedded masked image in latent space. Denoising diffusion implicit model
4

---

Published as a conference paper at ICLR 2024
Depth
Fine-tred
2
MCUA-M
0
Pose
Convense
Arrpt leatton
F
4/11 V.
Fineided
Loss
Skelich
Denotsing sampling with CMB
Musked image
Input masi
PM
WLeN
Conoot
Figure 3: Illustration of our method. We initiate the inference process with randomly initialized latent 2y. This latent is denoised times, with the concatenation of the masked image and mask acting as conditioning for both MCU-Ner and frozen U-Net denoiser. Through CMB, we fuse diverse modality guidance signals, aiding the frozen original U-Net 0, to iteratively produce the desired content. The content is finally transformed into pixel space via decoder network, resulting in the completed RGB output.
(DDIM) (Song et al. same training objective as DDPM. Accordingly, the sampling process formulated as,
2021 defines each step of denoising as non- Non-Markovian process while retaining
2 vI E2, mi, XmL) + VI 0111 a--1--1-- -
my, Xm1) + oís, e,
(3)
Var
where the noise & follows the standard normal distribution N(O,) and independent of x,, and o, 17 v( a0-1)/ a,) v1 a,/a,-1 By gradually denoising over 7 timesteps, the content of the missing region is hallucinated the latent space, producing a conditional sample Zo p(zr|m Xm1). zn is then transformed into the pixel space & D(za) via the left-inverse decoder network D corresponding to the autoencoder E(), finally resulting in the completion outcome & E BHWWx3
3.2 MCU-NET: MODALITY-SPECIRE CONDITIONAL. U-NET
The first stage in MaGIC is to learn image completion under single-modalle guidance. For this purpose, we propose a simple yet effective modality-specific conditional U-Net (MCU-Net). Particularly, for the auxiliary guidance e' C (C (ct) denotes the of N auxiliary guidance), MCU-Net consists of a standard U-Net denoiser 0, (Rombach et al. (2022) and an encoding network Tul For simplicity, we will omit in the following sections.
Auxiliary Guidance
Wr,c
K
Text
Sketch Map
Semantic
K
Map
Depth
V
The encoding network T, employed extract multi- scale guidance signals, represented as F!. where I (O...., L. and L denotes the number of times the feature map scale is reduced within the U-Net denoiser. Afterwards, F' is injected to the latent in MCU-Net obtain modality- guided feature. In spe- cific, we denote the latent MCU-Net as Wxe (c E C) to distinguish it from the original diffusion model's z, As illustrated in Fig. 4 to inject guidance signals into the latent Wie, add F! intermediate feature maps Fl. of the encoder of MCU-Net, resulting
Pose
Map
LiNoe Danolise
Map
Canny Edge
Map
A elemert-on ise
additoon
CrOSS attenion
Figure 4: Illustration of MCU-Net.
in guided feature map ftl Fl ene F. [0, I.J. And we incorporate the text modality in manner
5

---

Published as a conference paper at ICLR 2024
consistent with SD, which integrates its information into intermediate features via cross-attention mechanism.
To utilize the generative capability of pre-trained SD, we freeze the original U-Net denoiser when training MCU-Net, allowing the unlocked encoding network t, to learn guidance signal extraction and fit the pre-trained denoiser.
3.3 CMB: CONSISTENT MODALITY BLENDING
Despite achieving image completion under single-modality with MCU-Net, it is not trivial to integrate multiple MCU-Nets for multi-modality image completion. A naive way to jointly re-train these learned MCU-Nets, which is cumbersome and inflexible for multi-modality image completion. To deal with this, we propose the novel consistent modality blending (CMB), training-fres algorithm to integrate guidance signals from different auxiliary modalities without requiring additional joint re-training. great benefit of CMB that, the multi-modality guidance latent code in MCU-Net remains aligned with the internal knowledge of SD model, without affecting its original ability. As shown in Fig. [3 the guidance signals from arbitrary combination of independent models (i.e., MCU-Nets) in gradient aspect gradually control the image completion process with input modalities.
Specifically, given series of MCU-Nets trained independently multiple modalities C, extract the guidance signals F.. A simple way for integrating different modalities to directly update intermediate feature maps F by adding accumulated guidance signals Fc Fene + Eeec Fe. However, we argue that this simple manner we called feature-level addition is impractical, as the denoiser is trained solely on the distribution of f. Fene F... Drawing inspiration from recent udvancements in classifier- dassifier-guidance guidance diffusion (Dhariwal & Nichol [2021). we introduce converse amplification strategy. This technique enables the intermediate feature maps F, of original U-Net to more closely approximate each guided feature map P...
Converse Amplification. We use the intermediate features from the original U- Net 6, which is not equipped with guidance encoding network, while F. to denote guided features from MCU-Net 9. of modality c. No- tably, U-Net 0. and MCU-Net Og undergo par- allel denoising process. At each step f. every la- tent is denoised using the DDIM sampler (Song et al. 2021). the original U-Net 0,, we denote the denoised latent the intermediate latent 2
denote
Algorithm 1 Usage of CMB in MaGIC
Require: Given the input masked image Xm, mask m, series of MCU-Net parameters 6,,. the number of times converse amplification P. the number of iteration steps of back- proppaappppaannnnooo Q.
mj downsample(m))
2: Aat = E(xw)
3: N(O,
4: Wr. N(O, CC
5: for T...., do
WT then
7:
Eg. F. < 0, (z, f. mm, sampler(s..............mmmmmm frr
We bias/ F. towards f. by calculating their Eu- clidean distance in each scale /:
8:
(Eq. 3
continue
end if
+
aFf. F.)
F.I.F. F'1l.. TED
(4)
11: for 1.... do do
12:
6, Fc - Oolwic. m.
13:
Wi-1C sampler(w.c.e))
(Eq.
R
where 6, are scale factors to weight the strength
14:
F. 0lla, mI, Xm11)
leads to either improved alignment guidance
15:
द = sampler(t, 6rr 2 = o V.V.
(Eq.
modality e or greater diversity in the outputs.
16:
(Fq.
N |C] indicates the modality number of aux-
17: end for
iliary guidance set. We then apply the distance energy loss, similar the classifier guid-
18: end for
19: return D(2))
ance methods in Chen et al. 22033; ); Dhariwal &
Nichol (202T). to adjust latent code of the original SD model. Specifically, each denoising step, we obtain Fc and F, firstly, then the gradient of their distance is calculated through back pack-propagation to update the denoised latent i
2 2 o.
(5)
Owing to CMB, is not necessary to jointly re-train the learned MCU-Nets, making MaGIC flexible in merging arbitrary multi-modality for completion. Alg. ssoows the procedure of CMB for MaGIC.

---

Published as a conference paper at ICLR 2024
onaKE
dnald
onam
LAMA
MAT
ControlNet
T21-Adapter+
Ours
Guidance Input
Figure 5: Qualitative comparison for image completion using single modality as guidance. indicates the use of latent-level blending (Avrahami et al. 2023) to preserve pixels in unmasked regions.
4 EXPERIMENTS
In this work, we study three research questions, RQ1, RQ2 and RQ3:
RQ1: Can our MCU-Net effectively perform image completion guided by various modalities?
RQ2: Can our MaGIC with CMB seamlessly integrate guidance from multiple modalities to produce
credible completion results?
RQ3: How do different module designs (e.g., adjustments in hyperparameters and inference pro- cesses) impact the overall effectivenes??
4.1
EXPERIMENTAL SETTINGS
In our experiments, we select several adge-based image completion methods, including EC (Nazeri
al., 2019), CTSDG (Guo et al., 2021), ZITS (Dong et al. [2022), and state-of-the-ee (SOTA)
techniques such as LAMA (Suvorov et al., 2022), LDM (Rombach et al., 2022), and MAT (Li ctt al. 2022). We also include controllable image generation baselines such as ControlNet Zhang
& Agrawala 2023) and T2I-Adapter (Mou al. 2023) in qualitative comparison, they
be easily adapted to the image completion task with the concept of Blended Diffusion (Avrahami al. 2022 2023). For fair comparison, we apply the same set of image mask pairs across all tests, and, for comparisons involving auxiliary guidance, we ensure that each method receives identical guidance map instructions. The masks used in testing designed to uniformly span masking ratio range from 0 to 100%. The evaluation adopts both image metrics (i.e., FID and P/U-IDS (Zhao et al. 202T) and text-to-image metric (i.e., PickScore (Kirstain al. 2023) which gauges the fidelity of generated content based on learned human preferences. Acknowledging the pluralistic outcomes of our method, we conduct tests on a total of five images to determine mean scores and standard deviations. For all diffusion-basee methods, the denoising step 7 is set to 50. For further details on experimental configuration, please see the supplementary material.
4.2 IMAGE COMPLETION WITH SINGLE-MODALITY GUIDANCE USING MCU-NET
To answer RQ1, we compare our approach with sfate-of-the art (SOTA) inpainting methods (Suvorov) et al. 2022 Li et al.) 2022) and SOTA single modality guidance image generation methods. We employ levent-level blending (Avrahami al. 2023) to preserve pixels in unmasked regions for image generation methods such ControlNet (Zhang & Agrawala, 2023) and T21-Adapte (Mou et al.) 2023). As depicted in Fig. 5 our method generates content without noticeable artifacts, maintaining stronger spatial context consistency. Conversely, T2I-Adapter generates stone house the road (Ist row in Fig. 5) and ControlNet puts a dancer on the soccer field (2nd row in Fig. 5).
Quantitatively, the scores of edge-based methods COCO and Places2 are displayed in Tab. I Across all metrics, our method demonstrates significant improvements, indicating that our MCU-Net effectively generate content under the guidance of various single modalities.
7

---

Published as a conference paper at ICLR 2024
COCO
Places2
Method
FIDJ
PickScore /
FIDJ
U-IDSt %
P-IDST %
EC (Nazeri et al.
2019)
76.64
23.14
25.08
12.89
2.86
CTSDG (Guo 2021
97.05
24.03
42.81
0
ZITS (Dong al.
2022
61.27
28.09
18.96
18.75
7.20
Our MCU-Netf
47.70=0.29
30.79=0.10
10.74+0.07
23.83=0.30
10.18+0.48
Our MCU-Net
39.43=0022
37.12:0.11
9.09=0.04
25.34:0.29
10.64:0.46
Our MCU-Net &
01.21
34.96=0.17
10.27=0.06
24.21+0.24
9.93=0.38
Our MCU-Net
41.15=0.27
34.94=0.06
8.32=0.02
26.23=0.07
10.96:0.33
Table I: Comparison of using single auxiliary modality guidance for image completion. *: ground
truth edge map as guidance, v: estimated depth map as guidance, 4: segmentation map as guidance, I: the higher the better, the lower the better, t: completion without any guidance.
COCO
COCO
Method
MMG
FID
PickScore 1/ %
Method
MMG
FID
PickScore f
CoMad
68.01
25.12
MaGIC w/ FLA (35 steps)
07.78+0.32
44.19=0.23
TFill
58.55
24.63
FeF
48.92
26.43
MaGIC w/ FLA (50 steps)
41.53+0.19
35.85=0.08
LAMA
48.63
29.06
MAT
45.51
27.10
MaGIC+
47.7010.29
30.7910.10
MaGIC
47.70.0.29
30.7910.10
MaGIC w/ CMB
37.65x0.22
49.57.0.17
MaGIC
37.63+0.22
49.57-0.17
(a) Comparison CMB with simple FL.A
(b) Comparison of MaGIC with SOTA methods.
Table 2: Comparisons of CMB and FL.A and MaGIC with others. MMG: multi-modality guidance.
4.3
IMAGE COMPLETION WITH MULT-MODALITY GUIDANCE USING MAGIC
We further decompose RQ2 into two smaller questions, RQ2.1 (Is CMB effective?) and RQ2.2 (How does MaGIC perform?)
Answering RQ2.1. CMB aims to integrate different modalitics as guidance for image completion in training-free fashion. Compared with CMB, simple way is to aggregate feature maps F. (c C) by addition (i.e., feature-level addition or FLA for short) to produce Fc as Fc + Fenc + Evec Fe. To show the effectiveness of CMB, we compare it with FLA on COCO in Tab. 2a Note that, test
FLA with 30 and 50 steps, respectively. To guarantee an equitable assessment across all auxiliary modalities, we opt for wide-ranging set of modalities. Given that specific modality (e.g., pose) may not be applicable to all test images (e.g., certain landscape images), we ensure that our test suite incorporates diverse range of modalities. This includes segmentation map, depth map, Canny edge map, sketch map, and a prompt text. As displayed in Tab. 2a the proposed CMB significantly surpasses FLA with naive addition, evidencing the effectiveness CMB in merging multi-modality for completion. Interestingly, the performance of FLA with 50 steps is counter- counter-intuitively lower than that with 35 steps, suggesting that this simple method may overly manipulate the latent code. This indicates that the direct addition of different MCU-Net feature maps for multi-modality guidance is impractical. By contrast, our CMB efficaciously integrates the signals from multi- modtil guidance.
Answering RQ2.2. To validate the effectiveness of our MaGIC, compare it with state-of of the-art image completion methods, including LAMA (Suvorov et al. (022) and MAT (Li et al.) (022). on COCO. As in Tab. 2b our guidance- free inpainting model denoted MaGICT comparable to SOTA inpainting baselines MAT and LAMA. When employing multi-modality (segmentation, canny edge, sketch, depth, and text) as guidance, to MaGICt, we obtain gains of 21 % in FID and 61 % in PickScore. Notably, the PickScore implies that, from the perspective of learned human preference, completed images have 49.57% chance of being more faithful to the ground truth image caption than the original images.
MaGIC gains significant improvements. In comparison
4.4 ABLATION STUDY
To answer RQ3, conduct rich ablations COCO follows.
8

---

Published as a conference paper at ICLR 2024
seg depth canny sketch
FID
PickScore 17%
P
Q
FIDI
PickScore1 %
35,42-0.10
47.81+0.34
30
1
38.50=0.23
48.11:0.21
20
34.9620.17
30
10
38.20+0.28
48.60
39.43=0.26
37.12+0.11
/
41.15=0.27
34.94+0.06
10
40.40=0.25
45.98=0.12
20
38.37=0.34
48.72+0.15
41.38+0022
35.99+0.13
42.59-0.19
34.98+0.16 16
30
5
37.65+0.22
49.57
39.9620.12
48.5820.25
40
5
36.78+0.22
50.64+0.20
37,65x0,22
49.57=0.17
50
36.60=0.12
50.52:0.07
(a) Ablation on different modalities.
(b) Hyperparameter analysis of CMB.
Table 3: Ablation studies on the multi-modality complementary and the hyper-parameters of CMB.
Impact of modalities. To delve into auxiliary modalities, we investigate their individual contributions. We distinguish among five modalities used in our experiments: edge and sketch for fine-grained structural control, segmentation and depth for coarse-grained spatial-semantic control, and text for content-specific cue. As in Tab. 3al the guidance from text significantly enhances image quality (FID) and generated content (PickScore). Interestingly, excluding text, the performance of com- bined modalities appears balanced, suggesting optimal generation quality when modalities provide complementary information. When using all modalities, the performance is the best.
Joint multi-modality re-training. Our method allows multi- modiitm guidance without the need for additional joint training. However, exploring the joint re-training of all modality-specifie conditional U-Nets with classifier-free guidance style can help identify the upper bound performance.
Building such a model necessitates fuser mechanism to blend diverse input modalities. To ensure effectiveness, we integrated CoAdapterFuser (Mou et al. 2023). aligning with our design goals. Addressing the lack of paired dataset with extensive labels actoss various modalities was also essential. We extracted 650,000 images from the Laion dataset and generated four modalities (canny edge map, depth map, sketch map, and semantic map) using open-source tools. During joint re- training, we randomly dropped out each modality at a 0.5 probability. This training process is memory-intensive, necessitating reduction in batch size a quarter of single-modality training. The model underwent 180,000 iterations. As evidenced by its lower FID, shown in the first row of Tab. [3a the unified model achieves higher fidelity than our training-fire -free method. However, encounters issues such as the need for paired training data, difficulty in adding new modalities, and substantial computational requirements for joint training.
Guidance in iteration. The proposed CMB algorithm involves two important hyperparameters, i.e. the number P of denoising steps incorporating CMB and the iteration times of gradient descent performed in each CMB operation. We study the impact of different P and Q on the multimodal conditioning completion task as in Tab. 3b From Tab. 3b we can observe that with Q fixed, the performance is almost consistently improved by increasing the number P of denoising steps (from 10 to 50) equipped for CMB. Interestingly, given the fact in Tab, 2a that incorporating guidance through simple FLA could impair the performance of completion, the results further demonstrate the effectiveness of CMB.
5 CONCLUSION AND LIMITATION
In this paper, we propose a novel, simple yet effective method, named MaGIC, for multi-modality image completion. Specifically, we first introduce the MCU-Net that is used to achieve single- modality image completion by injecting the modality signal. Then, we devise novel CMB algorithm that integrates multi-modality for more plausible image completion. On extensive experiments, we show that MaGIC shows superior performance. Moreover, is generally applicable to various image completion tasks such as in/out-painting and local editing, and even the image generation task.
MaGIC is proposed to facilitate image completion with multi-modality. Yet, there exist two limitations. First, the ability to generate high-freyuency details is tied to the backbone completion model, which means even with ample detailed guidance, achieving the desired fidelity may not be guaranteed. This can be improved by adopting more powerful backbones if necessary. In addition, our MaGIC is less efficient than current single-step completion models, with inference time increasing in line with guidance modalities. This is common issue for diffusion models, and we leave it for future research.

---

Published as a conference paper at ICLR 2024
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In CVPR, 2022.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. SIGGRAPH, 2023.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. 1rX1::2302.07121, 2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
John F. Canny, A computational approach to edge detection. IEEE TPAMI, 8(6):679-698, 1986.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Trans. Graph., 42(4):148:11114:10 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. arXiv:2304.03373; 2023.
MMPose Contributors. Openmmlab pose estimation toolbox and benchmark. https://gi thub http:/ppen-mmaaa/mmmng 2020.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Object removal by exemplar- semmpla-bbssed inpainting. In CVPR, 2003.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In NeurlPS, 2021.
Qiaole Dong, Chenjie Cao, and Yanwei Fu. Incremental transformer structure enhanced image inpainting with masking positional encoding. In CVPR, 2022.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. CVPR, 2022.
Xiefan Guo, Hongyu Yang, and Di Huang. Image inpainting via conditional texture and structure dual generation. In ICCV, 2021.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 2020.
Daichi Horita, Jiaolong Yang, Dong Chen, Yuki Koyama, and Kiyoharu Aizawa. structure-guided diffusion model for large-hole diverse image completion. arXiv:2211.1044 2022.
Jacseok Jeong, Mingi Kwon, and Youngjung Uh. Training-frr ffre style transfer emerges from h-space in diffusion models. In ICLR, 2023.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. CVPR, 2023.
Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye. Diffusionclip: Text-guided diffusion models for robust image manipulation. In CVPR, 2022.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. arXiv:2305.01566, 2023.
Anat Levin, Assaf Zomet, Shmuel Peleg, and Yair Weiss. Seamless image stitching in the gradient domain. In ECCV, 2004.
10

---

Published as a conference paper at ICLR 2024
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. MAT: mask-aware transformer for large hole image impainting. In CVPR, 2022.
Liang Liao, Jing Xiao, Zheng Wang, Chia- Wen Lin, and Shin ichi Satoh. Guidance and evaluation: Semantic-aware image inpainting for mixed scenes. In ECCV, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, 2014.
Andreas Lugmayr, Martin Danelljan, Andrés Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, 2022.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv:2302.0844 2023.
Kamyar Nazeri, Eric Ng. Tony Joseph, Faisal Z. Qureshi, and Mehran Ebrahimi. Edgeconnect: Structure guided image inpainting using edge prediction. In ICCVW, 2019.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew. Ilya Sutskever, and Mark Chen. GLIDE: towards photorcalistic image generation and editing with text-guided diffusion models. In JCML, 2022.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv:2204.06122 2022.
René Ranfil, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI, 44 (3):1623-1637, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NearlPS, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b; An open large-scale dataset for training next generation image-text models. arXiv preprint:2210.00842 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
Zhuo Su, Wenzhe Liu, Zitong Yu, Dewen Hu, Qing Liao, Qi Tian, Matti Pietikäinen, and Li Liu. Pixel difference networks for efficient edge detection. In ICCV, 2021.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. In WACV, 2022.
Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Bringing old photos back to life. In CVPR, 2020.
Su Wang. Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J. Fleet, Radu Soricut, Jason Baldridge, Mohammad Norouzi, Peter Anderson, and William Chan. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In CVPR, 2023.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. Smartbrush: Text and shape guided object inpainting with diffusion model. In CVPR, 2022.
11

---

Published as a conference paper at ICLR 2024
Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowledge distillation for semantic segmentation. In CVPR, 2022.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. Free-form image inpainting with gated convolution. In ICCV, 2019.
Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training-free energy-guu mnergy-guided conditional diffusion model. arXiv:2303.09833 2023.
Yongsheng Yu, Dawei Du, Libo Zhang, and Tiejian Luo. Unbiased multi-modality guidance for image impainting. ECCV, 2022.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv:2302.05554, 2023.
Shengyu Zhao, Jonathan Cui, Yilun Sheng. Yue Dong, Xiao Liang, Eric I-Chao Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In ICLR, 2021.
Chuanxia Zheng, Guoxian Song, Tat-Jen Cham, Jianfei Cai, Dinh Q. Phung, and Linjie Luo. High- quality pluralistic image completion via code shared VQGAN. arXiv:2204.0193/ 2022a.
Haitian Zheng, Zhe Lin, Jingwan Lu, Scott Cohen, Eli Shechtman, Connelly Barnes, Jianming Zhang, Qing Liu, Yuqian Zhou, Sohrab Amirghodsi, and Jiebo Luo. Structure-guided image completion with image-level and object-level semantic discriminators. arXiv:2212.06311 2022b.
Bolei Zhou, Ágata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE TPAMI, 40(6): 1452-1444, 2018.
12

---

Published as a conference paper at ICLR 2024
MaGIC: Multi-modality Guided Image Completion
Appendix
CONTENTS
Introduction
2
Related Work
3 MaGIC: Multi-modality Guided Image Completion
3.1 Preliminaries
3.2 MCU-Net: Modality-specific Conditional U-Net.
3.3
CMB: Consistent Modality Blending
4
Experiments
4.1
Experimental settings
4.2
Image Completion with Single-Modality Guidance using MCU-Net
4.3
Image Completion with Multi-Modality Guidance using MaGIC
8
4.4
Ablation Study
Conclusion and Limitation
A
Implementation Details
14
A.I Different Image-based Conditions And Hyper-parameters.....
14
A.2 Acquisition of Conditions
14
A.3 Architecture of Encoding Network Tr
14
A.4 Experimental Setting
15
Application Results
15
B.1 Real User-input Image Editing
15
B.2 Image Outpainting
15
C More Experimental Results And Studies
15
C.1 Quantitative Comparisons with Conditional Text- Methods
15
C.2 Qualitative Comparisons in Multimodal Conditioning
17
C33 Study in Feature-level Addition
17
C.4 Failure Cases
17
C.5 Selection of 6c
18
C.6 Study in Adaptability and Image Generation Application
18
13

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 DUTERENT IMAGE-BASED CONDITIONS AND
Our experiments include 6 types of image-based conditions:
Canny edge & Sketch. We utilize the training set of COCO (Lin et al. 20144, which contains 123K images, the training data to train MCU-Net separately under canny and sketch guidance. The corresponding canny edge and sketch are generated by Canny algorithm (Canny 1986) with default thresholds, and PiDiNet (Su al. 2021 with threshold of 0.5, respectively.
Segmentation. We utilize training set of COCO-Stuff (Caesar et al. 2018) as training data, which includes 123K images and corresponding semantic segmentation annotations. It covers 80 thing classes, 91 stuff classes and 1 "unlabeled" class, providing a comprehensive range of semantic information for MCU-Net training.
Depth. In order to obtain sufficient volume of data to train MCU-Net under this conditions with abstract representation, we select 650K images from LAION-AESTHETIC dataset (Schuhmann et al., 2022). And we adopt MiDaS (Ranftl et al. 2022 on them to generate depth maps.
Pose. We also pick images from LAION-
(Schuhmann al. 2022 to
construct training data for MCU-Net under pose guidance. The key distinction from building training dataset for depth guidance is that the selected images must contain least one person for pose generation. To achieve this, we employ MM-Pose (Contributors) 2020), an open-source toolbox for pose estimation, to filter out images that do not meet the requirement, and generate pose for the retained images. In the end, we gather a total of 600k image-pose pairs train MCU-Net under this condition.
Text. Within our default backbone, the SD-2.I Impainting. the prompt text is conditioned as the key and value of the cross- -oss-attention mechanism in the U-Net denoiser. It's noteworthy that this backbone is pretrained with the prompt text in the classifier-free way (Ho & Salimans, 2021). Consequently, in this work, we opt to use the backbone directly, thus bypassing the necessity to fine-tune an MCU-Net for text guidance.
All our experiments are conducted using 8 NVIDIA A100-40G GPUs. We set the batch size to 64 and employed the Adam optimizer (Kingma & Ba 2015) with the learning rate of 1e-5 for training 10 epochs. These settings remain consistent across all conditions.
A.2 ACQUISITION OF CONDITIONS
To facilitate a reliable and convenient comparison of model performance, we employed the conditions provided by the dataset directly or leveraged existing tools (Yang et al. 2022. Contributors 2020
Ranftl et al. 2022) to estimate them. We then evaluated the model performances using quantitative
metrics completing the corresponding masked RGB images. It is important to note that
method also supports the input of manually designed guidance conditions (as shown in Fig. [ Fig. 15 Fig. T6 and Fig. [7). However, when manually design dense guidance conditions like segmentation and depth maps. it's crucial to ensure their consistency with the information retained in the unmasked regions, particularly in the case of depth maps where values represent the distance between pixels and the camera. Fortunately, sparse conditions like sketch or pose maps can offer sufficient guidance information. We intend to release our code for condition generation, enabling users to obtain modalities including sketch, pose and segmentation maps effortlessly for image editing purposes.
A.3 ARCHITECTURE OF EXCODING NETWORK T
The condition encoding network is designed to be simple and lightweight, and serves the purpose of extracting the multi-scale guidance signals from the input condition image. These guidance signals are aligned in size with the intermediate feature maps of the MCU-Net's encoder. As this not the main focus of our work, we have referred to the design of T21-Adapter (Mou et al. 2023). Specifically,
14

---

Published as a conference paper at ICLR 2024
it consists of four feature extraction blocks with downsample module placed between each pair of adjacent blocks, and each feature extraction block composed of one convolution layer and two residual blocks. Notably, cannot simply use the denoiser of T21-Adapter our backbone for single modality guided image completion. Our work involves gathering multi-modality datasets and training MCU-Net with masked images via Classifier-free guidance loss follows,
arg min Neo.lll cg(2,, Mi, Ami cm),
(6)
This training enables our model to adeptly fill masked regions while ensuring spatial consistency with the unmasked areas, shown in Figure and 11 Figure of Section 4.2
This finding aligns with the observations in
A.4 EXPERIMENTAL SETTING
In order to evaluate all baselines and our proposed method in fair manner, the same image-mask pairs used in quantitative experiments. Additionally, testing mask sam ples obtained based a uniform distribution ranging from 0% to 100% to encompass the majority of mask ra- tios encountered in real-worf world scenarios. The testing mask is randomly generated based on the algorithm from (Li al. 2022), with the histogram of the testing mask ratio of COCO dataset visualized in Fig.
Number of Sampds
150 100
0.0
0.6
Mack
Figure Illustration of mask ratio.
All quantitative experiments are conducted on the COCO (Lin et al. 2014) and Places (Zhou et al. 2018) datasets. Evaluation of the methods involves using the first 1000 images in the COCO validation set and the first 5000 images
Places validation set. Masks from COCO are replicated for the Places dataset. For auxiliary guided completion, the Canny algorithm (Canny (986) and PiDiNet (Su et al.] (2021) are employed to obtain the canny edge map and sketch map, respectively. MiDaS (Ranful al.] (2022) is adopted acquire depth maps for both datasets. COCO serves as a dataset with semantic segmentation and prompt text annotations. As the Places dataset lacks ground-truth labels, the semantic segmentation map is estimated using CIRKD (Yang al. 2022
times
APPLICATION RESULTS
B.1 REAL USER-INPUT IMAGE EDITING
We highlight the adaptability of our method in handling user-input image editing tasks designed to manipulate real-world images based on user intention, as demonstrated in Fig. [5] This figure emphasizes method's capacity to modify the structure semantics of local regions using user- input guidance such as scribble, pose map and prompt text, while fully maintaining the integrity of the unmasked region.
B.2
IMAGE OUTPAINTING
Our method can also be used extend image, like generating a panorama from small part of the image content. As demonstrated in Fig. 16 and Fig. 17 our method showcases its capability outpaint photograph
painting guided by text and sketch map. Remarkably, method exhibit the ability to generate suitable content that is harmonious even with the broader context of panoramic image.
MORE EXPERIMENTAL RESULTS AND STUDIES
QUANTITATIVE COMPARISONS WITH CONDITIONAL TEXP-TO-IMAGE METHODS
Contemporary methods like ControlNet and T21-Adapter have demonstrated remarkable achievements controllable image generation. For direct comparison, we employ latent-level blending to utilize these methods for image completion, maintaining the experimental settings of earlier Experiments.
15

---

Published as a conference paper at ICLR 2024
COCO
Places2
Method
FIDI
PickScoret/ / %
FIDJ
U-IDS1 / %
P-IDST /
ZITS &
61.27
28.09
18.96
18.75
7.20
T2I- T2I-Adapter
48.23
30.10
10.39
19.44
5,66
ControlNet
37.17
37.30
10.35
18.45
4.58
Ours
41.15
34,94
8.32
26.23
10.96
T2I- Adapter
50.92
30.22
18.10
14.91
4.56
ControlNet
46.13
32.52
15.96
14.46
3.18
Ours
39.43
37.12
9.09
25.34
10.64
T2I-Adapter
50.65
28.10
15.36
15.99
4.30
ControlNet &
58.27
26.11
18.13
13.68
3,24
Ours 41.91
34.96
10.27
24.21
9.93
T2I-Adapter
39.08
34.26
14.27
14.76
3,30
Ours
37.65
49.57
8.98
25.30
10.90
Table Quantitative comparisons with conditional image completion and text-to-image methods. :: ground truth edge map as guidance, v: estimated depth map as guidance, & segmentation map as guidance, o: using segmentation, depth, canny, sketch, and text (on COCO) for guidance simultaneously.
COCO
Places2
Method
CLIP1 / %
PSNRT
SSIM
LPIPS_
PSNRT
SSIMI
LPIPS_
7ITS
28.33
14.31
0.2767
0.5382
21.07
0.6888
0.2614
T2I-Adapter
28.59
18.26
0.6272
0.3409
18.34
0.6537
0.3208
ControlNet
28.97
19.22
0.6871
0.3183
18.59
0.6647
0.3220
Ours
29.37
18.13
0.6188
0.3467
19.00
0.6569
0.3111
T21-Adapter
28.05
17.84
0.5894
0.3729
17.57
0.5765
0.3805
ControlNet R
28.22
18.16
0.6275
0.3583
17.49
0.5967
0.3703
Ours
29.11
17.47
0.5960
0.3628
17.91
0.6109
0.3432
T21-Adapter
28.10
17.55
0.5635
0.3830
17.33
0.5529
0.3923
ControlNet
26.48
16.98
0.5587
0.4023
17.22
0.5568
0.3948
Ours 28.87
17.01
0,5681
0,3799
17.44
0.5860
0,3591
T2I-Adapter 30.23
19.45
0.6748
0.3217
19.17
0.6626
0.3255
Ours
31.29
17.49
0.5921
0.3717
17.85
0.6085
0.3439
Table 5: Additional quantitative comparison results in terms of CLIP score and traditional recon- struction metrics. *: ground truth edge map as guidance, v: estimated depth map as guidance, segmentation map as guidance, using segmentation, depth, canny, sketch, and text (on COCO) for guidance simultancously.
As Table reveals, our MaGIC significantly surpasses baseline models in FID, U-IDS, P-IDS, and PickScore for most guidance types. In multi-modality guidance, enhance T21-Adapter with multi-adapter controlling Mou et al. (2023) (feature-level addition), resulting in T21-Adaptero, For the COCO dataset, we employ five modalities: canny edge. depth. segmentation, sketch map, and text. For the Places dataset, we utilize canny edge, depth, segmentation, and skeich map, as it lacks manually-crafted captions. Our MaGIC outperforms T21-Adaptero by 44.68% in PickScore on COCO, and shows improvements of 37.07%, 71.40%, and 230.30% in FID, U-IDS, and P-IDS, respectively, on Places, as detailed in the last two rows of Table
While traditional reconstruction metrics, such PSNR, SSIM, and LPIPS, rely pixel-wise similarity to the ground truth and tend to favor blurry outputs, as noted by Zhao et al. (2021) and Li et al. (2022), they not optimal for quantitatively assessing image completion. Nevertheless, include these traditional metrics for reference. Additionally, we provide the CLIP Score as an extra measure for more comprehensive evaluation in the Table
16

---

Published as a conference paper at ICLR 2024
C.2 QUALITATIVE COMPARISONS MULTIMODAL CONDITIONING
As illustrated in Figure 9 perform qualitative, side-by-side comparisons with T2I-Adapters, For our MaGICo, we produce five diverse results. Under the guidance four modalities, MaGIC demonstrates strong controllability and high-fidelity outputs, aligning with our quantitative findings. In contrast, while 121-Adapter effectively adheres the layout or shape to guidance, it fails to generate images of above-average quality with realistic details. This shortfall is attributed the feature-level addition approach, leading to an out-of-distribution effect in the SD U-Net.
C.3 STUDY IN FEATURE-LEVEL. ADDITION
Although we claim that FLA (feature-level addi- tion) a simple yet imperfect method for com- bining multiple modalities, and demonstrate the effectiveness of our CMB by comparative exper- iments, a comprehensive understanding of these two methods remains elusive. To this end, we opt to visualize the feature distributions stem- ming from T2I-Adapter with dingle-modalle training and multi multi-modality utilization strate- gies, including FLA or our proposed CMB.
In specific, choose the feature from the mid- dle denoising step (i.e., the 25th step of DDIM sampler) and output from U-Net encoder. The t-SNE visualization result is shown in Figure 7 and different colors represents features from dif- ferent sources, while the associated numbers indicate the index and cluster center of each feature type. Numbers ranging from small to large represent features obtained from 121-Adapter-Can Canny (0), T21-Adapter-Des Depth (I),
Figure 7: t-SNE visualization of features output from U-Net encoder.
(2), T2I-Adapter-Skete -ketch (3), T2I-Adapter-CMl CMB (4) and 221-Adapter-FLA. FLL (5), respectively, where the first four indicate the trained single-modality while the last two are two methods of combining these four modalities.
We can draw two conclusions from Figure
1. Features derived from different tribution disparities, and FLA (5) directly adds modality features resulting in the distribution deviation of obtained feature from all others. This observation aligns with assertion in the main manuscript that "we called feature-level addition is impractical, as the denoiser is trained solely on the distribution of F. enc Fe".
models (0, 1, and 3) show significant dis-
2. In contrast to FLA, the distribution of features obtained through CMB (4) is surrounded by other single-modality distributions. This phenomenon is coherent with Equation and [5] where the distribution of obtained features is "pulled" by the distributions of the other four lingle-moallitis.
C.4 FAILURE CASES
Figure 8 shows two failed instances of applying CMB T21-Adapter for multi-modalities guidance. Here we present a more complex testing scenario involving non-overlapping information between two modalities. In the first case, we use Anything-4.0 as the backbone and there are two mistakes in the generated image: the misshapen cat and the incorrectly positioned bench under the girl. The former discrepancy possibly arises from the pose adapter contributing stronger features compared to those from the depth adapter, consequently affecting the representation of the latter information, which is not accurately reflected in the generated image. The issue might be alleviated by training depth adapter more stronger or increasing 6depth while decreasing 6 pose (see Equation 4 for details). While the latter mistake is the inherent challenge in SD, and many related works Chen et al. (003); Chefer et al. (222)) could be referenced for potential mitigation steategies. In the second example, the generated image depicts a wall as the background, leading to the complete loss of depth information from the depth map.
17

---

Published as a conference paper at ICLR 2024
Depth Pose Guidance
lovely sits the bench, and beautiful is singtngg
Depth Fose Guidance
storntrooper standing kitchen with 1adder.
Figure 8: Failed cases when adopting CMB on T2I-Adapter.
These instances underscore that the inherent problems of SD persist despite employing CMB. Fur- thermore, in scenarios where information correlation between different modalities low, evident issues such as information loss and errors in generated images become more pronounced. Through these failed cases, it can be seen that the inherent challenges SD cannot be eliminated by CMB. Furthermore, in scenarios where information correlation between different modalities is low, evident issues such as information loss and errors in generated images become more pronounced.
C.5 SELECTION OF óc
our quantitative and qualitative comparative experiments, natural to assign constant weight of to each 6,- However, it also could be considered ö, hyperparameter. Notably, although adopting introduces a potential hyperparameter requiring tuning, our approach, focused on training- free scenarios, makes this addition not overly burdensome compared to the process of training a large, multimodal guided model.
C.6 STUDY IN ADAFTABILITY AND IMAGE GENERATION APPLICATION
Our proposed MaGIC has the ability adapt variety of backbone diffusion models, including but not limited to, the image generation model Anything-4.0, Stable Diffusion-Le 1.5 (also employed by T2I-adapter and ControlNet), and the image completion model Stable Diffusion Inpainting-2. (the default in MaGIC). In order to elucidate the differences among these backbone diffusion models, a qualitative experiment was carried out, focusing primarily on anime-style image generation model Anything-4.00 image generation model Stable Diffusion (SD), the mask-aware T21-adapter
(oou et al. 2023 Avrahami et al. 2023), and our own MaGIC.
As portrayed in Fig. 12 a) and (b), our MaGIC method exhibits exceptional generalizability image generation backbones. These backbones can produce convincing results guided by factors such as sketch, depth, segmentation, and the canny edge map. T21-Adapter (Mou al. 2023) is conditional image generation framework based on Stable Diffusion-11.5 To equip 12I-Adapter with CMB for the completion of masked image, (Avrahami et al. 2023). As evidenced in Fig. c2(c) and (d), incorporating blending into T21-Adapter can preserve the unmasked region remains while the generated masked region does not perceive the unmasked region, given the fact that there are two sheep heads in single sheep.
implemented technique known latent-level blending
We further adapt CMB T21-Adapter and ControlNet (Zhagg & Agrawala, 2023) to verify its
effectiveness on multi-modality guided image generation. The results are shown in Fig. 13 and Fig.
4
18

---

Published as a conference paper at ICLR 2024
Masked
LAMA
MAT
T2I-Adapter
Ourso Ressul
Ours RResul2
Ourso-Result3
Ourso-Result
Guidance
Masked
LAMA
MAT
T21-Adapters
RerselResultl
Ocrso-Result3
Ourso-Result
Oussu-R
Guidance
Masked
LAMA
MAT
T21-Adapter
Ours Result
Ousso-Resul12
Ourso Resuul3
Oursu-Resul14
Ousso-ResultS
Guidance
Figure 9: Qualitative results of MaGIC with four guidance compared baselines.
19

---

Published as a conference paper at ICLR 2024
Masked
MAT
T21 IZI-Adapter+
MCU-Netf
MCU-Net4
Masked
MAT
T2I-Adapter
MCU-Netf
MCU-Net4
Masked
MAT
T21-Adaptcra
MCU-Nett
MCU-NetA
Masked
MAT
T21- Adapter
MCU-Neti
MCU-NetO
Masked
MAT
T21- Adaptero
MCU-Nett
MCU-Neto
30 O
Masked
MAT
T21-Adaptero
MCU-Nett
MCU-Neto
Figure 10: Qualitative results of MCU-Net compared to T21-Adapter and MAT. &: segmentation map as guidance, o: sketch map as guidance. MCU-Neth denotes the completion without any guidance.
20

---

Published as a conference paper at ICLR 2024
Masked
MAT
T21-A -21-Adapter
MCU-Netf
MCU-Net -Net
Masked
MAT
T21- Adapter
MCU-Nett
MCU-Netw
Masked
MAT
T21- -21-Adr
MCU-Nett
MCU-Netv
caaa 1 mo
Masked
MAT
T21-Adapter
MCU-Neti
MCU-Net4
Masked
MAT
T21-Adapter
MCU-Nett
Net
Masked
MAT
T2I-Adaptera
MCU-Neti
MCU-Neta
Figure 11: Qualitative results of MCU-Net compared to T21-Adapter and MAT. depth map guidance, 4: canny edge as guidance. MCU-Net" denotes the completion without any guidance.
21

---

Published as a conference paper at ICLR 2024
Masked
(b)
(c)
Guidance
Figure 12: Results from different backbone models. (a) Anything-4.0, (b) Stable Diffusion-le (c)
T2I-adapter with blending, (d) Stable Diffusion Inpainting-2.1 (default in MaGIC).
Generation Results
Cuidance
Generation Results
Cuidance
Figure 13: Adapting the proposed CMB T2I-Adapter (Mou al. 2023 with Anything-4.0 backbone for multi-modality guided image generation.
Canny Edge
Depth Canny Edge
DAY
Figure 14: Adapting the proposed CMB to ControlNet (Zhang & Agrawala, 2023 for multi-modality guided image generation.
22

---

Published as a conference paper at ICLR 2024
Masked Image
Guidance
Editing Results
"Three GROOT are standing"
"The STORMTROOPER skiing"
"The laatman is standing"
"a man is wearing red-and- golden armor, IRONMAN style, lighting on the chest"
man holding CAPTAIN AMERICA'S SHIELD"
"A big rocket stands behind a man"
Figure 15: Application examples: local editing.
23

---

Published as a conference paper at ICLR 2024
"masterpiece, ultra-detailed, 27 RLLA extremely detailed CG unity 8k wallpaper"
"masterpiece, ultra-detailed, flying
Velanne on the top, extremely detailed CG unity 8k wallpaper"
Figure 16: Application examples: sketch and guided image outpainting.
24

---

Published as a conference paper at ICLR 2024
"best illustration, MORDOR, masterpiece, high resolution, THE LORD OF THE RINGS."
"best illustration, Aerial view of the City near rivers."
"best illustration, Skyrim, Cloudy, an extremely delicate and beautiful, island."
"best illustration, immortals island, an extremely delicate and beautiful, game." "
Figure 17: Application examples: sketch and text guided image outpainting.
25


Reviews:
[{"review_id": "OLfDpGaEy0", "reviewer": "Reviewer_c8RF", "summary": "The paper MaGIC: Multi-modality Guided Image Completion introduces a novel framework for image completion that supports various guidance modalities, such as text, edge, sketch, and more. The proposed method, MaGIC, enables flexible and scalable multi-modality guidance without the need for retraining the model. The paper demonstrates consistent improvements in image quality over existing approaches.", "questions": "I wonder why the authors did not provide CLIP score evaluation as well as reconstruction performances.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Innovative and Flexible Approach** The paper addresses the challenging problem of multi-modality-guided image completion. It proposes a new simple training-free procedure, allowing for various guidance modalities, such as text, edge, sketch, segmentation, depth, and pose. \n\n** Large Consistent Gains** The paper shows consistent and significant improvements over state-of-the-art approaches, particularly in image quality.", "weaknesses": "**Clarity and Typos** The paper is challenging to follow and contains multiple typos, which can impede understanding. Improved clarity in the presentation and thorough proofreading would enhance the paper's quality.\n\n**Non-standard Update Scheme** The update scheme presented in equation (5) appears inhomogeneous, as it involves gradient descent with respect to $z_t$ but updates $z'_{t-1}$. This choice could be a reasonable heuristic but is not discussed or justified, which leaves questions about its validity.\n\n**Lack of Quantitative Evaluation** The paper only qualitative results without quantitative evaluation metrics when compared with recent baselines such as ControlNet and T2I-adapter. In particular, the performances with respect to ControlNet should be carefully assessed.  \n\n**Inadequate Dataset and Modality Descriptions** The datasets used and the conditioning modalities are briefly presented. A more detailed description of the datasets, along with the rationale for their selection, would be beneficial. \n\n**Missing Ablations** A more in-depth exploration of the impact of the weights $\\delta_c$ in equation (4) would offer valuable insights. The stability of these parameters is critical as their tuning could rapidly be cumbersome.\n\n**Inconsistent Results** The results in Table 3.b appear to be inconsistent, with FID scores not following the expected pattern. This raises questions about the efficiency of the CMB method and its need for complex hyperparameter tuning. In particular, for a fixed P=30, FID(Q=1)>FID(Q=10)>FID(Q=5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vLkE1vQGhl", "reviewer": "Reviewer_jMgg", "summary": "MaGIC or Multi-modality Guided Image Completion can merge text, canny edge, sketch, segmentation, depth, pose, or any arbitrary combination as guidance for image completion. The authors aim to design a framework that is scalable and flexible. MaGIC has two core components -- a modality-specific U-Net (MCU-Net), and a consistency modality blending (CMB). The MCU-Net will be individually fine-tuned under each single modality, in the first stage. Then, to achieve multi-modality guidance, the CMB algorithm flexibly aggregates guidance signals from any combination of previously learned MCU-Nets. The MCU-Net is similar to T2I-Adapter, composed of a standard U-Net denoiser from the pre-trained Stable Diffusion (SD) and an encoding network which injects a single modality guidance into the U-Net to attain single-modality guidance. The CMB leverages guidance loss to gradually narrow the distances between intermediate features from SD pre-trained U-Net and multiple MCU-Nets during denoising sample stage. This ensures that the SD U-Net features do not deviate too much from the original feature distribution during multi-modality guidance. CMB is training-free and allows for the flexible addition or removal of guidance modalities, avoiding cumbersome re-training and preserving the feature distribution of the original SD U-Net denoiser.\n\nTo verify MaGIC, the authors conduct experiments on image inpainting, outpainting, and editing using the COCO, Places2, and in-the-wild data.", "questions": "Since the only contribution of this paper is CMB, I would suggest to have a very detailed comparison with respect to T2I-Adapter, ControlNet, and many more (for multiple modalities).\n\nApart from just a few qualitative results and some incremental metrics improvement, why do you think Converse Amplification (or simply a variant of classifier guidance) a better approach than zero-convolutions with ControlNet?\n\nAlso, can you add some failure cases of CMB? This is important to give a better idea of where ControlNet lacks and where CMB lacks (I understand CMB can be coupled with ControlNet or T2I-Adapter).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "I like the extension of classifier guidance to multiple modalities that too training-free. Similar techniques has been explored in other single-modality context like in Sketch-Guided Text-to-Image Diffusion Models, but extending to multi-modal case is a nice extension.\n\nThe qualitative comparisons are very intuitive (especially with T2I-Adapter and ControlNet). The overall presentation is reasonable and easy to follow.\n\nThe authors included substantial appendix sections, detailing several architectural details like the design of MCU-Net.", "weaknesses": "While this is an interesting piece of work, I have some big gripes (please let me know if I understood it wrong):\n\nIn related work section (page 3 last paragraph), the authors give the impression that T2I-Adapter (and ControlNet) \"fails to simultaneously use multi-modality as guidance\". This is completely wrong. I understand that T2I-Adapter do not explicitly train jointly for multiple modalities, but they can combine multiple modalities (see section 4.3.2 in T2I-Adapter).\n\nSecond, the authors need to provide a solid justification why a simple \"feature-level addition\" mentioned in Page-6 paragraph-2 is not good. T2I-Adapter (in broad terms) does exactly that. I would need more detailed experiment and intuition comparing both T2I-Adapter and ControlNet for multi-modality case.\n\nThe argument given by authors \"denoiser is trained solely on the distribution of $$\\hat{F}_{c} = F_{enc} + F_{c}$$ \" makes even less sense when you consider ControlNet with its zero-convolutions. In ControlNet we get plausible generation from the starting iterations, thanks to zero-convolutions and with training the conditioning branch becomes good. Hence, the distribution mismatch should not really be an issue to begin with.\n\nApart from my major concerns above, there are some minor corrections/concerns:\n1. I think Eq. 4 should be $$ l (\\hat{F}_{C}, F_{*}) = \\frac{1}_{L} \\sum_{l=0}^{L} \\sum_{i=1}^{N} \\delta_{c} || \\hat{F}_{c}^{l} - F_{*}^{l} ||_{2}^{2} $$ (Note: subscript is C and not c?)\n\n2. Typos e.g., 2nd last paragraph just after equation 2 \"Denoising\"\n\n3. The MCU-Net is basically T2I-Adapter. I do not see any reason to have a new name for it (only to rebrand something and create more confusion). On the other hand, given MCU-Net is same as T2I-Adapter, the only merit of this paper is CMB.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GGg7CqEjtS", "reviewer": "Reviewer_VxR9", "summary": "This paper proposes a multi-modal approach for image completion with LARGE missing regions. The different modalities, such as depth, edge, sketch, pose, provide complementary information for plausible completion. The approach does not require training.", "questions": "As in weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Dealing with LARGE missing regions is a critical task in image completion. This topic is of broad interest in the ML and image processing community.\n2. The idea of leveraging multiple resources is nice though not ground-breaking novel. Making is scalable and flexible is the key, which is solved by two stage approach: modality oriented conditional network and across-modality blending.\n3. The approach is integrated into the diffusion process neatly and training-free.\n3. The paper is very well written and easy to follow, with good illustrations. \n4. The results are convincing with well-planned experiments, which also demonstrate good image generation results beyond completion", "weaknesses": "1. I'm not fully convinced that different image channels/features, such as depth, sketch, edge, could be called modality.\n2. The fair comparison is not easy since most SOTA are not considering multiple resources in the same time. It'd be nice to share some insight into this, and share failure cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)