EdiBERT: a generative model for image editing
Authors: Thibaut Issenhuth
Ugo Tanielian
Jeremie Mary
David Picard
Venue: Accepted by TMLR ()

Abstract: Advances in computer vision are pushing the limits of image manipulation, with generative models sampling highly-realistic detailed images on various tasks. However, a specialized model is often developed and trained for each specific task, even though many image edition tasks share similarities. In denoising, inpainting, or image compositing, one always aims at generating a realistic image from a low-quality one. In this paper, we aim at making a step towards a unified approach for image editing. To do so, we propose EdiBERT, a bidirectional transformer that re-samples image patches conditionally to a given image. Using one generic objective, we show that the model resulting from a single training matches state-of-the-art GANs inversion on several tasks: image denoising, image completion, and image composition. We also provide several insights on the latent space of vector-quantized auto-encoders, such as locality and reconstruction capacities. The code is available at https://github.com/EdiBERT4ImageManipulation/EdiBERT.

Published in Transactions on Machine Learning Research (12/2022)
EdiBERT: a generative model for image editing
Thibaut Issenhuth
Lissenhuthherricco.com
Criteo Al Lab, Paris, Pronce LIGM, Ecole des Ponts, Unio Gustave Biffel, CNRS, Morne-la- Vallée, France
Ugo Tanielian
u. w.temiekkaneerite.ccom
Criteo AI Lab, Paris, Prance
Jérémie Mary
j. m.ary@eriteo..om
Criteo Al Lab, Paris, Prance
David Picard
david picard
LIGM, Boole des Ponts, Univ Gustove biffel, CNRS, Morne-la- Vallée, France
Reviewed on OpenReview: https: // openreview. net/ per/forum? id= CRBb CRBbbb3331 tkW3Lp
Abstract
Advances in computer vision are pushing the limits of image manipulation, with generative models sampling highly-realistic detailed images on various tasks. However, specialized model is often developed and trained for each specific task, even though many image edi- always
tion tasks share similarities. In denoising, impainting, aims at generating a realistic image from low-quality one. In this paper, we aim at making a step towards unified approach for image editing. To do so, we propose Ed- iBERT, a bidirectional transformer that re-samples image patches conditionally to a given image. Using one generic objective, we show that the model resulting from a single training matches state-of-the-art GANs inversion on several tasks: image denoising, image com- pletion, and image composition. We also provide several insights the latent space of voetor-quantized auto-cncoders, such as locality and roconstruction capacitios. The code is available at https:/gghhllloooo////BBBBttllle
image compositing,
1
Introduction
Significant progress image generation has been made in the past few years, thanks notably Generative Adversarial Networks (GANs) (Goodfellow et al.) 2014). For example, the StyleGAN architecture (Karras ct al.
2019 2020b) yiclds state-of-tho-art results in data-driven unconditional generative image modeling.
Empirical studies have also shown the usefulness of GANs' architecture when it comes to image manipulation. By following specific directions in the latent space, one can modify an image attribute such gender, age, the pose of a person (Shen et al. (020),, or the angle (Jahanian et al. 2019). However, since the whole picture is generated from a Gaussian vector, changing some undesired elements while keeping the others frozen is difficult. To solve this problem, edition algorithms involving optimization procedures have been proposed (Abdal et al., 2019 2020) but with one main caveat: the results are not convincing when manipulating complex visuals (Niemeyer & Geiger 2021 ) (cf. experimental section for visual results).
Independently, Van den Oord et al. 2017 propose VQVAE, promising latent representation by training /dcoder using discrete latent space. The authors demonstrate the possibility to embed images in sequences of discrete lokens borrowing ideas from vector quantization (VQ), paving the way for the generation of images with autoregressive transformer models (Ramesh et al., 2021 Esser et al.) 2021b). Building on this littcrature, we argue that one of the benefits of this representation that each token the sequence is mostly coding for localized patch of pixels (see section 3.5 thus opening the possibility for an efficient localized latent. edition.

---

Published in Transactions on Machine Learning Research (12/2022)
Denoising
Completion
Compositing
Scribble-edit
Crossover
Figure 1: Using single and straightforward training, EdiBERT can tackle wide variety of different tasks in image editing. In this image, the top row the input, while the second and third rows are different samples from EdiBERT, showing realism, consistency, and variety.
Aiming to build unified approach for image manipulation, we propose a method that leverages both the spatial property of the discrete vector-quantized representation and the use of model that performs attention on the whole image. To we train a bi-directionnal transformer network based on ideas from the language model BERT (Devlin et al. 2019), naming EdiBERT the resulting model. During training, EdiBERT tries to recover the original Lokens of a perturbed sequence through a bidirectional attention schema. In computer vision, this approach has mainly been studied the context of self-super representation learning (Bao et al. 2022 He al. (022). We advocate that training a single model using this generic objective provides sounded way obtain model able tackle several editing tasks. Finally, to practically handle these tasks, also derived two different sampling algorithms: dedicated for image denoising and editing, and second one for inpainting.
To better visualize the abilities of EdiBERT after single training, we show in Figure that the same model can now be used many different image manipulation tasks such as denoising, inpainting (or completion), compositing, and craseble-based editing.
To sum up, our contributions are the following:
We analyze the VQ latent representations and illustrate their spatial properties, and show how improve the reconstruction capabilities of VQGAN, using post-processing procedure that better recovers the pixel content outside of the edited region.
2

---

Published in Transactions on Machine Learning Research (12/2022)
We show how to derive two different sampling algorithms from single bidirectional transformers: one for the task of image denoising where the locations of the edits are unknown, and a second one for inpainting or image compositing where the mask specifying the area to edit is known.
Finally, we show that using this generic simple training algorithm along with its companion post- processing allow us to achieve competitive results on various image manipulation tasks.
2
Related work
We start this section by introducing transformer models for image generation. Then, we motivate the use of the VQ representation and bidirectional models for image manipulation.
2.1 Autoregressive image generation
The use of autoregressive transformers in the field generative modeling (Parmar et al. 2018 has been made possible by two simultaneous research branches. First, the extensive deployment, of attention mechanisms such as non-local mcans algorithms (Buades al.
2005 non-local neural networks (Wang et al.) 2018), and
also attention layers in GANs (Zhang al.) 2019 Hudson Zitnick] 2021). Second, the development. of
both classifiers and generative models sequentially inferring pixels via autoregressive convolutional networks
such as PixeICNN (Van den Oord et al. 2016a b). The self-attention mechanism (Vaswani et al. 2017).
which now become ubiquitous in computer vision, is quickly recalled here: sequence X E RL*a, where L is length of the sequence, is mapped by position-wise linear layer to a query Q E Rxdd, key K E IR Lxdx and a value gLxd, The self-attention layer is then:
QK* IV E Rxd,
aQt K, V) = softmax
(1)
If autoregressive transformers allow a principled log-likelihood estimation of the data, attention layers have a complexity scaling with the square of the sequence length, a clear bottleneck to scale to high-rcsolution images. To reduce the size of these sequences, Van den Oord et al. (2017) proposed the use of discrete representation. In this framework, an encoder F., a decoder D, and a codebook /alebook//ictioonary Z are learned simultaneously represent images with single sequence of tokens. [sser et al. (2021b) later trained autoregressive model on these token sequences, stressing that high-capacity transformers generate realistic high-resolution images. The framework consists of three steps:
1. Training simultaneously set of encoder /dt////ee//odebooodeoooo (E, D. Z), by combining reconstruction. commitment and adversarial losses. The reconstruction loss is a perceptual distance (Zhang et al. 2018). The commitment loss (Van den Oord et al. 2017) pushes the codebook towards the output of the encoder using a quantization loss. The adversarial loss is the Vanilla GANs loss defined (Goodfellow et al. 2014). The training objective becomes
E D"D arg min [Iree. (E, D, + Leommit. (E, + XLnav. (eEav.((E,D (E, E,D.Z
(2)
Training an autoregressive transformer to maximize the log-likelihood of the encoded sequences.
3. At inference, sampling a sequence with the transformer and decoding it with the docoder D.
This vector-queet qurtti- representation was later improved by Yu et al. 2021a) and used by Yu et al. 2022 to create PARTI, statc-of-tho-art text-to-image generative model. Interestingly, our work EdiBERT builds on top of the first step of VQGAN, and also requires the training of the triplet (E, 2) following equation 2
2.2 Bidirectional attention
The main property of autoregressive models that they only perform attention on previous tokens, making them inadequate when dealing with image manipulation (Esser et al. 2021a) Some works alleviate this bias
3

---

Published in Transactions on Machine Learning Research (12/2022)
2019 learn an autoregressive model random permutations of the ordering. Cao et al. ) propose model where missing tokens are inferred autoregressively, conditionally to the set of kept tokens. Similarly, Wan et al. (2021) use an auto-regressive procedure conditioned on the masked image, while Yu et al. (2021b) use BERT training with [MASK] tokens and Gibbs sampling. If this setting is ideal for lasks with masked lokens such as inpainting, makes ill-posed for scribble-editing and insertion without existing paired datasets. On the opposite, our EdiBERT tackles all tasks without any need for supervision. Finally, [sser al. (2021a) train ImageBART, a multinomial diffusion process (Hoogeboom) et al. 2021) in the discrete latent space of VQGAN. Each generated sequence is conditioned on the previous one and performs attention to the whole image. However, this method is computationally heavy since it requires making N x L inferences, where N is the number of generated sequences and L is the mumber of tokens in the sequence. A more efficient. way to perform bidirectional allention for image generation ion has been proposed in MaskGIT |Chang et al. 2022). MaskGIT consists of training with a BERT-like objective (Devlin et al. 2019) on sequences randomly perturbed with [MASK] tokens, and generating images with parallel decoding scheme. Similarly, Zhang et al. (2021) propose to use a masking-based strategy to perform conditional image editing with bidirectional attention mechanisms. However, they still require specific conditional data to learn their model editing model. In this paper, argue that by performing bidirectional attention over all the tokens and learning with denoising objective (tokens perturbed by randomization, not with [MASK] tokens), it is possible to train a single model tackling many editing tasks.
in different ways. Yang et al.
2.3 Unifying image manipulation
Initially, image manipulation methods were implemented without any trainable parameters. Image comple- tion was tackled using nearest-neighbor techniques along with large dataset of scenes (Hays & Efros) 2007) As to image insertion, blending methods were widely used, such as the Laplacian pyramids (Burt & Adelson 1987). In recent years, image manipulation has benefited from the advances of deep generative models. A first line of work has consisted of gathering datasets of corrupted and target images to train conditional gencrative models. By doing so, one can therefore learn a mapping from any corrupted image to a real onc. For example, Liu et al.) (2021) proposes an encoder-decoder archilecture for skelch-guided image inpainting. However, in all cases, dataset with both types of images is required, therefore limiting the applicability.
To avoid this dependency, a second idea known as GAN inversion methods leverages pre-trained un- conditional GANs. They work by projecting edited images on the manifold of real images learned by the pre-trained GAN. It can be solved either by optimization (Abdal et al. 2019 2020 Daras et al.) 2021], 1, or with an encoder mapping to the latent space (Chai et al.) 2021 Richardson et al. 2021 Tov et al. 2021 Pros of these GAN-based methods are that one benefits from the outstanding properties of StyleGan, state- of-the-art in image generation. However, these methods rely a. task-specific loss function that needs to be defined and optimized. More recently, another line of rescarch is based on the development of scoro-bascd models (Song et al. 2020): Meng el al.] (2022) use Langevin's 's dynamics for image edition and, {Esser el. al., 202 combine discrete diffusion models (Hoogeboom et al. 2021 Austin et al. 2021 with the discrete vector-quantized reprosentations from VQGANs.
3
Motivating EdiBERT for image editing
This section gives global description of the proposed EdiBERT model. We start with notations before describing the different steps leading to the BERT-based edition.
3.1 Discrete auto-encoder VQGAN
Let be an image with width w, a height h, and a number c of channels. I thus belongs to hxxxxe Let (E, D, Z) be respectively the encoder, decoder, and codebook defined in VQVAE and VQGAN (Van den Oord al. [2017] Esser et al., (2021b). The codebook consists of a finite number of tokens with fixed vectors in an embedding space: Z (4,,-.,,4>) (N) with l E Rd and N being the cardinality of the codebook.
4

---

Published in Transactions on Machine Learning Research (12/2022)
For any given image the encoder E outputs vector E(I) R lxd, which is then quantized and reshaped into sequence 8 of length I. as follows:
8 = (arg min [[[() - ........ arg min [E([) - =
ZEZ
=E2
where E(T), = I( E Rd is the feature vector of E(T) at position I, and Qz refers to the quantization operation using the codebook Z. Recall that, after the quantization step, one gets a sequence composed of L codebook elements, thus 8 Zh. After we feed the codebook embeddings to the decoder D, the reconstructed image becomes i P(Qz(E(T)))
Let's note D, the available image dataset. From a pre-trained encoder E and codebook Z, one can transform the image dataset D into a dataset of -okunnseeuences Dş == {Qz (P()), I D). When learning transformers on sequences of tokens, the practitioner is directly working with Dz-
3.2 Learning sequences with autoregressive models
The following sections aim at motivating the training objective for the EdiBERT model. To begin with, let Po be a transformer model parameterized with trained on Ds. For each position i in 8, we note (-1s), the modeled distribution of tokens conditionally to s.
When training autoregrossive transformer the discrete sequences of tokens Ds (Esser al., 2021b one needs to compute the likelihood po(s) of each given sequence ($1,...,81)) E Ds as follows:
po(s) = II ph (s.(se))) with Sex S41,1..,$-11..
(4)
Conditional distributions (s:(s<))) are computed using a causal left-to-right attention mask. The final objective of the autorogrossive model is to find the best set of parameters within e:
arg max ESED, logro(s).
(5)
lee
Limitations of the model. If this setting is well suited for unconditional image generation, it is ill-posed for image manipulation tasks, as shown by Esser et al. (0211a). the case ccribble-based editing, or inpainting, one wants to resample tokens conditionally to the whole image, so that the model has all the information its disposal.
3.3 A unique training objective for EdiBERT.
Let define the training objective for EdiBERT. For a sequence s (s1, 82), function oo randomly selects subset of & indices (fP,,..,, } where yp < L. At each selected position yi; perturbation is applied the token Sp., We attribute random token with probability p, or keep the same token with probability Consequently, the perturbed token 8p, becomes:
&pp = U(Z) with probability p.
&p. = Spc with probability 1-p.
where U(Z) refers to the uniform distribution the space of tokens Z. Similarly to Bao al. (022 the sampling function vp is defined with a 2D selection strategy, and the positions are selected by drawing random 2D rectangles, see in Figure 2 Contrarily to Bao et al. (2022) and Devlin et al. (2019), we only use random tokens from the codebook but no [MASK] tokens. We argue this setting corresponds more to the cases of denoising and editing, where tokens have to be sampled conditionally to an entire perturbed sequence.
Let us now call 5 (s1,...., S S s1.) the perturbed sequence, and D. (8, 8 E DD the perturbed dataset. The training of EdiBERT optimizes the following objective
k
arg max ESED, k Bee
Σ
log p4 (spp18))
(6)
5

---

Published in Transactions on Machine Learning Research (12/2022)
2
3
16
3
16
7
Encoder quantization
6
31
24
randomization
8
31
24
Flattening
16
15
Bidirectional transformer
Figure 2:
The 2D selection and randomization strategy for the training of our bidirectional transformer: EdiBERT is trained on sequences where localized patch of tokens have been perturbed.
Contrary to equation note that the objective in equation does not require causal left-to-right attention. Instead, the attention can be performed over the whole input sequence.
Wang & Cho 2019 show that it is possible to generate realistic samples with a BERT model starting with random initialization. However, compared with standard autoregressive language models, the authors stress that BERT generations are more diverse but of slightly worse quality, Building on the findings of Wang & Cho (2019), we do not aim use BERT for pure unconditional sequence generation but rather improve already existing sequence of tokens. In our defined EdiBERT model, for any given position i s, token will be sampled according to the multinomial distribution p (-|s).
Sampling from EdiBERT:
3.4 On the locality Vector Quantization encoding
In this paper, we argue that one of the main advantages of EdiBERT comes from the VQ latent space proposed by Van den Oord et al.] (2017) where each image encoded in a discrete sequence of tokens. Tn this section, we illustrate with simple visualizations the property of this VQGAN encoding. We explore the spatial correspondence between the position of the token in the sequence and set of pixels for the encoded image. We aim at answering the following question: do local modifications of the image lead to local modifications of the latent representation and vice versa?
Modifying the image. To answer this question, images are voluntarily perturbed with grey masks (i - im). Then, we encode the two images, quantize their representation using pre-trained codebook, and plot the distance between the two latent representations: [[Q)(E(1)) (z(((()))) The results are shown the first row in Figure 3 Due to the large receptive field of the encoder, tokens can be influenced by distant parts of the image: the down-sampled mask does not recover all of the modified tokens. However, tokens that are largely modified are either inside, very close to the down-sampled mask.
Modifying the latent space. To understand the correspondence between tokens and pixels, we stress how one can easily manipulate images using the discrete latent space. In Figure 3 we show that cutting a specific area of a source image to insert it in a different location another image possible only by replacing the corresponding tokens in both sequences. This spatial correspondence between VQGANs' latent space and the image interesting for localized image editing tasks, í.e. tasks that require modifying only subset of pixels without altering the other ones.

---

Published in Transactions on Machine Learning Research (12/2022)
Modifying the latent space via the image
Modifying the image via the latent space
Figure 3: Each token in the sequence is tied to small spatial area in the decoded image. In the 1st row: we voluntarily perturb images and display the variations among the tokens in the latent space. The heatmaps represent the distance (red is high) between the tokens of the original image and the tokens of the perturbed image. In the 2nd row: we stress how collages of images can easily be done with this discrete latent representation: third and fourth images are generated by the decoder from latent space collage.
3.5
On the reconstruction capabilities of Vector Quantization encoding
A limit of the framework resides in the of the vector quantization operation and the induced loss of information. Indeed, we observe in Figure g that VQGAN struggles to reconstruct high-frequency details, for example complex backgrounds on FFHQ dataset (Karras et al. 2019). To improve the reconstruction capabilities of VQGANs, we propose a simple optimization procedure over the latent space vectors.
The objective is to find the latent vectors that minimize the LPIPS (Zhang et al. 2018 between the larget image and the decoded reconstruction. We initialize the procedure from the output of the encoder E(T), and optimize the objective with gradient descent. Figure f shows how this procedure improves the inversion capabilities of VQGAN to make it better than GAN inversion methods (Abdal et al. 2020 A potential explanation of the limited reconstruction capabilities of VOGAN is displayed in Figure the latent vectors of the codebook might suffer from a very low rank. The optimization procedure seems to solve this since the latent vectors span much more dimensions of the embedding space after a few hundred optimization steps.
4 Image editing with EdiBERT
Baselines. For each task, we run comparisons with baselines and state-of-the-art models based on GANs inversion mothods. On FFHQ, we compare ImageStyleGAN2++ (Abdal et al., 2020) on pre-trained StyleGANs: StyleGAN2 (Karras et al. 2020b) and StyleGAN2-ADA (Krrras et. al. 2020a). Besides, we run the solution proposed by Chai et al. (2021) where StyleGAN2 model is inverted using trained encoder. Finally, we use In-Domain GAN (Zhu et al. 2020). hybrid method combining an encoder with an optimization procedure minimizing reconstruction losses. We also compare to Co-Modulated GANs ZZao conditional GAN for inpainting.
et al. 2020
Metrics. We follow the work of Chai et al. (2021) and use metrics assessing both fidelity and distribution fitting. The masked L1 metric (Chai et al. 2021) mcasures the closeness between the generated image and the source image outside the edited areas. The density/coverae metrics (Naeem et al. 2020 robust

---

Published in Transactions on Machine Learning Research (12/2022)
Original.
VQGAN + optim.
VQGAN.
Id-GAN (av.LPIPS 0.164)
12SG 14 (av. LPIPS= 0.345)
(av. LPIPS= 0.085) (av. LPIPS= 0.232)
Figure Comparison of reconstruction capabilities of VQGAN optimization to two GANs inversion methods such Id-GAN (Zhu et al.) 2020 and 12SG! Abdal et al. 2020 Averaged LPIPS are computed the validation set FFHQ.
1.0
0100
Cumulative variance
0.8
018
L91S
Loss
016
0.090
0.4
012
044
0.080
Codebook latent vectors
010
optim. steps
0.08
75
100
125
150
175
200
100
Optimization steps
Num. dimension
Figure 5: Analysis reconstruction capabilities VQGAN. On the left, we see that both the L1 and percep- tual loss (LPIPS) between original and reconstructed images significantly docrease when optimizing LPIPS the latent, vectors of VQGAN. This may be consequence of higher number of dimensions spanned by the latent vectors (on the right), after the optimization (allowing for more complex reconstructions).
versions precision/recall metrics. Intuitively, density measures fidelity while coverage measures diversity. Finally, the FID (Heusel al.] (207)) quantifics the distance between generated and target distributions. Moreover, we perform user study on FFHQ image compositing, More details and quantitative results on LSUN Bedroom presented in Appendix.
4.1 Localized image denoising
Image denoising aims to improve the quality of a pre-generated image or improve a locally perturbed one. The model has to work without information the localization of the perturbations. This means need
8

---

Published in Transactions on Machine Learning Research (12/2022)
Figure 6: Image denoising with EdiBERT: the color in the 4 different heatmaps proportional to the negative likelihood of the token. Tokens with lower likelihood appear in red in the heatmap and have higher probability of being sampled and edited. Consequently, conditional distributions output by EdiBERT are an efficient tool to detect anomalies and artifacts in the image.
to find and replace the perturbed tokens with more likely ones to recover realistic image. Thus, given a sequence 8 = (81,...,81.) we want to:
1. Detect the tokens that do not fit properly in the sequence
2. Change them for new tokens increasing the likelihood of the new sequence.
We desire a significantly more likely sequence with few possible token amendments. To do so, we measure the likelihood of each token s; based the whole sequence s, aiming to compute p(s,ss, and replace the least-probable tokens considering them independently. That is, propose to use the conditional probability output by the model in order to detect and sample the less likely odd tokens. Some examples of image denoising are presented in Figure 0 and we observe that EdiBERT is able to detect artifacts and replace them with more likely tokens. The full algorithm is given in the Algorithm
Algorithm Image denoising
Requires: Sequence (81,...,88.) BERT model pe, number of iterations T; for iterations in [0, Тj do
Compute P; logit( (ogit(-Pp(sssss (s.)s)), Vi [1, [.]
Sample p (Pı,...,Pl) Pi) (less likely position):
Sample E Z ~ pb((-s))
Insert sampled token: + t end
Image + Decoder(s);
Result: Image
4.2 Image inpainting
In this setting, we have access to a masked image im Rhxuxe along with the location of the binary mask m E B im has been obtained by masking an image i E gixwxe as follows: im = i m with O pointwise multiplication. The goal of image inpainting is to generate an image i that is both realistic (high density) and conserves non-masked parts, that is i (1 (1- m) = 1 (1 m).
Among the different tasks in image manipulation, image inpainting stands out. Indeod, when masking a specific area of an image, shall not consider the pixels within the mask to recover the target image.
9

---

Published in Transactions on Machine Learning Research (12/2022)
Ref. image
EdiBERT
LC
12SG 1++
Com-GAN
Figure 7: Image inpainting comparisons on FFHQ. EdiBERT performs better than inversion methods such as LC (Chai al. (021) and I2SG (Abdal al. 2020 Note that Com-GAN (Zhu et al. 2020 is specialized for image inpainting and trained on pair datasets (masked image, target image), can not perform other image editing tasks.
10

---

Published in Transactions on Machine Learning Research (12/2022)
The image impainting task thus requires specific care to reach statc-of-the-art performance; this is why we added five different elements to our approach, and validated these elements with visual results in Figure 13.
1. Randomization: to crase the mask influence, the tokens within the mask are given random values.
2. Dilation of the mask: as shown in Figure 3 some tokens outside of the down-sampled mask in the latent space are also impacted by the mask on the image. Modifying only tokens inside the down-sampled mask might not be enough and could lead to images with irregularities on the borders. As a solution, we apply a dilation on the down-sampled mask and show in Figure 13 that helps better blend the larget image's completion since the boundaries are removed.
3. Spiral ordering: since there is pre-defined ordering of positions in EdiBERT, one can look for an optimal sampling of positions. We argue that by sampling positions randomly, one does not fully leverage the spatial location of the mask. Instead, we propose spiral ordering that goes from the border to the inside of the mask. Qualitative and quantitative results in Figure 13 and Table 2 confirm the advantage of this ordering.
Periodic image collage: to preserve fidelity to the original image, we periodically perform a collage between the masked image and the decoded image. We observed in Figure 13 that without this collage trick; the reconstruction can diverge too much from the input image.
5. Online optimization on latent sequences: to improve fidelity to the masked image im. the final stage of the algorithm consists in an optimization procedure on the latent sequence Rixwxd The objective function is defined as:
- 0m) + L L= (m)
-((ss))))(((()) @ m))
(7)
where Ip is a perceptual loss (Zhang et al. 2018), and 8 the initial sequence from EdiBERT. Intuitively, the first term enforces the decoded image to get closer to the masked image im, while the second term is regularization enforcing the decoded image to stay similar to the completion pro- posed by the transformer's s likelihood. We illustrate in Figure and Figure 13 that the optimization leads to a better- presered source image.
Analyzing the results: we see from Table I and Figure 7 that the specialized method com-GAN (Zhao et al. 2020) outperforms non-specialized methods on image inpainting. This was expected since it is the only method that has been trained specifically for this task. Note that the trained model co-mod GAN cannot be used any other image manipulation task. Compared with the non-specialized method, EdiBERT always provides better fidelity to the source image (lower Masked LI) and realism (best FID and top-2 density). An ablation study is available in Table 2 and validates our choices. Finally, more details regarding the sampling algorithm for the task of inpainting are given in Appendix.
4.3 Image composition
In this setting, we have access to a non-realistically edited image ig E hxxxx, The edited image ie is obtained by a composition between a source image ig E hxwxe and target image in E kxwxx The target image can be a user-drawn scribble or another real image in the case of image compositing. Besides, pixels are edited inside a binary mask m e k which indicates the areas modified by the user. Thus, the final edited image is computed pointwise as:
ie - m).
(8)
Image composition aims to transform an edited image ie to make it more realistic and faithful without limiting the changes outside the mask. We note the source image i, outside the mask and the edits of the target image im for the inserted elements in the edition mask. Three tasks fall under this umbrella: scribble-based ediling, image compositing, and image crossovers.
11

---

Published in Transactions on Machine Learning Research (12/2022)
Masked
EdiBERT
(c)
Figure 8: Ablation study for inpainting. Components removed are (a) optimization, (b) dilation, (c) ran- domization, (d) collage, (e) spiraling (random order instead). Optimization improves fidelity to the source image, while the other components help increase image quality.
Table 1: Image inpainting and compositing on FFHQ 256 256. Com-GAN model specific for image inpainting, ID-GAN handles several editing tasks but not inpainting, while other methods handle both. We remove 12SG++ from the user study, since 2SGG++ is the same method with better GAN backbone, d.e. StyleGAN2-ADA (Karras et al. 2020a). Bold: 1 rank, blue: 2nd rank.
Inpainting
Compositing
User study
Masked L1
Dens. Cover. Masked Dens.
FID
L1
12SG++
Abdal et al.
2020
0.0767
23.6
0.99
0.88
0.0851
0.77
I2SG
Abdal et al.
2020
0.0763
22.1
1.25
0.91
0.0866
1.07
8.3%
I.C
Chai
2021
0.1027
27.9
1.12
0.84
0.1116
1.00
14.8%
EdiBERT
0.0290
13.8
1.16
0.98
0.0307
0.94
61.2%
Com-GAI GAN
Zhu et al. 2020
0.0086
10.3
1.42
1.00
ID-GAN
Zhu et al.
2020
0.0570
0.75
15.7%
12

---

Published in Transactions on Machine Learning Research (12/2022)
Results of image compositing FFHQ are presented in Table 1 and Figure EdiBERT always has the lowest. masked L1. We also present the results from a user study in Table 1 30 users were shown 40 original and edited images, along with four results (EdiBERT and baselines). They were asked which one is preferable, accounting for both fidelity and realism. The survey shows that on average, users prefer EdiBERT over competing approaches. We give more visual results along with the detailed answers of the user study in Appendix.
Table 2: Inpainting: Ablation study on the components of EdiBERT sampling algorithm. EdiBERT (Ist row) shows the best tradeoff between fidelity (masked L1) and quality (FID, density density/coveragge /coverage). Bold: 1 rank, blue: 2nd rank.
Optin-- Random-
Collage Dilation Masked
FID
Density Coverage
ization ization
L1
f
Spiral
0.0201
19.4
1.14
0.96
Random
0.0206
20.7
1.13
0.95
Spiral
X
/
V
0.0299
20.3
1.20
0.94
Spiral
X
V
V
0.0198
20.5
1.26
0.92
Spiral
V
0.0252
19.9
1.11
0.95
Spiral
0.0197
23.3
0.96
0.91
5
Discussions
EdiBERT is a bidirectional transformers model that can lackle multiple editing lasks from one single training, One of the key elements of the proposed method is that it does not require having access to paired datasets (source, target), or unpaired image datascts. This property shows how flexible EdiBERT is and why it can easily applied to different lasks. Overall, the proposed Iramework simple and tractable: 1) train a VQGAN Esser et al. (2021b), 2) train an EdiBERT model following the objective defined in equation
Interestingly, for simple applications, by the VQGAN pre-trained on ImageNet. However, for more complex data when dealing with multiple domains, one might have to train specialized codebook, which requires a large auto-encoder and lot of data. Another EdiBERT's drawback is related to the core interest of image editing, Since the tokens are predominantly localized, EdiBERT is perfectly suited for small manipulations that only require amending few numbers of tokens. However, some manipulations such as zooms or rotations require changing large areas of the source image. In these cases, modifying a large number of tokens might be more demanding.
directly train EdiBERT based the representations output
Broader Impact Statement
Similarly to other image generative models, EdiBERT might be used to create and propagate fake beliefs via deepfakes, as discussed in Fallis (2020).
6 Conclusion
In this paper, we demonstrated the possibility to perform several editing tasks by using the same pre-trained model. The proposed framework simple and aims al making a step towards a unified model able do any conceivable manipulation task on images. An exciting direction of research would be extend the editing capabilities of EdiBERT to global transformations (e.g. zoom, rotation).
13

---

Published in Transactions on Machine Learning Research (12/2022)
DD
Source
Composite
EdiBERT (*)
ID-GAN
12SG1+
Source
Composite
EdiBERT (+)
ID-GAN
12SC
Figure 9: Scribblo-based editing and image compositing: comparison with ID-GAN Zhu ct al. 2020 and I2SG AAbal el al. 2019). EdiBERT preserves better the fidelity the source image while being also able fit the inserted object properly. This confirms the quantitative results in Table EdiBERT seems to be leading in both fidelity and realism.
14

---

Published in Transactions on Machine Learning Research (12/2022)
References
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4432-
4441, 2019.
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan+++ How to edit the embedded images? In Proceedings of the IEEE/CV /CVF Conference on Compuler Vision and Pollern Recognition, pp. 2296-8300 2020.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Newral Information Processing Systems, 34:17981 17993, 2021.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pro-training of image transformers. International Conference on Learning Represenlations, 2022.
Antoni Buades, Bartomeu Coll, and J-M Morel. non-local algorithm for image denoising. In 2005 IEEE Computer Sociely Conference on Computer Vision and Pattern Recognition (CVPR 005, volume 2, pp. 60 65. LEEÉ, 2005.
Peter Burt and Edward H Adelson. The laplacian pyramid as compact image code. Readings in computer vision, pp. 671-679. Elsovier, 1987.
Chenjie Cao, Yuxin Hong, Xiang Li, Chengrong Wang, Chengming Xu, Yanwei Fu, and Xiangyang Xue. The image local autoregressive transformer. In Advances in Neural Information Processing Systems, 2021.
Lucy Chai, Jonas Wulff, and Phillip Isola. Using latent space regression analyze and leverage composi- tionality in gans. arXiv:2103. 10426, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022.
Giannis Daras, Joseph Dean, Ajil Jalal, and Alex Dimakis. Intermediate layer optimization for inverse problems using deep generative models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2421 24322. PMLR, 18 24 Jul 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Votume 1 (Long and Short Popers), pp.
4171 4186, 2019.
Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Information Processing Systems, 34, 2021a.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 12873-
12883, 2021b.
Don Fallis. The epistemic threat of deepfakes. Philosophy & Technology, pp. 1 21, 2020.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
James Hays and Alexci A Efros. Scene completion using millions of photographs. ACM Transactions on Graphics (ToG), 26(3), 2007.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-160000 2022.
M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 66266666666 6637,667774 2017.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. Advances in Neural Information Processing Systems, 14:12454-12465 2021.
Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In Marina Meila and Tong Zhang (ods.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 4487 4499. PMLR, 18 24 Jul 2021. URL http:://rrooo/s mlr.
(0sss//139/nudso21aa.. hatml
Ali Jahanian, Lucy Chai, and Phillip Isola. On the 'steerability' of generative adversarial networks. In International Conference on Learning Representations, 2019.
T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative ad adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Tero Karras, Miia Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12104- 12114, 2020a.
Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pattern Recognition, pp. 1110-8119, 2020b.
Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, Jing Liao, Bin Jiang, and Wei Liu. Defloc- net: Deep image editing via flexible low-level controls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10765-10774, 2021.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2022.
Muhammad Ferjad Nacem, Scong Joon Oh, Youngiung Uh Yunjey Choi, and Jacjun Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference Machine Learning, pp. 7176- 7185. PMLR, 2020.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11453 -1464,, 2021.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pp. 4055-4064 PMLR, 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv:2102. 12092, 2021.
Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen- Or. Encoding in style: a stylegan encoder for image-to-image translation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp.
2021.
16

---

Published in Transactions on Machine Learning Research (12/2022)
Yujum Shen, Jinjin Gu, Xiaoou Tang, and Bolci Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognation, pp. 9243 9252, 2020.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.18456, 2020.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashnik, and Daniel Cohen-Or, Designing an encoder for stylegan image manipulation. ACM Transactions on Graphics (TOG), 40(4):1-14, 2021.
Aaron Van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International Conference on Machine Learning, pp.
PMLR, 2016a.
Aaron Van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu. Conditional image generation with pixelenn decoders. In Proceedings of the 30th Inter- national Conference on Neural Information Processing Systems, pp. 4797-4805, 2016b.
Aaron Van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. Proceedings of the 31st, International Conference on Neural Information Processing Systems, pp. 6309- 6318, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Ziyu Wan, Jingbo Zhang, Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4692- 4701, 2021.
Alex Wang and Kyunghyun Cho. Bert has a mouth, and must speak: Bert as a markov random field language model. arXiv:1902.04094, 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhuldinov, and Quoe V Le. Xlnet: Gen- eralized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. orXiv preprint 2rXiv:21 10. 00.04627, 2021a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to- image generation. XiXiv preprint arXiw:2206.100 2022.
Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse image inpainting with bidirectional and autoregressive transformers. In Proceedings of the 29th ACM International Conference on Mullimedia, pp. 69 78, 2021b.
Han Zhang, Tan Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversar- ial networks. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7354-7363, 2019.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 586 595, 2018.
17

---

Published in Transactions on Machine Learning Research (12/2022)
Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. M6-ufc: Unifying multi-modal controls for conditional image synthesis. or Xio preprint ar11::2105.1221,, 2021.
Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Erie, Chao Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. In International Conference Learning Represeniations, 2020.
Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. European conference compuler vision, pp- 592 608. Springer, 2020.
18

---

Published in Transactions on Machine Learning Research (12/2022)
Appendix
A Implementation details
The code for the implementation of EdiBERT is available on GitHub at the following link FAiBBER
A pre-trained model on FFHQ available on a linked Google Drive. Notebooks to showcase the model have also been developped.
A.1 Training syper-parameters
We use the same architecture than Esser et al. (2021b) for both VQGAN and transformer. On LSUN Bedroom and FFHQ, we use codebook size of 1024. For the transformer, we use model with 32 layers of width 1024.
To train the transformer with 2D masking strategy, generate random rectangles before flattening Q(E(T)) The height of rectangles is drawn uniformly from [0.2 x h, 0.5 x hj. Similarly, the width of rectangles is drawn uniformly from [0.2 x w, 0.5 x w] In our experiments, since we work at resolution 256 x 256 and follow the from Esser et al. (2021b); we have = w = 256/16 = 16.
downsampling factor
Tokens outside the rectangle are usod as input, give context to the transformer, but not for back- used for back-propagation. Prand 90% of tokens inside
propagation. Tokens inside the rectangle the mask are put to random tokens, while Psame 1 Prand 10% are given their initial value. Although we did not perform large hyper-parameter study this paramcter, we feel it is an important one. The lower Prand, the more the learned distributions p-(-ss) will be biased towards the observed token 8;. However, setting Prand = leads to a model that diverges too fast. from the observed sequence.
A.2
Inference
A.2.1 Image inpainting.
We sel. the number of epochs to 2, collage frequency per epoch, top-k sampling to 100, dilation to 1, and number of optimization steps to 200. We apply gaussian filter on the mask for the periodic image collage,
Additionally, we use these two implementation details. 1) We use two latent masks: the latent down-sampled mask latent maskj, and the dilated mask latent. mask2, obtained by a dilation of latent maskj- The randomization is done with latent_maskj, such that no information from the unmasked parts of the image is crased. However, the selection of positions that are re-sampled by EdiBERT done with latent mask2. 2) At the second epoch, we randomize the token value, the position that is being replaced. This is only done for image inpainting.
A.2.2 Image compositing.
We the number of epochs to 2, collage frequency 4 per epoch, top-k sampling to 100, dilation to 1, and number of optimization steps to 200. We apply gaussian filter on the mask for the periodic image collage. Contrarily to inpainting, we do not randomize such that EdiBERT samples stay closer to the original sequence.
The full algorithm is presented below in Algorithm
B Additional experimental results
We give additional comparisons on FFHQ and LSUN Bedroom, for the following tasks: image inpainting in Table 3 image crossovers in Table [ and image composition in Table 5 All these experiments are run on the test-set of EdiBERT. Note that concurrent methods based on StyleGAN2 were trained on the full dataset, which advantages them.
19

---

Published in Transactions on Machine Learning Research (12/2022)
Algorithm 2: Image impanntin/cooppostioon.
Requires: Masked (or edited) image km, mask m, Encoder E, Decoder D, BERT model po, epochs e, periodic collage optimization steps optim steps;
+ QiE(m)))
latent mask < get_gessk mask in latent spacc(()]
if task is impainting then
+ latent mask rand (1 latent mask); end
for in [0,epochs] do
for p in chosen order(latens mask) do Sample token E 7 ~ r-(-11):
Insert sampled token: Sp f
if p%c=0 (collage) then
Encode image post-collage: Elim @lim@mm- mm+ D(s) (1 m));
end end
end
8
s
for i in 10, oplim sleps]: do
== L(D((( im) mm) + Lp ((D(s) D(5") @ (1 m))
t s+
end
Image Decoder(s);
Result: Image
Inpainting. We use 2500 images. On FFHQ, provide results for free-form masks and reclangular masks. The height of rectangular masks is drawn uniformly from [0.4 x h, 0.6 x hi with / = 256, and similarly for the width. For non-rectangular masks generations, we follow the procedure of Chai et al. (2021): we draw a binary mask at low-resolution 6 and uspsample it to 256 x 256 with bilinear interpolation.
The ablation study Table 2 of main paper is performed on free-form masks. Results in Table of main paper are on rectangular masks. On LSUN Bedroom, we provide results for rectangular masks.
Crossovers. We generate 2500 crossovers from random pairs of images, both FFHQ and LSUN Bedroom.
Editing/ /Coiiosiiio...iing........ We create small datasets of 100 images from the test set of EdiBERT for FFHQ scribble- based editing, FFHQ compositing and LSUN Bedroom compositing. A user study on FFHQ compositing is presented in main paper with statistically significant number of votes. We also provide some metrics 5 Because of the small size of the dataset, only report masked L1 and density. For density, the support of the real distribution estimated with 2500 real points, and density is averaged over the individual density of the 100 generated images.
20

---

Published in Transactions on Machine Learning Research (12/2022)
Masked
Density Coverage
FID +
L1 +
FFHQ: rect. masks
I2SG (Abdal ct al.
2020
0.0767
23.6
0.99
0.88
I2SG ++ [Abdal et al
2020
Karras et al.
2020a
0.0763
22.1
1.25
0.91
LC Chai et al. 2021
0.1027
27.9
1.12
0.84
EdiBERT (*)
0.0290
13.8
1.16
0.98
Co-mod. GAN (Zhao et al.
2020
0.0128
4.7
1.24
0.99
FFHQ:
masks
+ (Abdal et al.
2020
0.0440
22.3
0.92
0.89
12SG ++ (Abdal a
2020
Karras
2020a
0.0435
21.1
1.17
0.91
LC Chai et al. 2021
0.0620
27.9
1.22
0.85
EdiBERT (*)
0.0201
19.4
1.14
0.96
Com-GAN Zhao et al
2020
0.0086
10.3
1.42
1.00
LSUN Bedroom: rect.
masks
I2SG (Abdal et al.
2019
0.1125
50.2
0.04
0.04
MaskGTT Chang et al.
2022
0.0209
11.4
1.09
0.96
EdiBERT
0.0288
11.3
0.89
0.97
Table 3: Image inpainting.
Masked
Density Coverage
FID 1
L1 4
FFHQ
12SG++
Abdal et al.
2020
0.1141
29.4
0.97
0.78
T2SG ++ Abdal et al
2020
Karras et al.
2020a
0.1133
26.9
1.35
0.82
ID-GAN
Zhu et al.
2020
0.0631
23.2
0.88
0.83
LC (Chai
el al. 2021
0.1491
31.9
1.17
0.77
EdiBERT
0.0364
19.7
1.05
0.88
LSUN Bedroom
I2SG (Abdal et al. 2019
0.1123
45.7
0.12
0.20
ID-GAN (hu 2020
0.0682
21.4
0.35
0.57
EdiBERT
0.0369
12.4
0.64
0.84
Table 4: Image crossover.
21

---

Published in Transactions on Machine Learning Research (12/2022)
Masked Density
L1 $
FFHQ scribble-edits
12SG++ Abdal et al.
2020
0.7811
0.91
12SGt ++ Abdal et al.
2020
Karras et al.
2020a
0.0777
1.11
ID-GAN Zhu et al.
2020
0.0461
0.79
LC (Chai et al. 2021
0.1016
1.14
EdiBERT (*)
0.0281
0.96
FFHQ compositing
12SG++ (Abdal et al.
2020
0.0851
0.77
12SGI Abdal et al.
2020
Karras et al.
2020a
0.0866
1.07
ID-GAN Zhu al.
2020
0.0570
0.75
LC CChai et al. 2021
0.1116
1.00
EdiBERT
(*)
0.0307
0.94
LSUN Bedroom compositing
12SG (Abdal et al. 2019
021285
0.25
ID-GAN (Zhu et al.
2020
0.0484
1.45
EdiBERT
0.0247
1.49
Table 5: Image editing.
22

---

Published in Transactions on Machine Learning Research (12/2022)
C Baselines
We use the implementation and pre-trained models from the following repositories.
ID-GAN Zhu et al. 2020): http://iihb..cmmmgggeoor/////noo
pytorch, which has pre-trained models
on FFHQ 256x256 and LSUN Bedroom 256x256.
I2SG+ and 12SG++ + (Karras et al. 2020b Abdal al. 2020
We tested projections with the following pre-trained models FFHQ: StyleGAN2 (Karras al. 2020b) at resolution 256x256, and StyleGAN2-Ada (Karras For evaluation, we downsample the 1024x1024 generated
hutps://github.cc
et al. 2020a) at resolution FFHQ 1024x1024.
images to 256x256.
LC (Chai ct al. 2021):
We use the pre-trained encoder and StyleGAN2 generator, for FFHQ at resolution 1024x1024. For evaluation, we downsample the 1024x1024 generated images 256x256.
Com-GAN Zhao al. (2020) https///gitubb.cm//yyyeee /a/p//////zttttoomooaaaaa for image inpainting on FFHQ at resolution 512x512. We downsample the generated images to 256x256 for
We use the pro-trained network
evaluation.
maskgit We use the tokenizer and transformer trained for conditional image general ion and editing on ImageNet 256x256. To perform compar- isons with EdiBERT on L.SUN Bedroom Image Inpainting, we condition the transformer to the ImageNet class '843: studio couch, day bed'.
MaskGIT (Chang et al. 2022
maskitt
D Qualitative results on image compositon
We present more examples of image compositions, with image compositing and scribble-based editing on FFHQ and LSUN Bedroom in Figure T0 and 11
Preservation of non-masked parts. Thanks to its VQGAN auto-encoder, EdiBERT generally better conserves areas outside the mask than GANs inversion methods. This is particularly visible for images with complex backgrounds on FFHQ (Figure 10 5th and last rows).
Insertion of edited parts. Since EdiBERT is probabilistic model and the tokens inside the modified area are resampled, the inserted object can be modified and mapped to more likely object given the context. It thus generates more realistic images, but alter the fidelity to the inserted object. For example, of Figure II the green becomes lighter and the perspective of the inserted window is improved. Although it can be a downside for image compositing, note that this property is interesting for scribble-based editing, where the scribbles have to be largely transformed to get realistic image. Contrarily, GANs inversion methods tend to conserve the inserted object too much, even if results a highly unrcalistic generated image. We can observe this phenomenon on last row Figure 10
23

---

Published in Transactions on Machine Learning Research (12/2022)
Source
Composite
EdiBERT
ID-GAN (2200)
Zhu
al. 22S ++
Abdal al.
2020
Karras
al.
2020a
Figure 10: Image compositing and scribble-based editing on FFHQ.
24

---

Published in Transactions on Machine Learning Research (12/2022)
Source
Composite
EdiBERT
ID-GAN
Zhu
2SG
Abdal
2020
2019
Figure 11: Image compositing on LSUN Bedroom.
25

---

Published in Transactions on Machine Learning Research (12/2022)
Source
Composite
EdiBERT
ID-GAN
Zhu
al. 2S2
Abdal
al.
2020
2019
Figure 12: Image compositing on LSUN Bedroom.
26

---

Published in Transactions on Machine Learning Research (12/2022)
Masked
EdiBERT
MaskGIT
Figure 13: Comparisons of image inpainting on LSUN Bedroom between EdiBERT and MaskGIT.
27

---

Published in Transactions on Machine Learning Research (12/2022)
E Survey on FFHQ image compositing
The survey was presented as a Google Form with 40 questions. For each question, the user was shown 6 images: Source, Composite, EdiBERT, ID-GAN Zhu el al. (2020) 12SG Abdal et al. (2019); Karras et al. (0200a), LCC Chai et al. (2021). The different generated images were referred as Algorithm 1, Algorithm The user was asked to vote for its preferred generated image, by taking into account realism and fidelity criterions. The user had no time limit for the poll. users answered our poll. We provide the detailed answers for each image in Table 6
28

---

Published in Transactions on Machine Learning Research (12/2022)
EdiBERT
ID-GAN
Zhu et al. (2020) LC Chai et al.
(2021) 12SG ++ Abdal et al.
(2019): : Karras et al. (2020a)
17
5
15
1
8
22
4
3
19
4
5
2
22
0
4
4
6
7
8
9
21
1
5
3
23
1
4
20
5
11
13
6
0
27
0
3
0
12
3
3
12
16
4
6
4
25
2
1
2
18
8
1
3
8
13
9
26
0
4
7
0
21
2
14
9
1
6
27
0
1
2
11
19
0
14
9
4
3
16
14
0
21
1
3
5
8
2
18
2
19
3
3
5
22
7
0
23
2
1
18
0
2
10
27
2
1
22
2
1
24
0
5
3
25
2
28
0
2
24
0
6
27
0
3
27
1
2
22
7
9
15
6
14
0
14
2
Total 735 (61.25%)
189
(15.75%)
177
(14.75%)
99 (0.0825%)
Table 6: Detailed results of the user study. Each line corresponds to image, with the associated number of votes per method.
29


Reviews:
[{"review_id": "NBUncnF7Y6", "reviewer": "Reviewer_ANEH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper is well written and easy to follow, the proposed approach is simple. Further, the manuscript does a good job of motivating a bidirectional approach over AR models for image manipulation tasks. Interesting analysis of the latent space of a f16-VQGAN are presented.\n\n**Weaknesses:**\n\n- The proposed approach is effectively making an independence assumption for the masked tokens. The discussion on where this fails too short (only reference I can find is on p.14, \"However, some manipulations such as zooms or rotations require changing large areas of the source image. In these cases, modifying a large number of tokens might be more demanding.\"  \n- Further, the paper builds on AR factorization as a motivation to introduce bidirectional attention, i.e. a generative model that learns a \"full\" density, but itself presents an approach that makes a strong assumption on the density (i.e. independence of individual tokens). Again, advantages and disadvantages of such a modeling choice should be discussed (e.g., trading speed and control vs generative capabilities of complex content?). I also suspect that the approach will not work as well for larger masks, more complex scenes/datasets and VQ-Models with a downsampling factor of 8 oder 4 instead of 16.\n- Related: The paper should compare to approaches that use bidirectional models in latent space, most notably MaskGIT [1] and M6 [2] and to ones that build discrete diffusion [3, 4] models in latent space [5]. Furthermore, a discrete diffusion formulation formulation is highly related to the one described here and opens up a different view on the approach as a few-step decoding sampler of a multinomial diffusion process. The only reference to diffusion models is to SDEdit with a reference to decoding speed. \n- A discussion/ablation on the number of inference iterations T is missing. \n- The results are focused on FFHQ, a simple and highly structured dataset. How does the inpainting approach on more complex datasets. A common benchmark for inpainting models (such as the mentioned CoModGAN) is the Places dataset [6]\nmissing evaluation of applying the method in different latent spaces. E.g., is it possible to train an EdiBERT in an f=8 latent space? \n\n**Minor remarks/Questions:**\n\n- p.18, A.1. \"In our experiments, since we work at resolution 256 × 256 and follow the downsampling factor of 4 from Esser et al. (2021b), we have h = w = 256/4 = 16.\" --> My understanding from the main paper is that the downsampling factor is 16. \n- it would be nice to see more qualitative results on LSUN-Bedrooms, especially for inpainting.\n- a comparison to other models in terms of sampling speed is missing. While the approach is significantly faster than AR models, it aims to fulfill a different tasks (image manipulation vs full generation) and should thus be compared to related approaches. How expensive is the \"post-collage\" procedure described in Alg.2? How much quality is lost without post-hoc latent optimiation, and how much speed gained?\n- p.1: litterature -> literature\n\n\n_Refererences_\n- [1]: MaskGIT: Masked Generative Image Transformer, Chang et al, https://arxiv.org/abs/2202.04200\n- [2]: M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformers, Zhang et al, https://arxiv.org/abs/2105.14211\n- [3]: Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions, Hoogeboom et al, https://arxiv.org/abs/2102.05379\n- [4]: Structured Denoising Diffusion Models in Discrete State-Spaces, Austin et al, https://arxiv.org/abs/2107.03006\n- [5]: Vector Quantized Diffusion Model for Text-to-Image Synthesis, Gu et al, https://arxiv.org/abs/2111.14822\n- [6]: Places: A 10 million Image Database for Scene Recognition, B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents EdiBERT, a bidirectional transformer-based model trained with a simple masked token prediction task in the discrete latent space of a VQGAN and capable of inpainting, scribble-based image processing, and image harmonization/composition. Particular emphasis is placed on the ability to achieve these functions with a single model. The model is trained on 256x256 images of the FFHQ and LSUN datasets and achieves results that rival other more specialized models.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Detailed requests/clarifications can be found in the **weaknesses** section above. In general, I think that the paper should address more the possible shortcomings and discuss the related work mentioned above. I am also somewhat concerned about the applicability/relevance of the proposed approach. It seems to be limited to \"simple datasets\" such as FFHQ and LSUN and does not provide a clear advantage over other approaches. The more general discrete diffusion models seem more promising in this regard (while also providing a \"unified\" model), and the proposed approach could be interpreted as a particular denoising sampling scheme. \n", "broader_impact_concerns": "Ethical implications of the model are briefly mentioned. While the discussion is quite brief and only mentions deepfakes as a potential negative impact, a full discussion of the societal implications of powerful generative image models is likely beyond the scope of this paper and an ongoing community effort. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yxhx7lL4kN", "reviewer": "Reviewer_ucMa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is a good idea to design a single model for image manipulation tasks. Unified models are already proven to be useful for language tasks. And for high-level vision tasks, CLIP models are shown to generalize quite well to images with domain gap. This paper is inspired by the general idea and tries to propose a unified approach for image manipulation.\n2. According to experimental results, the proposed method achieves comparable subjective and objective performances, although only a single pretrained model is used.\n\nWeakness:\n1. The main contribution of this paper is the change of the optimization method of VQGAN, that is incorporating an additional optimization LPIPS loss over the latent space. The main contribution is a little bit trivial.\n2. The paper is not well-written. And a lot of desciptions are not self-contained. And a couple of them are listed below.\n    - The pipeline of the proposed framework is not clearly presented in the begining of this paper. A figure that describes the general pipeline would help to improve the readability of this paper.\n    - To adapt a single pretrained network to two different image manipulation task including denoising and image inpainting, the resampling strategy seems to be essential. But it is not explained well in the introduction why this resampling strategy is important.\n    - The equations and notations are not well explained. For example, what does $E(I)_{l}$ in Eqn. 1 mean? Only after reading another paper did I undertand the correct meaning. Again, what does the $Z^l$ in the following line mean?\n    - Some description is not self-contained. For example, after Eqn. 6, the authors mentioned that \"the sampling function φ is defined with a 2D masking strategy, and the training positions are selected by drawing random 2D rectangles in the 2D patch extracted from the encoder.\" Please explain explicitly what the meaning of 2D masking strategy is.\n    - Figures needs to be well explained. For example, in Figure 2: Modifying the image via the latent space, what does the four images mean? How are the third and fourth images formed is not explained.\n\n[1] Taming Transformers for High-Resolution Image Synthesis", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a generative model for image editing. It is claimed that a single pretrained model could be applied to image denoising, image inpainting, and image composition tasks. The problem of a previous method VQGAN is analyzed, which helps to identify the chance to improve that model. In particular, an additional optimization step is incorporated, namely, optimizing the LPIPS metric over the latent spaces. It is visualized that by adding this additional optimization step, more high-frequency details are recovered. Then the authors present how to adapt one single model to two different image manipulation tasks, namely image denoising and image inpainting. It is also show that competitive results can be achieved with the generic training algorithm.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Questions:\n1. The authors mentioned that \"An autoregressive model is trained to generate these token sequences directly, stressing that high-capacity\nexpressive transformers can generate realistic high-resolution images. To summarize, this framework consists of three main steps:\" Could the authors explain the difference between the three main steps mention here and the main steps used by the proposed method?\n2. Please revised the paper according to the suggestions given above. Please make the figures and contents more self-contained. Please explain each equation more clearly.", "broader_impact_concerns": "The authors base their experiments on exsiting public datasets. Thus, although the experimental results is conducted on human faces, there is no ethical issues at lease for the algorithm part.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1rKZp4FtXM", "reviewer": "Reviewer_DPNU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Use VQ-GAN for image denoising and manipulation\n- Quantitative evaluation shows competitive performance and user study gives the proposed method a clear advantage.\n- Overall the paper is clear and easy to follow\n- The ablation study shows the necessity of the different tricks proposed by the authors to achieve best results and avoid divergence from the input image.\n\n# Weaknesses\n\n1. The presentation of the model assumes the reader knows in detail of the VQ-GAN (Esser et al. 2021b). Not enough details are provided for the different distributions and attention mechanisms. This is important to judge on the novelty of the proposed model.\n2. Although the reconstruction is improved thanks to the latent space optimization, there is still an important remaining gap.\n3. Currently the proposed optimization algorithm relies on several tricks and guides to avoid divergence. It would be interesting to have some more comments on how feasible it would be to shape the latent space to render those tricks unnecessary. \n3. Limited visual comparisons: only the composition task has images and the images are provided in supplemental. \n4. Exploring an additional dataset would be interesting. What is the expected performance for a more general setting (beyond faces or interior scenes)?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work the authors propose a method to perform image edition/manipulation using VQ-GAN and attention models. \n\nStarting from the VQ-GAN model with its tokens, an image can be represented as a sequence of such tokens. During training, the sequence of tokens is randomly perturbed and the model is trained to recover the original sequence from the full perturbed input sequence. This objective is different from the more commonly used autoregressive model for unconditional image generation. \n\nAfter training the model, the authors analyze the effect of image space manipulation on the latent space. Another aspect that needs to be addressed is the reconstruction capabilities. Using the default token sequence estimation results in images that are further away from the input. To address this the authors use latent space optimization with the objective of reconstructing the input image. \n\nAfter all this, the image edition/manipulation is expressed as an optimization problem over the sequence of tokens. The exact steps and terms of this optimization vary depending on the exact problem (denoising vs inpainting).\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Currently I would recommend the paper for acceptance. The necessary changes are:\n1. More details about the model, in particular the different distributions and the attention mechanism.\n2. Visual example with other state of the art methods focusing on reconstruction gap\n3. Visual comparisons with existing methods on other tasks should appear in the main paper.\n\nAddressing the remaining points from the weaknesses section would strengthen the paper.\n", "broader_impact_concerns": "The authors acknowledge that the model might be used to create and propagate fake beliefs via deepfakes, a concern shared with other image generative models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a bidirectional transformer that re-samples image patches conditioned on an input image. The main advantage of the proposed method is that a *single* trained model can be applied to solve a wide variety of different image editing tasks. 

Reviewer ucMa and Reviewer DPNU are satisfied with the authors' detailed revision and response, including a discussion on missed references in the related work section and various clarifications on method exposition. After the discussions with the authors, both Reviewer ucMa and Reviewer DPNU are leaning toward acceptance. Reviewer ANEH, however, is leaning toward rejection. The main complaints were a lack of comparisons with MaskGIT and M6-UFC, simple, structured datasets (e.g., FFHQ), and some method exposition issues. 

The authors' responses partially address these concerns, including the revision of the related work to include relevant prior work and an explanation of the lack of other larger-scale datasets. The AE thinks the responses and revision sufficiently address the concerns of Reviewer ANEH. That being said, the AE agrees with Reviewer ANEH that a comparison with methods that also use bidirectional models in latent space, e.g,. MaskGIT and M6-UFC, would be beneficial for the readers to get a clearer picture of the literature. The AE thus recommends "Accept with minor revision". 

Specifically, the authors should
1) cite and discuss the differences between the proposed method and MaskGIT and M6-UFC
2) include at least one direct comparison with MaskGIT (code available here: https://github.com/google-research/maskgit).

Note that the acceptance is *not* tied to the performance comparison with MaskGIT. The MaskGIT requires training one specific model per task so there are still merits of the proposed method even if it underperforms MaskGIT on the compared tasks.
Decision: Accept (poster)