Semantic Image Manipulation with Background-guided Internal Learning
Authors: Zhongping Zhang
Huiwen He
Bryan A. Plummer
Zhenyu Liao
Huayan Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: semantic image manipulation
internal learning
scene-graph driven image editing

Abstract: Image manipulation has attracted a lot of interest due to its wide range of applications. Prior work modifies images either from low-level manipulation, such as image inpainting or through manual edits via paintbrushes and scribbles, or from high-level manipulation, employing deep generative networks to output an image conditioned on high-level semantic input. In this study, we propose Semantic Image Manipulation with Background-guided Internal Learning (SIMBIL), which combines high-level and low-level manipulation. Specifically, users can edit an image at the semantic level by applying changes on the scene graph. Then our model manipulates the image at the pixel level according to the modified scene graph. There are two major advantages of our approach. First, high-level manipulation requires less manual effort from the user compared to manipulating raw image pixels. Second, our low-level internal learning approach is scalable to images of various sizes without reliance on external visual datasets for training. We outperform the state-of-the-art in a quantitative and qualitative evaluation on CLEVR and Visual Genome datasets. Experiments show around 8 points improvement of SSIM (RoI) on CLEVR and around 25% improvement of user evaluation accuracy on Visual Genome, demonstrating the effectiveness of our approach.

Under review as a conference paper at ICLR 2023
SEMANTIC IMAGE MANIPULATION WITH BACKGROUND- INTERNAL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Image manipulation has attracted a lot of interest due to its wide range of appli- cations. Prior work modifies images either from pixel-level manipulation, such as image inpainting or through manual edits via paintbrushes and scribbles, or from high-level manipulation, employing deep generative networks to output image conditioned on bigh-level semantic input. In this study, we propose Semantic Im- age Manipulation with beokkroond-guudeee Internal Learning (SIMBIL), which combines high-level and pixel-level manipulation. Specifically, users can edit an image at the semantic level by applying changes on the scene graph. Then our model manipulates the image at the pixel level according to the modified scene graph. There are two major advantages of our approach. First, high-level ma- nipulation requires less manual effort from the user compared to manipulating image pixels. Second, our pixel-level internal learning approach is scalable to images of various sizes without reliance on external visual datasets for train- ing. We outperform the of-te-of-the-arl in quantitative and qualitative evaluation on CLEVR and Visual Genome datasets. Experiments show around 8 points im- provement of SSIM (Rol) on CLEVR and we found human users preferred our manipulated images over prior work by 9-33% on Visual Genome, demonstrating the effectiveness our approach.
INTRODUCTION
Image manipulation modifies the content of an image according to user guidance. The task can be solved in two primary ways: pixel-level manipulation on raw images and high-level manipulation on image semantics. Pixel-level manipulation spans image inpainting (Zhao al., 2019 Yeh al.) 2017). colorization (Zhang et al. 2016), object removal (Shetty et al., 2018). style transfer (Gatys et al. 2016), image extension (Teterwak al., 2019), etc. Pixel-level manipulation methods do not need to understand the semantic meanings of image. In contrast, high-level manipulation often uses deep generative networks conditioned on user inputs like semantic maps and language descrip- tions to identify the desired modifications. Most prior work for high-level image manipulation object-centric, such as human face transfer (Choi et al. 2018 Lee et al. [2200]Jo & Park] Park|2019 2019 ZZhoo al. 2018) and object appearance attribute modification (Li et al. 2020a Liang et al.) 2018). Recently, approaches modifying the entire scenes by instance maps ( (Wang et al. 2018), language de- scriptions (El-Nouby et al. 20199 [ NNichl et al. 2021 Avrahami et al. 2022) or scene graphs (Dhamo) ct al. 2020) are also proposed. Although high-level manipulation requires less manual effort from users, deep generative networks for high-level manipulation have drawbacks. First, high-level manipulation frameworks often only support outputting low-resolution images due to GPU memory requirements (Dhamo et al., 2020). Super-resolution modules (Saharia al.) [022) Nichol et al.) 2 are required to get higher-resolution images, introducing extra overhead. Second, generative models may result in the loss of attributes and details of the original images (Bau et al. 2020).
Ideally, good image manipulation method should satisfy the following requirements: (1) provide maximum convenience to users; for example, manipulating images by scene graphs or language description is more convenient than manually segmenting, replacing, removing the target object, (2) preserve the textures and details of the original image in appropriate areas, (3) correctly modify the target region of the image according to user instructions, (4) ability to generalize across input images without relying on specific external datasets. There are two major challenges to developing an approach that can satisfy these requirements. First, it is challenging for existing text-driven

---

scene graph Under review as a conference paper at ICLR 2023
(a) low level image maniputation
(b) high level image manipulation
This bird has yellow bill, blue head, wings and a yallow belly.
frumewurk: high level semantics low level munipulation
ridling
honse
odit target abject
mext to
girl
honse
Figure I: Prior work of image manipulation is either at pixel-level (e.g., EdgeConnect (Nazeri et al,, 2.19). shown in (a), or high-level (e.g., ManiGAN (Li al. 2020a)), shown in (b). In our work, shown in (c), we address the issues of prior work (see Section for discussion) by connecting high- level semantics with pixel-level manipulation, where the semantic level information encoded by an RNN-based seene-graph encoder. Then the pixel-level manipulation, background-guide internal learning, done according to the processed information.
image editing methods to accurately localize the Region of Interest (RoI) at complex scenes. E.g.. popular frameworks including GLIDE (Nichol al. 202T) and blended diffusion model (Avrahami et al. 2022) require users to manually select Rol. Methods (Li al. [020aa bb that do not require bounding boxes as input are mostly object-centric and the images do not contain complex semantic relationships between objects. The ambiguity of text makes developing
Rol prediction model challenging. For example, if there are multiple birds in an image, locating the target bird according to text descriptions would be challenging even for human. To solve this issue,
information to climinate the ambiguity of text, while still making manipulations easy. Second, most image inpainting methods (Nazeri et al. 2019 Yu et al. 2019 Rombach et al. (022) trained their models based on reconstruction task. In this case, as we will show in Section 4.2 and Section 4.3
these external learning methods tend to repair the target object even if the user command is to remove the object. We further introduce internal learning to avoid the object repair issue.
Specifically, we propose a Semantic Image Manipulation framework with Background-guided In- ternal Learning (SIMBIL). SIMBIL combines high-level image semantics with pixel-level manipu- lation. Figure[[illustrenn
the difference between SIMBIL and prior work by an object relationship change example. Figure 2 presents the overall structure of SIMBIL. First, the target object is de- termined by the scene graph of image. The users are able edit the nodes and edges of scene graphs for four operations, object removal, object replacement, semantic relationship change, and object addition. We use segmentation module outline the mask of target object. Recurrent Neural Networks (RNN)-based module further encodes the semantic modifications between the ob- jects and predicts the target Region Interest (Rol) according to editing operations. Finally, we improve Deep Image Prior (DIP) (Ulyanov al. 2018) by utilizing background pixels straint and propose the background-gen -gudded internal learning module.
In summary, the contributions of this paper are:
We propose a semantic image manipulation framework (SIMBIL) to combine high-level seman- ties with pixel-level image manipulation, reducing manual effort and alleviating the issues caused by prior work. Notably, compared to existing manipulation methods using scene graphs (Dhamo et al. 2020). SIMBIL can generate higher resolution images while accurately preserving the orig- inal details of the input images.
We develop background-ge guided internal learning algorithm based on DIP (Ulyanov et al. 2018) for image inpainting, which utilizes the average value of the background pixels around the missing part as guidance as opposed to only relying on the implicit prior captured by the neural network parameterization. boosting performance.
We use Rol to indicate the region that is supposed he edited in the image.
2

---

Under review as a conference paper at ICLR 2023
Inpat Image
Secne Graph
Targer Objjett Segmentation
bise cylinder
behind
rel cihh
sphere
fron:
cylimker
bluz sylinder
smy eylinder
I:fl of
blue eylande
ebject cobbaaking
gny sylinder red sphers brown sphere cylinder- mihe
predicate ersheoding
antrro
(subject oe nox)
inpplet encodar
Scene Graph Encoder
Background guidded Internal Learning
Figure 2: SIMBIL overview. Given an input image and its modified scene graph, our approach con- sists of three modules in sequence: (1) Object Segmentation Module: segmenting the target object according to the modified scene graph; (2) Rol Prediction Module: predicting the Region of Interst
by encoding the modified triplets of the scene graph; (3) Internal Learning Module: leveraging the average value of the background pixels around the target object to guide the inpainting results.
Quantitative and qualitative experiments on CLEVR JJonsonn et al., 2017 and Visual
Genome (Krishna et al.) (017) demonstrate SIMBIL outperforms the ttate-of-the-art Extensive experiments on high resolution images demonstrate SIMBIL's flexibility and scalability.
2 RELATED WORK
Image Manipulation. Many studies based on image synthesis focus on object-centric scenarios,
Yeh al., 2017] Zhao et al., (018)), text-guided attribute manipulation (I (Liang et al. 2018 Li et al. 2020a). manually
e.g., editing human face (Choi al.) 2018 Lee al.] 2020 Jo Park] 2019
editing with paintbrush and scribbles (Brock et al. 2016 Zhu et al. 2016). Recently there has
been some work modifying images in complex scenes consisting of multiple objects (Wang et al.,
2018
El-Nouby et al., 2019 Tan et al., [019 Dhamo et al. 2020 Nichol et al. 2021 Avrahami
al.
2022). These methods often require human effort to outline the region of interest (Wang
jet al.
2018; Nichol et al. 2021 Avrahami et al. 2022) and focus only on object replacement or
addition (El-Nouby al. 2019 Tan et al. 2019 Nichol et al. [021 Avrahami et al.) (2022). Dhamo
jet al. (2020) proposed a scene-graph based approach that can also edit the semantic relationships between objects in single image. Unlike methods based generative networks, SIMBIL adopts an automatic "photoshop" mechanism, directly manipulating raw pixels. Our method is applicable different input images while preserving the original image details.
Internal Learning. While external learning trains model on external datasets, internal learning methods such as Deep Image Prior (Ulyanov al., 2018) use generator network trained on a sin- gle image to address tasks like image denoising, inpainting, and super-resolution. Internal learning not limited image inverse problems (Guasch et al.) 2020) Zhang Lin 2020 Zhang et al.) 2019b), but has other applications including video motion transfer (Chan et al.) 2019), video in- painting (Zhang al.) (019a). and semantic photo manipulation (Bau et al. (020).. Gandelsman al. (2019) also applied coupled DIPs to unsupervised image decomposition. In this paper, we troduce background -egiddddd mechanism based on DIP, which uses the background pixels for more accurate guidance than the implicit prior used by DIP for the missing region of an image.
Scene Graphs and Visual Relationship Detection. Scene graphs (Johnson et al. 2015) describe the objects, attributes of objects, and relationships between objects in image, and methods that generate them can be divided into two categories: Convolutional Neural Network (CNN)-based
methods (Li al. 2017 2018: Yang al. 2018 Qi al. 2019) and Recurrent Neural Networks

---

Under review as a conference paper at ICLR 2023
(RNN)-based) methods (Herzig et al. 2018: Xu al. 2017 Zellers et al. 2018). At a high level, scene graph construction combines object/entity detection (Ren et al. 2015] Plummer et al. 2020)
and detecting their visual relationships (Dai et al.
2017 Lu et al. 2016 Plummer et al.) (017).
In our paper, users can apply changes to the relationships between objects to realize image manip- ulation. A relevant task is visual entity localization according to visual relationships. KKiishna et al. (2018) introduce an iterative model to localize the entities in the referring relationship. Plummer et al. (2017) combine linguistic cues with learned weights for phrase localization. In our project, the main difference is that the target object is invisible in the image. Therefore, we propose an RNN- method automatically predict a plausible position for the target object according to the existing information from the image and wwwpeeectppdiiaaee e- ject>> triplets.
3 SIMBIL
Given an input image I and its corresponding scene graph G, users can apply changes on G. Our task is to perform semantic manipulation according to the user-modified scene graph G and the original input image 1. In practice, G can be obtained by scene graph generation methods (Li et al. 2018 Qi et al. 2019 Herzig et al. 2018: Xu et al., 2017: Zellers et al.] 2018). Following (Dhamo et al, 2020). we use ground truth scene graphs in our experiments so the accuracy of current methods is not factor in evaluating our approach. Figure 2 provides an overview of our framework. We introduce four kinds of modifications on scene graphs in Section 3.1] In Section 3.2]. we discuss the details of our Rol prediction model. The background-guideer internal learning algorithm is then
presented in Section 3.3 Finally, we summarize the overall learning strategy in Section 3.4
3.1 MODIFICATIONS ON SCENE GRAPHS
Following (Dhamo et al. 2020), we perform four tasks of semantic manipulation: object addi- tion, object replacement, relationship change, and object removaf] These manipulations are re- flected on the nodes and edges of scene graphs. Take the blue cylinder in Figure as exam- adding a new object (node) and its corresponding relationships (edges) with other objects on the scene graph; (2) Object Replacement: replacing the node which repre- sents <blue cy cylinder>> to another object; (3) Relationship Change: changing one spatial rela- tionship of blue cylinder (edge) from <blue cylinder-from
ple, (1) Object Addition:
of-red cude> to <blue cude>. Other relationships of the blue cylinder can be changed simi- larly; (4) Object Removal: deleting the node of <blue cy inder> and its corresponding edges. Given the modified scene graph G, we extract the bounding box and mask of the target object by MaskFormer (Cheng et al. 2021), and then proceed with performing the alteration.
cylinder-behine-e
3.2 REGION OF INTEREST (Rol) PREDICTION
We introduce a Rol prediction module to support the object addition and semantic relationship change tasks. Benefiting from this module, SIMBIL can automatically predict Rol instead of man- ually outlining the bounding box (Nichol et al. 2021 Avrahami et al., (022). Unlike relationship detection (Krishna et al. 2017). which identifies relationships between visible reference and target object in a scene, our goal is to determine where to place the target object given the reference object and its semantic relationship to the target object. In addition, multiple relationships may need to be encoded as the target object may have relationships with multiple objects in the scene. Inspired by RNN-based methods on scene graph generation (Herzig et al.) 2018 Zellers et al. 2018), we develop an L-TMebasedd model that encodes all the triplets of the modified scene graph.
We define the categories of the entities (subject & object) in images as O = (or.... J, the cor- responding bounding boxes as B (bi...., b..,, and the relationships (predicates) between entities as R (ri....,PT*) }. Given the modified scene graph G and the target object o in an image, the triplets of the target object are referred as = {21, .... }r}. For t E {1...,?") Be is a triplet in < s - p - 0 > format, where 8, O E O and p E R. For each triplet yt, we devise two embed- ding layers that obtain the object embedding V. V. and predicate embedding V, separately. We
Following Dhamo al. 2020). we perform four operations on CLEVR (Johnson al. 2017). three oper- ations Visual Genome (Krihhna (207): object replacement, relationship change, and object removal.
4

---

Under review as a conference paper at ICLR 2023
deep Image
background-guldedd
Figure 3: Left: deep image prior: Right: background -uidedd internal learning. Our methods generate more reasonsble inpainting result by leveraging the background pixels as guidance.
also introduce binary indicator / to indicate whether the target object is subject or object in yt- Suppose the reference object corresponds to "subject" and the target object corresponds to "object", also consider the position of reference object b, part of the input to model. The con- catenation of these features are encoded by a LSTM model. Specifically, it can be expressed as: Tr concal (Va, Vo, Vp: ba, I); h2 LSTM( (22, h1-1), where he is the hidden state of LSTM triplet yt- hr is followed by an MLP to predict the Rol. We crop the final output to range (0~1) since the image size is normalized and train the model using Mean squared error (MSE).
3.3 BACKGROUND-GUIDER INTERNAL LEARNING
Our internal learning approach, SIMBIL, has two advantages over external learning methods (Nazeri jet al. 2019; Yu et al., 2019 Rombach et al. 2022). First, external learning methods rely on the training data that consists of image pairs before and after inpainting. Thus, inpainting model for CLEVR and Visual Genome needs to be trained separately and is not applicable to different input images. Second, rather than remove objects. This is because these models use the ground truth image instead of the modified image supervision (i.e., Visual Genome lacks before and after editing image pairs).
will show in Section 4 many external learning methods tend to repair
Given an image Ty and a binary mask m, the input to image inpainting is zu @ m, where m defines the known areas of image 2n (1 for the known regions and 0 otherwise) and @ denotes Hadamard's product. The standard image inversion problem can be formulated by
argmin E (z; xo) R(z)
where E(x; 2n) is task depe-dependent data term and R(x) is the regularization term. For deep image prior (DIP) (Ulyanov al., 2018), the regularization term R(x) is replaced by the implicit prior from the neural network parametrization. Specifically, the input of the generative neural network noise map a that has one channel and shares the same spatial size as image zg. The network parameters are updated to minimize E(x; zo). In terms of image inpainting. E(z; satisfies
E(x; m0) = ||(x zo) @ m|
(2)
where is the output of the generative model. Note that our "missing" regions are determined by a segmentation model, i.e., the boundary of the hole might not be accurate. Thus, the genera- tive network may be given noisy input, resulting in incorrect predictions by prior work like DIP. In addition, the object to remove can be large, in which case the relative performance of DIP is worse (Ulyanov et al.) 2018 Zhang et al. 2019a).
We address the aforementioned issue with noisy inputs in two ways. First, we use mask dilation alleviate the misinformation of the boundary. Second, we use the background pixels around the hole as guidance to the model. Specifically, suppose a consistent background is divided into two parts; the distributions of each part should still be consistent with the other. As shown in Figure 3 the segmentation of the target car is the missing region of the original image. Then the background pixels within the red bounding box are extracted from the image. We constrain the average of the missing region by the average of the specified background region. Denote the average value of background pixels as B, B E R our new objective function is:
Exx; xo) = 1|(x - mu) @ ml12 + A C |average(2 @ 01 (1- (1-m)) m)) -
(3)
5

---

Under review as a conference paper at ICLR 2023
where C' represents the number of image channels, and À is the hyper parammeee to control the two loss terms. In practice, the second loss term can be further divided into sub-terms, for example, constraining the average row by row.
3.4 OVERALL LEARNING STRATEGY
There are three components in our framework, which are segmentation module, Rol prediction mod- ule, and background-guice internal learning. The segmentation module (Section 3.1 is applied in two cases (1) removing the target object from the original position, (2) searching relevant objects (from the current image or query images) given the object category, which is mainly for object ad- dition and replacement" The Rol prediction model (Section 3.2 is designed for object addition and relationship change. To train this model, we extract modified triplets, bounding boxes of reference objects, and target objects to construct a dataset. Take CLEVR as an example; we extract 6688 im- ages from the training set to construct the training data, then the 818 images in the validation set used to evaluate the performance of the model. The target object is pasted according to the predicted Rol of the model. After this step, remember we still need to address the missing region ("hole"), which is accomplished using our proposed background-gund the final output. More implementation details are provided in Appendix A
internal learning (Section 3.3 to get
4 EXPERIMENTS
4.1 DATASETS AND EXPERIMENT SETTINGS
Datasets. We evaluate on CLEVR (Johnson et al. 2017 and Visual Genome (Krishna et al. 2017 CLEVR is a synthetic dataset that contains ground truth pairs for image editing. For fair com- parison, we use the test set provided by (Dhamo et al.) 2020). Visual Genome, on the other hand, lacks before and after image editing pairs. Therefore, we leverage human evaluation estimate the correctness of manipulation. Qualitative examples from both CLEVR and Visual Genome are presented to demonstrate the effectiveness of our model on synthetic and natural images.
Metrics. Following Dhamo et al. (2020), we report mean absolute error (MAE) and structural similarity index measure (SSIM) of Rol to evaluate the model. We defined the modified area of images as Rol so that it can directly reflect the manipulation accuracy. In addition, we also performed user study evaluate whether the edited images are consistent to editing commands.
Baselines. We use the code and pretrained model of SIMSG (Dhamo et al., 2020) as our scene- graph-guided image editing baseline. In addition, we apply baselines on separate tasks, Specifically, we use DeepFillv2 Yu et al. 2019)) EdgeConenct (Nazrri et al. 2019), and Latent Diffusion Model (LDM) (Rombach et al. (022) on object removal, use Blended Diffusion Model (BDM) (Avrahami et al. 2022) on object replacement and addition, use GLIDE (Nichol et al. [022] on object removal, replacement, and addition. Since GLIDE and BDM require manually outline Rol, we apply Rol prediction of SIMBIL to these methods.
4.2 PERFORMANCE ON CLEVR DATASET
Manipulation Experiments. Table reports quantitative results of different methods at 256 2 image resolution. From the table, we see that SIMBII. notably outperforms baselines on four image editing tasks, especially for object removal and relationship change. We draw three major conclu- sions from Table T First, external image inpainting approches (Yu et al. 2019 Nazeri et al. 2019
Rombach et al. 2022) do not perform well on our object removal task. As we discussed Sec- 3.33 these methods tend to introduce unexpected objects to repair the missing regions. Second, scene-graph based image editing approaches, SIMSG and SIMBIL, perform more accurate manipu- lation operations on object replacement and object addition compared to text-driven methods GLIDE and BDM. This is because scene graph contains semantic relationships and object positions in the
2 We adopt compositional scene representation mechanism for object addition and replacement following Text2Scene (Tan et al. (2099)
"Since SIMSG requires more than 48GB GPU memory to train model that outputs 256 256 images, apply USRNet (Zhang (020) the output of SIMSG avoid out-of-memory issue.

---

Under review as a conference paper at ICLR 2023
Inpur Image
Ground Teuth-Ral
Iopot Image
Ground Truth-Rol
Oldjcct Repiscement
Objjett
Object Additio
Rehatinskby
Figure 4: Comparing SIMBIL to SIMSG CLEVR. Rol is outlined by red bounding boxes. See Section 4.2 for discussion.
Table 1: Quantitative results of image manipulation CLEVR dataset. Empty value means the corresponding approach not capable of this task. See Section 4.2 for discussion.
Object
Object
Object Addition
Relationship
Method
Removal Replacement
Change
MAE | SSIM t MAE & SSIM
MAE SSIM MAE
SSIM
DeepFillv2 Yu et al. 2019
39.77
65.10
EdgeConnect (Nazeri et al.
2019)
29.26
66.70
LDM (Rombach al. 2022)
37.84
52.41
LAMA (uuvorov et al.
2022
33.54
71.19
MAT Li al. 2022)
31.46
74.80
BDM AArahami et al.
2022
42.32
50.12
59.25
52.64
GLIDE (Nichol et al.
2021
39.51
62.91
34.59
55.57
40.25
59.04
SIMSG Dhham
2020
30.38
85.89
33.76
67.41
44.58
65,80
33.31
85.95
SIMBIL (ours)
9.90
93.33
27.64
68.05
34.18
64.78
11.40
92.32
image which are difficult to be described by text guidance. The relevant information important for models to accurately predict manipulated images. Third, observe that though SIMSG achieves comparable SSIM scores with our method on object replacement and addition, the MAE loss of SIMSG is much higher than our model, which means the original attributes such as brightness likely to be modified by SIMSG. To validate our analysis, we present images of different manipula- tions in Figure The figure shows that SIMBIL can accurately edit images while still preserving the original background perfectly. In contrast, the brightness of SIMSG output is different from original images (e.g., output of object addition & relationship change), resulting higher MAE loss.
4.3 PERFORMANCE ON VISUAL GENOME DATASET
Manipulation Experiments. As with experiments CLEVR, apply popular inpainting meth- ods on object removal, text-driven editing methods on object replacement, and SIMSG on all ma- nipulation tasks. For each image editing task, randomly selected 30 images generated by each baseline. This results in 150 images for object removal, 120 images for object replacement, and 60 images for relationship changes. Each image is annotated three times by AMT workers and asked our annotators to judge whether the image is correctly manipulated according to the input guidance. In Table we report that SIMBIL significantly outperforms baselines, especially for object removal and relationship change tasks, which is consistent with our conclusions on CLEVR.
We the image resolution to 256 256 to fit the output for most baselines. For SIMSG. we apply USRNet to increase the resolution from 64 x 64 256 256. Figure (a) provides object replacement results, where SIMBIL clearly outperforms both scene-graph driven approach SIMSG and text-driven approaches GLIDE and BDM. Though BDM and GLIDE can generate some plausible objects some cases, the appearance and shape of the edited object still look odd. Figure 5 (b) presents images reports by relationship change. From these results, we can see that our Rol prediction module outputs reasonable values for the relationship change between objects, for instance, riding beside, near
We provide qualitative results in Figure 5
Figure (c() provides
object removal examples. Consistent with our discussion in Section 3.3 and Section 4.2 external approaches tend to introduce unexpected objects in
when inpainting. We helieve the

---

Under review as a conference paper at ICLR 2023
Coured Frath
HUM
GLIDE
SINEELL
(b)
LOoDE Frais
SIMBII
Spplice- pervos
depinant
gs Besdde
EdycConnnt
(c)
ICSRAVC
bivi
Figure 5: Qualitative results comparing SIMBIL with baselines on Visual Genome. (a): object replacement; (b): relationship change: (c): object removal. The edited regions are zoomed below each image. See Section 4.3 for discussion.
Table User evaluation judging the correctness of an image manipulation on Visual Genome. See Section 4.3 for discussion.
Object Removal
Object Replacement
Relationship Change
EdgeConnect (Nazeri al. 2019
27.8%
Latent Diffusion (Rombach et al. 2022
21.1%
Blended Diffusion (Avrahamı ci al. 2022
31.1%
GLIDE (Nichol et al. 2021)
23.3%
28.8%
SIMSG (Dhamo et al.
2020
22.2%
24.4%
15.6%
SIMBIL
50.0 %
40.0%
48.9%
model may recognize the appearance of some objects due to seeing similar images during training. contrast, internal learning approaches do not suffer from this issue because they only utilize the context and textures of the current input image. Additionally, our background-guider method provides higher quality images compared to DIP.
4.4 LIMITATIONS AND FUTURE WORK
In this section, we analyze the limitations of our method and provide suggestions to address these limitations. Four cases presented in Figure
8

---

Under review as a conference paper at ICLR 2023
original
ground truth
SIMBIL
original
SIIMBIL((ip))
SIMBIL
shadow removal
biased background
original
ground truth
SIMBIL
[b] artificial boundary
[d) human pose transfer
Figure 6: Limitations of SIMBIL. See Section 4.4 for discussion.
(a) Shadow removal: Since our segmentation module does not consider shadows part of an object, they typically remain an image. However, this can addressed by incorporating shadow detection methods (e.g.. (Wang al. 2020 in the removal module.
(b) Artificial boundary: some images, we can observe white edges we zoom out the boundary region of the modified object. We assume the artificial boundary is introduced when we resize the object from the query images.
(c) Biased background: Although background-guidee internal learning outperforms DIP in most cases, we do have some failures like the example in Figure ((c). We believe this because our segmentation model does not distinguish the tree from the background. As result, approach guides the missing region to be similar to the tree in some areas within the "hole".
(d) Human pose transfer:
limited to spatial relationships such as beside, near, on, but also span predicates like standing on, running, walking. Ideally, the appearance of the objects should also be changed according various predicates. We tried apply a human pose transfer model (Zhu et al.] 2019) pretrained on DeepFashion (Liu et al. 2016) to address this issue. However, there significant domain shift from DeepFashion Visual Genome that resulted in poor performance, which could be addressed in future work via domain adaptation techniques.
Relationships between objects
(e) Inference time: Internal learning methods train separate network to process each image. The inference time to process 256 256 image by RTX 3090 takes approximately 35 seconds. We observe that our background-guidane mechanism has negligible overhead compared to DIP. For comparison, text-driven methods BDM (Avrahami et al. 2022) takes approximately 28 seconds and GLIDE (Nichol et al. 202T) takes approximately 8 seconds to process each image. While our approach does take a little longer at inference time, it does not have the expensive training step (in terms of data and computational time) of BDM GLIDE.
CONCLUSION
In this paper, we proposed a semantic image manipulation method called SIMBIL that combines high-level image semantics with pixel-level manipulation. SIMBIL mainly consists of object seg- mentation module, an RNN-based Rol prediction module to predict the edited region for target objects, and nackground-guice internal learning module for image inpainting. SIMBIL outper- forms the otate-of-the-art in a qualitative and quantitative evaluation CLEVR and Visual Genome datasets. For example, our method outperforms SIMSG around points improvement of SSIM on CLEVR and around 25% improvement of user evaluation accuracy on Visual Genome. Exten- sive experiments on images with higher resolution, which prior work struggled to perform, further demonstrate the effectiveness of our method. Thus, argue the combination of high-lever seman- tics and pixel-level manipulation is a promising way to solve the image manipulation problem since it requires less human effort and accurately preserves the original image's attributes and details.

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
Our method, SIMBIL, is presented in detail in Section 3 2 the overall pipeline of our method by an example of semantic relationship change. Additionally, we provide implementation details in Appendix A We will also release our code to ensure reproducibility after the paper is
accepted.
7 ETHICS STATEMENT
In this paper, we develop a model named SIMBIL for semantic image manipulation. From the ex- periments, we see that SIMBIL can effectively manipulate images according user guidance. Our scene-graph driven editing technique not only make our system easy to use, but also help people with disabilities who cannot perform pixel-level image manipulation to manually segment, delete, and move the objects. However, just like other image editing approaches, SIMBIL can modify the content of the original images. Therefore, it is critical for practitioners to carefully review the mod- images
ified images to avoid spreading misinformation. Additionally, we found are difficult to be recognized and located by exisiting image manipulation detection methods like IDNet Wu & Zhou 2021). Thus, internal learning method also provides a potential improve- ment direction for image manipulation detection, ie.,, how to develop a more robust framework that recognize both the external learning manipulated images and the internal learning manipulated
images.
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
David Bau, Hendrik Strobelt, William Peebles, Bolei Zhou, Jun- Yan Zhu, Antonio Torralba, et al. Semantic photo manipulation with generative image prior. arXiv preprint arXiv:2005.07727, 2020.
Andrew Brock, Theodore Lim, James Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:
2016.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei Efros. Everybody dance now, Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 5933-5942, 2019.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems. 34:17864-17875, 2021.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8789-8797, 2018.
Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference computer vision and Pattern recognition, pp. 3076-3086, 2017.
Helisa Dhamo, Azade Farshad, Iro Laina, Nassir Navab, Gregory D Hager, Federico Tombari, and Christian Rupprecht. Semantic image manipulation using scene graphs. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5213-5222, 2020.
Alaaeldin El-Nouby, Shikhar Sharma, Hannes Schulz, Devon Hjelm, Layla El Asri, Samira Ebrahimi Kahou, Yoshua Bengio, and Graham W Taylor. Tell, draw, and repeat: Gen- erating and modifying images based on continual linguistic instruction. In Proceedings of the IEFF/CVF International Conference Computer Vision pp. 10304-10312, 2019.
10

---

Under review as a conference paper at ICLR 2023
Yosef Gandelsman, Assaf Shocher, and Michal Irani. "double-dip": Unsupervised image decompo- sition via coupled ceep-image-priie priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11026-11035, 2019.
Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference computer vision and patter recog- nition, pp. 2414-2423, 2016.
Lluís Guasch, Oscar Calderón Agudo, Meng-Xing Tang, Parashkev Nachev, and Michael Warner. Full-waveform inversion imaging of the human brain. NPJ digital medicine, 3(1):1-12, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Map- ping images to scene graphs with permutation-ine invariian structured prediction. arXiv preprint 10X.051802.05551,, 2018.
Ronghang Hu, Marcus Rohrbach, and Trevor Darrell. Segmentation from natural language expres- sions. In The European Conference Computer Vision (ECCV), 2016.
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, and Ross Girshick. Learning to segment every thing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4233 1241-424,2222 2018.
Youngjoo Jo and Jongyoul Park. Sc-fegan: face editing generative adversarial network with user's sketch and color. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1145-17555 2019.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 3668-3678, 2015.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 20012901129 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsoureed dense image annotations. International journal of computer vision, 123(1)332-73, 2017.
Ranjay Krishna, Ines Chami, Michael Bernstein, and Li Fei-Fei. Referring relationships. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6867-6876, 2018.
Cheng-Han Lee, Ziwei Liu, Lingyun Wu, and Ping Luo. Maskgan: Towards diverse and interactive facial image manipulation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5549-5558, 2020.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip HS Tor. Manigan: Text-guided image manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7880-7889, 2020a.
Bowen Li, Xiaojuan Qi, Philip Torr, and Thomas Lukasiewicz. Lightweight generative adversar- ial networks for text-guided image manipulation. Advances in Neural Information Processing Systems, 13:22020-22031 2020b.
Wenbo Li, Zhe Lin, Kun Zhou, Lu Qi, Yi Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10758-10768, 2022.

---

Under review as a conference paper at ICLR 2023
Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE International Conference Computer Viston, pp. 1261-1270, 2017.
Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. Factorizable net: an efficient subgraph- based framework for scene graph generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 335-351, 2018.
Xiaodan Liang, Hao Zhang. Liang Lin, and Eric Xing. Generative semantic manipulation with mask-contrasting gan. In Proceedings of the European Conference Computer Viston (ECCV), pp. 558-573, 2018.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European conference on computer vision, pp. 852-869 Springer, 2016.
Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edgeconnect: Struc- ture guided image inpainting using edge prediction. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision Workshops, pp. 0-0, 2019.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.1074 2021.
Bryan A Plummer, Arun Mallya, Christopher M Cervantes, Julia Hockenmaier, and Svetlana Lazeb- nik. Phrase localization and visual relationship detection with comprehensive image-language cues. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1928-1937, 2017.
Bryan Allen Plummer, Kevin Shih, Yichen Li, Ke Xu, Svetlana Lazebnik, Stan Sclaroff, and Kate Saenko. Revisiting image-language networks for open-ended phrase detection. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2020.
Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3957-3966, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint 1rXiv:1506001499 2015.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic eext-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205.1 / 1487, 2022.
Rakshith Shetty, Mario Fritz, and Bernt Schiele. Adversarial scene editing: Automatic object re- moval from weak supervision. arXiv preprint (XXi:18806009911 2018
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robuu large mask inpainting with fourier convolutions. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 2149-2159, 2022.
12

---

Under review as a conference paper at ICLR 2023
Fuwen Tan, Song Feng, and Vicente Ordonez. Text2scene: Generating compositional scenes from textual descriptions. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6710-6715 (0096699,2200 2019.
Piotr Teterwak, Aaron Sarna, Dilip Krishnan, Aaron Maschinot, David Belanger, Ce Liu, and William T Freeman. Boundless: Generative adversarial networks for image extension. In Pro- ceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10521-10530, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1446694544 2018.
Tianyu Wang, Xiaowei Hu, Qiong Wang, Pheng-Ann Heng. and Chi- Chi-Wing Fu. Instance shadow detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pp. 1880-1889, 2020.
Ting- Chun Wang, Ming- Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8798-8890 2018.
Haiwei Wu and Jiantao Zhou. lid-net: Image inpainting detection network via neural architecture search and attention. IEEE Transactions Circuits and Systems for Video Technology, pp. 1-1, 2021. doi: 10. 10.1199/CSVT::2011 30075039.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-4199 2017.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference computer vision (ECCV), pp. 670-685, 2018.
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh Do. Semantic image inpainting with deep generative models. In Proceedings the conference on computer vision and pattern recognition, pp. 5485-5485555 2017.
the IEEE
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 4471-4480, 2019.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831-5840, 2018
Haotian Zhang, Long Mai, Ning Xu, Zhaowen Wang, John Collomosse, and Hailin Jin. An internal learning approach to video inpainting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2720-2729, 2019a.
Kai Zhang, Luc Van Gool, and Radu Timofte. Deep unfolding network for image super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 3217-3226, 2020.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European confer- ence on computer vision, pp. 649-666 Springer, 2016.
Zhongping Zhang and Youzuo Lin. Data-driven seismic waveform inversion: A study on the ro- bustness and generalization. IEEE Transactions on Geoscience and Remote sensing, 58(10): 6900-6913, 2020.
Zhongping Zhang, Yue Wu, Zheng Zhou, and Youzuo Lin. Velocitygan: Subsurface velocity image estimation using conditional adversarial networks. In 2019 IEEE Winter Conference on Applica- fions of Computer Vision (WACV), pp. 705-714. IEFF, 2019b.
13

---

Under review as a conference paper at ICLR 2023
Bo Zhao, Bo Chang, Zequn Jie, and Leonid Sigal. Modular generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV). pp. 150-165, 2018.
Yinan Zhao, Brian Price, Scott Cohen, and Danna Gurari. Guided image inpainting: Replacing an image region by pulling content from another image. 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1514-1523. IEEE, 2019.
Jun-Yan Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A Efros. Generative visual manipu- lation on the natural image manifold. In European conference computer vision, pp. 597-613.
Springer, 2016.
Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2347-2356, 2019.
A IMPLEMENTATION DETAILS
A.1 SEGMENTATION MODULE
CLEVR. We apply the CLEVR framework (Johnson al., 7) generate dataset consisting of 1,000 images with object masks. Following (Dhamo et al. 2020). Mask R-CNN (He et al. 2017)) is trained classify the objects into 24 categories with 3 shapes (sphere, cylinder, cube) and 8 colors (blue, yellow, purple, green, red, gray, cyan, brown).
Visual Genome. We use MaskFormer (Cheng al. 202 pre-trained on COCO (Lin et 2014) for Visual Genome. In this paper, we focus on objects that the MaskFormer is trained to identify. However, these categories can be greatly expanded using methods like the transfer learning approach Mask R-CNN (Hu et al. 2018) or segmenting entities based on natural language queries (Hu et al. 2016
A.2 ROI PREDICTION
CLEVR. As we mentioned in the main paper, the modified triplets, bounding boxes of reference objects and target objects are extracted from 6,688 images as our training data. 818 images are adopted as the validation set. We set the maximum number of the modified triplets of each image to 5. The mean absolute error (MAE) validation set is 12.05 (computed over runs), where the image size is 256 x 256. should be mentioned that the gap between the predicted value and ground truth value does not necessarily indicate the predicted value is incorrect. Any points in the correct region can satisfy the relationships between objects. For instance, if a blue cube is on the right side of red sphere, then any positions the right side of the red sphere should be correct. Therefore, the accuracy of our model is underreported.
Visual Genome. Similar to CLEVR, uation. The MAE on validation set is 77.84+ 14251.25 (computed over 5 runs), where the image size is 512 512.
40,000 images for training and 2089 images for eval-
A.3 BACKGROUND-GUIDE INTERNAL LEARNING
We the encoder-decoder structure with skip connections (Ulyanov al. 2018 for inpainting model. The iteration number for each image is set to 2,000 and À in Eq. 5 is set to 0.1. For CLEVR, constrain the average of each missing region by a single value. For Visual Genome, constrain the average of missing regions row by row.
A.4 MODIFICATIONS ON SCENE GRAPHS
To precisely reflect the modified relationships between objects, users may need to modify many graph edges. This process be simplified by modifying only few graph edges. For example. in our Visual Genome experiments, we observe that modifying a single triplet can also render plausible editing results, though it may bring randomness Rol prediction.
14

---

Under review as a conference paper at ICLR 2023
Table Ablation study of SIMBIL on CLEVR (256x256 image). DIP, guide, dilation denotes deep image prior, background-guider internal learning, and mask dilation respectively. See Section B for discussion.
All pixels
Rol only
Method
MAE
SSIM 1
LPIPS +
FID L
MAE
SSIM 1
DIP
1.85
98.16
0.069
7.13
25.38
73.48
DIP+guide
1.73
98.19
0.069
8.07
19.85
75.97
DIP+dilation
1.72
98.26
0.065
7.21
22.02
75.60
1.71
98.30
0.061
6.55
18.86
79.48
Table 4: Comparison of evaluating modifications A and B. See Semantic correctness of edits of Section B for details.
1 (Rol)
SSIM(Rol)
(Rool)
SSIM(RoIf
A
18.86
79.48
B
41.04
65.56
B ADDITIONAL EXPERIMENTS
provide additional qualitative results that demon- strating SIMBIL outperforms prior work on Visual Genome to supplement the results from the main paper.
Additional Qualitative Results. Figures 7
9
Diverse Outputs. As we discussed in Section A.2 modified scene graph can point to different outputs with correct semantics based on the same input. We generate diverse outputs using the manipulated relationship's 's probability map shown in Figure IO
Semantic correctness edits. To illustrate that the incorrect semantics of manipulated images could be captured by our quantitative metrics in the main paper, performed comparison experi- ment in Table[4] We applied modified scene graph A and modified scene graph B on the same input and compared their outputs with the ground truth label of scene graph (note Ay/B). We the correctly modified image has significantly better results.
Ablation Study. Table 3 ments of SIMBIL are based on both the background-guidee mechanism and the mask dilation.
provides the ablation study of mask dilation, validating that the improve-
Image Editing using Predicted Scene Graphs. We performed an experiment using scene graphs predicted by F-Net (Li et al.) 2018) in Table 5 Consistent with our experiments in Table 3 we performed four image manipulation operations the test set, including object removal, object ad- dition, object replacement, and relationship change. We see that using predicted scene graphs has minor impact on the performance image editing results compared ground truth scene graphs. However, our method with predicted scene graphs still outperforms SIMSG (F-Net), which means ground truth scene graphs are not essential for the improvement of our model. For other baselines including GLIDE(Nichol
al., 2021) and BDM (Avrahami et al., 2022), since Rol as predicted by our model (otherwise, GLIDE and BDM require manually outlining Rol), the influence of predicted scene graphs should be the same.
the same
Iterations in optimization. We compare the iterations of DIP and SIMBIL in Figure to demon- strate the improvement of our tackground-guide mechanism. E.g., in the bottom figure, the moved object is blue cylinder and DIP uses the color of purple cube which is far from the blue cylinder to fill in the missing part. In contrast, SIMBIL uses the color of grey background fill in the missing part.
15

---

Under review as a conference paper at ICLR 2023
Table Image editing using predicted scene graphs by F-Net (Li et al. 2018 GT denotes ground truth scene graphs. See Image Editing using Predicted Scene Graphs of Section for details.
4 0
SSIM f
4 (Rol) +
SSIM(Rolf
SIMSG (F-Net)
8.52
96.92
27.81
74.74
SIMSG (GT)
8.30
96.97
24.59
78.61
ours (F-Net)
1.94
98.17
23.55
76.29
ours (GT)
1.71
98.30
18.86
79.48
Input
SIMSG+
SIMBIL(dip)
SIMBIL
remove "person"
remove "person"
remove 'elephant"
remove "dog"
remove "cow"
"car'
remove 'motorcycle'
Figure 7: Object Removal. Target objects are outlined by light yellow bounding boxes. Consistent with our conclusion in Section4..3 our backgound-guice internal learning mechanism notably im- proves the inpainting results, especially when the removed object is large.
16

---

Under review as a conference paper at ICLR 2023
SIMSG+ USR-Net
SIMBIL(dip)
SIMBIL
replace "person' with 'zebra'
replace 'person' with 'dog'
replace 'glirl' with 'sheap'
replace "dog" with 'sheep'
replace 'motorcycle" with buus
replace with "car"
replace with 'boat"
Figure 8: Object Replacement. Our method can effectively replace target objects according to editing commands.
17

---

Under review as a conference paper at ICLR 2023
Input
SIMSG+ USR-Net
SIMBIL(dii))
SIMBIL
man riding wave
man beside wave
man riding horse
beside
bird on chair
man riding horse
man next to horse
girl near bike
bike
bench sidewalk
on sidewalk
near
bench on free
Figure 9: Relationship Change. SIMBIL puts target objects to reasonable positions according to different scene-graph modifications, supplementing discussion in Section
18

---

Under review as a conference paper at ICLR 2023
noar
neXt TO
standing on
probebiltyy raap
th Rols oupul
Figure 10: Top: probability map of selected predicates; Bottom: example of diverse outputs, "bird-on-water
remova "parson"
cylinder"
Figure 11: The optimization iterations of Deep Image Prior (Ulyanov et al. 2018) (top row each image) and method (bottom of each image). We background-guided mechanism to constrain the optimization of deep image prior and generate plausible results.
19


Reviews:
[{"review_id": "jwTKuNc5_vr", "reviewer": "Reviewer_BfVZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new framework, named SIMBIL, for semantic image manipulation. The framework contains three components, including a segmentation module to extract the target object according to the modified scene graph, a RoI prediction module to determine the new location of the target object, and a background-guided internal learning to complete the “hole” in the original location of the target object. ", "summary_of_the_review": "The paper proposed an interested method with great results. But the writing of the paper should be improved, and more analysis may be included. I would like to see in the rebuttal (a) the authors' answers to the above questions and (b) the authors' ideas for improving the manuscript to remove incorrect or contradictory descriptions, which determine the final score.", "strength_and_weaknesses": "Strength\n1. The idea of manipulating image by modifying its scene graph is new and has potential.\n2. The proposed background-guided internal learning makes sense and alleviates the drawbacks of existing schemes.\n3. The results on the validation benchmark are promising and surpass prior semantic image manipulation methods.\n\n\nWeaknesses\n1. There are some inaccuracies or self-contradictions in the description of high-level ideas of the proposed method. For example, \n(a) In Sec.1, the authors state “low-level manipulation spans image inpainting, object removal… do not need to understand the semantic meaning of an image”, which should be wrong. First, addressing tasks like image inpainting and object removal via neural networks requires semantic information. For example, for inpainting, it is necessary to understand the semantic information around the missing area to perform meaningful completion.\nSimilarly, before removing an object, it is necessary to know its semantic information to accurately segment the entire object. Second, the word “low-level” may be confusing here. “low-level” usually refers to tasks that are not related to image content/semantics, such as image denoising and deblurring. These tasks do not need to know what the image content is. Therefore, it is more accurate to use words like \"pixel-level manipulation\" rather than “low-level manipulation”.\n\n(b) In Sec.2, the authors state “image manipulation can be seen a special case of image synthesis”, which may be inaccurate. Image manipulation not only includes synthesis, many other operations, such as image enhancement, cropping, and denoising, are all image manipulations. Besides, some statements like “Unlike methods based on generative networks, SIMBIL … directly manipulating raw pixels” are difficult to understand (as SIMBIL also uses background-guided internal learning to generate pixels).\n\n(c) In Sec.3, the authors state “we perform four tasks of semantic manipulation: object addition, …, and object removal”, which conflicts with the statement in (a) above because “object removal” becomes a “semantic manipulation” here.\n\n2. It is not clear how the proposed components form object replacement flow. Specifically, whether the proposed framework adding the target object before or after completing the missing background?\n\n3. The rationale for using RoI Prediction for Relationship Change may be further discussed. Specifically, for object Relationship Change, modifying the scene graph requires a huge effort because many graph edges need to be modified (e.g., in Fig.2, moving the blue cylinder needs to change five edges carefully). Instead, the operation of directly dragging/resizing the segmented object to the specified position should be faster and more accurate (e.g., in Fig.2, users may directly drag the blue cylinder to a certain position in a precise manner). After that, the scene graph may be updated according to the user interactions in some way.\n\n\n4. Since the discussed limitations like artificial boundary and biased background are also appear in other tasks like image inpainting / generation. More task-specific limitations may be discussed. For example, as the proposed methods require scene graph as an input. In practice, a complete scene graph requires huge efforts to create, especially scenes with lots of objects. Hence, how robust the proposed method when inputting an incomplete scene graph?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LBHhdHzx387", "reviewer": "Reviewer_8S2u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper edits images with the guidance of a modified scene graph. Building on image deep prior, this paper takes the foreground and background separation into account for better synthesis quality. Experiments on CLEVR and visual genome illustrate the performance of the proposed method.", "summary_of_the_review": "Considering the weak technical contributions and potential problematic implementation of other methods, I do not suggest this paper for acceptance.", "strength_and_weaknesses": "**Strength**\n- This paper makes attempts to edit an image with a modified scene graph.\n\n- Experiments compare with many recent methods and a user study is given.\n\n**Weakness**\n\n- The contributions of this paper could be strengthened. For now, it is hard to find significant novelty and insights in the scene graph modification and RoI prediction part. As for the background-guided learning strategy, it simply separates the foreground and background and uses them for image generation. The overall technical novelty is weak.\n\n- The deep image prior, as a single image-based method, performs much worse than the ones learned from a large-scale image dataset. Thus, it is not convincing that the authors mainly base on and compare with the deep image prior work in Sec. 3.3.\n\n- The scene graph modification only contains four simple forms, making the use of a scene graph (as a scene graph can model very complicated scenes) questionable. Maybe a simple input triplet or a referring expression will work.\n\n- I would like to see more relationship change results for complicated natural images. \n\n- The visual results are unsatisfactory and contain noticeable artifacts. For example, the *remove* results are much worse compared with the reported results in state-of-the-art papers (the results of LDM are obviously wrong). The authors should check their experiment settings and make sure the compared results are correct.\n\n- The organization and presentation of this paper could be improved. I am confused about the relationship between object removal and scene graph editing.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8NqpVi6M3of", "reviewer": "Reviewer_Pno2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes SIMBIL to manipulate the image in high-level and low-level aspects. High-level manipulation requires less manual effort from the user compared to manipulating raw image pixels. The low-level internal learning approach is scalable to images of various sizes without reliance on external visual datasets for training. The experiments show the good performance of this paper.\n", "summary_of_the_review": "See the weakness.", "strength_and_weaknesses": "# Strength\n1. This paper use scene graphs to build the relationship between objects. And the users can manipulate the image with the scene graphs, which is more flexible than the low-level control method.\n2. The SIMBIL can automatically predict RoI for the object addition and semantic relationship change tasks. This strategy helps the users not select the specific regions of the object and improves the robustness of SIMBIL.\n3. The background-guided internal learning is reasonable, which can help the complete content consistent with the background even with the large holes.\n\n# Weakness\n1. Is the correct scene graphs essential for the performance of image manipulation? The authors should compare the different scene graph prediction methods. \n2. If the image has no multi-object, is the SIMBIL can work well? For example, there is a sky in an image, and the sky has many clouds, dose the scene graphs can build the relationship between these clouds?\n3.  For object removal, the edge-connect is not the SOTA. The authors should compare the SOTA method [1,2]. \n4. The novelty of SIMBIL is lacking. The main difference between the proposed methods and SIMSG is background-guided internal learning. I think this is not enough for the ICLR. As I mentioned above, if SIMBIL uses the SOTA object removal method, I think the performance may be better. \n\n[1] Suvorov, Roman, et al. \"Resolution-robust large mask inpainting with fourier convolutions.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.\n\n[2] Li, Wenbo, et al. \"MAT: Mask-Aware Transformer for Large Hole Image Inpainting.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5_Ycbr2zWZS", "reviewer": "Reviewer_mRHW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a semantic manipulation framework, combining low-level and high-level conditions, for object removal, object replacement, semantic relationship change and object addition. Quantitative and qualitative experiments validate the effectiveness of this design.", "summary_of_the_review": "The paper develops a unified system for multiple image manipulation tasks, demonstrating its high flexibility. But the authors should compare different methods in a more rigorous manner, and the visual results should be further improved. More technical novelties are required.", "strength_and_weaknesses": "### Strengths\n1. Each of the four mentioned problems is well-defined and carefully processed in SIMBIL. Compared with existing methods, the method shows superior performance in semantic understanding and image manipulation.\n2. The resource requirement of SIMBIL is more relaxed, without reliance on human annotations and external large datasets, thus being more user-friendly.\n3. The paper is easy to follow and the system is easy to reproduce.\n\n### Weaknesses\n1. The comparison is not so convincing as the involved competing methods are not specifically trained on the datasets as SIMBIL does. It remains questionable whether SIMBIL will be still the best if other methods are well-finetuned or re-trained.\n2. The visualized results of SIMBIL in Fig.4 and Fig.5 are far from satisfying. We observe obvious unappealing artifacts or the prediction is quite different from the ground truth, e.g., the replaced object in Fig.4 (the 2nd case) should be a ball instead of a cube.\n3. The paper introduces a well-designed system, but the technical novelty is quite limited. The background-guided internal learning is quite ad-hoc for object removal. Besides, the system requires the GT scene graph for manipulation, which is impractical in real-world cases. Though scene graph generation methods could be adopted, we do not know the degree of performance drop so that the robustness of the system should be evaluated.\n4. Some typos should be fixed. For example, the 2nd line 'Benefits from ... outline ...' should be 'Benefiting from ... outlining ...'.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject