Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new framework, named SIMBIL, for semantic image manipulation. The framework contains three components, including a segmentation module to extract the target object according to the modified scene graph, a RoI prediction module to determine the new location of the target object, and a background-guided internal learning to complete the “hole” in the original location of the target object. 
strength_and_weaknesses: Strength
1. The idea of manipulating image by modifying its scene graph is new and has potential.
2. The proposed background-guided internal learning makes sense and alleviates the drawbacks of existing schemes.
3. The results on the validation benchmark are promising and surpass prior semantic image manipulation methods.


Weaknesses
1. There are some inaccuracies or self-contradictions in the description of high-level ideas of the proposed method. For example, 
(a) In Sec.1, the authors state “low-level manipulation spans image inpainting, object removal… do not need to understand the semantic meaning of an image”, which should be wrong. First, addressing tasks like image inpainting and object removal via neural networks requires semantic information. For example, for inpainting, it is necessary to understand the semantic information around the missing area to perform meaningful completion.
Similarly, before removing an object, it is necessary to know its semantic information to accurately segment the entire object. Second, the word “low-level” may be confusing here. “low-level” usually refers to tasks that are not related to image content/semantics, such as image denoising and deblurring. These tasks do not need to know what the image content is. Therefore, it is more accurate to use words like "pixel-level manipulation" rather than “low-level manipulation”.

(b) In Sec.2, the authors state “image manipulation can be seen a special case of image synthesis”, which may be inaccurate. Image manipulation not only includes synthesis, many other operations, such as image enhancement, cropping, and denoising, are all image manipulations. Besides, some statements like “Unlike methods based on generative networks, SIMBIL … directly manipulating raw pixels” are difficult to understand (as SIMBIL also uses background-guided internal learning to generate pixels).

(c) In Sec.3, the authors state “we perform four tasks of semantic manipulation: object addition, …, and object removal”, which conflicts with the statement in (a) above because “object removal” becomes a “semantic manipulation” here.

2. It is not clear how the proposed components form object replacement flow. Specifically, whether the proposed framework adding the target object before or after completing the missing background?

3. The rationale for using RoI Prediction for Relationship Change may be further discussed. Specifically, for object Relationship Change, modifying the scene graph requires a huge effort because many graph edges need to be modified (e.g., in Fig.2, moving the blue cylinder needs to change five edges carefully). Instead, the operation of directly dragging/resizing the segmented object to the specified position should be faster and more accurate (e.g., in Fig.2, users may directly drag the blue cylinder to a certain position in a precise manner). After that, the scene graph may be updated according to the user interactions in some way.


4. Since the discussed limitations like artificial boundary and biased background are also appear in other tasks like image inpainting / generation. More task-specific limitations may be discussed. For example, as the proposed methods require scene graph as an input. In practice, a complete scene graph requires huge efforts to create, especially scenes with lots of objects. Hence, how robust the proposed method when inputting an incomplete scene graph?

clarity,_quality,_novelty_and_reproducibility: The proposed method has a certain novelty and achieves promising results. However, insights and in-depth analytical discussions behind the proposed method are missing.
Besides, the writing of the article needs to be improved. There are some misinterpretations of basic knowledge, some sentences that are difficult to understand, and some self-contradictory descriptions.

summary_of_the_review: The paper proposed an interested method with great results. But the writing of the paper should be improved, and more analysis may be included. I would like to see in the rebuttal (a) the authors' answers to the above questions and (b) the authors' ideas for improving the manuscript to remove incorrect or contradictory descriptions, which determine the final score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper edits images with the guidance of a modified scene graph. Building on image deep prior, this paper takes the foreground and background separation into account for better synthesis quality. Experiments on CLEVR and visual genome illustrate the performance of the proposed method.
strength_and_weaknesses: **Strength**
- This paper makes attempts to edit an image with a modified scene graph.

- Experiments compare with many recent methods and a user study is given.

**Weakness**

- The contributions of this paper could be strengthened. For now, it is hard to find significant novelty and insights in the scene graph modification and RoI prediction part. As for the background-guided learning strategy, it simply separates the foreground and background and uses them for image generation. The overall technical novelty is weak.

- The deep image prior, as a single image-based method, performs much worse than the ones learned from a large-scale image dataset. Thus, it is not convincing that the authors mainly base on and compare with the deep image prior work in Sec. 3.3.

- The scene graph modification only contains four simple forms, making the use of a scene graph (as a scene graph can model very complicated scenes) questionable. Maybe a simple input triplet or a referring expression will work.

- I would like to see more relationship change results for complicated natural images. 

- The visual results are unsatisfactory and contain noticeable artifacts. For example, the *remove* results are much worse compared with the reported results in state-of-the-art papers (the results of LDM are obviously wrong). The authors should check their experiment settings and make sure the compared results are correct.

- The organization and presentation of this paper could be improved. I am confused about the relationship between object removal and scene graph editing.

clarity,_quality,_novelty_and_reproducibility: The authors are encouraged to improve the clarity and quality of this paper. The novelty of this paper is below the quality bar of an ICLR paper. As a straightforward improvement on DIP, this paper could be easily reproduced by modifying the DIP code.
summary_of_the_review: Considering the weak technical contributions and potential problematic implementation of other methods, I do not suggest this paper for acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes SIMBIL to manipulate the image in high-level and low-level aspects. High-level manipulation requires less manual effort from the user compared to manipulating raw image pixels. The low-level internal learning approach is scalable to images of various sizes without reliance on external visual datasets for training. The experiments show the good performance of this paper.

strength_and_weaknesses: # Strength
1. This paper use scene graphs to build the relationship between objects. And the users can manipulate the image with the scene graphs, which is more flexible than the low-level control method.
2. The SIMBIL can automatically predict RoI for the object addition and semantic relationship change tasks. This strategy helps the users not select the specific regions of the object and improves the robustness of SIMBIL.
3. The background-guided internal learning is reasonable, which can help the complete content consistent with the background even with the large holes.

# Weakness
1. Is the correct scene graphs essential for the performance of image manipulation? The authors should compare the different scene graph prediction methods. 
2. If the image has no multi-object, is the SIMBIL can work well? For example, there is a sky in an image, and the sky has many clouds, dose the scene graphs can build the relationship between these clouds?
3.  For object removal, the edge-connect is not the SOTA. The authors should compare the SOTA method [1,2]. 
4. The novelty of SIMBIL is lacking. The main difference between the proposed methods and SIMSG is background-guided internal learning. I think this is not enough for the ICLR. As I mentioned above, if SIMBIL uses the SOTA object removal method, I think the performance may be better. 

[1] Suvorov, Roman, et al. "Resolution-robust large mask inpainting with fourier convolutions." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[2] Li, Wenbo, et al. "MAT: Mask-Aware Transformer for Large Hole Image Inpainting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.


clarity,_quality,_novelty_and_reproducibility: Clarity: fair
Quality: fair
Novelty: poor 
Reproducibility: fair
summary_of_the_review: See the weakness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a semantic manipulation framework, combining low-level and high-level conditions, for object removal, object replacement, semantic relationship change and object addition. Quantitative and qualitative experiments validate the effectiveness of this design.
strength_and_weaknesses: ### Strengths
1. Each of the four mentioned problems is well-defined and carefully processed in SIMBIL. Compared with existing methods, the method shows superior performance in semantic understanding and image manipulation.
2. The resource requirement of SIMBIL is more relaxed, without reliance on human annotations and external large datasets, thus being more user-friendly.
3. The paper is easy to follow and the system is easy to reproduce.

### Weaknesses
1. The comparison is not so convincing as the involved competing methods are not specifically trained on the datasets as SIMBIL does. It remains questionable whether SIMBIL will be still the best if other methods are well-finetuned or re-trained.
2. The visualized results of SIMBIL in Fig.4 and Fig.5 are far from satisfying. We observe obvious unappealing artifacts or the prediction is quite different from the ground truth, e.g., the replaced object in Fig.4 (the 2nd case) should be a ball instead of a cube.
3. The paper introduces a well-designed system, but the technical novelty is quite limited. The background-guided internal learning is quite ad-hoc for object removal. Besides, the system requires the GT scene graph for manipulation, which is impractical in real-world cases. Though scene graph generation methods could be adopted, we do not know the degree of performance drop so that the robustness of the system should be evaluated.
4. Some typos should be fixed. For example, the 2nd line 'Benefits from ... outline ...' should be 'Benefiting from ... outlining ...'.
clarity,_quality,_novelty_and_reproducibility: This work gives us some new insights into image manipulation, which is interesting. The paper is well-organized and explained clearly.
summary_of_the_review: The paper develops a unified system for multiple image manipulation tasks, demonstrating its high flexibility. But the authors should compare different methods in a more rigorous manner, and the visual results should be further improved. More technical novelties are required.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission receives 1 slightly positive review and 3 negative reviews. The raised issues include unclear technical presentation, limited contributions to existing technical modules, unmotivated scene graph constructions, and unsatisfied experimental results. Although the authors try to address these issues during the rebuttal phase, the novelty concern still exists, and the AC feels the current work is not ready for reporting in this venue. The authors shall take the suggestions to further improve the current submission and welcome to the next venue.
justification_for_why_not_higher_score: The technical issue with limited novelty makes the main concern of the current work.
Decision: Reject