Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new image harmonization method. Image harmonization aims at compositing regions of different images in a way the observer can not tell they were coming from different original images. The main novelty of this work is the allowance of interaction by the user. In the proposed model the user can select which part of the background image should be used for harmonization. Furthermore, thae authors also propose a new loss based on luminance inspired by professional users.
strength_and_weaknesses: Strenghts:

- The paper is easy to follow.
- The results overtake the state-of-the-art.
- Allowing the user to select the region of interest is interesting.

Weaknesses:
- Luminance Matching Loss: The Luminance matching loss is not perceptual. It is well known that to be able to perceive a change of luminace value, we need a much larger difference in the highlight regions that in the shadow regions (Weber's law). For this reason, I stringly believe than the three different terms in the luminace loss shall be weighted according to this fact.

- The authors use a ranking experiment for the user study. These type of experiments are known to be very noisy. Also, there is not a reference in how statistical significant is the distance of 0.19 between IPH and Bargain-Net. For this reason, the authors should instead go for a pair-wise comparison, and then use the Thurstone Case V law (See [1]) to also study the statistical significance of the results.

- Authors need to discuss why when including the Shadow loss and the Highlight loss the method improves PSNR but get worse in SSIM. I believe it might be related to my first comment.

- Also, it is not make sense to put both PSNR and MSE in the Tables. They are completely correlated as PSNR is directly computed from MSE. Also, authors need to include some Color metric, in order to prove that no color artifacts, such as Hue shifts appear in the result. I recommend CIE DElta E.

[1]  Color Gamut Mapping, Ján Morovič, 2008, John Wiley & Sons, Ltd

clarity,_quality,_novelty_and_reproducibility: Clarity:

- The paper has not been fully edited. For example, section 2 is almost impossible to read, as the name of the authors in many papers is repeated twice (included in the text, and also as the reference). Also, Table 2 does not make sense if the ranking are going between 1 and 3; I am guessing the authors put the decimal point in the incorrect location.

Quality:
- The paper seems to have enough quality for ICLR. The idea is good and the results improve over the state of the art.

Novelty:
- The paper seems to be novel enough; the idea of interactive harmonization deserves publicatiion.

Reproducibility:
- Authors will publish all the codes once the paper is accepted.
summary_of_the_review: This is an interesting paper with good results. Some issues need to be addresses (see my detailed explanation in the Weaknesses section). For this reason, my rating is somehow between 5 and 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces interactive image harmonization that enables the user to select a specific object/person to guide. The paper also proposes a new technique of using luminance matching loss and a new dataset for validating portrait harmonization. Experimental results show the effectiveness over the previous methods.
strength_and_weaknesses: Strength

1. Experimental results show the effectiveness significantly.
2. The ablation study shows the effectiveness of using the proposed luminance matching loss.

Weakness
1. In term of deep learning model, it looks like very similar to the previous model of DoveNet. The paper does not discuss the difference in details.

2. If two objects are in different colors, such as red car as a reference region and blue car as a foreground image, what would happen?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written besides a typo of "a interactive fashion" on page 6. The novelty of selecting a region as a reference area may not be very strong because it would be user interface issue rather than research issue of image harmonization.
summary_of_the_review: The paper is introducing interactive image harmonization that allows the user to select a region as a guide. When two or more people/objects are in the same image, if the user uses a person/object as a reference image, the generated image would be better especially for portrait photos taken in a studio with a solid background. Experimental results show the effectiveness of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a new framework, namely interactive portrait harmonization, to allow users to select a reference region in the background image to guide the harmonization. The key to this framework is a luminance matching loss to ensure the luminance consistency between the selected reference region and the foreground region. Two datasets are also constructed for training and testing, respectively. Extensive experiments on natural and synthetic data demonstrate the effectiveness of the proposed framework.

strength_and_weaknesses: Strength
+ A novel framework with a luminance matching loss is proposed to solve the interactive portrait harmonization.
+ The paper is well-written and easy to follow.

Weaknesses
- The harmonized foreground is only suitable for the reference region, not the foreground placement region. In fact, in the real world, the illumination at different locations in the background might be different, which results in appearance differences of the foreground object. Besides, is it reasonable to RANDOMLY choose an instance mask for dataset construction as a reference guide mask? How to ensure the luminance/color consistency between the foreground placement location and the random reference location?
- The performance of the interactive image harmonization heavily depends on the user’s selected reference guide region. Compared with those automatic image harmonization methods, the reference guide region selected by the user is not necessarily optimal. In addition, in some cases, the partial region of the background is inadequate to infer the luminance/color conditions of the foreground placement region.
- For the luminance matching loss, the author claimed three new losses are used to match the highlight, mid-tone, and shadow between the reference region and foreground region. But these equations (1)-(3) indicate that the losses only relate to the predicted harmonized image and the ground truth, which is unrelated to the reference region.

clarity,_quality,_novelty_and_reproducibility: The paper is well organized and appears to be technically sound.
summary_of_the_review: The paper introduced a new setting, interactive portrait harmonization, to allow users to select a reference region in the background image to guide the harmonization. However, the luminance/color conditions of the reference region do not necessarily match that of the foreground region, which limits the realism of interactive harmonization. In addition, the reference guide region is subjective and not necessarily optimal. Thus, I give the paper a borderline reject rating.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new framework for interactive harmonization. A new network is proposed to allow users to pick certain regions of the background and perform harmonization on the foreground. A novel luminance matching loss and a new synthetic dataset and a real test dataset is proposed. The proposed network is shown to perform better than previous generic harmonization methods on the new datasets as well as on some “in the wild” edited portrait images.
strength_and_weaknesses: -> The  idea of interactive harmonization is interesting and could meet the practical demands of concise human harmonization in the real world. This is the first work to look into this problem.
	
-> One of the main contributions is a large-scale dataset with reference regions annotated. And they also build a benchmark test with professionally edited harmonization results. This dataset could be helpful if it is being released publicly. The paper has relatively extensive evaluations on the dataset and also vivid examples.

-> The loss function introduced seems to help well in achieving a good performance while looking at the ablation study. The inspiration of the new loss function being what is done in manual harmonization is sound.

->The paper shows results on some “in the wild” edited images which look impressive. 

-> The framework is shown to be robust and have different results for different reference regions and also shown to have multiple other use-cases like color transfer and easy switching to automatic harmonization.

clarity,_quality,_novelty_and_reproducibility: 
-> This paper focuses on harmonization but one other task very relevant to it is re-lighting. The paper should discuss more on where this method stands when compared to relighting or why it is different. The proposed luminance loss seems to focus more on the light aspect; so more details  are needed.

-> Comparison with previous methods on “in the wild” edited images for background harmonization is shown only on Fig 1. More examples of how the background harmonization fails would be interesting to see.

->  PortraitTest dataset proposed in the paper is said to have  ground-truth annotations collected from professional experts. There aren't any more details about this. How were these GTs created? Using manual harmonization techniques? If yes, what parameters were changed? More details are needed.

-> The increase in performance seems high for the real dataset when compared to the synthetic dataset which is weird considering the network is trained on synthetic dataset. Any explanations for it?

-> Figure 1 Caption. “Bottom row- Casual portrait.” The term casual can be replaced with Non-studio.

summary_of_the_review: Based on the strength and weakness mentioned above, I recommend to accept this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach for portrait harmonization - adjusting the appearance of a foreground object pasted into a photo, to make for a realistically looking composite. In particular, the paper studies a new setting, interactive harmonization, where not the full background image is used as a reference for harmonization, but a (user-selected) region. The method seems to work well and outperform the alternatives.

The reviewers are overall quite positive about the paper, with three of them arguing for acceptance and one against. The main points are as follows:

Pros:
1. An interesting and new setting
2. Good experimental results, including comparisons to baselines and an ablation study
3. A new dataset that will be released

Cons:
1. Somewhat limited technical contribution
2. Doubts about random selection of reference mask being a good choice and generally putting work of reference selection on the user being a good idea

Overall, the paper seems technically sound, introduces a new potentially practically relevant problem setting, a corresponding dataset, and a working solution. So I recommend accepting the paper. I encourage the authors to address the reviewers' comments where possible, in particular perhaps run some quantitative experiments on the effect of region selection on the result.
justification_for_why_not_higher_score: It's a good paper, but not groundbreaking.
justification_for_why_not_lower_score: the paper seems technically sound, introduces a new potentially practically relevant problem setting, a corresponding dataset, and a working solution
Decision: Accept: poster