Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Paper Summary**
The paper proposed a carefully designed neural rendering pipeline to realistically insert one image fragment into another image via deep image prior and the consistency in the albedo, shading, and gloss in the rendered image. Experiments show great improvement over baselines both qualitatively and quantitatively (via user study). 

**Strength**
1. The inverse partial convolution is a neat idea. This idea effectively helps the network to learn the shading of the target image and to avoid directly copying the inserted objects.
2. The shading consistency loss and the designed experiments to pre-train the image decomposition network are also carefully designed and thoughtful. The analysis in Fig 5 also provides insights for the relationship between WHDR and the actual rerendering effects. This could inspire many followups on how we should design the algorithm to better composite an image.
3. The experiments are also well performed. To quantitatively evaluate the performance, this paper conducted user studies with detailed descriptions of the whole procedure. And qualitatively, the results the paper showed are better than baselines. 

**Weakness**
1. Speed, the overall pipeline relies on the training Deep Image Prior (DIP), making the whole pipeline slow during the real application.
2. Some notation in this paper is not clear. E.g.  in Eq. 6,  what does the second term means? 
3. The paper didn't provide the exact method for Image Harmonization baseline, it would be better for the readers to understand what's the exact method to be compared. 
4. It's not clear how does the proposed method generalize to high-order lighting effect since the lighting model the paper used is Spherical Harmonic. 

**Overall**
Overall, I think this paper proposed many neat ideas for image composition and performed solid experiments, but considering the limitation of the method (which has been described in Sec. 5), I vote for a weak accept initially.
review_title: Carefully designed pipeline with well conducted experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
This paper introduces a reshading method for cut-and-paste image composition. It uses a modified deep image prior as the rendering networking, and trains with a novel shading consistency loss. The results are plausible. Inserted fragments have proper shading with the source image and unchanged albedo.

Pros:
- an improved DIP that produces consistent image decomposition inferences (albedo, shading, gloss).
- a novel shading consistency loss that accounts for the spatial similarity and past inferences.
- Extensive experiments showing advantages over CP and IH.

Cons:
- Is scene geometry not needed for this task?

Cut-and-paste methods may bring ambiguity for the inserted object. Cars in Fig 7 have a fixed depth because it is grounded in human perception. However, the circle cut-outs in Fig 11 are floating, which makes these disks have ambiguous depth. For example, in the second last row of Fig 11, a floating disk closer to the camera will have darker shading while a disk with further depth (under the lamp) will be lit up and has brighter lighting. In this paper, the only supervision on scene geometry is the surface normal in the shading loss. The inserted fragment may have an ambiguous depth in human perception. Therefore, in the inference phase, the shading factor for the cut-out might be ambiguous.

- Quantitative evaluation on albedo

One way to evaluate the reshading method is to measure how much the inserted fragment's albedo changed. Results from CP, RS and IH could be passed into the image decomposition network, and the albedo values could be compared with the ones in the source image.

- Timing for inference

How much time does it take for DIP to render a 1024p image? The paper mentions minutes, but could be more precise.

Comments:
I don't quite follow the spherical harmonics coefficients and how they are calculated from normals and shading. There could be a formulation equation in Sec 3.2 or in supplementary.

Updates: Thanks for the author's response. My concerns are mostly addressed. But I still believe explicit geometry modeling should be included for this task. This could be added in future works. Overall, I am positive on the submission and keep my original rating.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes cut-and-paste neural rendering that allows to insert objects into a target scene in a plausible manner, i.e., in terms of shading plausibility. At the core of the approach is a deep image prior that allows to match the shading and albedo fields based on shading and albedo consistency losses. A normal estimation network that is trained based on synthetic data is used to further inform shading estimation. The approach is interesting and shows plausible results.

The submission performs many ablation studies that give a better understanding of the inner workings of the approach. The approach is compared to several other methods that are outperformed visually and in a user study. The results obtained by this approach look plausible.

There is one statement in the paper I disagree with: “We cannot quantitatively evaluate our reshading method, because we do not know ground truth”. It is possible in two ways: Completely synthetic scenes that are rendered using computer graphics or real scene where images are taken before and after inserting an object. This would allow access to ground truth pairs of images with and without the object. It would be interesting to see how close the outputs of these methods are to the actual ground truth, both qualitatively and quantitatively. For all shown results, I was always unsure how it would look in reality, which makes it quite hard to judge how good the method actually performs. For sure this approach produces more plausible results than previous approaches, but how close are these results actually to how it would look in reality. I think this question is unanswered so far.

In summary, I like the submission and I think the proposed approach is novel, but I would have hoped for an actual evaluation against GT data. Therefore, I am currently a bit on the fence to suggest acceptance, but would like to hear the opinion of the other reviewers and see the author response.
review_title: Cut-and-paste Neural Rendering

Meta-review: Overall the review is borderline: R2 and R4 are slightly positive and R3 is slightly negative. All the reviewers like the novel shading consistency loss proposed in the paper and, improved DIP that produces consistent image decomposition inferences, and good experimental results. However, reviewers also shared concerns about speed and the thoroughness of the evaluation, and human tolerance of shading inaccuracy. These points were addressed in details in the rebuttal, and reviewers didn’t change their initial scores.

The AC is concerned about the cut-and-paste neural rendering results. Because there are no cast shadows, the rendering doesn’t look realistic under the lighting conditions in the new image. It’s unclear that the proposed method would lead to a promising direction of copying and pasting contents into images for photorealistic editing. Consequentially, the paper is not ready for publication at its current form.

Decision: Reject