Cut-and-Paste Neural Rendering
Authors: Anand Bhattad
David Forsyth
Venue:  (2021)
Keywords: Neural Rendering
Reshading
Relighting
Computational Photography
Image Decomposition

Abstract: Cut-and-paste methods take an object from one image and insert it into another.   Doing so often results in unrealistic looking images because the inserted object's shading is inconsistent with the target scene's shading. Existing reshading methods require a geometric and physical model of the inserted object, which is then rendered using environment parameters. Accurately constructing such a model only from a single image is beyond the current understanding of computer vision.

We describe an alternative procedure -- cut-and-paste neural rendering, to render the inserted fragment's shading field consistent with the target scene.  We use a Deep Image Prior (DIP) as a neural renderer trained to render an image with consistent image decomposition inferences.  The resulting rendering from DIP should have an albedo consistent with composite albedo; it should have a shading field that, outside the inserted fragment, is the same as the target scene's shading field; 
and composite surface normals are consistent with the final rendering's shading field. 
The result is a simple procedure that produces convincing and realistic shading. Moreover, our procedure does not require rendered images or image-decomposition from real images in the training or labeled annotations. In fact, our only use of simulated ground truth is our use of a pre-trained normal estimator. Qualitative results are strong, supported by a user study comparing against state-of-the-art image harmonization baseline.

Under review as a conference paper at ICLR 2021
CUT-AND-PASTE NEURAL RENDERING
Anonymous authors
Paper under double-blind review
Target Scene
Cut-and-Paste
Cong et al. (0200)
Ours
We generate realistic renderings of cut-and-paa aand-patte images. Our method is entirely image- based and can convineingly reshade/relight fragments with complex surface properties (a lego dozer, plant and chair in top row) and matte, glossy and specular fragments (a set of 16 different materials bottom row) added to a spatially varying illuminated target scene (indoor-outdoor and day-night) without requiring the geometry of the inserted fragment the parameters of the target scene.
Figure 1:
ABSTRACT
Cut-and-passe methods take an object from one image and insert it into another. Doing so often results unrealistic looking images because the inserted object's shading is inconsistent with the target scene's 's shading. Existing reshading methods require a geometric and physical model of the inserted object, which is then rendered using environment parameters. Accurately constructing such a model only from a single image is beyond the current understanding of computer vision.
We describe alternative procedure sutt-and-patte neural rendering, to render the inserted fragment's 's shading field consistent with the target scene. We use Deep Image Prior (DIP) neural renderer trained render image with consistent image decomposition inferences. The resulting rendering from DIP should have albedo consistent with cut-and-paste albedo: should have shading field that, outside the inserted fragment, is the same as the target scene's 's shading field; and cut-and-paste surface normals are consistent with the final rendering's 's shading field. The result is simple procedure that produces convincing and realistic shading. Moreover, our procedure does not require rendered images or image decompasition from real images any form of labeled annotations in the training. fact, our only use simulated ground truth is our use of a pre-trained normal estimator. Qualitative results are strong, supported by user study comparing against ofate-of art image harmonization baseline.

---

Under review as a conference paper at ICLR 2021
INTRODUCTION
Cut-and-Paste rendering involves creating a new image by cutting fragments out of one or more souree images and pasting them into target image; the idea originates with Lalonde al. (2007). Results are often unrealistic, because of the difference in illumination between the source and target images. But the procedure is useful to artists, and there is consistent evidence that such procedures can be used to train detectors (Liao et al. 2012 Dwibedi et al. (017). When the geometry and material of the inserted object are known, it is enough to infer an illumination model from the target, render and composite. But current procedures for recovering shape and material from a single fragment simply can't 't'tt deal with most realistic fragments (think of, say, a furry cat).
This paper describes an alternative method, Cut-and- Paste Neural Rendering, that can render convinc- ing composite images by adjusting the cut-and-part images so that some simple image inferences consistent with cut-and-pase paste predictions. So the albedo from the adjusted image should look like cut-and-paste albedo; the shading should look like a shading field; and the image should look like an image. A simple post-processing trick produces very high-resolution composites. Note that all our rendered images are 1024x 1024 pixels resolution and are best viewed on screen. Evaluation is mostly qualitative, but we show that our method fools recent method for detecting tampering.
Our contribution is a method that can realistically correct shading in composite images, without requiring labeled data; our method works for matte, glossy and specular fragments without an explicit geometric or physical model; and human subjects prefer the results of our method over cut-and-pass and image harmonization.
2 RELATED WORK
Object Insertion starts with Lalonde et al. (2007), who insert fragments into target images. Lalonde et al. (2007) control illumination problems by checking fragments for compatibility with targets;
Bansal al. (2019) do so by matching contexts. Poisson blending (Pérez al., 2003 Jia et al.) 2006) can resolve nasty boundary artifacts, but significant illumination and color mismatches will cause cross-talk between target and fragment, producing ugly results. Karsch et al. (20TT) show that computer graphics (CG) objects can be convincingly inserted into inverse rendering models got with a geometric inference or with single image depth reconstruction (Karsch et al.| [2014). Inverse rendering trained with rendered images can produce excellent reshading of CG objects (Ramachandran 1988). However, recovering a renderable model from an image fragment extremely difficult, particularly if the fragment has an odd surface texture. Liao et al. showed that weak geometric model of the fragment can be sufficient to correct shading if one has strong geometric information about the target scene (Liao et al. 2015 2019). In contrast, our work is entirely image-based: one takes fragment from one image, drops it into another, and expects a system to correct it.
We use image harmonization (IH) methods strong baseline. These procedures aim to correct corrupted images. IH methods are trained to correct images where a fragment has been adjusted by some noise process (made brighter; recolored; etc.) the original image (Sunkavalli et al. 2010 Tsai et al., 2017] Cong et al. (2020), and so could clearly be applied here. But we find those image harmonization methods very often change the albedo of an inserted object, rather than its shading. This is because they rely on ensuring consistency of color representations across the image. For example, in the iHarmony dataset from Cong al. (020), they change pink candy to brown (an albedo change; see Fig 12 in Appendix). In contrast, we wish to correct shading alone.
Image Relighting. With appropriate training data, for scenss,seenes, one can predict multiple spherical harmonic components of illumination (Garon et al. (019)). or parametric lighting model (Gardner et al. 2019) or even full radiance maps at scene points from images (Song & Funkhouser Sriniiannn et al.) 2020). For outdoor scenes, the sun's position is predicted in panoramas using a learning- based approach (Hold-Geoffroy et al., 2011). One can also construct a volumetric radiance field from multi-view data to synthesize novel views (Mildenhall et al. 2020). However, we maps or multi-view
do not have access to either training data with lighting parameters/enviromnet data to construct such radiance field. Our renderings entirely image-based. Recent single- image relighting methods relight portrait faces under directional lighting (Sun et al.) 2019 Zhou et al. 2019 Nestmeyer al. 2020). Our method can relight matte, gloss and specular objects with
2

---

Under review as a conference paper at ICLR 2021
Consistent Inferences
Image
Decumposition
Network
Doep Image Prior
Input
Mask
Rendered
Albedo
Shading
Gloss
Figure 2: Cut-and-Paste Neural Renderer. Given two images, target scene (top row) and new fragment added to this target scene (bottom row), our approach generates a plausible, realistic rendering of the composite image by correcting the fragment's shading. Our method uses DIP as a neural renderer trained to produce consistent image decomposition inferences. The resulting rendering from DIP should have an albedo same the cut-and-paste albedo; it should have a shading and gloss field that, outside the inserted fragment, is the same as the target scene's 's shading and gloss field. This simple process produces convincing composite rendering for any cut-and paste images.
complex material properties like cars (Fig 7 for both indoor and outdoor spatially varying illuminated environments only from single image and without requiring physics-based BRDF (Li et al. (2020)
Image decomposition, Land's influential Retinex model assumes effective albedo displays sharp. localized changes (which result in large image gradients), and that shading has small gradients (Land] 1959a b 1977 Land & McCann 1971). These models require no ground truth. An alternative is
use CG rendered images for image decomposition training (Li & Snavely 2018), particularly
with specialized losses (Bi ct al. 2015; Fan et al., 2018) One can also train using rendering constraints to produce form of supervuuer training (annerr al. (017). Current image decomposition evaluation uses the weighted human disagreement rate ( (WHDR) (Bell et al., 2014); current champions are (Fan et al.) 2018). We use an image decomposition method built around approximate statistical models of albedo and shading (paradigms) to train our image decomposition network without requiring real image ground truth decompositions. Our method has reasonable, but not SOTA, WHDR; but we show that improvements in WHDR do not result in improvements in reshading (Fig(5).
CUT-AND-PASTE NEURAL RENDERER
We synthesize reshaded composite image containing a fragment transferred from source image into a target scene image. We use a deep image prior (DIP) (Ulyanov et al. (0188) as a neural renderer to produce a reshaded image that produces consistent image decomposition inferences. We use an image decomposition trained on paradigms (statistical samples of albedo, shading and gloss; Fig 4p) and not real images described in section 333 and normals inferred by the method Nekrasov et al.) (2019) to meet the shading consistency tests (section 3.2). The final reshaded image's s albedo must be like the cut-and-paste albedo; the reshaded image's 's shading must match the shading of the target scene outside the fragment; and the shading of the reshaded image must have reasonable spherical harmonic properties and meet a consistency test everywhere Fig summarizes our method.
3.1 DEEP IMAGE PRIOR FOR RENDERING CUT-AND-PASTE IMAGES
Assume we have noisy image I, and wish to reconstruct the original. Write for random vector, and fa for CNN with parameters and E(fa (z); for loss comparing the image fa(z) to I, The Deep Image Prior seeks
- argmin (f)
and then reports fz)... We modify this formulation by requiring that the E(::I, impose inferential consistency. In particular, write Iø for some inference network(s) and (I,, for inferences constructed out of I, and the source image I,- We seek
For us, I6 is an image decomposition network (pretrained and fixed), and tu creates target albedos (At), shading (S,) and glosses (G₂) fields. We then train DIP to produce image that has reasonable intrinsic image properties. For DIP, the input 2 is the cut-and- -aste image and fo is optimized to inpaint inserted fragment and also meet satisfactory intrinsic image properties.
3

---

Under review as a conference paper at ICLR 2021
We use U-Net with partial convolutions (Liu et al. 2018 Shih al. 2020 Dundar al. 2020).
However, we find the standard partial convolution often convergence to trivial solution, producing images close to cut-and- cut-and-paste and without convincing reshading. To prevent this overfitting to out-and-paste images, we flip the context for the partial convolution; that is, we consider the inserted fragment as the context and hallucinate/outpannt the entire target scene around it. We can view this as an inverse partial convolution.
We use CP (T, s) for an operator that cuts the fragment out of the source image Ie. scales it by s, and places it in the relevant location in the target image I, .M for a mask with the size of the target image that is 0 inside the fragment and 1 outside. Reconstruction loss for background and is given by:
Lrecons I (I)
We then pass the DIP rendered image through the image decomposition network making A Svender and Grender for the albedo, shading and gloss respectively. Our consistent image decomposi- tion inference losses to train DIP are:
Ldecomp (AP(ZE Arenderl | Srender M/ + + G.M
(4)
3.2 SHADING CONSISTENCY LOSSES
We use two shading consistency losses to make the very strong structure of a shading field apparent to DIP's rendering. There is good evidence that shading is tied across surface normals (this underlies spherical harmonic models (Liao et al. 2019) Li et al. 2018 Yu & Smith 2019)), and one should think of a surface normal latent variable that explains shading similarities in images. We assume that the resulting illumination, approximated with the first 9 spherical harmonics coefficients (SHC; a 9-dimensional vector) does not change when new objects are added to a target scene. We get SHC by solving the least square regression between normals (N) and shading (S) for both the target and the resulting rendered image. We breifly explain the least square regression formulation between normals and shading. Consider we have k = m pixels in image, our N E R4x3 and S We estimate first 9 spherical harmonics basis (B(N) E R from W. We can now write S SHC B(N). The solution for SHC is then B(N) We get normal estimates from Nekrasov et al. (201) and then minimize loss (CsHc) between the target and rendered image's SHC(S:N). We use Huber loss for LSHc-
0.5 x (SHC(S;;N,) SHC(SrnneriNer)) for|SHC, SHCrenddrr S 1, Lsuc
SHC(S;; (HC( 0.5 otherwise.
Spherical harmonic shading fields have some disadvantages: every point with the same normal must have the same shading value, which results in poor models of (say) indoor shading walls. To control this effect, we use a neural shading consistency loss (CNsc) that allows the shading field to depart from spherical harmonic shading field, but only in ways consistent with past inferences. Inspired from a pixel-wise discriminator in Schönfeld et al. 2020). we use a shading consistency network U-Net; ((S: trained to discriminate real and fake shading-normal pair. Our shading- normal consistency network (c(S:W)) produces two outputs; one a pixel level map, yielding the first loss term in Eq. 6 which measures per-pixel consistency; the other image level value, yielding the second term in Eq. 6 which measures consistency for the whole image. The LN loss is a binary cross-yntropy loss. We train DIP using LNsC to produce shading field that achieves pixel-wied wise consistency score when stacked with the cut-and-pasee normals (see Fig 3). Let m x be the resolution of our renderings, then LN is given by
Lxsc
log ((Srender (1, fl Nepli.)) logs (SrenderNcP)
(6)
Figure 3: Shading Consistency U-
Real/Fake
Reae/Pake guanl wete scowe)
Net trained separately to discrim- inate consistent and inconsistent pixel-level shading-normal pairs.
In summary, the overall rendering losses that we minimize to update DIP are Lvender Lracons + Litcoompp + LsHc + Lnse
"B(A)" is the pseudo-inverse of B(N).
4

---

Under review as a conference paper at ICLR 2021
Figure Image Decom- position. Left: sam- ples drawn from our paradigms that are used to train our image decom- position network. Right: examples showing MS COCO image decompo- sitions after training on paradigms. Note our im- age decomposition net- work is trained without real image labeled data.
Albedo
Shuding (a) Paradigms
Gloss
Image,Albedo, Albedo, Shading, (from rop bottom)
Image Decomposition Real Images (MS COCO)
3.3 INTRINSIC IMAGE DECOMPOSITION
We believe we could use any competent image decomposition network's s inferences for reshading. However, our experience suggests accurate albedo recovery (measured by strong WHDR performance) results in poor reshading outcomes (Fig (5). It is also helpful to have small and efficient network to reduce the overall back propaagatn time when training a DIP. Therefore, we trained small U-Net to produce albedo, shading and gloss layers from images using paradigms, samples from statistical models intended to capture the spatial statistics of albedo, gloss and shading. This is simple extension of the Retinex model (Land 1977). Albedo paradigms are Mondrian images. Shading paradigms are Perlin noise (Perlin) 1985). and gloss paradigms comprise light bars on a dark background. These are used produce fake images by simple composition (AS G) that in turn are used to train the image decomposition network. Figure 4 shows some samples from each. As Figure 4 illustrates, the resulting intrinsic image models are satisfactory on MSCOCO real images (Lin et al., 2014). For our experiments, we trained two models (Paradigm I and Paradigm II); samples drawn from different statistical models, to investigate the consequences of strong WHDR recovery (II has significantly better WHDR than I). The major difference between the two is that Paradigm I has higu-freeuency, finengrained details albedo and not shading, and for Paradigm the opposite.
3.4 POST-PROCESSING
Removing DIP artifacts. In our approach (see Fig 2) DIP sees both the target image and the cut-and-paste image. We require that DIP inpaints the shading field of the original target image and also the cut-and-paste composite image. This process is analogous to rendering scenes once with an object added and once without an object (see (Debevec) 1998; Karsch et al., 201T)). Doing so means the target scene's shading field acts like regularizer that prevents DIP copying out-and-paste Furthermore, we can remove DIP specific noisy artifacts from the reconstructed image. Using the notation above and writing Toky for the target image rendered by DIP with the object and Inoobj for the target scene rendered by DIP (i.e. no object), we form
Ifsnal
M) @ Indy + M @ (I, Iabs Inooby)
High-resolution Rendering. Our image decomposition network can decompose very high-resolution albedos reliably. Since our shading and gloss are both locally smooth for Paradigm (fine details in albedo), we can easily upsample them to high-resolution. This allows resolution final reshaded images (1024p) with no additional computational budget. Note that we train DIP render images with 256p resolution only, and all our final rendered images in this paper are of 1024p resolution. The IH baseline (Cong et al. 2020) can only reshade 256p resolution images.
render very-high-
4 EXPERIMENTS
Dataset. We collected about 75 diverse images with spatially varying illumination, both indoors- outdoors, dighnight to act as target scenes in our experiments. We tested our aut-and-paste rendering circle cutouts spheres (Fig 6) plant, chair, lego and set of sixteen materials used in NeRF
5

---

Under review as a conference paper at ICLR 2021
Target Scene
Cut & Paste
Paradigm (Our)
Paradigm II (Our)
Li & Snavely 2018
Figure 5: Better WHDR does not mean better reshading. Reshaded images when using target infer- ences from different image decomposition networks. Paradigm I achieves (relatively weak) WHDR of 22%; Paradigm II achieves 19%, close to state-of-the-art (SOTA) for an image decomposition network that does not see rendered images (Liu et al., 2020)). However, Paradigm II produces signifi- cantly worse reshading results. Furthermore, reshading using CGIntrinsics (Li & Snavely) 2018) (a supervised SOTA method) is also qualitatively worse than using Paradigm I. This reflects that better recovery of albedo, as measured by WHDR, does not produce better reshading. The key issue is that methods that get low WHDR do so by suppressing small spatial details the albedo field (for example, the surface detail the lego), but the normal inference method cannot recover these details, and so they do not appear in the reshaded image. From the perspective of reshading, it is better to model them as fine detail in albedo than in shading.
Target Scene
CP
CP-Depth
IH
TH-Depth
Ours
Ours-Depth
Figure 6: Our method has some implicit notion of the 3D layout of the scene, which is required to choose the appropriate shading. Our method shades the white discs as spheres (rather better than IH, implying it "knows" about shape). Furthermore, depth network network|Rantil et al. (2020) applied to reshaded results recovers depths that span the volume of the scene rather more than for others. Note the curious fact that the depth network can choose depths for CP and IH images confidently, too.
(Mildenhall et al. 2020) (Fig- We also show reshading results for cars (Fig 7). Many other objects can found in our Appendix. We use ADF20k validation set /Zhou al. (2017) for supplying real residual loss to image decomposition network and also to train our shading consistency network. Note that the ADE20k does not have ground truth surface normals and we use normals from
pretrained network (Nekrasov et al. 2019) described in section 3.2
Training Details. We used U-Net for DIP, Image Decomposition and Shading Consistency Network. Network architecture and other training details in supplementary. We update our DIP for fixed 10k iterations and this takes about 1200 seconds using our image decomposition network and 1600 seconds when using CGIntrinsic (Li & Snavely 2018

---

Under review as a conference paper at ICLR 2021
Target Scene
CP
Ours
CP
IH
Ours
Figure 7: Glossy effects in car paint (with glitter in them) make reshading cars particularly challenging case with their complex reflective properties. Our method successfully reshades cars without distinct shift in object and background color produced by IH.
Target Scene
CP
IH
Ours
CP
IH
Ours
Figure 8: Our method reshades objects with complex surface geometry and complex material properties, and appears preserve the identity of the material without a distinct shift in object and background color produced by IH.
IH Baseline. We use DoveNet from Cong et al. 2020 as our IH baseline. We use their provided pretrained model for our evaluation
Intrinsic images. Figure 5 explains how better albedo recovery can lead to weaker reshading. Paradigm I achieves realistic reshading and we use image decomposition model trained on Paradigm I for all our experiments.
Fooling users. We wish to know whether reshaded images are reliably preferred to and paste images. We have conducted a user study to compare our reshading method (RS), cut-and-paste (CP), and image harmonization (IH). We collected data from total of 122 unique users in 500 studies from Amazon Mechanical Turk. Each study consists of a prequalifying process, followed by pair-wiss wise comparisons, where the user asked which of two images are more realistic. The prequalifying process presents the user with five tests; each consists of an image with inserted white spheres which are not reshaded (i.e. bright white disks) and an image with inserted spheres which have been reshaded (see Fig 6). We ignore any study where the user does not correctly identify all five reshaded images, on the grounds that the difference very obvious and the user must not have been paying attention. The result is 109 prequalified studies. The comparisons are balanced (each study is 3 RS-IH pairs, CP-IH pairs and RS-CP pairs, in random order and presentation).
The simplest analysis strongly supports the idea that RS is preferred over both alternatives. One compares the probability that RS is preferred to IH (.673, over 327 comparisons, so standard error is .026, and the difference from 0.5 is clearly significant); RS is preferred to CP (.645, over 327 comparisons, so the standard error is .026, and the difference from 0.5 is clearly significant); IH is
preferred CP 1,51 over 327 comparisons, so standard error is .027, and there no significant difference from 0.5). An alternative is a Bradley-Terry model (Tsai et al 2017 Cong et al. 2020
hhtpps: / http://s/////ttbbt./o/op thub. com homm///mmm///m Imaee/mmaaaaaeeeae Harrage__arroiiiee zation Datasets attasess/tre/mmstterr
DoveNet

---

Under review as a conference paper at ICLR 2021
Target Scene
CP
IH Eraams
Leteomp
+Ssc
+Lsxo Esnc
Figure 9: Ablation study showing how different components of our loss helps correcting overall shading. Results improve moving from left to right. We observe that our LsNC helps in improving local changes based on the near-by surroundings and Lshc takes into account the overall illumination and hence provides complementary solutions.
used in image harmonization evaluation, regressing the quality predicted by the Bradley-Tery model against the class of algorithm. This yields coefficients of 0 for IH, -0.347 for CP and 0.039 for RS, implying again that RS is preferred over IH and strongly preferred over CP.
Quantitative Analysis. We cannot quantitatively evaluate our reshading method, because we do not know the ground truth. But we tried scoring images based on a quality when reference is available (Mittal et al. a 2012). Quite surprisingly, NIQE gets significantly worse score for target scenes without any fragments added. This shows that getting reliable quantitative evaluation for reshading is extremely hard. We also tried the recent CNN detector of image fakes described by Wang al. (2020). We find that does not detect any of our synthesized images fake; but also fails to detect cut-and-pase images and image harmonization images.
Images
NIQEI
CNN-D
B-TT
IH
38.39
0
CP 9.67
0
-0.35
Ours
9.78
0.04
Targel Scene
12.51
NA
Table 1:
Quantitative Analysis without ground truth is unreliable for reshading. We report numbers for no-reference image quality estimator (NIQE), a CNN- based tam- pering detector (CNN-D) and Bradley-Terry (BT) model's s analysis from the user-study. Surprisingly, NIQE is worse for target scenes, CNN-D cannot detect any image (CP, IH or ours) as fake, and from the user study be- tween three methods, BT analysis shows our reshaded images are preferred over CP and IH.
Cut & Paste
Our
Cut & Paste
Our
Figure 10: Failure Examples. The wrongly shaded regions are marked in red. In the first exam- ple, DIP aggressively copies background shading onto the chair. However, the lego and the plant are reshaded accurately. In the second, the reason for failure surface normals. The scene has two dominant normal directions ground (pointing up- wards) and the sky (pointed towards viewer). The lack of third direction forces DIP to copy shading either from the sky or the ground plane.
5 DISCUSSION
Evaluation on synthetic scenes. One could build ground truth data on both synthetic (easy, but possibly misleading) and real (experimentally hard, see Karsch et al. (201I) scenes. But evaluating using these will be misleading. The problem is that it is not possible representation from an image fragment using current computer vision technology. This means the inserted object will be rendered incorrectly, and so evaluating using a metrie like Mean Squared Error (MSE) may be bad. Worse, MSE says nothing about the realism of the reshaded image. One could have good MSF, but poor qualitative results; one could also have bad MSE, but strong qualitative results. Only user study remains
recover renderable
evaluation.
Why could this work? It should clear that corrections to inserted fragments are not veridical. As Liao et al.) (2015) 2019) noted, corrected fragments often fool humans more effectively than physically accurate lighting, likely because humans attend to complex materials much more than to consistent illumination (Berzhanskaya et al. (005). The alternative physics theory by [Cavanahh & Alvarez (2005) argues that the brain employs a set of rules that are convenient, but not strictly physical, when interpreting scene from retina image. When these rules violated, perception
8

---

Under review as a conference paper at ICLR 2021
alarm is fired, or recognition is negatively affected (Gauthier et al. 1998). Otherwise, the scene "looks right". This means that humans may tolerate a considerable degree of estimation error, as long as it is of the right kind. By insisting that the image produces consistent inferences, we appear to be forcing errors to be "of the right kind".
Need For Speed. A desirable cut-and- paste reshading application would correct shading on inserted objects requiring no explicit 3D reasoning about either fragment or scene. It would do so in ways that produce consistent inferences of image analysis. Finally, it would be quick. We described method that meets two of these three desiderata (DIP still requires minutes to render an image).
Casting Shadows. Our renderer can adjust the background with cast-shadows but often unrealistically. Therefore, we only show shading corrections on the fragment in this paper. Having better inferences (for eg.reg., normal prediction network) should improve our results significantly. Also, better illumination modeling like Lighthouse (Srinivasan et al. (2020) and residual appearance networks (Sengupta et al. 2019) should also improve background shading changes. We also believe GAN based losses should help in correcting background shadows. This can also be combined with a few recent promising work in this direction (Izadinia & Seitz 2020 Zhan et al., 2020).
Extending to Videos. Our method can reshade multiple frames at a time quite convincingly (in Fig 14 of Appendix). Extending to videos is natural next step. Once the speed of our renderings is improved, we can also use our techniques for improving frame interpolation.
REFERENCES
Aayush Bansal, Yaser Sheikh, and Deva Ramanan. Shapes and context: In-the-wild image synthesis manipulation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 2317-2326, 2019.
Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions on Graphics (TOG), 2014.
Julia Berzhanskaya, Gurumurthy Swaminathan, Jacob Beck, and Ennio Mingolla. Remote effects of highlights on gloss perception. Perception, 2005.
Sai Bi. Xiaoguang Han. and Yizhou Yu. An image transform for adge-preserving smoothing and scene-level intrinsic decomposition. ACM Transactions Graphics (TOG), 2015.
Patrick Cavanagh and George A Alvarez. Tracking multiple targets with multifocal attention. Trends in cognitive sciences, 97):349-354, 2005.
Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li. and Liqing Zhang. Dovenet: Deep image harmonization via domain verification. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 1394-8403, 2020.
Paul Dehevec. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998.
Aysegul Dundar, Karan Sapra, Guilin Liu, Andrew Tao, and Bryan Catanzaro. Panoptic-based image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8070-8079, 2020.
Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1301-1310, 2017.
Qingnan Fan, Jiaolong Yang, Gang Hua. Baoquan Chen, and David Wipf. Revisiting deep intrinsic image decompositions. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
Mare-Andre Gardner, Yannick Hold-Geoffroy, Kalyan Sunkavalli, Christian Gagné, and Jean- Bean-François Lalonde. Deep parametric indoor lighting estimation. In Proceedings of the IEEE International Conference on Computer Vision 2019.
Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean- Gean-François Lalonde. Fast spatially- varying indoor lighting estimation. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, 2019.

---

Under review as a conference paper at ICLR 2021
Isabel Gauthier, Pepper Williams, Michael J Tarr, and James Tanaka. Training "greeble'experts: framework for studying expert object recognition processes. Vision research, 38(15-16)22401-4422 1998.
Yannick Hold-Geoffroy, Akshaya Athawale, and Jean- SannFrangois Lalonde. Deep sky modeling for single image outdoor lighting estimation. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, 2019.
Hamid Izadinia and Steven M Seitz. Scene recomposition by learning-based icp. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 900-939099990 (0009939,200 2020.
Michael Janner, Jiajun Wu, Tejas D Kulkarni, Ilker Yildirim, and Josh Tenenbaum. Self-supervised intrinsic image decomposition. Advances Neural Information Processing Systems, 2017.
Jiaya Jia, Jian Sun, Chi-Keung Tang, and Boung-Yeung Shum. Drag-and-drop pasting. ACM Transactions on graphics (TOG), 25(3)633-663 2006.
Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem. Rendering synthetic objects into legacy photographs. ACM Transactions Graphics (TOG), 2011.
Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, and David Forsyth. Automatic scene inference for 3d object compositing. ACM Transactions on Graphics (TOG), 2014.
Jean-Françur Lalonde, Derek Hoiem, Alexei A Efros, Carsten Rother, John Winn, and Antonio Criminisi. Photo clip art. ACM transactions graphics (TOG), 2007.
Edwin H Land. Color vision and the natural image. part i. Proceedings of the National Academy of Sciences of the United States of America, 1959a
Edwin H Land. Color vision and the natural image part ii Proceedings of the National Academy of Sciences of the United States of America, 1959b.
Edwin H Land. The retinex theory of color vision. Scientific american, 1977.
Edwin H Land and John J McCann. Lightness and retinex theory. Josa, 1971
Zhengqi Li and Noah Snavely. Cgintrinsics: Better intrinsic image decomposition through physically-based rendering. In Proceedings of the European Conference on Computer Vision (ECCV), 2018
Zhengqin Li, Zexiang Xu, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Learning to reconstruct shape and spatially-varying reflectance from single image. ACM Transactions on Graphics (TOG), 2018
Zhenggin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. Inverse rendering for complex indoor scenes: Shape, spatially- -arrying lighting and svbrdf from a single image. 2020.
Zicheng Liao, Ali Farhadi, Yang Wang, Ian Endres, and David Forsyth. Building dictionary of image fragments. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3442-3449 -449, IEEE, 2012.
Zicheng Liao, Kevin Karsch, and David Forsyth. An approximate shading model for object relighting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Zicheng Liao, Kevin Karsch, Hongyi Zhang, and David Forsyth. An approximate shading model with detail decomposition for object relighting. International Journal of Computer Vision, 2019.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 2014.
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. Image inpainting for irregular holes using partial convolutions. In Proceedings of the Eumpean Conference on Computer Vision (ECCV), 2018
Yunfei Liu, Yu Li, Shaodi You, and Feng Lu. Unsupervised learning for intrinsic image decomposition from single image. 2020.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. arXiv preprint arXiv:2003. 08934, 2020.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEEE Signal processing letters, 2033):00-212, 2012.
10

---

Under review as a conference paper at ICLR 2021
Vladimir Nekrasov, Thanuja Dharmasiri, Andrew Spek, Tom Drummond, Chunhua Shen, and Ian Reid. Real- time joint semantic segmentation and depth estimation using asymmetric annotations. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.
Thomas Nestmeyer, FrançoFs Lalonde, lain Matthews, Epic Games, Andreas Lehrmann, and Al Borealis. Learning physics-guided face relighting under directional light. 2020.
Patrick Pérez, Michel Gangnet, and Andrew Blake. Poisson image editing. In ACM SIGGRAPH 2003 Papers. pp. 313 318.318.22 2003.
Ken Perlin. An image synthesizer. ACM Siggruph Computer Graphics, 1985.
Vilayanur Ramachandran. Perceiving shape from shading. Scientific American, 1988.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2020.
Edgar Schönfeld. Bernt Schiele, and Anna Khoreva. A u-net based discriminator for generative adversarial networks. arXiv preprint arXiv:2002.1 12655, 2020.
Soumyadip Sengupta, Jinwei Gu, Kihwan Kim, Guilin Liu, David W Jacobs, and Jan Kautz. Neural inverse rendering of an indoor scene from a single image. In Proceedings of the IEEE International Conference on Computer Vision, pp. 8598 8607, 2019.
Meng-Li Shih, Shih- Su. Johannes Kopf. and Jia-Bin Huang. 3d photography using context-aware layered depth inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8028 8038,00888200 2020.
Shuran Song and Thomas Funkhouser. Neural illumination: Lighting prediction for indoor environments. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2019.
Pratul P Srinivasan, Ben Mildenhall, Matthew Tancik, Jonathan T Barron, Richard Tucker, and Noah Snavely. Lighthouse: Predicting lighting volumes for cohheret illumination. 2020.
Tiancheng Sun, Jonathan Barron, Yun-Ti Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Trunsactions on Graphics (Proceedings SIGGRAPH), 2019.
Kalyan Sunkavalli, Micah K. Johnson. Wojciech Matusik, and Hanspeter Pfister. Multi-scale image harmonization. ACM Transactions Graphics (TOG), 29(4):1-10, 2010.
Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming- Hsuan Yang. Deep image harmonization. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7789-3797, 2017.
Dmitry Ulyanov, Andrea Vedaldi. and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2018.
Sheng-Yu Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnn-generated images surprisingly easy to spot...for now. In CVPR, 2020.
Ye Yu and William AP Smith. Inverserendernet: Leariing single image inverse rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Fangneng Zhan, Shijian Lu, Changgong Zhang, Feiying Ma, and Xuansong Xie. Adversarial image composition with auxiliary illumination. arXiv preprint arXiv:2009.00 08255. 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait relighting. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
11


Reviews:
[{"review_id": "uYEUBUknKRY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThis paper introduces a reshading method for cut-and-paste image composition. It uses a modified deep image prior as the rendering networking, and trains with a novel shading consistency loss. The results are plausible. Inserted fragments have proper shading with the source image and unchanged albedo.\n\nPros:\n- an improved DIP that produces consistent image decomposition inferences (albedo, shading, gloss).\n- a novel shading consistency loss that accounts for the spatial similarity and past inferences.\n- Extensive experiments showing advantages over CP and IH.\n\nCons:\n- Is scene geometry not needed for this task?\n\nCut-and-paste methods may bring ambiguity for the inserted object. Cars in Fig 7 have a fixed depth because it is grounded in human perception. However, the circle cut-outs in Fig 11 are floating, which makes these disks have ambiguous depth. For example, in the second last row of Fig 11, a floating disk closer to the camera will have darker shading while a disk with further depth (under the lamp) will be lit up and has brighter lighting. In this paper, the only supervision on scene geometry is the surface normal in the shading loss. The inserted fragment may have an ambiguous depth in human perception. Therefore, in the inference phase, the shading factor for the cut-out might be ambiguous.\n\n- Quantitative evaluation on albedo\n\nOne way to evaluate the reshading method is to measure how much the inserted fragment's albedo changed. Results from CP, RS and IH could be passed into the image decomposition network, and the albedo values could be compared with the ones in the source image.\n\n- Timing for inference\n\nHow much time does it take for DIP to render a 1024p image? The paper mentions minutes, but could be more precise.\n\nComments:\nI don't quite follow the spherical harmonics coefficients and how they are calculated from normals and shading. There could be a formulation equation in Sec 3.2 or in supplementary.\n\nUpdates: Thanks for the author's response. My concerns are mostly addressed. But I still believe explicit geometry modeling should be included for this task. This could be added in future works. Overall, I am positive on the submission and keep my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MwYNvtLyOJB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes cut-and-paste neural rendering that allows to insert objects into a target scene in a plausible manner, i.e., in terms of shading plausibility. At the core of the approach is a deep image prior that allows to match the shading and albedo fields based on shading and albedo consistency losses. A normal estimation network that is trained based on synthetic data is used to further inform shading estimation. The approach is interesting and shows plausible results.\n\nThe submission performs many ablation studies that give a better understanding of the inner workings of the approach. The approach is compared to several other methods that are outperformed visually and in a user study. The results obtained by this approach look plausible.\n\nThere is one statement in the paper I disagree with: “We cannot quantitatively evaluate our reshading method, because we do not know ground truth”. It is possible in two ways: Completely synthetic scenes that are rendered using computer graphics or real scene where images are taken before and after inserting an object. This would allow access to ground truth pairs of images with and without the object. It would be interesting to see how close the outputs of these methods are to the actual ground truth, both qualitatively and quantitatively. For all shown results, I was always unsure how it would look in reality, which makes it quite hard to judge how good the method actually performs. For sure this approach produces more plausible results than previous approaches, but how close are these results actually to how it would look in reality. I think this question is unanswered so far.\n\nIn summary, I like the submission and I think the proposed approach is novel, but I would have hoped for an actual evaluation against GT data. Therefore, I am currently a bit on the fence to suggest acceptance, but would like to hear the opinion of the other reviewers and see the author response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AjTiVcsEck", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper Summary**\nThe paper proposed a carefully designed neural rendering pipeline to realistically insert one image fragment into another image via deep image prior and the consistency in the albedo, shading, and gloss in the rendered image. Experiments show great improvement over baselines both qualitatively and quantitatively (via user study). \n\n**Strength**\n1. The inverse partial convolution is a neat idea. This idea effectively helps the network to learn the shading of the target image and to avoid directly copying the inserted objects.\n2. The shading consistency loss and the designed experiments to pre-train the image decomposition network are also carefully designed and thoughtful. The analysis in Fig 5 also provides insights for the relationship between WHDR and the actual rerendering effects. This could inspire many followups on how we should design the algorithm to better composite an image.\n3. The experiments are also well performed. To quantitatively evaluate the performance, this paper conducted user studies with detailed descriptions of the whole procedure. And qualitatively, the results the paper showed are better than baselines. \n\n**Weakness**\n1. Speed, the overall pipeline relies on the training Deep Image Prior (DIP), making the whole pipeline slow during the real application.\n2. Some notation in this paper is not clear. E.g.  in Eq. 6,  what does the second term means? \n3. The paper didn't provide the exact method for Image Harmonization baseline, it would be better for the readers to understand what's the exact method to be compared. \n4. It's not clear how does the proposed method generalize to high-order lighting effect since the lighting model the paper used is Spherical Harmonic. \n\n**Overall**\nOverall, I think this paper proposed many neat ideas for image composition and performed solid experiments, but considering the limitation of the method (which has been described in Sec. 5), I vote for a weak accept initially.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall the review is borderline: R2 and R4 are slightly positive and R3 is slightly negative. All the reviewers like the novel shading consistency loss proposed in the paper and, improved DIP that produces consistent image decomposition inferences, and good experimental results. However, reviewers also shared concerns about speed and the thoroughness of the evaluation, and human tolerance of shading inaccuracy. These points were addressed in details in the rebuttal, and reviewers didn’t change their initial scores.

The AC is concerned about the cut-and-paste neural rendering results. Because there are no cast shadows, the rendering doesn’t look realistic under the lighting conditions in the new image. It’s unclear that the proposed method would lead to a promising direction of copying and pasting contents into images for photorealistic editing. Consequentially, the paper is not ready for publication at its current form.

Decision: Reject