Cut-and-Paste Neural Rendering

Cut-and-paste methods take an object from one image and insert it into another . Doing so often results in unrealistic looking images because the inserted object ’ s shading is inconsistent with the target scene ’ s shading . Existing reshading methods require a geometric and physical model of the inserted object , which is then rendered using environment parameters . Accurately constructing such a model only from a single image is beyond the current understanding of computer vision . We describe an alternative procedure – cut-and-paste neural rendering , to render the inserted fragment ’ s shading field consistent with the target scene . We use a Deep Image Prior ( DIP ) as a neural renderer trained to render an image with consistent image decomposition inferences . The resulting rendering from DIP should have an albedo consistent with cut-and-paste albedo ; it should have a shading field that , outside the inserted fragment , is the same as the target scene ’ s shading field ; and cut-and-paste surface normals are consistent with the final rendering ’ s shading field . The result is a simple procedure that produces convincing and realistic shading . Moreover , our procedure does not require rendered images or image decomposition from real images or any form of labeled annotations in the training . In fact , our only use of simulated ground truth is our use of a pre-trained normal estimator . Qualitative results are strong , supported by a user study comparing against state-of-the-art image harmonization baseline . 1 INTRODUCTION . Cut-and-Paste rendering involves creating a new image by cutting fragments out of one or more source images and pasting them into a target image ; the idea originates with Lalonde et al . ( 2007 ) . Results are often unrealistic , because of the difference in illumination between the source and target images . But the procedure is useful to artists , and there is consistent evidence that such procedures can be used to train detectors ( Liao et al. , 2012 ; Dwibedi et al. , 2017 ) . When the geometry and material of the inserted object are known , it is enough to infer an illumination model from the target , render and composite . But current procedures for recovering shape and material from a single fragment simply can ’ t deal with most realistic fragments ( think of , say , a furry cat ) . This paper describes an alternative method , Cut-and-Paste Neural Rendering , that can render convincing composite images by adjusting the cut-and-paste images so that some simple image inferences are consistent with cut-and-paste predictions . So the albedo from the adjusted image should look like cut-and-paste albedo ; the shading should look like a shading field ; and the image should look like an image . A simple post-processing trick produces very high-resolution composites . Note that all our rendered images are 1024x1024 pixels resolution and are best viewed on screen . Evaluation is mostly qualitative , but we show that our method fools a recent method for detecting tampering . Our contribution is a method that can realistically correct shading in composite images , without requiring labeled data ; our method works for matte , glossy and specular fragments without an explicit geometric or physical model ; and human subjects prefer the results of our method over cut-and-paste and image harmonization . 2 RELATED WORK . Object Insertion starts with Lalonde et al . ( 2007 ) , who insert fragments into target images . Lalonde et al . ( 2007 ) control illumination problems by checking fragments for compatibility with targets ; Bansal et al . ( 2019 ) do so by matching contexts . Poisson blending ( Pérez et al. , 2003 ; Jia et al. , 2006 ) can resolve nasty boundary artifacts , but significant illumination and color mismatches will cause cross-talk between target and fragment , producing ugly results . Karsch et al . ( 2011 ) show that computer graphics ( CG ) objects can be convincingly inserted into inverse rendering models got with a geometric inference or with single image depth reconstruction ( Karsch et al. , 2014 ) . Inverse rendering trained with rendered images can produce excellent reshading of CG objects ( Ramachandran , 1988 ) . However , recovering a renderable model from an image fragment is extremely difficult , particularly if the fragment has an odd surface texture . Liao et al . showed that a weak geometric model of the fragment can be sufficient to correct shading if one has strong geometric information about the target scene ( Liao et al. , 2015 ; 2019 ) . In contrast , our work is entirely image-based : one takes a fragment from one image , drops it into another , and expects a system to correct it . We use image harmonization ( IH ) methods as a strong baseline . These procedures aim to correct corrupted images . IH methods are trained to correct images where a fragment has been adjusted by some noise process ( made brighter ; recolored ; etc . ) to the original image ( Sunkavalli et al. , 2010 ; Tsai et al. , 2017 ; Cong et al. , 2020 ) , and so could clearly be applied here . But we find those image harmonization methods very often change the albedo of an inserted object , rather than its shading . This is because they rely on ensuring consistency of color representations across the image . For example , in the iHarmony dataset from Cong et al . ( 2020 ) , they change pink candy to brown ( an albedo change ; see Fig 12 in Appendix ) . In contrast , we wish to correct shading alone . Image Relighting . With appropriate training data , for indoor-scenes , one can predict multiple spherical harmonic components of illumination ( Garon et al. , 2019 ) , or parametric lighting model ( Gardner et al. , 2019 ) or even full radiance maps at scene points from images ( Song & Funkhouser , 2019 ; Srinivasan et al. , 2020 ) . For outdoor scenes , the sun ’ s position is predicted in panoramas using a learning-based approach ( Hold-Geoffroy et al. , 2019 ) . One can also construct a volumetric radiance field from multi-view data to synthesize novel views ( Mildenhall et al. , 2020 ) . However , we do not have access to either training data with lighting parameters/environment maps or multi-view data to construct such a radiance field . Our renderings are entirely image-based . Recent singleimage relighting methods relight portrait faces under directional lighting ( Sun et al. , 2019 ; Zhou et al. , 2019 ; Nestmeyer et al. , 2020 ) . Our method can relight matte , gloss and specular objects with complex material properties like cars ( Fig 7 ) for both indoor and outdoor spatially varying illuminated environments only from a single image and without requiring physics-based BRDF ( Li et al. , 2020 ) . Image decomposition . Land ’ s influential Retinex model assumes effective albedo displays sharp , localized changes ( which result in large image gradients ) , and that shading has small gradients ( Land , 1959a ; b ; 1977 ; Land & McCann , 1971 ) . These models require no ground truth . An alternative is to use CG rendered images for image decomposition training ( Li & Snavely , 2018 ) , particularly with specialized losses ( Bi et al. , 2015 ; Fan et al. , 2018 ) . One can also train using rendering constraints to produce a form of self-supervised training ( Janner et al. , 2017 ) . Current image decomposition evaluation uses the weighted human disagreement rate ( WHDR ) ( Bell et al. , 2014 ) ; current champions are ( Fan et al. , 2018 ) . We use an image decomposition method built around approximate statistical models of albedo and shading ( paradigms ) to train our image decomposition network without requiring real image ground truth decompositions . Our method has reasonable , but not SOTA , WHDR ; but we show that improvements in WHDR do not result in improvements in reshading ( Fig 5 ) . 3 CUT-AND-PASTE NEURAL RENDERER . We synthesize a reshaded composite image containing a fragment transferred from a source image into a target scene image . We use a deep image prior ( DIP ) ( Ulyanov et al. , 2018 ) as a neural renderer to produce a reshaded image that produces consistent image decomposition inferences . We use an image decomposition trained on paradigms ( statistical samples of albedo , shading and gloss ; Fig 4a ) and not real images described in section 3.3 , and normals inferred by the method of Nekrasov et al . ( 2019 ) to meet the shading consistency tests ( section 3.2 ) . The final reshaded image ’ s albedo must be like the cut-and-paste albedo ; the reshaded image ’ s shading must match the shading of the target scene outside the fragment ; and the shading of the reshaded image must have reasonable spherical harmonic properties and meet a consistency test everywhere Fig 2 summarizes our method . 3.1 DEEP IMAGE PRIOR FOR RENDERING CUT-AND-PASTE IMAGES . Assume we have a noisy image It , and wish to reconstruct the original . Write z for a random vector , and fθ for a CNN with parameters θ and E ( fθ ( z ) ; It ) for a loss comparing the image fθ ( z ) to It . The Deep Image Prior seeks θ̂ = argminθE ( fθ ( z ) ; It ) ( 1 ) and then reports fθ̂ ( z ) . We modify this formulation by requiring that the E ( · ; It ) impose inferential consistency . In particular , write gφ for some inference network ( s ) and tψ ( Is , It ) for inferences constructed out of It and the source image Is . We seek θ̂ = argminθE ( gφ ( fθ ( z ) ) ; tψ ( Is , It ) ) . ( 2 ) For us , gφ is an image decomposition network ( pretrained and fixed ) , and tψ creates target albedos ( At ) , shading ( St ) and glosses ( Gt ) fields . We then train DIP to produce an image that has reasonable intrinsic image properties . For DIP , the input z is the cut-and-paste image and fθ is optimized to inpaint inserted fragment and also to meet satisfactory intrinsic image properties . We use a U-Net with partial convolutions ( Liu et al. , 2018 ; Shih et al. , 2020 ; Dundar et al. , 2020 ) . However , we find the standard partial convolution often convergence to a trivial solution , producing images close to cut-and-paste and without convincing reshading . To prevent this overfitting to cut-and-paste images , we flip the context for the partial convolution ; that is , we consider the inserted fragment as the context and hallucinate/outpaint the entire target scene around it . We can view this as an inverse partial convolution . We use CP ( Is ; It ; s ) for an operator that cuts the fragment out of the source image Is , scales it by s , and places it in the relevant location in the target image It.M for a mask with the size of the target image that is 0 inside the fragment and 1 outside . Reconstruction loss for background and is given by : Lrecons = ||It M− ( fθ ( CP ( Is ; It ; s ) ; M ) ) ||2 ( 3 ) We then pass the DIP rendered image through the image decomposition network gφ making Arender , Srender and Grender for the albedo , shading and gloss respectively . Our consistent image decomposition inference losses to train DIP are : Ldecomp = ||ACP ( Is ; It ; s ) −Arender|| 2 + ||St M−Srender M||2 + ||Gt M− Grender M||2 ( 4 )

Summary: This paper proposes cut-and-paste neural rendering that allows to insert objects into a target scene in a plausible manner, i.e., in terms of shading plausibility. At the core of the approach is a deep image prior that allows to match the shading and albedo fields based on shading and albedo consistency losses. A normal estimation network that is trained based on synthetic data is used to further inform shading estimation. The approach is interesting and shows plausible results.