Image-guided Neural Object Rendering
Authors: Justus Thies
Michael Zollhöfer
Christian Theobalt
Marc Stamminger
Matthias Nießner
Venue:  (2020)
Keywords: Neural Rendering
Neural Image Synthesis

Abstract: We propose a learned image-guided rendering technique that combines the benefits of image-based rendering and GAN-based image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours and sightseeing, the digital inspection of historical artifacts). A core component of our work is the handling of view-dependent effects. Specifically, we directly train an object-specific deep neural network to synthesize the view-dependent appearance of an object.
As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of a captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realistic results. Using this image-guided approach, the network does not have to allocate capacity on ``remembering'' object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantitatively on synthetic as well as on real data.

Published as a conference paper at ICLR 2020
IMAGE-GUIDED NEURAL OBJECT RENDERING
Justus Thies', Michael Zollhöfer", Christian Theobalt', Maae Stamminger', Matthias Nießner Technicaa University of Munich, "Stanford University, "Max-Planck-Insituue for Informatics, 'University of Erlangen-Nuremberg
ABSTRACT
We propose a learned image-guided rendering technique that combines the bene- fits of image-based rendering and GAN-basea image synthesis. The goal of our method is to generate photo-realistic re-renderings of reconstructed objects for virtual and augmented reality applications (e.g., virtual showrooms, virtual tours & sightseeing, the digital inspection of historical artifacts). core component our work the handling of view-dependent effects. Specifically, directly train an object-specific deep neural network to synthesize the view-dependent appear- ance of an object. As input data we are using an RGB video of the object. This video is used to reconstruct a proxy geometry of the object via multi-view stereo. Based on this 3D proxy, the appearance of captured view can be warped into a new target view as in classical image-based rendering. This warping assumes diffuse surfaces, in case of view-dependent effects, such as specular highlights, it leads to artifacts. To this end, we propose EffectsNet, a deep neural network that predicts view-dependent effects. Based on these estimations, we are able to convert observed images to diffuse images. These diffuse images can be projected into other views. In the target view, our pipeline reinserts the new view-dependent effects. To composite multiple reprojected images to a final output, we learn a composition network that outputs photo-realist approach, the network does not have allocate capacity on "remembering" object appearance, instead it learns how to combine the appearance of captured images. We demonstrate the effectiveness of our approach both qualitatively and quantita- tively on synthetic as well as on real data.
results. Using this image-guided
INTRODUCTION
In recent years, large progress has been made in 3D shape reconstruction of objects from pho- tographs or depth streams. However, highly realistic re-rendering of such objects, e.g., in virtual environment, is still very challenging. The reconstructed surface models and color information often exhibit inaccuracies strong view-dependent appearance effects, such as specularities. These effects not only frequently cause errors already during image-based shape reconstruction, but are also hard reproduce when re-rendering an object from novel viewpoints. Static diffuse textures are frequently reconstructed for novel viewpoint synthesis, but these textures lack view-dependent appearance effects. Image- based rendering (IBR) introduced variants of view-dependent texturing that blend input images on the shape (Buehler et al. 200T: Heigl al. 1999 Carranza al. 2003) Zheng et al. 2009). This enables at least coarse approximation of view-dependent effects. However, these approaches of- ten produce ghosting artifacts due view blending on inacourate geometry, or artifacts at occlu- sion boundaries. Some algorithms reduce these artifacts by combining view blending and optical flow correction (Eisemann et al. 22088) Casas al. [2055] Duu al. 2018), or by combining view- dependent blending with view-spec geometry (Chaurasia et al. 2013; Hedman et al.) (016) or geometry with soft 3D visibility like Penner & Zhang (0177). Hedman et al. (2018) reduces these artifacts using deep neural network which is predicting per-pixel blending weights.
comparably coarse (e.g., Izadi et al. (201T)). Many objects also exhibit
In contrast, our approach explicitly handles view-dependent effects to output photo-realistic images and videos. It neural rendering approach that combines masee-based rendering and the advances in deep learning. As input, we capture a short video of an object to reconstruct the geometry using multi-view stereo. Given this 3/) reconstruction and the set of images of the video, we are able

---

Published as a conference paper at ICLR 2020
Lniinnted Vor
Greed Treth
e
Figure 1: Overview of our image-gun -ggdedd rendering approach: based on the nearest neighbor views, we predict the corresponding view hee-dependent effects using our EffectsNee architecture. The view- dependent effects are subtracted from the original images to get the diffuse images that can be re- projected into the target image space, In the target image space we estimate the new vien-dependent effect and add them to the warped images. An encoder-decoder network is used to blend the warped images to obtain the final output image. During training, we enforce that the output image matches the corresponding ground truth image.
to train our pipeline in self-supervised manner. The core of approach is neural network called EffectsNer which is trained in a Siamese way to estimate view-dependent effects, for exam- ple, specular highlights reflections. This allows to remove view-dependent effects from the input images, resulting in images that contain view-independent appearance information of the ob- ject. This view-independent information can be projected into a novel view using the reconstructed geometry, where new vieeddependent effects can be added. CompositionNet, a second network, composites the projected K nearest neighbor images to a final output. Since CompositionNer is trained to generate photo-realistic output images, it is resolving reprojection errors as well as filling regions where no image content is available. We demonstrate the effectiveness of our algorithm using synthetic and real data, and compare to classical computer graphics and learned approaches.
To summarize, we propose a novel neural image-guided rendering method, hybrid between clas- sical image-based rendering and machine learning. The core contribution is the explicit handling of view-dependent effects the source and the target views using EffectsNel that can be learned in a self-supervised fashion. The composition of the reprojected views to a final output image without the need of hand-crafted blending schemes is enabled using our network called CompositionNer.
RELATED WORK
Multi-view 3D Reconstruction Our approach builds on coarse geometric proxy that is obtained using multi-view 3D reconstruction based on COLMAP (Schönberger Frahm 2016). In the last decade, there has been lot of progress in the field of image-hased 3D reconstruction. Large-scale 3D models have been automatically obtained from images downloaded from the internet (Agarwal eet al. 2011). Camera poses and intrinsic calibration parameters are estimated based structure- from-motion Jebara et al. 1999; Schnberger & Frahm 2016), which can be implemented based on global bundle adjustment step TTiggs 22000) Afterwards, based on the camera poses and calibration, a dense three- dimenssiiai pointcloud of the scene can be obtained using a multi-view stereo reconstruction approach (Seitz et al.) 2006 Goesele et al., 2007] Geiger et al. (01I)) Finally, a triangulated surface mesh is obtained, for example using Poisson surface reconstruction (Kazhdan al. 2006). Even specular objects can be reconstructed (Godard al. 2015).
Learning-based Image Synthesis Deep learning methods can improve quality in many realis- tic image synthesis tasks. Historically, many of these approaches have been based on generator networks following an encoder-decoder architecture (Hinton & Salakhutdinov 2006 Kingma & Welling. 2013), such as a U-Net (Ronneberger et al. (015a) with skip connections. Very recently, adversarially trained networks (Goodfellow et al.) [ et al. 20.1)217)) Mirza & Osindero 2014]
Radford et al. 2016) have shown some of the best result quality for various image synthesis tasks. For example, generative CNN models to synthesize body appearance (Esser et al. 2018), body ar-
ticulation (Chan et al. 2018), body pose and appearance (Zhu et al. 2018 Liu et al. 2018). and face rendering [Kim et al. 2018 Thies al. 2019) have been proposed. The DeepStereo approach of
Flynn et al. (2016) trais a neural network for view synthesis based on a large set of posed images.
Tulsiani et al. 2018 employ view synthesis proxy task learn layered scene representation.
2

---

Published as a conference paper at ICLR 2020
View synthesis be learned directly from light field data shown by Kalantari al. (016)) Appearance Flow (Zhou et al. 2016) learns an image warp based on a dense flow field to map in- formation from the input to the target view. [hoou al. (2018) learn to extrapolate stereo views from imagery captured by a narrow-baseling stereo camera. Park et al. (2017) explicitly decouple the view synthesis problem into an image warping and inpainting task. In the results, we also show that CNNs trained for image-to-image translation [Isola et al. (2016)) could be applied to novel view synthesis, also with assistance of shape proxy.
Image-based Rendering Our approach is related image-based rendering (IBR) algorithms that cross-project input views to the target via a geometry proxy, and blend the re-projected views (Buehler et al. 200T Heigl et al. 1999. Carranza et al. 2003) Zheng al. 2009). Many previous IBR approaches exhibit ghosting artifacts due to view blending on inaccurate geometry, or exhibit artifacts at occlusion boundaries. Some methods try to reduce these artifacts by combining view
blending and optical flow correction (Fisemann et al. 2008; Casas et al. 2015 Du et al.) 2018). by using view-specific geometry proxies (Chaurasia et al. [2013 Hedman et 2016), or by encoding uncertainty in geometry as soft 3D visibility (Penner & Zhang 2017). Hedman et al. (2018) propose a hybrid approach between IBR and learning-base view blending function for image-based rendering with view-dependent shape proxies. In contrast, our learned IBR method learns to combine input views and to explicitly separate view-dependent effects which leads to better reproduction of deeeddependent appearance.
image synthesis. They use a CNN to learn a
Intrinsic Decomposition Intrinsic decomposition tackles the ill poo-pd problem of splitting im- age into a set of layers that correspond to physical quantities such as surface reflectance, diffuse shading, and/or specular shading. The decomposition of monocular video into reflectance and shad- ing is classically approached based on a set of hand-crafted priors (Bonneel et al.. 2014 Ye et al., 2014 Meka et al., 2016). Other approaches specifically tackle the problem of estimating (Lin al., 2002 or removing specular highlights (Yang et al. 2015)). A diffuse//ppeclar separation can also be obtained based on a set of multi-view images captured under varying illumination (Takechi & Okabe, 2017). The learning-basee basen approach of Wu et al. (2018) converts a set of multi-view images of specular object into corresponding diffuse images. An extensive overview given in the survey paper of Bonneel et al. (2177).
3 OVERVIEW
We propose a learning- based image-guided rendering approach that enables novel view synthesis for arbitrary objects. Input to our approach is set of N images I { k of object with constant illumination. In preprocess, we obtain camera pose estimates and a coarse proxy geometry using the COLMAP crructure-frommmotion approach (Schönberger & Frahm (2016); Schönberger et al. (016)). We use the reconstruction and the camera poses to render synthetic depth maps Dr for all input images Iş to obtain the training corpus T = {[Ix, { see Fig. 8 Based on this input, our learning-based based approach generates novel views based on the stages that are depicted in look-up to select a small number n & N of fixed views from a subset of the training corpus. In our experiments, we are using number of n 20 frames, which we call reference images. Per target view, we select K = nearest views from these reference images. Our EffectsNet predicts the view-dependent effects for these views and, thus, the corresponding view-independent components can be obtained via subtraction (5. The view- www-independent component is explicitly warped to the target view using geometry- guided cross- projection (Sec. (). Next, the view-dependent effects of the target view are predicted and added on top of the warped views. Finally, our CompositionNel is used optimally combine all warped views to generate the final output (Sec. (6) In the following, we discuss details, show how our approach can be trained based on our training corpus (Sec. 4. and extensively evaluate our proposed approach (see Sec. and the appendix).
Fig. T First, we employ a coverage-
4 TRAINING DATA
Our approach is trained in an object-specific manner, from scratch each time. The training corpus T [[I,, N consists of N images I₃ and depth maps D& per object with constant light.
3

---

Published as a conference paper at ICLR 2020
Laput
Iuput Depth
View-dependint
Vien-dependent De-projecte)
Imagen (re-grojected)
Taaget taaage
Losa betweum Diffise Imagos
Souree Taaagg
ElffectsNet is trained in a self-supervised fashion. In a Siamese scheme, two random images from the training are chosen and fed into the network to predict the view-dependent effects based on the current view and the respective depth map. After re-projecting the source image the target image space we compute the diffuse color via subtraction. We optimize the network by minimizing the difference between the two diffuse images in the valid region.
Figure 2:
Synthetic Training Data To generate photo-realistic synthetic imagery we employ the Mitsuba Renderer (Jakob 2010) to simulate global illumination effects. For each of the N views, ray- trace a color image Z& and its corresponding depth map Dr. We extract a dense and smooth temporal camera path based on spiral around the object. The camera oriented towards the center of the object. All images have a resolution of 512 x 512 and are rendered using path tracing with 96 samples per pixel and maximum path length of 10. The size of the training sequence is 920, the test set contains 177 images.
Real World Training Data Our real world training data is captured using a Nikon D5300 at resolution of 1920 1080 pixels. Since we rely on a sufficiently large set of images, we record videos of the objects at frame rate of 30Hz. Based COLMAP (Schönberger & Frahm 2016 Schönberger et al. 2016), we reconstruct camera path and dense point cloud. We manually iso- late the target object from other reconstructed geometry and & Hoppe 2013) step to extract the surface. We use this mesh to generate synthetic depth maps Dj corresponding to the images I& (see Fig. 8 Finally, both, the color and depth images cropped and re-scaled to a resolution of 512 x 512 pixels. The training corpus ranges from 1000 to 1800 frames, depending the sequence.
Poisson reconstruction (Kazhdan
EffectsNet
A main contribution of our work is a convolutional neural network that learns the disentanglement of view-dependent and view-independent effects in self-supervised manner (see Fig. 2 Since our training data consists of a series of images taken from different viewing directions, assuming constant illumination, the reflected radiance of corresponding points in two different images only differs by the view hiew-dependent effects. Our self-supervised training procedure is based on Siamese network that gets a pair of randomly selected images from the training set as input. The task of the network is extract view-dependent lighting effects from image, based on the geometric information from the proxy geometry.
Using fixed projection layer, we back-project the input depth image D; to world space using the intrinsic and extrinsic camera parameters that known from the photogram- metric reconstruction. Based on this position map we generate normal maps via finite differences as well map of the reflected viewing directions. These inputs inspired by the Phong illumina- tion model (Phong 1975) and are stacked along the dimension of the channels. Note, the network input is only dependent on the geometry and the current camera parameters, i.e., the view. Thus, can also be applied to new target views based on the rendered depth of the proxy geometry.
Network Inputs:
Network Architecture: Our network & is encoder-decoder network with skip connections, similar to U-Net (Ronneberger et al., 2015b). The skip connections can directly propagate low- level features the decoder. The encoder is based on 6 convolution layers (kernel size 4 and stride 2). The convolution layers output 32, 32, 64, 128, 256 and 512-dimensional feature maps, respectively. We use the ReLUU activation function and normalize activations based batchnorm.
4

---

Published as a conference paper at ICLR 2020
The decoder mirrors the encoder. We use transposed convolutions (kernel size 4 and stride 2) with the same number of feature channels as in the respective encoder layer. As final layer we use a 4 x 4-convolution with stride of 1 that outputs a 3-dimensional tensor that is fed to Sigmoid to generate an image of the view-dependent illumination effects.
Self-supervised Training: Since we assume constant illumination, the diffuse light reflected by a surface point is the same in every image, thus, the appearance of a surface point only changes by the view-dependent components. We train our network in a self-supervised manner based on a Siamese network that predicts the view-dependent effects of two random views such that the difference of the diffuse aligned images is minimal (see Fig. 2). To this end, we use the re-projection ability (see Sec. 66 to align pairs of input images, from which the view-dependent effects have been removed (original image minus deeendepnd effects), and train the network to minimize the resulting differences in the overlap region of the two images.
Given a randomly selected training pair (Ip, In) and let (X)),, t E {p. q) denote the output of the two Siamese towers. Then, our self-supervised loss for this training sample can be expressed as:
() = M (I, - (X())) W -
t +x())))
(1)
Here, denotes the Hadamard product, O are the parameters of the decoder network Φ, which is shared between the two towers. M is a binary mask that is set to one if surface point visible in both views and zero otherwise. We regularize the estimated view-dependent effects to be small w.r.t an f--ormm This regularizer is weighted with 0.01 in our experiments. The cross- projection WP from image to image q is based on the geometric proxy.
IMAGE-GUIDED RENDERING PIPELINE
To generate a novel target view, we select subset of K 4 images based on a coverage-based nearest neighbor search in the set of reference views (n = 20). We use EffectsNet to estimate the new-dependent effects of these views, to compute diffuse images. Each diffuse image cross- projected to the target view, based on the depth maps of the proxy geometry. Since the depth map of the target view is known, we are able to predict the view-dependent effects in the target image space. After adding these new effects to the reprojected diffuse images, we give these images as input to our composition network Compositionnee (see Sec. 6) CompositionNer fuses the information of the nearest neighbor images into single output image. In the following, describe the coverage- based sampling and the cross projsssro,, and we show how to use our EffectsNee to achieve a robust re-projection of the view-dependent effects.
Coverage-based View Selection The selection of the K nearest neighbor frames is based on sur- face coverage w.r.t. the target view. The goal to have maximum coverage of the target view to ensure that texture information for the entire visible geometry is cross-projected. View selection is cast as an iterative process based on a greedy selection strategy that locally maximizes surface cov- erage. To this end, we start with 64 x 64 sample points on a uniform grid on the target view. In each iteration step, we search the view that has the largest overlap with the currently 'uncovered" region in the target view. We determine this view by cross- coss-projecting the samples from the target view to the captured images, based on the reconstructed proxy geometry and camera parameters. A sample point in the target view is considered covered, if it is also visible from the other view point, where visibility is determined based on an occlusion check. Each sample point that is covered by the finally selected view is invalidated for the next iteration steps. This procedure is repeated until the K best views have been selected. To keep processing time low, we restrict this search to small subset of the input images. This set of reference images is taken from the training corpus and contains n = 20 images. We chose these views also based on the coverage- based selection scheme described above. Le.., we choose the views with most (unseen) coverage among all views in an iterative manner. Note that this selection is done in a pre-processing step and is independent of the test phase.
bbsxy pross-projection We model the cross pooss-projection WA from image p to image g based on the reconstructed geometric proxy and the camera parameters. Let K, E 13423 denote the matrix of intrinsic parameters and T, = [R,,It,] E 4 the matrix of extrinsic parameters of view p. A similar notation holds for view q. Then, a homogeneous 2D screen space point s, = (u, v, d)" e R3 in view p. with depth being d, can be mapped to screen space of view q by: 8.y Wp((p), with
5

---

Published as a conference paper at ICLR 2020
Sp. We employ this mapping to cross-projeet color information from source view to a novel target view. To this end, we map every valid pixel (with a depth estimate) from the target view to the source view. The color information from the source view is sampled based on bilinear interpolation. Projected points that are occluded in the source view or are not in the view frustum Applying the cross-pir prosspproectiio to the set of all nearest neighbor images, we get multiple images that match the novel target view point.
invalidated. Occlusion is determined by depth test w.r.t. the source depth map.
View-dependent Effects lmage-based rendering methods often have problems with the projection of view-dependent effects (see Sec. 7. In our image-guided pipeline, we solve this problem using EffectsNet. Before re-projection, estimate the view-dependent effects from the input images and subtract them. By this, view-dependent effects are excluded from warping. View- dependent effects are then re-inserted after re-projection, again using EffectsNet based on the target view depth map.
CompositionNet: Image compositing The warped nearest views are fused using a deep neural network called CompositionNet. Similar to the EffectsNet, our CompositionNes is an encoder- decoder network with skip connections. The network input is a tensor that stacks the K warped views, the corresponding warp fields as well as the target position map along the dimension of the channels and the output is a three channel RGB image. The encoder is based on 6 convolution layers (kernel size 4 and stride 2) with 64, 64, 128, 128, 256 and 556-dimensional feature maps, respec- tively. The activation functions are leaky ReLUS (negative slope of 0.2) in the encoder and Rel.Us in the decoder. both cases, we normalize all activations based on batchnorm. The decoder mirrors the encoder. We use transposed convolutions (kernel size 4 and stride 2) with the same number of feature channels as in the respective encoder layer. As final layer we use a 4 x 4-convolution with a stride of and Sigmoid activation function that outputs the final image.
We are using an & loss and an additional adversarial loss to measure the difference between the predicted output images and the ground truth data. The adversarial loss is based the conditional PatchGAN loss that is also used in Pix2Pix (Isola et al., 2016). In our experiments, we are weighting the adversarial loss with a factor of 0.01 and the -loss with factor of 1.0.
Training Per object, both networks are trained independently using the Adam optimizer (Kingma Ba 2014) built into Tensorflow (Abadi et al. 2015). Each network trained for 64 epochs with a learning rate of 0.001 and the default parameters B1 0.9, B2 0.999, e
7 RESULTS
The main contribution of our work is to combine the benefits of IBR and 2D GANS, a hybrid that is able to generate temporally-stable view changes including view-dependent effects. We analyze our approach both qualitatively and quantitatively, and show comparisons to IBR as well as to 2D 20
GAN-based methods. For all experiments we used K = 4 views per frame selected from reference views. An overview of all image reconstruction errors is given in Tab. in the appendix. The advantages of our approach can best be seen in the supplemental video, especially, the temporal coherence.
Using synthetic data we are quantitatively analyzing the performance of our image-based rendering approach. We refer to the Appendix A.2. for detailed ablation study w.r.t. the training corpus and comparisons to classical and learned image-based rendering techniques. Since the EffectsNet is a core component of our algorithm, we compare our technique with and without EffeectsNet (see Fig. 3} using synthetic data. The full pipeline results in smoother specular highlights and sharper details. On the test set the MSE without EffectsNer is 2.6876 versus 2.3864 with EffectsNet.
The following experiments are conducted on real data. Fig. 4 shows the effectiveness of our Effect- sNet to estimate the specular effects in an image. The globe has specular surface and reflects the ceiling lights. These specular highlights are estimated and removed from the original image of the object which results in diffuse image of the object. In Fig, 5 we show comparison to Pix2Pix trained on position maps. Similar to the synthetic experiments in the appendix, our method results in higher quality.

---

Published as a conference paper at ICLR 2020
Without EffectsNet
With EffectaNet
Ground Truth
255
Color Difference
Ablation study w.r.t the EffectsNet. Without EffectsNet the specular highlights are not as smooth as the ground truth. Besides, the EffectsNet leads to a visually consistent temporal animation of the view-dependent effects. The close-ups show the color difference w.r.t. ground truth.
Figure 3:
Original Image
View-dependent Effects
Diffuse Image
Figure 4: Prediction and removal of view-dependent effects of a highly specular real object.
We also compare to the state-of-the image-based based rendering approach of Hedman al. (2018) The idea of this technique is to use neural network to predict blending weights for amage-based rendering composition like InsideOut (Hedman et al. 2016). Note that this method uses the per frame reconstructed depth maps of 20 reference frames in addition to the fused 3D mesh. As can be seen in Fig. 6 the results are of high-quality, achieving an MSE 45.07 for DeepBlending and an MSE of 51.17 for InsideOut. Our object-specific rendering results in error of 25.24. Both methods of Hedman et al., do not explicitly handle the correction of view-dependent effects. In contrast, our approach uses EffectsNet to remove the view-dependent effect in the source views (thus, enabling the projection to a different view) and to add new view-dependent effect the target view. This be seen in the bottom row of Fig. (6, where we computed the quotient between reconstruction and ground truth showing the shading difference.
8 LIMITATIONS
Our approach is trained in an object specific manner. This is limitation of our method, but also ensures the optimal results that can be generated using our architecture. Since the multi-view stereo reconstruction of an object is an offline algorithm that takes about an hour, we think that training the object specific networks (EfffectsNet and CompositionNet) that takes similar amount of time is practically feasible. The training of these networks be also includes the appearance of the object. At test time our approach runs at interactive rates, the inference time of EffectsNer is 50Hz, while CompositionNer runs at 10Hz an Nvidia 1080Ti. Note that our approach fails, when the stereo reconstruction fails.
reconstruction refinement that
Similar to other learning based approaches, the method is relying reasonable large training dataset. In the appendix, we conducted an ablation study regarding the dataset size, where our approach gracefully degenerates while pure learning-based approach shows strong artifacts.

---

Published as a conference paper at ICLR 2020
PixePix
Ours
Ground Truth
Figure 5: Comparison to Pix2Pix on real data. can be seen that Pix2Pix can be used to synthesize novel views. The close-up shows the artifacts that occur with Pix2Pix and are resolved by our approach leading higher fidelity results.
InsideOut
DeepBlending
Ours
Ground Truth
Shading Quotient
Figure 6: Comparison to the IBR method InsideOut of Hedman et al. (016)) and the learned IBR blending method DeepBlending of Hedman et al. (2018). To better show the difference shading, computed the quotient of the resulting image and the ground truth. A perfect reconstruction would result in quotient of 1. As can be seen our approach leads to more uniform error, while the methods of Hedman al. show shading errors due the view-dependent effects.
CONCLUSION
In this paper, propose novel image-guided rendering approach that outputs photo-realistie im- ages of an object. We demonstrate the effectiveness of our method in a variety of experiments. The comparisons to competing methods show on-par or better results, especially, in the presence of view-dependent effects that can be handled using our EffectsNet. We hope to inspire follow-up work in supe-suue re-rendering using deep neural networks.
ACKNOWLEDGEMENTS
We thank A for providing scanned 3D models and Angela Dai for the video voice over. This work funded by a Google Research Grant, supported by the ERC Starting Grant Scan2CAD (804724), ERC Starting Grant CapReal (335545), and the ERC Consolidator Grant 4DRepLy (770784), the Max Planck Center for Visual Computing and Communication (MPC-VCC), a TUM- IAS Rudolf Mößbauer Fellowship (Focus Group Visual Computing), and Google Faculty Award. In addition, this work funded by Sony and the German Research Foundation (DFG) Grant Making Machine Learning Static and Dynamic 3D Data Practical, and supported by Nvidia.
https: www. http://wwwwttetddd./o mode13
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogencous systems, 2015. URL hitps: /www www.wefwoso.....oooomo org/ Software available from tensorflow.org.
Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski. Building rome in a day. Commun. ACM, 14110)1005-112 October 2011. ISSN 0001-0782, doi: 1..1140/200029 2690000129 UtI.ttpp://doi..... http:///oo..aa..oo/.........
2001269. 200 1293
Nicolas Bonneel, Kalyan Sunkavalli, James Tompkin, Deqing Sun, Sylvain Paris, and Hanspeter Pfister. Interactive Intrinsic Video Editing. ACM Transactions on Graphies (SIGGRAPH Asia 2014), 33(6), 2014.
Nicolas Bonneel, Balazs Kovacs, Sylvain Paris, and Kavita Bala. Intrinsic Decompositions for Image Editing. Computer Graphics Forum (Eurographies State of the Art Reports 2017), 36(2), 2017.
Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstruc- tured lumigraph rendering. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '01, pp. 425-432, New York, NY, USA, 2001, ACM. ISBN 1-58113-374-X, doi: 10. 10.1145/383259
URL http: acm. org/10.
1145/383259. 383309
Joel Carranza, Christian Theobalt, Marcus A. Magnor, and Hans-Patee Seidel. Free- viewpoint video of human actors. ACM Trans. Graph. (Proc.
July 2003. ISSN 0730- 0301. doi: 10.1145/882262.882300 URL http:://doooo acm. http:///0........oooo.0..111111.
882309
Dan Casas, Christian Richardt, John P. Collomosse, Christian Theobalt, and Adrian Hilton. 4d model flow: Precomputed appearance alignment for real-time 4d video interpolation. Comput. Graph. Forum, 34(7):173-182, 2015.
C. Chan, S. Ginosar, T. Zhou, and A. Efros. Everybody Dance Now. ArXiv e-prints, August 2018.
Gaurav Chaurasia, Sylvain Duchene, Olga Sornine and George Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Graph., 32(3):30:1330:1 July 2013. ISSN 0730-0301.
Paul Debevec, Yizhou Yu, and George Boshokov. Efficient view-dependent IBR with projective exxure-mapping EG Rendering Workshop, 1998.
Ruofei Du, Ming Chuang, Wayne Chang, Hugues Hoppe, and Amitabh Varshney. Montage4d: Interactive seamless fusion of multiview video textures. In Proceedings the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, I3D '18 pp. 5:1-5:11, New York. NY, USA, 2018. ACM. ISBN 978- --500--70000 doi: 10.1145/3100834.199888 URL http://doi. acm. wto/10..//.o. 111111111300 90834499134... 01115/3119994441998444444 3190843
M. Eisemann, B. De Decker, M. Magnor, P. Bekaert, E. De Aguiar, N. Ahmed, C. Theobalt, and A. Sellent. Floating Textures. Computer Graphics Forum (Proc. EUROGRAHICS, 2008. ISSN 1467-8659. doi: 10.1111/.1467-8653
138.x.
Patrick Esser, Ekaterina Sutter, and Björ Ommer. A variational u-net for conditional appearance and shape generation. CoRR, abs/ 804.04694, 2018. URL wwwp:xii.ogg/a///ooaaoeea 04694

---

Published as a conference paper at ICLR 2020
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning to predict new views from the world's imagery. In The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016. URL http: http:://www../ cv- www.datooooooooooooooooooooooooo
htt.:/aat..oo..cogooppeaaaeeesse content_ _eppt___pr_____ 201 1at0r_2016/htmir F ynn yyw__ppettttrroo.aarrnnnnn...oo. paper. html
Andreas Geiger, Julius Ziegler, and Christoph Stiller. Stereoscan: Dense 3d reconstruction in real- time. In Intelligent Vehicles Symposium, pp. 963-968. IEEE, 2011. ISBN 978-1-4577-0890--9
Clment Godard, Peter Hedman, Wenbin Li, and Gabriel J. Brostow, Multi-view Reconstruction of Highly Specular Surfaces in Uncontrolled Environments. In 3DV, 2015.
Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. Multi-view stereo for community photo collections. In ICCV, pp. 1-8. IEEE Computer Society, 2007. ISBN 978-1-4244-1631-8
lan J.
Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. 2014.
Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. Scalable Inside-Out Image- Based Rendering. 35(6):231:1-231: 2016.
Peter Hedman, Julien Philip. True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Bros- tow. Deep blending for free-viewpoint based rendering. ACM Transactions on Graphics (SIGGRAPH Asia Conference Proceedings). 37(6), November 2018. URL hntp::/p://wwt sop.
inria. fr reves / Basi /tc.///1111 8 HPPFDB18
Benno Heigl, Reinhard Koch, Marc Pollefeys, Joachim Denzler, and Luc J. Van Gool. Plenoptic modeling and rendering from image sequences taken by hand-held camera. In Proc. DAGM, pp. 94-101, 1999.
Geoffrey E. Hinton and Ruslan Salakhutdinov. Reducing the dimensionality data with neural networks. Science, 313(5786):504-500 July 2006. ISSN 0036-8075. doi: 10.1 126/sciecee 1127647.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. pp. 5967-5976, 2017. doi: 10.1109/VVR..2017..633
Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, and Andrew Fitzgib- bon. Kinectfusion: Real-time 3d reconstruction and interaction using moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST 11, pp. 559-568, New York, NY, USA, 2011, ACM, ISBN 978-1-4503-0716-1. doi: 10. 1145/2047196.2047777 URL
acm. http://ff.....ooo.ooo..//////2/1
Wenzel Jakob. Mitsuba renderer, 2010. http://www.miissubereedddrrr.
Jebara, A. Azarbayejani, and A. Pentland. 3d structure from 2d motion. IEEE Signal Processing
Magazine, 16(3):66-84, May 1999. ISSN 1053-5888. doi: 10.1 1991709997775774
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi, Learning- baased view synthesis for light field cameras. ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia 2016),
35(6), 2016.
Michael Kazhdan and Hugues Hoрре Screened poisson surface reconstruction. ACM Trans. Graph. 321 July 2013. ISSN 0730-0301. doi: 10. 10.1145/448228.2487755 URL http: // //d.ii.cm..omm www./rg///0 1145 2tt.://0.000...11111///822228210
Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. Poisson surface reconstruction. In Pro- ceedings of the Fourth Eurographics Symposium Geometry Processing, SGP '06. pp. 61-70, Aire-la- Ville, Switzerland, Switzerland, 2006. Eurographics Association. ISBN 3-905673-36-3
J http::///// //11. cm .r.. cttggiitationn www.attio.........mmmmmmmmmmmm wwwmic=miiioo.mmmm....mmmmmm 1 281957. 1281 965
10

---

Published as a conference paper at ICLR 2020
H. Kim, P. Garrido, A. Tewari, W. Xu, J. Thies, N. Nießner, P. Pérez, C. Richardt, M. Zollhöfer, and C. Theobalt. Deep Video Portraits. ACM Transactions on Graphics 2018 (TOG), 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, 0bss/141.66900, 2014.
6980
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013. URL http:://bbb/t uni-tuiiitr...... wtw.eerrdeee/bbeee http://ww.............////aaa/aa
hhmml
K KinggaW133
Stephen Lin, Yuanzhen Li, Sing Bing Kang, Xin Tong, and Heung- Yeung Shum. Diffuse-specular separation and depth recovery from image sequences. In Proceedings of the 7th European Conference on Computer Vision-Parl III, ECCV '02, pp. 210-224, Berlin, Heidelberg, 2002. Springer- Verlag. ISBN 3-540-43746-0. URL http: acm. wtt//orggttteoooooo http:/iaatiio... cfmii
045317.649349
Lingjie Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bernard, Marc Haber- mann, Wenping Wang, and Christian Theobalt. Neural animation and reenactment of human actor videos, 2018.
Abhimitra Meka, Michael Zollhoefer, Christian Richardt, and Christian Theobalt. Live intrinsic video. ACM Transactions on Graphics (Proceedings SIGGRAPH), 35(4), 2016.
Mehdi Mirza and Simon Osindero, Conditional generative adversarial nets. arXiv:1411.178. 2014.
URL httpt: /taptttss///axii... http:/r//aaabbagg.eee
M. Nießnner M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG), 2013.
Eunbyung Park, Jimei Yang, Ersin Yumer, Duygu Ceylan, and Alexander C. Berg. Transformation- grounded image generation network for novel 3d view synthesis. CoRR, abs/1703.0299 2017.
Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Trans. Graph., 36(6): 235:1-235: 1, November 2017. ISSN 0730-0301.
Bui Tuong Phong. Illumination for computer generated pictures. Commun, ACM, 186))311-317, June 1975. ISSN 0001-0782, doi: 0..1144/360825.660883 URL httt:://ttoot.... acm .rm. 10. 1144/ 360825.0.. 360839
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. 2016.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomed- ical image segmentation. In International Conference Medical Image Computing and Cosputer-Assisted Intervention (MICCAI), pp. 234-241, 2015a. ISBN 978-3-319-24574-4, doi:
10.10077998833311922577444
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Nassir Navab, Joachim Hornegee, William M. Wells, and Alejan- dro F. Frangi (eds.), Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241, Cham, 2015b. Springer International Publishing. ISBN 978-3-319-24574-4
Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-Motioo Revisited. In Confer- Computer Vision and Pattern Recognition (CVPR), 2016.
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference Computer Vision (ECCV), 2016.
J. L. Schnberger and J. Frahm. Structurefrom-mmotee revisited. In 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 4104-4113 June 2016. doi: 10.11091 CVPR.2016.445.
11

---

Published as a conference paper at ICLR 2020
Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. A com- parison and evaluation of multi-view stereo reconstruction algorithms. In Proceedings of the 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition Volume
I, CVPR '06, pp. 519-528, Washington, DC, USA, 2006. IEEE Computer Society. ISBN 0- 7695-2597-0. doi: 10.1109/CVPR 2006..
UtLpttp: /x doi wtw./ooroo...... 1009/VVRR.
2006.19
K. Takechi and T. Okabe. Diffuse-specular separation of multi-view images under varying illumina- tion. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 2632-2636, Sept 2017. doi: 10. 10.1109/C1122001.889677
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Trans. Graph., 58(4))66:1166:11) July 2019. ISSN 0730-0301. doi: 10.1 10.1145/330644633329000
Bill Triggs, Philip F. McLauchlan, Richard Hartley, and Andrew W. Fitzgibbon. Bundle adjust- ment modern synthesis. In Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ICCV '99, pp. 3-540-67973-1.
London, UK, UK, 2000. Springer- Vereae. ISBN
Shubham Tulsiani, Richard Tucker, and Noah Snavely. Layer-structured 3d scene inference via view synthesis. In ECCV, 2018.
Shihao Wu, Hui Huang, Tiziano Portenier, Matan Sela, Daniel Cohen-Or, Ron Kimmel, and Matthias Zwicker. Specular-to-diffuse translation for multi-view reconstruction. In Vittorio Fer- rari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, doi: 10.1007/ 978- 978-3-030-01225-00 12. URL https: /ooi wwg. 10c.1100000000 978 - 12
volume 11208 of Lecture Notes in Computer Science, pp. 193-21121111 Springer, 2018.
Qingxiong Yang. Jinhui Tang, and Narendra Ahuja. Efficient and robust specular highlight re- moval. JEEE Trans. Pattern Anal. Mach. Intell., 37(6):1304-1311 2015. URL http::/dbblp
ami-trier.co de hwwpdd///oooaaaaaeeoooooaaaaeeee hpw.aaaaaaaaaaaaaaaaaaaaaaaaalla 37 hrml Yangmmiiaannnn5.
Genzhi Ye, Elena Garces, Yebin Liu, Qionghai Dai, and Diego Gutierrez. Intrinsic video and ap- plications. ACM Trans. Graph., 33(4): July 2014. ISSN 0730-0301. doi: 10.1 2601097.2601 135. /ttptt/:///ooi acm. org 10. 0.......11211110.0000 2601135
Ke Colin Zheng, Alex Colburn, Aseem Agarwala, Maneesh Agrawala, David Salesin, Brian Curless, and Michael F. Cohen. Parallax photography: creating 3d cinematic effects from stills. In Proc. Graphics Interface, pp. 111-118. ACM Press, 2009.
Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A. Efros. View synthesis by appearance flow. In ECCV. 2016.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans, Graph., 37:4):65:--65:12, July 2018. ISSN 0730-0301. doi: 10. 10.1145/3197517:2201333 URL http:://ooo...... htt.://di..ccccooooommmmooo. acm acm.0 org/10.
1145 0745/31975111 320112223313333
Hao Zhu, Hao Su, Peng Wang, Xun Cao, and Ruigang Yang. View extrapolation of human body from single image. CoRR, abs/1804.04211 043114 2018.
12

---

Published as a conference paper at ICLR 2020
A APPENDIX
A.1 TRAINING CORPUS
In Fig. we show an overview of synthetic objects that we used to evaluate our technique. The objects differ significantly in terms of material properties and shape, ranging from nearly diffuse materials (left) the highly specular paint of the car (right).
Figure 7: 2010), we generate images of various objects that significantly differ in terms of material properties and shape.
Renderings of ground truth synthetic data. Based the Mitsuba Renderer (Jakob)
To capture real-world data we record short video clip and use multi-view stereo to reconstruct object depicted in Fig.
Observed Images
3D Reconstruction
RGB-D Training Corpus
Figure 8: Based on set of multi-view images, we reconstruct coarse 3D model. The camera poses estimated during reconstruction and the 3D model are then used render synthetic depth maps for the input views.
A.2 ADDITIONAL, RESULTS
An overview of all image reconstruction errors of all sequences used this paper is given in Tab. MSE values reported w.r.t. a color range of [0, 255). Note that the synthetic data contain ground truth depth maps, while for the real data, we are relying on reconstructed geometry. This is also reflected in the photo-metric error (higher enror due misalignment).
Sequence
N-IBR
IBR (Dehevec al. 1998
Pix2Pix Isola
2016))
Ours
Synthetic
Fig.
Car
72.16
39.90
12.62
3.61
Fig.
Statue
44.27
25.01
5.40
2.38
Fig.
Vase
38.93
17.31
18.66 66
1.12
Fig.
12
Bust
35.58
20.45
4.43
1.52
Fig. 4
Globe
152.21
81.06
154,29
30.38
Real
Fig.
4
Shoe
98.89
59.47
116.52
56.08
Fig.
Bust
397.25
72.90
45,23
25.24
Table 1: MSE of photometric re-rendering error of the test sequences (colors in [0-255]). Pix2Pix trained on world space position maps. N-IBR the naïve blending approach that gets our nearest neighbors as input.
13

---

Published as a conference paper at ICLR 2020
A.2.1 ADDITIONAL EXPERIMENTS ON SYNTHETIC DATA
Using synthetic data we are quantitatively analyzing performance of our image-based rendering approach.
EffectsNet In Fig. we show a qualitative comparison of our predicted diffuse texture to the ground truth. The figure shows the results for Phong rendering sequence. As can be seen, the estimated diffuse image is close the ground truth.
Estimated Specular
Estimated Diffuse
Phong Rendering
Ground Truth
Figure 9: Comparison of the estimated diffuse images based EffectsNer and the ground truth renderings. The input data has been synthesized by a standard Phong renderer written in DirectX. The training set contained 4900 images.
Comparison to Image-based Rendering We compare our method to two baseline image-based rendering approaches. A naive IBR method that uses the nearest neighbor of our method and com- putes per pixel average of the re-projected views, and the IBR method of Debevec et al. (1998) that uses all reference views and a per triangle view selection. In contrast our method, the classical IBR techniques are not reproducing view-dependent effects as realistically and smoothly which can be seen in Fig. I0 As can seen the naive IBR method also suffers from occluded regions. Our method is able to in- in-paint these regions.
Naive IBR
Debevec et al.
Ours
Ground Truth
Figure 10: Comparison of our neural object rendering approach to IBR baselines. The naive IBR method uses the same four selected images as our approach as input and computes a wixel-wise average color. The method Debevec et al. (1998) uses all reference views (n 20) and per triangle view selection. The training set contained 1000 images.
14

---

Published as a conference paper at ICLR 2020
Comparison to Learned Image Synthesis To demonstrate the advantage of approach, also compare to an image-to-image translation baseline (Pix2Pix (Isola et al. 2016)). Pix2Pix is trained translate position images into color images of the target object. While it is not designed for this specific task, we want to show that it is in fact able to produce individual images that look realistic (Fig. [T] however, is unable produce temporally-coherent video. On test set with 190 images, our method has a MSE of 1.12 while Pix2Pix results in a higher MSE of 18.66. Pix2Pix trained on pure depth maps from our training results in MSE of 36.63 since the input does explicitly contain view information.
Pix2Pix
Ours
Ground Truth
Figure 11:
In comparison to Pix2Pix with position maps as input, we can see that our technique able generate images with correct detail well without blur artifacts. Both methods trained on a dataset of 920 images.
Evaluation of Training Corpus Size In Fig. 12 show the influence of the training corpus size on the quality of the results. While our method handles the reduction of the training data size well, the performance of Pix2Pix drastically decreases leading significantly higher MSE. When comparing these results to the results in Fig 11 it becomes evident that Pix2Pix has significantly lower error on the bust sequence than on vase sequence. The vase has much more details than the bust and, thus, is harder to reproduce.
MSE w.r.t. Training Dataset Size
20
15
Pix2Pix
Ours
1 train set
/ 4 train set
1 / 2 train set
full train set
Ground Truth Truth
this graph we compare the influence the training corpus size on the MSE for our approach and Pix2Pix trained on position maps. The full dataset contains 920 images. We gradually half the size the training set. As can seen, the performance of our approaches degrades more gracefully than Pix2Pix.
Figure 12:
15

---

Published as a conference paper at ICLR 2020
A.2.2 ADDITIONAL EXPERIMENTS ON REAL DATA
Comparison Texture-based Rendering Nowadays, most reconstruction frameworks like COLMAP (Schönberger & Frahm 2016). KinectFusion (lzai et al. 2011). or VoxelHash- ing (Nießner et al.J (003)) output a mesh with per-vertex colors or with texture, which is the de facto standard in computer graphics. Fig. [3] shows side-by-sid comparison of our method and rendering using per-vertex colors as well as using static texture. Since both the vertex colors as well as the texture are static, these approaches are not able to capture the view-dependent ef- fects. Thus, view-dependent effects are baked into the vertex colors or texture and stay fixed (seen close-ups in Fig. 13
Vertex Colors
Static Texture
Ours
Ground Truth
Figure 13: Image synthesis on real data in comparison to classical computer graphics rendering approaches. From left to right: Poisson reconstructed mesh with per-vertex colors, exture-based rendering, our results and the ground truth. Every texel of the texture is a cosine-weighted sum of the data of four views where the normal points towards the camera the most.
Comparison to Image-based Rendering Fig. 14 shows comparison of our method to image- based rendering. The IBR method of Debevec et 1998) uses per triangle based view selection which leads to artifacts, especially, in regions with specular reflections. Our method is able to reproduce these specular effects. Note, you can also see that our result is sharper than the ground truth (motion blur), because the network reproduces the appearance of the training corpus and most images do not contain motion blur.
Reconstrueted Geometry
Debevec et al.
Ours
Ground Truth
Figure 14: Image synthesis on real data: we show a comparison to the IBR technique of Debevec jet al. (1998.. From left to right: reconstructed geometry of the object, result of IBR, our result, and the ground truth.
16


Reviews:
[{"review_id": "rJgQmJS6qB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of re-rendering the appearance of an object from novel viewpoints, given reference images of the same object as input. Multi-view reconstruction and camera poses are used to generate depth images to be used as additional input to the network. The authors propose an EffectsNet architecture to derive view-independent Lambertian reflectance maps and train the network in a self-supervised way through re-projection. From the target viewpoint, the image is re-textured by a blending network and view-dependent lighting effects are added back, to generate a novel-view rendering.\n\nApplication may be limited due to the fact that the network is re-trained from scratch for each object instance. Another weakness is insufficient evaluation against other deep-learning based IBR approaches. It is nice to see comparison with Hedman et al. (2016 and 2018) but I feel that details are missing. Figure 6 and L3-4 of page 8 claim a better MSE and qualitative results for a single example, but the difference is not obvious and to validate the claim that the proposed approach \"leads to superior reproduction of view-dependent appearance\" (Page 3 under Image-based Rendering), I think a more systematic evaluation on multiple objects and views is needed.\n\nDespite the weaknesses, I am leaning towards Weak Accept because I think the benefits (requiring less training examples and being able to generate a more temporally coherent images) are convincing. Using surface reflectance decomposition to reduce photometric inconsistency in projection-based self-supervision is also novel for this task. But I think a more thorough comparison with Hedman et al. would be crucial in justifying the design choices.\n\nOne potential improvement for future work is that it may be possible to compute confidence maps in the diffuse image estimation step and use that to improve blending of the warped images.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "rJeAFcPpFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method for rendering of reconstructed objects. At its core, it works by distinguish between diffuse images and \"view-dependent effects\" (in essence, every element related to the scene instead of the object like specularities). The approach leverages two distinct neural networks. First, EffectsNet, which infers the view-dependent effects from an image and their depth map. Second, CompositionNet, which learns to combine multiple warped view into the final image and to remove the related artifacts. Results show improvements with respect to previous methods, especially for temporal coherence and dealing with a low number of images.\n\nThe proposed approach is well described, and does seem to fix a few issues with current methods. The comparisons are extensive and comprehensive for what I can tell. The results on static images are not significantly better than previous work, however, and the need to train the whole pipeline for every object is quite a limitation. Apart these considerations, I do not have much complaints about this work. The few others I came up with are provided below.\n\nI am not completely sure here, but it seems like CompositionNet could be used with other approaches than the one described in the paper, in particular with IBR. How would it improve the performance of previous approaches, especially regarding temporal coherence?\n\nFor the EffectsNet training, we have to assume that two random images from the training set can be reprojected onto one another without loosing too much of the surface. If the views provided by these images are too different, than the number of pixels actually providing a loss to the network will be very low, which would be detrimental to the learning process. Could you comment on that? Is there a quick fix to this issue?\n\nOne thing that was not discuss was about the effects integrating a neural network into the rendering pipeline have on the output resolution. Classical methods are not really limited on this regard, but neural networks cannot easily provide high resolution outputs. I understand that EffectsNet is not directly used as image (only to subtract the \"view-dependent\" parts), but could there be issues if we use a resolution of, say, 2048x2048 instead of 512x512? Or even higher? Given the aim of the paper at targeting high quality rendering, such consideration has important practical consequences.\n\nIn summary, this paper does a reasonable job at improving the current IBR methods. The idea of making the network learns the scene effects instead of the whole object appearance is sound and could probably be translated to other related problems.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Byln3WvatS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a method to perform neural rendering. From a set of images taken of a static object under constant illumination, the proposed method first selects the four viewpoints nearest to the requested novel viewpoint. Then, the images corresponding to those viewpoints are blended together using an encoder-decoder network to produce the novel view image. The key contribution of the submission over previous work is the handling of view-dependent effects such as specular highlights. Those view-dependent effects are first removed from the nearest retrieved images using the proposed EffectsNet. The resulting estimated diffuse images are then re-projected to the target viewpoint and view-dependent effects from this viewpoint are added before blending the images together.\n\nThe proposed method improves the robustness of existing neural rendering methods to materials that are not roughly diffuse. \n\nThe process to select the 20 reference images is based on a coverage scheme that is never presented, hindering reproducibility. Would it be possible to describe this scheme? Similarly, the hyperparameters used for the adversarial loss are not explicitly stated, are they exactly the same as the cited Pix2Pix? Would the source code be shared publicly?\n\nI would have appreciated an ablative experiment where the proposed pipeline is kept as-is, but a state-of-the-art intrinsic decomposition technique (as discussed in sec. 2) was applied instead of EffectsNet.\n\nI would recommend using subsections to prevent confusion in references (for example, sec. 6, p. 5 referring to sec. 6).\n\nIn fig. 6, the quotient image is hard to interpret as it is not linear. A colormap representing the percentage of error wrt. the ground truth might be easier to interpret.\n\nEffectsNet is used for both removal and addition of view-dependent effects, which makes part of the paper confusing. Maybe adding the mathematical symbols of eq. 1 to fig. 2 might help the reader to understand the training steps?\n\nThe impact of the regularizer weight of 0.01 (p. 5) is not discussed. I suspect this value might be important, as too much regularization might give underestimated view-dependent effects and too little might provide strong and incoherent effects. An analysis of this hyperparameter would be welcome.\n\nIn my opinion, the proposed abstract is slightly hard to read, maybe it would benefit from being shorter?\n\nMinor details\n- p. 3 “An extensive overview is give[n] in [...]”\n- p. 8 I believe the “z” of the unit “Hz” is wrongly stylized as a mathematical variable.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkewAg0n9r", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a novel neural image-guided rendering method, a hybrid between classical image-based rendering and machine learning. They learn EffectsNet in a self-supervised manner to capture the view-dependent component using a reprojection loss. To render a novel view they subtract a view-dependent component from K closest train views, reproject the diffuse component to the novel view, add view-dependent component, use a CompositionNet to render a final image. They use both synthetic and real data for the experiments and compare to state-of-the-art image-based rendering methods of Hedman et al.    \n\n\nI think the method discussed in the paper is a nice contribution to the neural rendering area. The paper is written clearly and the provided amount of technical details is enough to reproduce the method. \n\n\nI only have some concerns about the experiments. \n\n1) The method relies on the depth maps and the main contribution of the paper is related to the rendering of view-dependent effects. However, in realistic scenarios, the presence of the view-dependent effects leads to severe degradation of photogrammetry pipeline: imprecise camera poses and geometry. To my understating, imprecise geometry and camera poses should affect the method heavily. I think, the paper lacks a thorough evaluation on more realistic data, having only a single example in Fig. 14. \n\n2) Authors compare to DeepBlending of Hedman et al. and although the proposed model achieves the best quantitative result, to my mind, the qualitative difference is not very noticeable, while it is known that MSE error does not correlate with human perception good enough. To justify the presented method it would be great to see experiments with a more clear difference to Hedman et al. baseline. \n\n3) Authors of Pix2Pix (Isola et al., 2016) do not provide experimental results on novel view-point synthesis. Although it is completely fine to refer to Pix2Pix as to the closest and simplest neural-based baseline I do not find it convincing to compare the proposed method to Pix2Pix without any other baselines as in the figures 5, 11, 12. Pix2pix had never been claimed to be developed or validated for this particular problem so it is a kind of \"artificial\" baseline. Moreover, I believe a stronger \"artificial\" baseline can be easily constructed. In particular, it is hard to believe that for the \"Vase\" scene, which has almost no view-dependent effects, a network that takes a world space position map cannot learn a direct mapping from (x,y,z) to (r, g, b). That is why, I believe, a stronger \"artificial\" baseline could be easily constructed by some kind of tweaking of Pix2Pix, say, turning off GAN or changing neural networks' architecture slightly.  \n\nConsidering the issues mentioned above my I tend to recommend this paper for rejection. \n\n\nQuestions:\n\n1) Video shows examples of novel view synthesis for a predefined camera trajectory. How this trajectory is created for both synthetic and real objects? How does it differ from the train trajectory?\n\n\nSome minor comments:\n\n1) In Fig. 2 it reads \"Input color\", whereas the RGB color image is not used as input to EffectsNet. It confuses on the first read. In \"... based on the current view and the respective depth map\", it is also easy to think that \"current view\" refers to RGB image.\n\n2) The model from Figure 8 is never used.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a new variation of neural (re) rendering of objects, that uses a set of two deep ConvNets to model non-Lambertian effects associated with an object. The paper has received mostly positive reviews. The reviewers agree that the contribution is well-described, valid and valuable. The method is validated against strong baselines including Hedman et al., though Reviewer4 rightfully points out that the comparison might have been more thorough. 

One additional concern not raised by the reviewers is the lack of comparison with [Thies et al. 2019], which is briefly mentioned but not discussed. The authors are encouraged to provide a corresponding comparison (as well as additional comparisons with Hedman et al) and discuss pros and cons w.r.t. [Thies et al] in the final version.
Decision: Accept (Poster)