Object-Centric Neural Scene Rendering
Authors: Michelle Guo
Alireza Fathi
Jiajun Wu
Thomas Funkhouser
Venue: ICLR 2022 Submitted (2022)

Abstract: We present a method for composing photorealistic scenes from captured images of objects. Traditional computer graphics methods are unable to model objects from observations only; instead, they rely on underlying computer graphics models. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitted radiance of a scene. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object-centric neural scattering functions (OSFs), a representation that models per-object light transport implicitly using a lighting- and view-dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering light transport effects including occlusions, specularities, shadows, and indirect illumination, both within individual objects and between different objects. We evaluate our approach on synthetic and real world datasets and generalize to novel scene configurations, producing photorealistic, physically accurate renderings of multi-object scenes.

Under review as a conference paper at ICLR 2022
OBJECT-CENTRIE NEURAL SCENE RENDERING
Anonymous authors Paper under double-blind review
ABSTRACT
We present method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitte collection of images. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, propose to learn object-centric neural scattering functions (OSFs), rep- resentation that models per-object light transport implicitly using lighting- and niew-dependen neural network. This enables rendering scenes even when ob- jects or lights move, without retraining. Combined with a volumetric path tracing procedure, framework is capable of rendering light transport effects including occlusions, specularities, shadows, and indirect illumination, both within individ- ual objects and between different objects. We evaluate OSFs on synthetic and real world datasets, and on generalizing to new scene configurations. Learning OSFs leads photorealistic, physically-accuraa renderings multi-object scenes.
radiance of a scene from a
INTRODUCTION
Synthesizing images of dynamic scenes an important problem in computer vision and graphics, with applications in AR/VR and robotics (Savva et al. 2019 Xia et al. 2020). For synthetic scenes, a user typically designs set of 3D objects separately, then composes them into scenes to be rendered with specified camera, material, and lighting parameters. While this traditional graphics approach allows for flexible scene compositions, requires detailed models of geometry, lighting, materials, and cameras, which can be difficult to obtain for real-world scenes.
To render real-world scenes without computer graphics models, recent works have explored using neural implicit methods (Lombardi al. 2019 Sitzmann et al. [2019 b). Most notably, Milden- hall et al. (2020) proposed neural radiance fields (NeRF), which achieve photorealistic quality by implicitly modeling the volumetric density and directional emitted radiance of scene.
However, shown in Figure 1 NeRF cannot gen- eralize beyond the scene it was trained on, because it assumes static scenes and fixed illumination and learns a radiance field, which estimates only the resulting radiance along ray after all light trans- port has occurred in a scene. Thus, for dynamic scenes where lights and objects can move, sep- arate NeRF-bbaee model is needed for each new scene configuration.
NeRF (Baseline)
(b) OSF (Our Method)
Figure 1: (a) NeRF. (b) Our method.
To address this issue, we propose Object-Centric Neural Scattering Functions (OSFs) to synthesize dynamic scenes of objects learned from 2D images (Figure We represent each object learned 7D scattering function with inputs (x, y, z, di, 02, do, 0.), where (z, y, z) is the spatial location, (ø,, 0.) is the incoming light direction, and (Ø., fo)) is the outgoing light direction. The function outputs the volumetric density as well as the fraction of light arriving from direction (ø,, that scatters in outgoing direction (Ø.. 0.).
Each OSF models all light bounces (reflections) and occlusions (shadows) within an object. Since each object's scattering function radiance transfer function rather than radiance field, it is

---

Under review as a conference paper at ICLR 2022
(4) Panama
(9) Object Library
(c) Cempose Scene
(d) Move Light
(6) Move Camera
Mova Objects
Figure 2: We propose an object-centric neural scene representation for image synthesis. Given scene description (a), and a repository of neural object-centric scattering functions (OSF) trained independently from images and frozen for each object (b), we can compose the objects into scenes (c), and render photorealistic images as we move lights (d), cameras (e), and/or objects (f). Our framework is capable of rendering occlusions, specularities, shadows, and indirect illumination.
intrinsic to the object (independent of the scene it is in) and can be reused across different object placements and lighting conditions without retraining. We emphasize that because NeRFs are radi- ance fields, they cannot be composed, and cannot generalize beyond one scene. In contrast, we can render infinitely many scenes. We can build a library of OSFs trained independently for different objects to be composed into scenes with different object placements, camera, and lighting.
To model light transport between objects, we integrate our implicit object functions with volumetric path tracing. Like NeRF, we evaluate the radiance and volumetric density at 5D samples along every primary ray to camera and composite them with an over operator. However, unlike NeRF, estimate the radiance for each 5D sample by integrating our 7D OSF across the 2D sphere of incoming light directions. We estimate the integral with Monte Carlo path tracing (Kajiyal 1986) to reproduce shadows and indirect illumination effects.
Our key idea is to decompose the rendering problem into (i) a learned component (per-object asset creation), and (ii) non-learned component (per-scene path tracing). The learned component mod- els intra-object light transport (e.g.. bounces from the seat of a chair to the back of the chair). The non-learned component handles inter-object light transport (e.g., bounces from a wall to chair). To- gether, they model the full rendering equation (Kajiya 1986) (except for occluders or light sources that intrude the object's convex hull (Sloan al. 2002)). Since only the inter-object light trans- port changes as objects and lights move, no re-training required for different scene arrangements. Experimental results indicate that our method is capable of rendering images with novel scene com- positions and lighting conditions better than alternative learned approaches.
In summary, our contributions are:
1. Learning Object-Centric Neural Scattering Functions (OSFs) that model intra-object light trans- port implicitly using a lighting- and view-dependent neural network.
2. Integrating implicitly learned object scattering functions with volumetric path tracing to model inter-object light transport.
A rendering algorithm that enables rendering scenes with moving objects, lights and cameras, using implicit functions.
2 RELATED WORK
Classical object-centric representations. Factoring light transport into intra- and inter-object il- lumination has long history in traditional computer graphies (Dutre et al. 2018). In most cases, the motivation is to improve rendering efficiency by approximating intra-object lighting factors with simple transfer functions (e.g.. linear) for simple radiance fields (e.g., spherical harmonies) de- rived from from computer graphics models, as in precomputed radiance transfer (PRT) (Sloan et al.) 2002], ambient occlusion (Miller 1994), or virtual walls (Arnaldi et al.) 1994). In other cases, the motivation is to insert captured, real-world radiance fields into synthetic scenes, as in Light Field Transfer (Cossairt et al., 2008). These methods generally store the radiance field for objects a discrete representation (c.g., sampled 2D or 4D grid). As result, they cannot reproduce accurate inter-object light transport, especially for objects with intersecting bounding volumes. In contrast, focus learning radiance transfer from images in order to model complex real-world scattering accurately, and utilize volumetric rendering techniques to account for inter-object illumination.
2

---

Under review as a conference paper at ICLR 2022
Novel view synthesis. Traditional methods for synthesizing novel views of a scene from cap- tured images include using Peructre-FFoom-oonn
(Hartley & Zisserman 2003) and bundle adjust- ment Triggs et al. 1999) to predict a sparse point cloud and camera parameters of the scene. More recently, a number of learning-
novel view synthesis methods have been presented but require 3D geometry as inputs (Hedman al. 2018: Thies et al. 2019; Meshry al. 2019 Aliev al.) 2020 Martin-Bruall et al., 2018). Others use multiplane images as proxies for novel view synthe- sis, but their viewing ranges are limited to interpolated input views (Flynn al. 2016 Zhou al., 2018 Srinivasan et al. 2019 Mildenhall et al. 2019). Some works represent scenes as coarse voxel grids and use CNN- based decoder for differentiable rendering, but lack view consistency due to the use of 2D convolutional kernels (Nguyen- Phuoc et al.. 2018 2019 2020).
Recently, volume rendering approaches have been used to render scenes represented as voxel grids that are more view-consistent (Lombardi et al. 2019;; Sitzmann et al. 2019a). However, the render- ing resolution of these methods are limited by the time and computational complexity of discretely sampled volumes. To address this issue, Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) directly optimizes a continuous radiance field representation using a multi-layer perceptron. This allows synthesizing novel views of realistic images at an unprecedented level of fidelity. To make NeRF more efficient, Neural Sparse Voxel Fields (Liu et al. (2200) have been proposed as a sparse voxel octree variant of NeRF and demonstrate the ease of composing learned NeRFs with their voxel representation. See (Dellaert & Yen-Chen (2220) for survey. While these implicit methods produce high-quality novel views of scene, their models assume a static scene with fixed illumi- nation. Our method enables synthesizing dynamic scenes with novel viewpoint, lighting, and object configurations.
Relighting. Learning-baat -baeed methods that relight images without explicit geometric reasoning have been proposed, but lack the ability to recover hard shadows (Sun et al. 2019 Xu et al. 2018 Zhou ctt al. 2019). Other works use geometric representations that facilitate shadowing computation,
but require 3D geometry as input (Philip et al. 2019 ZZhnn
al., 2021 Oechsle al.. 2020
Rematas & Ferrari 2020). Deep Reflectance Volumes (Bi et al. (020b) reconstructs a voxelized representation of scene and predict per-voxel BRDFs, but the fixed resolution of voxel grids limits the quality in the rendered images. Similarly, Neural Reflectance Fields (Bi et al.) (020a)) predicts the parameters of a BRDF model, but demonstrate higher fidelity rendering by learning a continuous scene representation. However, Neural Reflectance Fields focuses on relighting single objects, and requires manual specification of the BRDF model. Parametric BRDF models are unable handle complex scattering functions, including real-worke scattering phenomena that are difficult to model. In contrast, our method is capable of learning all scattering functions, and can render multiple objects in dynamic scenes.
3 PRELIMINARIES
3.1 VOLUME RENDERING
To render an image of a scene with arbitrary camera parameters, camera rays sent into the scene, through each pixel on the image plane. The expected color of each pixel is computed as the radiance along each camera ray.
Volume rendering is an approach for computing the radiance traveling along rays traced in a volume. Let rtt) 3o + wot be point along a ray with origin To and direction wa where IR. is ID location along the ray, and the in wo denotes "outgoing" direction. For our purposes, we assume non-emissive and non- absorptivee volumes. From Novák et al. (2018), the volume rendering equation to compute the radiance L( wo) of the ray is defined as:
L(zo, wo) = 1.' r(t)a(r(t)) w.) dt, where T(t) = exp (-/," (((((()))dd))) du (1)
exp
where in and ty are near and far integration bounds, (r((t) denotes the volume density of point r(t), and 7(t) denotes the accumulated transmittance from bn The term (r(r(t),,,, w.) is the light scattered at point rtt) along direction Wo. defined as the integral over all incoming light directions:
L.(w, = L(x, wr), wi, w.) dwi,
where S is a unit sphere and f, is a phase function that evaluates the fraction of light incoming from direction wi at point a that scatters out in direction wa. In NeRF Mildenhall al. (2020) assume
3

---

Under review as a conference paper at ICLR 2022
We consider more general form of the volume rendering equation that explicitly models light paths within and between objects. This is important for dynamic scenes, where lighting and objects can move with respect to one another.
fixed illumination and do not consider any form of Equation 2
3.2 RAY MARCHING
The continuous integrals in Equation [ can be estimated with quadrature (Kniss al., 2003] Max
1995), as done in NeRF (Mildenhall et al. 2020). For each ray, stratified sampling is used to obtain N samples [1,) along the ray, where 1, E [n, lr]. The rendering equation is approximated by:
L(wo, = Σ Tea;; Lo (2;;,
where
La Wo)
1
Σ LEC
La wi)p
[C]
where Ti = [I ag) and a; -- To compute the average over incoming light
paths Li, we discretize over the domain S in Equation 2 by sampling a set of incoming light paths
L (41,..., 1,), where p = (a wr,wo) E [0, Il. the fraction of light incoming from light path that is scattered in direction Wo-
3.3 NEURAL RADIANCE FIELDS
NeRF represents continuous scene volumetric radiance field, approximated with multilayer perceptron Fe. The model Fe takes spatial location æ (at, y, 2) and viewing direction d = (ω. 0) as input, and outputs the density and color (r, g. b). where g. E [0, 1]. Frequency-basee positional encoding (Rahaman et al. 2019: Vaswani et al. 2017) is applied to the inputs to better capture high-frequency variation in appearance and geometry.
hierarchical volume sampling procedure (Mildenhall al., 2020 Levoy 1990) then employed to more efficiently allocate samples along each ray. This technique biases sample allocation to favor the visible parts the scene that contribute the most to the final render, avoiding occluded or free space in the scene. NeRF simultaneously optimizes two radiance fields, where the sample weights Tú a; from a coarse model are used to bias samples for a fine model. The L2 loss used to optimize both models: EreRl(C.(r) - C(r) + (rč)(r) - C(r)lB, where R is the set of all camera rays, Cr(r) and (r) denote the radiance along ray r predicted by the coarse and fine models respectively, and C(r) is the ground truth pixel color for r.
METHOD
4.1
OBJECT-CENTRES NEURAL SCATTERING FUNCTION
We represent each object as 7D object-centric neural scattering function (OSF), depicted in Fig- ure 3al For each object, we learn an implicit function Fe: : (z, (t,Wi:W)) - (a, p) that receives a 3D point in the object coordinate frame, the incoming light direction, and the outgoing light direction, and predicts the volumetric density as well as fraction of incoming light that is scattered in the out- going direction. @ are learned weights that parameterize the neural network, = (x,, y, 2) denotes the spatial location, Wi = (or, 0:) denotes the incoming light direction, wo = (do,, 0.) denotes the outgoing light direction, denotes the volumetric density, and p (Pr: Pg: Pb) denotes the frac- tion of light arriving at x from direction W/ that is scattered and leaving in direction Wo- The final color of point is the integral of p multiplied by the incoming radiance over all incoming light directions in unit sphere S (Équation] 2). Following NeRF, we similarly apply positional encoding to our inputs (æ, wi, w.) and employ hierarchical sampling procedure recover higher quality appearance and geometry of learned objects.
During training, we assume a single point light source with radiance of 11, This simplifies Lg from Equation 2 to (x, w.) = L(x ( Wi, w,) fp(æ,, wi, Wo). To learn per-object NeRFs independent of object rotation and translation, the inputs to Fe must be in the object's canon- ical coordinate frame. Given a object transformation T. for object o,, we apply T," to (r. Wt, w.) before feeding the inputs the network.
4.2 RENDERING MULTIPLE OSFs
Once we have learned an OSF for each object, we aim at composing the learned objects into scenes. An overview of procedure is visually depicted in Figure
4

---

Under review as a conference paper at ICLR 2022
Canara
Fe
Dieed Barthalhon
||
Pantra
Suurce
Light Source
Camera
W
wo
Dianie Dap
(a) We represent each object as an object-centric neural scattering function (OSF), which models how light entering at point æ on the object, from direc- tion wi where corresponds to light path, undergnes multiple bounces within the object and exits along di- rection wa with some fractional amount of light p. approximate the scattering function with a mul- tilayer perceptron Fe where are Icarned weights that parameterize the neural network. Given single point incoming light direction wi, and an out- going direction Wo. Pe outputs the volume density a of that point, as well as the fraction of light arriving at a from direction W/ that is scattered in direction Wo-
(b) Our procedure for rendering arbitrary scene consisting of multiple objects, light sources, and cam- eras. Given a set of objects, we compute direct illu- mination by shooting rays from cach light source to each object (brown arrows). Shadows are computed by sending shadow rays back each light source (purple arrow). The shadow ray from the desk is oc- cluded by the mug, so the mug casts a shadow on the desk. We send secondary rays between objects to ren- der indirect illumination effects, such between the desk and the kettle (green and blue dashed arrows). Finally, rays are sent back to the camera to render the final image (dark blue arrows).
Figure Using our method (OSFs) to render: (a) single and (b) multiple objects.
(a) Primary ray sampling
(b) Shadow ray sampling
Figure 4: Sampling procedure. (a) Scene with camera, light source, and object bounding boxes. Primary rays are sent from the camera into the scene. Rays that do not intersect with objects are pruned. Of the intersecting rays, we sample points within intersecting regions. (b) Shadow rays from each sample are sent to the light source, and samples within intersecting regions are evaluated.
Let O = io be a set of N objects we wish to render. For simplicity, we first describe the rendering process for each object o,, then explain the process combine results across all objects to render the final scene, Let O E O denote object i with transformation T; E I and bounding box dimensions D, E 3 Further let r be camera ray with origin e E R and direction wo E R which we define with parameters 7 [c, wol RR. Our goal is to compute L(c, wo) as described in Equation 3 We compute the ray-box intersection between the ray and the object to obtain near bound f and far bound I such that r(t,, and r(f,)) each intersect a box plane, as shown in Figure Note that rays that do not intersect with o, are excluded from our computation. We sample M points between t and ty along ray r to obtain a sample X {ar I Mm where X MM3 Given
light source I, we evaluate the object's model Fe, (X', Wi, wo) to obtain alpha values a' E R and phase function values p² E Mx3
It is not always possible for a light ray from light source to reach the object o, Any of the other objects in O' {oz E O | + i} in the scene may occlude the incoming light, casting a shadow on object o, We compute shadows by sending shadow ray r. from each of the M samples in X
5

---

Under review as a conference paper at ICLR 2022
to the light source I. Evaluating the shadow ray enables us to determine the amount of light blocked along the ray by other objects. We define the parameters of the M shadow rays as I R 2Mxx x6
For each object O, E O', we compute ray-box intersections between shadow rays I' and o, 's bound- ing box. This allows us to compute the amount of light traveling towards o that is blocked by Oj. Similar to primary rays, we sample M points along each shadow ray obtain set of points X RMMMM We then evaluate the object model Fe, (X') to obtain alpha values A RM
For each shadow ray Tm- combine samples A across the N objects in O' by sorting cording sample distance to obtain alpha values Am IM)(--), The fraction of unobstructed light traveling along the shadow ray T' is computed as the transmittance:
M(N-1)
II (1- Amn) n=1
Thus, the adjusted incoming radiance from light source when accounting for occlusions is com- (am, puted as (Zm, wi) ( wt).
We follow the scattering equation in Equation 2 and now consider all incoming light directions over the unit sphere S. This accounts for secondary light rays traveling to object o, indirectly from another object o, (indirect illumination). We approximate the integral over the unit sphere S by sampling K directions on the unit sphere uniformly at random. For each direction wk randomly sampled for point we send secondary ray r from æ in direction w/ and evaluate the radiance L(x, wx) traveling along the ray. To compute the radiance of the secondary ray ((x),))) we employ the same technique used to compute the radiance of a primary ray L(c, wo) (described at the beginning of Section 4.2). The incoming radiance L(x, wx) is multiplied with the phase function value p p= fp(#; (z, wky ) to determine the outgoing radiance L(x wo), where is evaluated using Fe.. Note that this is possible due to the recursive nature of our formulation. Only secondary rays are described here (two bounces), but method supports arbitrary number of bounces.
Rendering. We sample and evaluate all objects in O obtain alpha values fa' M and phase function values {p' } for a set of sampled points {X*
) along ray r. We sort the samples across all objects produce final set of P M N samples {mmm m== {am ) m and {pm P
Given light paths L containing both direct and indirect illumination, we render the final radiance of ray with origin Do and direction Wo with the following equation:
P
1
L(xo, Wo)
[
amp'n Im ampP'.. Le(am, wI). where
Tm
II (1 an), 1=1
(5)
LECm=1
and La wi) is the radiance from light path I arriving point Tmn-
Runtime. In total, the cost of rendering a single image with Npisel pixels and Notjeat objects is O(P Note that P is an upper bound on number of samples that need be evaluated. In practice, single ray often only intersects with at most one object in the scene, which means that the proposed rendering procedure is not significantly more expensive than the single object setting. We also note that compared to Bi et al. (2020a) or traditional volumetric path tracing methods, OSF crucially does not require running path tracing within each object to simulate intra-object light bounces. This is because OSF learns the object-level scattering function that directly predicts the effects after all light bounces (reflections) and occlusions (shadows) within an object have occurred. Thus OSF is significantly faster than NRF which relies on simulating intra-object light bounces while querying its learned BRDF model.
In our experiments, rendering single image with single OSF resolution of 256 x 256 takes roughly 3.7 seconds. While the computation cost is high, there are efforts to reduce the rendering speed of NeRF that are orthogonal to this work. For instance, KiloNeRF (Reiser al. 202 can easily adapted to this work by utilizing thousands of tiny MLPs instead of one single large MLP to represent each OSF to obtain 1-2 orders of magnitude speed up.
EXPERIMENTS
Datasets and evaluation metrics. We evaluate our approach on several image datasets:
FURNITURE-SINGLE: 15 objects rendered with random object pose, point light, and viewpoint.

---

Under review as a conference paper at ICLR 2022
FURNITURE- RANDOM: 25 dynamic scenes, each containing random layout of multiple objects,
point light, and viewpoint.
BERNITURE-REALITTTER Scenes containing realistic arrangements of objects rooms.
REAL-NRF: Real-world objects from Bi et al. (2020a), captured in dark room under varying viewing and lighting directions.
REAL-OUTDOOR: Real-world outdoor scenes from Mildenhall al. (2020).
For FURNITURE datasets, der images at 256 x 256 resolution for different object arrangements, camera views, and lighting configurations. We report PSNR, SSIM (Wang et al. 2003), and LPIPS (Zhang al., 2018 metries. Baselines and ablations.
Blender's Cycles path tracer (Blender Foundation, 1994) to
We compare our method to the following baselines:
1. o-NeRF: A variant of the NeRF model, but with NeRF trained per object. When o-NeRFs are composed into scenes, they are rendered separately.
o-NeRF + S: An extension o-NeRF with inter-object shadows; reduces the light arriving at each o-NeRF by the cumulative opacity of shadowing objects along the ray from the light ($4.2)
These baselines represent what could be achieved by combining separately trained NeRFs into scene. Of course, since o-NeRFs produce radiance fields (not scattering fields), we do not expect them to perform well in novel lighting environments or object placements.
5.1 NOVEL LIGHTING
In the first experiment, investigate how OSF method handles novel lighting conditions.
We train one model per object in FURNITURE- SINGLE. For each object model, we train on 400 images with randomized viewpoint and lighting, and test on 20 images of novel view-
Ground Truth
point and lighting. As can be seen in Fig- 3 our method produces more accurate ap- pearance of the objects in comparison to NeRF when tested on novel illumination con- ditions. In particular, o-NeRF fails pre- dict self-shadows for the couch and chair cor- rectly. Additionally, o-NeRF fails to disen- tangle viewpoint versus pearance, producing incorrect shadows for the couch and chair, and fails to capture the specu- lar details of the ottoman. Quantitative results can be found in Table
OSF
ap-
Couch
Chair
Table
Otisman
Figure 5: Novel lighting results.
5.2 SCENE COMPOSITION
In a second experiment, we conduct scene composition task on FURNITURE- -ANDRR,- where multiple object models are combined into scenes in random pose, lighting, and viewpoint configu- rations. For this task, use the same object models trained in Section 5.1 Results shown in Table I and Figure 6 While not shown the main text, results for CURNITURE-REALISTICC can be found in Appendix
Table 1: Quantitative results for novel lighting (FURNITURE-SII) SINGLE) and scene composition SENTTURE-RANDME Rows denote different methods: our full model (OSF), a variant NeRF where one NeRF is trained per object (o-NeRF), and o-NeRF with shadows (o-NeRF + S).
Dataset
FURNITURE-SINGLA
FURNITURE-RANE
Method
PSNRI
SSIMI
LPIPS
PSNR1
SSIM
LPIPS
o-NeRF
33.22
0.980
0.021
12.17
0.690
0.280
O-NERF +
14.70
0.697
0.267
OSF (Our Method)
44.07
0.998
0.002
19.02
0.793
0.135

---

Under review as a conference paper at ICLR 2022
OSF (ours)
O-NeRF + S
O-NeRF
Ground Truth
Figure 6:
Scene composition results on TURNITURE-RANDON The models OSF, o-NeRF, and o-NeRF + S are explained in $ Compared to o-NeRF, our model (OSF) is able disentangle and view-dependent appearance and can render shadows.
These results suggest that OSF outperforms all baselines and ablations, both quantitatively and qual- itatively. As the previous experiment (Section 5.1). we find that OSF reproduces object appear- ances and self-shadows more accurately than the baselines. The difference is especially apparent in the couches in scenes (a) and (b), where the couches predicted by o-NeRF are extremely dark. This is due to the fact that o-NeRF is unable to disentangle riew-dependence appearance from light- dependent appearance, and simply interpolates the radiance field learned another different lighting configuration. Please note that OSF is able model inter-object light transport effects by rendering shadows cast by one object onto another and on the ground plane. Plus, it is able to render indirect illumination of one object reflecting light onto another. For example, light reflected from the left wall causes the left of the couch and table in scenes (a) and (b) to be brighter. Neither of these lighting effects are present in the o-NeRF results.
5.3
SCENES
In this section we evaluate our method on real world objects and scenes from the REAL--NRF and REAL-OUTDOOR datasets. experiments, we train one OSF for each object in REAL-NRE and each scene in REAL- OUTDOOR.
Goond That
OSF (D4m))
NRF (R 61
For these
Figure 7 shows a comparison between ground truth, our method (OSF), and Neural Re- flectance Fields (NRF) (Bi et al. 2020a). We show that OSF recovers stronger, more accurate specular highlights compared to NRF OSF also produces more detailed appearances (see pony logo). This comparison demonstrates the main advantage of OSF: the ability handle complex scattering functions.
Comparison of OSF (ours) to Neu- ral Reflectance Fields (NRF) (Bi al. 2020a OSF produces stronger, more accurate specular highlights the legs (see zoomed view) and recovers more detailed appearances (see pony logo).
Figure 7:
For scene composition, the OSFs trained on each object are composed with a synthetic floor OSF in Figure row (a). Our method is able to compute accurate shadows, such as the shadow cast by the pony onto the two other objects in the scene. The indirect reflections from the floor allow the shadowed objects to be slightly visible as shown in the "OSF" panel.
Figure rows (b) and (c) show results inserting REAL-NRE objects into real outdoor scenes OUTDOOOU Shadows and reflections are rendered with randomized lighting directions to approximate the environment lighting. Our method accurately renders occlusions between the in- serted objects and the vase in Figure g row (c). Due to the compositional nature of OSFs, we are able to insert the learned pinecone from Figure (b) into (c).
In Figure X each column shows ablated versions of OSF to study the impact of computing shadows and indirect illumination with our path tracing algorithm. "No Shadows, No Indirect" represents a version of our model containing only direct illumination (without modeling inter-object lighting effects). We additionally show "No Indirect" and "Indirect Only" variants of our model which
8

---

Under review as a conference paper at ICLR 2022
Our Methooi Object-Centric Neural Scattaring Functions (OSF)
NeRF (21)
Shadows, No Inchrect
Indi
Full Model
(b)
(c)
Figure 8: Real-world results. NRF (Bi al. 2020a) and NeRF (Mildenhall al. 2020 learn on individual static scenes or objects. In contrast, we compose real-world objects and scenes using OSFs. The objects are composed with a (a) synthetic floor and (b, c) real outdoor scenes from REAL- OUTDOUR. Columns show different ablated versions of our model: "No Shadows, No Indirect" which considers only direct illumination; "No Indirect" which includes both direct illumination and shadows; "Indirect Only" which considers only indirect illumination. Our OSFs show the most realistic renderings, with accurate shadows (e.g., pony shadowing the two other objects (row a) and indirect illumination (i.e., the ground and environment illuminating the objects).
represent computing shadows and indirect illumination, respectively. As illustrated by Figure g our full model containing both shadows and indirect illumination effects is the most realistic. Additional results on real-world scenes, including complex shadows, can be found in Appendix
6 DISCUSSION
We have proposed Object-Centric Neural Scattering Functions (OSFs), method that enables com- posing objects captured only from photographs into photorealistic renderings of dynamic scenes. We demonstrated that decomposing scene into implicit object functions that are view- and light- dependent enables reusabiliy of objects across scenes where objects, camera, and lighting can change. We presented method for integrating our learned implicit functions with volumetric path tracing, and showed inter-object light transport effects such as shadow and indirect illumination for real-world objects where computer graphics model is available. We believe our work is step towards a graphics pipeline where real-world scenes are modeled by a composition of implicit func- tions combine the flexibility of object-centric neural modeling with the photorealism of graphics rendering algorithms.
There few main limitations to OSF. First, the computational complexity of method is high, but there are several works tackling the orthogonal issue of improving NeRF efficiency (as discussed in Section 4.2) that easily be applied to OSFs. Second, while learning intra-object light transport means that intra-object path tracing is not needed, this formulation assumes that at test time, there are no occluders or light sources that intrude the object's convex hull (Sloan al.) 2002) (e.g.. person sitting in chair). However, OSFs can still be rendered even if their bounding boxes are intersecting, as long as this assumption is not violated. Finally, acquiring datasets of real world objects with varying point light sources and viewpoints is challenging, but we hope that in the future such acquisition real world datasets will become easier to capture and more widely available.
REPRODUCIBILITY STATEMENT
We describe our method (Section 4 and experimental setup (Section 5) in detail to maximize repro- ducibility. We will release our code upon publication to facilitate future research.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. Neural point-based graphics. European Conference on Computer Vision, 2020.
Bruno Arnaldi, Xavier Pueyo, and Josep Vilaplana. On the division of environments by virtual walls for radiosity computation. In Photoreulistic Rendering in Computer Graphics, pp. 198-205. Springer, 1994.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint arXiv:2008. 2000024008.,3824, 2020a.
Sai Bi. Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images. European Conference on Computer Vision, 2020b.
Blender Foundation. Blender 3d modelling and rendering package. http:///we /ttp:///www..//// http://wwwbbbleede.e blender. org 1994.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236. 2016.
Oliver Cossairt, Shree Nayar, and Ravi Ramamoorthi. Light field transfer: global illumination between real and synthetic objects. ACM Transactions on Graphics, 27(3):1-6, 2008
Frank Dellaert and Lin Yen-Chen. Neural volume rendering: Nerf and beyond. arXiv preprint 1xXiv:2001.05244. 2020.
Philip Dutre, Philippe Bekaert, and Kavita Bala. Advanced global illumination. CRC Press, 2018.
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. Deepstereo: Learning predict new views from the world's imagery. In Conference on Computer Vision and Pattern Recognition, pp- 5515-5524, 2016.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Peter Hedman, Julien Philip. True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Bros- tow. Deep blending for free-ite image-based rendering, ACM Transactions on Graphics, 37(6):1-15, 2018.
James T Kajiya. The rendering equation. In Conference on Computer Graphics and Interactive Techniques, pp. 143-150, 1986.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412::1412.6980 2014.
Joe Kniss, Simon Premoze, Charles Hansen, Peter Shirley, and Allen McPherson. A model for volume lighting and modeling. JEEE Transactions on Visualization and Computer Graphics, (2):150-162, 2003.
Mare Levoy. Efficient ray tracing of volume data. ACM Transactions on Graphics 933:2455-21, 1990.
Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems, 2020.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics, 38(4):1-14 2019.
Ricardo Martin Martin-Brualla, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, et al. Lookingood: Enhancing performance capture with real-time neural re-rendering. ACM Transactions on Graph- ics, 2018.
10

---

Under review as a conference paper at ICLR 2022
Nelson Max. Optical models for direct volume rendering. IEEE Transactions Visualization and Computer Graphics, 1())99-108, 1995.
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Bruall Brualla. Neural rerendering in the wild. In Conference on Computer Vision and Puttern Recognition, pp. 6878-6887, 2019.
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics, 1844):1-14,200
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020.
Gavin Miller. Efficient algorithms for local and global accessibility shading. In Conference on Computer Graphics and Interactive Techniques, pp. 319-326, 1994.
Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsupervised learning of 3d representations from natural images. In International Conference Computer Vision, pp. 7588-7597, 2019.
Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra. Blockgan: Learning 3d object-aware scene representations from unlabelled images. Advances in Neural Information Processing Systems, 2020.
Thu Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep convo- lutional network for differentiable rendering from 3d shapes. In Advances in Neural Information Processing Systems, pp. 1991-7901, 2018.
Jan Novák, Ilyyan Georgiev, Johannes Haníka, and Wojciech Jarosz. Monte carlo methods for volu- metric light transport simulation. In Computer Graphics Forum, volume 37, pp. 551-576. Wiley Online Library, 2018
Michael Oechsle, Michael Niemeyer, Lars Mescheder, Thilo Strauss, and Andreas Geiger. Learning implicit surface light fields. International Conference on 3D Vision, 2020.
Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A Efros, and George Drettakis. Multi-view relighting using geometry- network. ACM Transactions Graphics, 38(4):1-14, 2019.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning, pp. 5301-5310. PMLR, 2019.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Con- ference Computer Vision (ICCV), pp. 14335-14345, October 2021.
Konstantinos Rematas and Vittorio Ferrari. Neural voxel renderer: Learning accurate and control- lable rendering tool. In Conference on Computer Vision and Pattern Recognition, p 5417-5427, 2020.
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In International Conference on Computer Vision, 2019.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. Conference on Computer Vision and Pattern Recognition, pp. 2437-2446, 2019a
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Con- tinuous ad-structure-awand neural scene representations. In Advances in Neural Information Pro- cessing Systems, pp. 1121-1132, 2019b.
11

---

Under review as a conference paper at ICLR 2022
Peter-Pike Sloan, Jan Kautz, and John Snyder. Precomputed radiance transfer for rendering in dynamic, low-frequency lighting environments. Conference on Computer Graphics and Interactive Techniques, pp. 527-536, 2002.
Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng. and Noah Snavely, Pushing the boundaries of view extrapolation with multiplane images. In Conference on Computer Vision and Pattern Recognition, pp. 175-184, 2019.
Tiancheng Sun, Jonathan T Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fyffe, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions on Graphics, 38(4):79-1, 2019.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics, 38(4):1-12 2019.
Bill Triges, Philip F McLauchlan, Richard Hartley, and Andrew W Fitzgibbon. Bundle ad- justment-a modern synthesis. In International Workshop on Vision Algorithms, pp. 298-372. Springer, 1999.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In Asilomar Conference Signals, Systems & Computers, 2003, volume 2, pp- 1398-1402. IEEE, 2003.
Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martin, and Silvio Savarese. Interactive gibson benchmark: benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):
713-720, 2020.
Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image- relighting from optimal sparse samples. ACM Transactions on Graphics, 37(4):1-13, 2018
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Conference on Computer Vision and Pattern recognition, pp. 586 006-599,222 2018.
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, et al. Neural light transport for relighting and view synthesis. ACM Transactions on Graphics, 2021.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrait re- lighting. In International Conference on Computer Vision, pp. 7194-7202, 2019.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Transactions on Graphics, 37 2018.
12

---

Under review as a conference paper at ICLR 2022
A REAL- WORLD SCENE COMPOSITION
Different scene configurations of composed objects from REAL-NRE are shown Figure 9 We show the effect of moving the light, camera, or objects. Notice how the the appearance and shadows of the objects are updated across different scene configurations. Also notice that even when parts of the palm tree object and the cartoon object are cast under the pony's shadow, they do not appear completely dark due to the indirect illumination from the floor.
Analyzing the effect of different numbers of indirect (sccondary) rays per primary sample, Figure I0 shows the result. As can seen from the figure, the noisiness of the indirect illumination render decreases the number of samples increase. Results in this paper contain between and five randomly sampled secondary ray for each primary ray sample.
Move Light
Move Camera
Move Objects
Figure 9: Composing real-world objects from REAL-NRE using our OSF method. We demonstrate the effect of moving the light, camera, objects. Note how the appearance and shadows of the ob- jects are updated across different scene configurations. Also notice that even when parts of the palm tree object and the cartoon object cast under the pony's shadow, they do not appear completely dark due to the indirect illumination from the floor.
=
N
Figure 10: Visualizing the effect of different numbers of indirect (secondary) rays (N) per primary sample for our OSF model (the brightness of these images has been increased only for visualization purposes). Note that the noisiness the render decreases as N increases. We find that we are able to achieve relatively non-noisy results with approximately five samples.
13

---

Under review as a conference paper at ICLR 2022
Single-object renderings from REAL-NRF shown in Figure II The objects captured in dark room with one- aae-light-at-a-tine setup. After training OSF on each object in this dataset, we are able to render the objects from novel viewpoints and lighting directions.
Original
SURF
SNOP
Move Light
SURF SNOP
Move Camera
Figure 11: Learned OSFs on objects from REAL-NRF. The objects were captured in a dark room with a me-lighh-at----imee setup. After training OSF on each object in this dataset, we able to render the objects from novel viewpoints and lighting directions.
14

---

Under review as a conference paper at ICLR 2022
B
ABLATION EXPERIMENTS
Direct Only
Indirect Only
Direct + Shadows
OSF (Ours)
Figure 12: Ablation results our OSF model.
Figure 12 shows ablation results on FURNITURE-REALISTE We evaluate different variants of our model: "Direct Only" which considers only direct illumination; "Indirect Only" which considers only indirect illumination; "Direct Shadows" which includes both direct illumination and shad- ows. Our full model (OSF) shows the most realistic rendering, with accurate shadows and indirect illumination effects such the left side of the couches and tables appearing brighter due indirect lighting from the left wall. Note that the white area on the right of the images represent rays with zero density that composited onto white background (and therefore do not contribute indirect illumination to the scene).
OSF (ours)
O-NeRF
O-NeRF
Ground Truth
(a)
(b)
Figure 13: Comparisons on scene composition FURNITURE-REALISTTE The models OSF, o- NeRF, and o-NeRF + S are explained in $5) Compared to o-NeRF, our model (OSF) is able to disentangle lighting-dependent appearance from view-dependent appearance for individual objects, and is able to render shadows cast by objects onto the ground correctly.
15

---

Under review as a conference paper at ICLR 2022
C COMPLEX ILLUMINATION
In this experiment, we investigate how scenes composed of OSF objects can be rendered with com- plex illumination from an environment map.
Specifically, we apply the combination of a point light source and the environment map shown in the top-left corner of Figure 14 to light one of our scenes in FURNITURE- REALISTIC... This simulates the appearance of the scene as if the scene were inserted into complex lighting environment, which stresses the benefits of the OSF path tracing framework.
For each OSF sample point, project the equirectangular coordinates of the environment map into spherical coordinates, sample 20 direc- tions on the unit sphere uniformly at random, evaluate the OSF function for each incoming di- rection, and integrate them outgoing radiance us- ing [q Please note that green- blue tint is slightly apparent in the scene rendering, due to the contribution of green and blue lighting from the environment map.
Figure 14: Complex illumination results.
16

---

Under review as a conference paper at ICLR 2022
D IMPLEMENTATION DETAILS
A flowchart of our method is shown in Figure
15
We approximate our model Fe with multilayer perception (MLP) with rectified linear activations. The predicted density a is view-invariant, while the scattering function value is dependent on the incoming and outgoing light directions. We use an eight-layer MLP with 256 channels to predict and four-layer MLP with 128 channels to predict p. For positional encoding, we use W 10 to encode the position a W to encode the incoming and outgoing directions (Wi, Wa), where W is the highest frequency level. To avoid p from saturating in training, adopt scaled sigmoid (Brock et al. 2016) defined as (p) = 8(S(p) 0.5) + 0.5 with å = 1.2. We use a batch size of 4,096 rays.
woal_object
rglo,
rays
sont by_t
integrate
RGB,
eval_object
rgb,
intersect idks
intersect
sample
OSF w, W,
rgb, a
scatter
t rgb a
light direction
secondary
render
RGB, trans
evail_ secondary
eval_abject
Figure 15: Flowchart of our method. See for for more details.
For synthetic datasets, we sample N: = 64 coarse samples and N, = 128 fine samples per ray. For
real world datasets, we sample N₂ 64 coarse samples and N, 64 fine samples per ray. We
the Adam optimizer (Kingma & Ba 2014) with a learning rate of 0.001, 31 0.9, .2 = 0.999, and 10-7.
17


Reviews:
[{"review_id": "vDz-N-_in3", "reviewer": "Reviewer_TPuY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This paper introduces 7D object-centric neural scattering functions (OSFs) using a lighting- and view-dependent network. The additional light direction input makes this representation support novel lighting rendering.\n+ Given multiple optimized OSFs, this paper proposes to render a compositional scene using volumetric path tracing, light transport effects like occlusions, specularities, shadows, and indirect illumination are considered.\n+ Some of the visual results in the paper look visually pleasing.\n\nWeaknesses:\n- The proposed OSFs sound interesting. However, it requires hundreds of images captured under different point light sources and viewpoints for optimizing the network, which is difficult to achieve in the real world.\n- The proposed OSFs representation is not well validated. The baseline (o-NeRF) compared in the paper is not very convincing. Comparison with existing appearance modeling methods (e.g., Neural Reflectance Fields, NeRV) on single object rendering under novel views and lightings should be provided. \n- Volumetric path tracing for multiple objects rendering is not new.\n- The rendering time complexity seems to be very high. However, no discussion is provided in the paper.\n", "summary_of_the_paper": "This paper proposes a NeRF based method for composing photo-realistic scenes from captured images. The proposed method learns object-centric neural scattering functions (OSFs) to implicitly model per-object light transport using a lighting- and view-dependent neural network. Multiple objects can be rendered with volumetric path tracing. The proposed method has been evaluated on both synthetic and real-world datasets.", "summary_of_the_review": "This paper introduces object-centric neural scattering functions for modeling light transport of an object, allowing novel lighting rendering and scene composition. However, the data requires to optimize the OSFs is restricted and the rendering time complexity is high. Also, the experiment is not very convincing due to the lack of comparisons. I would like to give a negative rating at the current stage.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hfbUkqiPRyf", "reviewer": "Reviewer_KXhU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The idea of representing object as an light transportation function is a novel contribution, prior works are operating on predifined BRDF function with limited capability.\n2. The paper also proposed an approach to render with implicit function with secondary lighting effect and shadows while compositing the scene.\n3. The paper is well written and easy to follow.\n\nWeakness:\n\n1. In the paper, to train the OSFs, the authors assumes the object is captured with the point light with radiance of (1, 1, 1), is the position of this point light is known as well? or the pipeline can also backwards to estimate the position of point light? This assumption (even just the point light with fixed radiance and unknown position)  is hard for real object, as the real illumination can be arbitrary complex. \n\n2. For OSFs, The author argues \"our method is capable of learning all scattering functions, and can render multiple objects in dynamic scenes.\" I agree with this argument, as using a MLP to represent light scattering function have a higher capacity than pre-defined BRDF formulation. However, it is missing in the experiment to demonstrate this, as there is no comparison on why using MLP to parameterize lighting scattering function is better than others (e.g. Nerual reflectance field). There are also other works to estimate the light scattering function: Nerd [a], PhySG [b], which uses spherical Gaussian to represent the light, and Disney BRDF. I suggest the author to compare with Neural Reflectance Fields (Bi et al., 2020a) or PhySG on experiment Sec. 5.1 to demonstrate this point.\n\n3. Computation cost of the whole process is huge. In particular, for each pixel, the paper sample M points, and for each point, the paper sample K directions in a sphere and for each direction, the the author further sample M points to estimate the intensity for each object, this gives M*M*K*(N-1)*(N_pixel) in total, could the author provide the computation cost comparison for it? (e.g. memory and rendering time)\n\n4. minor points: in Fig. 7, it's very hard to see the green-blue tint as mentioned by the author, it would be better if the author can show a comparisons with point light to see the effect of rendering with environment map. \n\n5.  The proposed the rendering technique for secondary lighting effect (shadow) is great, however, in principle, the proposed approach should also be able to backwards through this? it would be better if the paper can show some results of using it to run backward optimization. \n\n[a] NeRD: Neural Reflectance Decomposition from Image Collections\nMark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, Hendrik P. A. Lensch\n\n[b]PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting\nCVPR 2021\nKai Zhang*  Fujun Luan*  Qianqian Wang  Kavita Bala  Noah Snavely", "summary_of_the_paper": "The paper proposed a method to composite objects parameterized using implicit functions into realistic scenes. The idea is to first capture the representation for each object separately, and each object is represented using neural scattering function, which predicts the outgoing lighting transport conditioned on the input lighting direction, viewing direction and 3d location. The objects are composed into the full scene by doing volume rendering along the ray, and the radiance of each sampled point in the ray is calculated via integrating on a sphere to obtain the lighting radiance (including secondary (indirect) light effect, the shadow effect is obtained from iterating the ray from light position to the  sampled point). The results on both synthetic scene and real scene shows improvement over baselines (o-Nerf, o-Nerf S) that didn't consider the lighting transportation. ", "summary_of_the_review": "In summary, the proposed method using lighting transportation function is a novel approach to represent an object and extends previous works on learning BRDF of an object, considering the missing comparisons and the limitations, I vote for the board line initially, and I'm happy to listen to the authors and other reviewers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "68fiyJrrY9A", "reviewer": "Reviewer_6eq6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n\nThe method is well-designed and models even indirect illumination. This is a significant and important contribution. It enables a decent level of control over neural scene representations, namely rigid transforms of objects, while properly/explicitly modelling the resulting appearance changes via path tracing. While extending neural radiance fields to object-level rigid transformations is rather straightforward, accounting for the illumination is not.\n\nThere are some results on real scenes.\n\n------\n\nCons:\n\nColored illumination:\n- Direct: Shadows rays (Equation 4) assume that all color channels are absorbed/occluded equally? So colored glass for example would cast a grey shadow and not a colored shadow? Were there any experiments to account for such effect, e.g. per-color-channel densities?\n- Indirect: Indirect illumination should be colored if I understand the method correctly? I cannot see any result though that exhibits colored indirect illumination, not even for the yellow sign in Fig. 12.\n- In both cases, a result showing that colored illumination is possible or an explicit statement that color is not taken into account is necessary I believe (e.g. in limitations or as a stated assumption when introducing the method).\n\nRuntime:\n- An evaluation of the runtime (training time and inference/rendering time) is missing. It could be measured in dependence on the number of objects (1,2,3,5 for example) and number of bounces (1,2,3 for example), M could be kept fixed, rays could be sampled randomly instead of rendering full images if full images are too expensive. I believe that a comparison to NeRF in that regard is also necessary to give an understanding of how much impact ray tracing has on runtime (number of objects=1 and bounces=1 should cover this case, I believe?). Such an experiment would also provide experimental support for the claim at the end of page 6 regarding runtime.\n- Related to that, an experiment (qualitative and quantitative) where the number of randomized light directions for indirect lighting (and also environment map sampling; number of directions: few, normal, many), number of points on each secondary ray (M; few, normal, many) and the recursion depth/number of bounces (few, normal, many) varies would be ideal. This could also be done with smaller image patches (say 50x50 pixels) if full images are too slow. That would give an idea on what the trade-offs are between fidelity, runtime, and noise. If compute is still too high, each of the three axes could be explored independently (with the other two axes set to normal) and then also jointly (all three axes set to few, all three set to normal, all three set to many).\n\nConsistency:\nA supplemental video would have been useful, e.g. for switching between different lighting conditions. It could also have shown whether the scene transforms smoothly/nicely/consistently when moving the light around or when moving the camera. Still images would be insufficient to demonstrate this.\n\nLimitations:\nLimitations are missing. Could mention again that objects' bounding boxes are not allowed to intersect. Runtime and only grey-scale/uncolored direct shadows are also likely limitations.\n\nExperiments:\nExperiments are only on a few scenes. \n\nReproducibility:\nThere is no mention of a code release.\n\n------\n\nNot relevant for the decision:\n\nMinor questions:\n\n- Fig. 12: Why are the floor and walls black for the indirect-only setting? Are they not just treated like objects? Also, is the visualization of the indirect-only result exaggerated or why is the side of the white chair visibly white in indirect-only but not in OSF?\n- The training settings are not sufficiently specified, the new parameters are missing: number of bounces and number of points sampled on secondary and shadow rays. The number of training iterations per object is also missing. The resulting training time both on synthetic and real-world objects is also not mentioned.\n- first line page 3: \"especially for objects with intersecting bounding volumes.\" -- This method shares this weakness, right?\n- Sec. 4.2: just for clarity, is each object's bounding box (for ray intersection) computation axis-aligned with the object coordinate system or the world/scene coordinate system?\n- Is there anything that constrains (in a soft or hard manner) the outgoing fractions to sum up/integrate to 1 or at most 1 for a given incoming light direction?\n- Fig. 10: What exactly is N in this figure? N is used in the main text to refer to the number of objects and to the number of point samples along a ray, neither of which seems like the right parameter here.\n\nMinor suggestions for improvements:\n\n- Fig. 7: I currently cannot see much in this figure, a comparison to a white/grey environment map would make it easier to tell that there is an effect.\n- I'm not a fan of the equation two lines after Eq. 4. I understand what it's trying to say but I believe this needs to be changed to be mathematically correct, unless that makes a bunch of other equations messy. Also, why is it L_l instead of just L? That notation should be introduced beforehand.\n- Fig. 8: Switching out columns 2 and 3 would make the difficult comparison between No Indirect and Full Model easier.\n- There's a typo at the end of page 2: from from", "summary_of_the_paper": "The paper proposes a decomposition of a 3D scene into object-level neural radiance fields. This allows to apply rigid transforms to each object independently and to rearrange the scene. Crucially, light transport is modelled, which turns the radiance fields into scattering functions. Direct lighting and shadowing, as well as indirect illumination from several light bounces are taken into account, such that the illumination of the modified scene looks correct. As long as the scattering fields are trained on a sufficient number of light positions, the lights can also be moved around at test time.", "summary_of_the_review": "The paper proposes a well-designed method and is promising. However, experiments are not yet quite sufficient for acceptance. That's why I will go with borderline reject for now. \n\nExperiments on more real scenes, for example from the same dataset as currently, should be added.\n\nFurthermore, the experiments suggested under \"Runtime\" above should be added, though possibly in a reduced form since the method might be too slow to feasibly obtain these numbers. In that case, I'd prefer an experimental setup wherer the number of pixels is reduced (e.g. randomly picking 1000 rays from a full image, and doing that for, say, 100 images, ideally across different scenes in terms of objects and object positions) rather than fewer settings tried. \n\nDemonstrating plausible consistency while smoothly changing light position/object positions also strikes me as beneficial but I don't believe that acceptance should hinge on it. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "0iUh1Na18tn", "reviewer": "Reviewer_4NH2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion, the strengths of the paper include:\n\n1. It can be beneficial to represent objects as neural scattering functions rather than neural reflectance fields. As pointed out in the paper, this may allow the modeling of more complex materials. Besides, it may have the potential to support fast rendering of in-object indirect illumination. However, I feel both benefits are not clearly demonstrated in the paper. \n2. The proposed method demonstrates clear improvement with respect to classical baseline Nerf. \n3. The rendering pipeline considers relatively complex light transport such as soft shadows and indirect reflection, which can be considered as an improvement compared to many prior works. \n\nThere are probably some questions needed to be classified, which are listed below:\n1. The benefit of the representation may not be fully demonstrated. \nThe method that is most closely related to the proposed method should be neural reflectance fields by Bi et al. The major benefit of the proposed method may include the potential to handle more complex materials and the ability to fast render in object indirect illumination. However, both benefits are not clearly demonstrated in the paper, which makes the arguments weak.  \n\n2. The rendering equation may be biased.\nI am a little concerned about how shadows are rendered in this paper. According to equation 4, it simply computes the product of the alpha values of all samples. However, here the samples are not uniformly created for every ray or every object because authors assign the same number of samples for every object, which means smaller object will have higher sampling rate . This may lead to bias when rendering shadows, especially soft shadows. I feel this is an issue that should probably be corrected. \n\n3. The novelty of the paper may be limited. \nThe three contributions mentioned in the introduction can probably be summarized into 2 major contributions. One is the novel representation. However, there is no comparison with the prior representation (neural reflectance fields) to demonstrate that the new representation is better. Therefore this contribution may not be fully supported. The other is the rendering algorithm. However, the proposed algorithm seems to be a simplified version of standard volume path tracing, without importance sampling of the density field and the light sources.  Besides, prior work (neural reflectance fields) has already demonstrated that their representation can be rendered by a standard renderer using volume path tracing (See Fig 9 in neural reflectance fields). Therefore, the second contribution may not be considered as very novel and significant either. \n\n4. Necessary comparisons may be missing. \nBesides neural reflectance fields, authors may consider comparisons with other neural implicit representations that support relighting, including PhySG by Zhang et al. NerD by Boss et al., and NeRFactor by Zhang et al. In addition, it will be better to compare with traditional representation using mesh and BRDF. The state-of-the-art can be Luan et al. EGSR 2021. \n\nMinor issues:\n1. Notations can be cleaned. \nSeveral notations may never be used. That may include $\\gamma$ in the second paragraph of section 4.2 and $\\Gamma$ in the third paragraph of section 4.2. Meanwhile, $\\mathbf{r}(t)$ in the second paragraph of section 4.2 comes out without being explained first. \n\n2. Citations and related works can probably be improved. Since this paper focuses on scene compositing, it may be better to have one more paragraph in related work discussing recent scene compositing methods. That includes neural implicit methods like GIRAFFE by Niemeyer et al. and more traditional methods like Lighthouse by Srinivasan et al. and Inverse rendering by Li et al. \n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper proposes a new neural implicit representation called object-centric scattering function for scene compositing application. The major extension is to add the lighting direction as an input so that the new representation can be used for relighting. To train this representation, the authors propose minimizing the rendering loss of images rendered from different views under different point lighting. Given this representation, the authors propose a standard volume path tracing framework to render different objects and scene structures together, with indirect illumination and shadow being modeled. Experiments show that compared with nerf, the proposed method achieves better accuracies in object compositing and can handle changes of illumination. ", "summary_of_the_review": "Given the above issues, my current review leans towards negative. My major concern is the lacking of novelties and the technique correctness of some design choices. I will appreciate it if authors can address my concerns in the rebuttal and I will change my reviews accordingly. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommend borderline rejection based on limited novelty, missing comparisons with other methods, and runtime inefficiency. The authors’ response helped clarify other questions but did not eliminate the main concerns about the paper. The AC agrees with the reviewers that, in its current form, the paper does not pass the acceptance bar of ICLR. The reviews have detailed comments and suggestions that should help the authors to improve the work for another conference.
Decision: Reject