OBJECT-CENTRIC NEURAL SCENE RENDERING
Year: 2021

Abstract:
We present a method for composing photorealistic scenes from captured images of objects. Our work builds upon neural radiance fields (NeRFs), which implicitly model the volumetric density and directionally-emitted radiance of a scene from a collection of images. While NeRFs synthesize realistic pictures, they only model static scenes and are closely tied to specific imaging conditions. This property makes NeRFs hard to generalize to new scenarios, including new lighting or new arrangements of objects. Instead of learning a scene radiance field as a NeRF does, we propose to learn object-centric neural scattering functions (OSFs), a representation that models per-object light transport implicitly using a lightingand view-dependent neural network. This enables rendering scenes even when objects or lights move, without retraining. Combined with a volumetric path tracing procedure, our framework is capable of rendering light transport effects including occlusions, specularities, shadows, and indirect illumination, both within individual objects and between different objects. We evaluate OSFs on synthetic and real world datasets, and on generalizing to new scene configurations. Learning OSFs leads to photorealistic, physically-accurate renderings of multi-object scenes.

heading: 1 INTRODUCTION
text: Synthesizing images of dynamic scenes is an important problem in computer vision and graphics, with applications in AR/VR and robotics (Savva et al., 2019; Xia et al., 2020). For synthetic scenes, a user typically designs a set of 3D objects separately, then composes them into scenes to be rendered with specified camera, material, and lighting parameters. While this traditional graphics approach allows for flexible scene compositions, it requires detailed models of geometry, lighting, materials, and cameras, which can be difficult to obtain for real-world scenes.
To render real-world scenes without computer graphics models, recent works have explored using neural implicit methods (Lombardi et al., 2019; Sitzmann et al., 2019a;b). Most notably, Mildenhall et al. (2020) proposed neural radiance fields (NeRF), which achieve photorealistic quality by implicitly modeling the volumetric density and directional emitted radiance of a scene.
However, as shown in Figure 1, NeRF cannot generalize beyond the scene it was trained on, because it assumes static scenes and fixed illumination and learns a radiance field, which estimates only the resulting radiance along a ray after all light transport has occurred in a scene. Thus, for dynamic scenes where lights and objects can move, a separate NeRF-based model is needed for each new scene configuration.
To address this issue, we propose Object-Centric Neural Scattering Functions (OSFs) to synthesize dynamic scenes of objects learned from 2D images (Figure 2). We represent each object as a learned 7D scattering function with inputs (x, y, z, φi, θi, φo, θo), where (x, y, z) is the spatial location, (φi, θi) is the incoming light direction, and (φo, θo) is the outgoing light direction. The function outputs the volumetric density as well as the fraction of light arriving from direction (φi, θi) that scatters in outgoing direction (φo, θo).
Each OSF models all light bounces (reflections) and occlusions (shadows) within an object. Since each object’s scattering function is a radiance transfer function rather than a radiance field, it is
intrinsic to the object (independent of the scene it is in) and can be reused across different object placements and lighting conditions without retraining. We emphasize that because NeRFs are radiance fields, they cannot be composed, and cannot generalize beyond one scene. In contrast, we can render infinitely many scenes. We can build a library of OSFs trained independently for different objects to be composed into scenes with different object placements, camera, and lighting.
To model light transport between objects, we integrate our implicit object functions with volumetric path tracing. Like NeRF, we evaluate the radiance and volumetric density at 5D samples along every primary ray to the camera and composite them with an over operator. However, unlike NeRF, we estimate the radiance for each 5D sample by integrating our 7D OSF across the 2D sphere of incoming light directions. We estimate the integral with Monte Carlo path tracing (Kajiya, 1986) to reproduce shadows and indirect illumination effects.
Our key idea is to decompose the rendering problem into (i) a learned component (per-object asset creation), and (ii) a non-learned component (per-scene path tracing). The learned component models intra-object light transport (e.g., bounces from the seat of a chair to the back of the chair). The non-learned component handles inter-object light transport (e.g., bounces from a wall to a chair). Together, they model the full rendering equation (Kajiya, 1986) (except for occluders or light sources that intrude the object’s convex hull (Sloan et al., 2002)). Since only the inter-object light transport changes as objects and lights move, no re-training is required for different scene arrangements. Experimental results indicate that our method is capable of rendering images with novel scene compositions and lighting conditions better than alternative learned approaches.
In summary, our contributions are:
1. Learning Object-Centric Neural Scattering Functions (OSFs) that model intra-object light transport implicitly using a lighting- and view-dependent neural network.
2. Integrating implicitly learned object scattering functions with volumetric path tracing to model inter-object light transport.
3. A rendering algorithm that enables rendering scenes with moving objects, lights and cameras, using implicit functions.

heading: 2 RELATED WORK
text: Classical object-centric representations. Factoring light transport into intra- and inter-object illumination has a long history in traditional computer graphics (Dutre et al., 2018). In most cases, the motivation is to improve rendering efficiency by approximating intra-object lighting factors with simple transfer functions (e.g., linear) for simple radiance fields (e.g., spherical harmonics) derived from from computer graphics models, as in precomputed radiance transfer (PRT) (Sloan et al., 2002), ambient occlusion (Miller, 1994), or virtual walls (Arnaldi et al., 1994). In other cases, the motivation is to insert captured, real-world radiance fields into synthetic scenes, as in Light Field Transfer (Cossairt et al., 2008). These methods generally store the radiance field for objects in a discrete representation (e.g., a sampled 2D or 4D grid). As a result, they cannot reproduce accurate inter-object light transport, especially for objects with intersecting bounding volumes. In contrast, we focus on learning radiance transfer from images in order to model complex real-world scattering accurately, and utilize volumetric rendering techniques to account for inter-object illumination.
Novel view synthesis. Traditional methods for synthesizing novel views of a scene from captured images include using Structure-From-Motion (Hartley & Zisserman, 2003) and bundle adjustment (Triggs et al., 1999) to predict a sparse point cloud and camera parameters of the scene. More recently, a number of learning-based novel view synthesis methods have been presented but require 3D geometry as inputs (Hedman et al., 2018; Thies et al., 2019; Meshry et al., 2019; Aliev et al., 2020; Martin-Brualla et al., 2018). Others use multiplane images as proxies for novel view synthesis, but their viewing ranges are limited to interpolated input views (Flynn et al., 2016; Zhou et al., 2018; Srinivasan et al., 2019; Mildenhall et al., 2019). Some works represent scenes as coarse voxel grids and use a CNN-based decoder for differentiable rendering, but lack view consistency due to the use of 2D convolutional kernels (Nguyen-Phuoc et al., 2018; 2019; 2020).
Recently, volume rendering approaches have been used to render scenes represented as voxel grids that are more view-consistent (Lombardi et al., 2019; Sitzmann et al., 2019a). However, the rendering resolution of these methods are limited by the time and computational complexity of discretely sampled volumes. To address this issue, Neural Radiance Fields (NeRF) (Mildenhall et al., 2020) directly optimizes a continuous radiance field representation using a multi-layer perceptron. This allows synthesizing novel views of realistic images at an unprecedented level of fidelity. To make NeRF more efficient, Neural Sparse Voxel Fields (Liu et al., 2020) have been proposed as a sparse voxel octree variant of NeRF and demonstrate the ease of composing learned NeRFs with their voxel representation. See (Dellaert & Yen-Chen, 2020) for survey. While these implicit methods produce high-quality novel views of a scene, their models assume a static scene with fixed illumination. Our method enables synthesizing dynamic scenes with novel viewpoint, lighting, and object configurations. Relighting. Learning-based methods that relight images without explicit geometric reasoning have been proposed, but lack the ability to recover hard shadows (Sun et al., 2019; Xu et al., 2018; Zhou et al., 2019). Other works use geometric representations that facilitate shadowing computation, but require 3D geometry as input (Philip et al., 2019; Zhang et al., 2021; Oechsle et al., 2020; Rematas & Ferrari, 2020). Deep Reflectance Volumes (Bi et al., 2020b) reconstructs a voxelized representation of a scene and predict per-voxel BRDFs, but the fixed resolution of voxel grids limits the quality in the rendered images. Similarly, Neural Reflectance Fields (Bi et al., 2020a) predicts the parameters of a BRDF model, but demonstrate higher fidelity rendering by learning a continuous scene representation. However, Neural Reflectance Fields focuses on relighting single objects, and requires manual specification of the BRDF model. Parametric BRDF models are unable to handle complex scattering functions, including real-world scattering phenomena that are difficult to model. In contrast, our method is capable of learning all scattering functions, and can render multiple objects in dynamic scenes.

heading: 3 PRELIMINARIES

heading: 3.1 VOLUME RENDERING
text: To render an image of a scene with arbitrary camera parameters, camera rays are sent into the scene, through each pixel on the image plane. The expected color of each pixel is computed as the radiance along each camera ray.
Volume rendering is an approach for computing the radiance traveling along rays traced in a volume. Let r(t) = x0 + ωot be a point along a ray r with origin x0 and direction ωo, where t ∈ R is a 1D location along the ray, and the o in ωo denotes “outgoing” direction. For our purposes, we assume non-emissive and non-absorptive volumes. From Novák et al. (2018), the volume rendering equation to compute the radiance L(x0,ωo) of the ray is defined as:
L(x0,ωo) = ∫ tf tn τ(t)σ(r(t))Ls(r(t),ωo) dt, where τ(t) = exp ( − ∫ t tn σ(r(u)) du ) , (1)
where tn and tf are near and far integration bounds, σ(r(t)) denotes the volume density of point r(t), and τ(t) denotes the accumulated transmittance from tn to t. The termLs(r(t),ωo) is the light scattered at point r(t) along direction ωo, defined as the integral over all incoming light directions:
Ls(x,ωo) = ∫ S L(x,ωl)fp(x,ωl,ωo) dωl, (2)
where S is a unit sphere and fp is a phase function that evaluates the fraction of light incoming from direction ωl at a point x that scatters out in direction ωo. In NeRF, Mildenhall et al. (2020) assume
fixed illumination and do not consider any form of Equation 2. We consider a more general form of the volume rendering equation that explicitly models light paths within and between objects. This is important for dynamic scenes, where lighting and objects can move with respect to one another.

heading: 3.2 RAY MARCHING
text: The continuous integrals in Equation 1 can be estimated with quadrature (Kniss et al., 2003; Max, 1995), as done in NeRF (Mildenhall et al., 2020). For each ray, stratified sampling is used to obtain N samples {ti}Ni=1 along the ray, where ti ∈ [tn, tf ]. The rendering equation is approximated by:
L(x0,ωo) = N∑ i=1 τiαiLs(xi,ωo) where Ls(xi,ωo) = 1 |L| ∑ l∈L L(xi,ωl)ρ l i, (3)
where τi = ∏i−1 j=1(1−αj) and αi = 1− e−σi(ti+1−ti). To compute the average over incoming light paths Ls, we discretize over the domain S in Equation 2 by sampling a set of incoming light paths L = {l1, . . . , lK}, where ρli = fp(xi,ωl,ωo) ∈ [0, 1], the fraction of light incoming from light path l that is scattered in direction ωo.

heading: 3.3 NEURAL RADIANCE FIELDS
text: NeRF represents a continuous scene as a volumetric radiance field, approximated with a multilayer perceptron FΘ. The model FΘ takes spatial location x = (x, y, z) and viewing direction d = (φ, θ) as input, and outputs the density σ and color c = (r, g, b), where r, g, b ∈ [0, 1]. Frequency-based positional encoding (Rahaman et al., 2019; Vaswani et al., 2017) is applied to the inputs to better capture high-frequency variation in appearance and geometry.
A hierarchical volume sampling procedure (Mildenhall et al., 2020; Levoy, 1990) is then employed to more efficiently allocate samples along each ray. This technique biases sample allocation to favor the visible parts of the scene that contribute the most to the final render, avoiding occluded or free space in the scene. NeRF simultaneously optimizes two radiance fields, where the sample weights τi · αi from a coarse model are used to bias samples for a fine model. The L2 loss is used to optimize both models: ∑ r∈R‖Ĉc(r) − C(r)‖22 + ‖Ĉf (r) − C(r)‖22, where R is the set of all camera rays, Ĉc(r) and Ĉf (r) denote the radiance along ray r predicted by the coarse and fine models respectively, and C(r) is the ground truth pixel color for r.

heading: 4 METHOD

heading: 4.1 OBJECT-CENTRIC NEURAL SCATTERING FUNCTION
text: We represent each object as a 7D object-centric neural scattering function (OSF), depicted in Figure 3a. For each object, we learn an implicit function FΘ : (x,ωl,ωo) → (σ,ρ) that receives a 3D point in the object coordinate frame, the incoming light direction, and the outgoing light direction, and predicts the volumetric density as well as fraction of incoming light that is scattered in the outgoing direction. Θ are learned weights that parameterize the neural network, x = (x, y, z) denotes the spatial location, ωl = (φl, θl) denotes the incoming light direction, ωo = (φo, θo) denotes the outgoing light direction, σ denotes the volumetric density, and ρ = (ρr, ρg, ρb) denotes the fraction of light arriving at x from direction ωl that is scattered and leaving in direction ωo. The final color of a point x is the integral of ρ multiplied by the incoming radiance over all incoming light directions in unit sphere S (Equation 2). Following NeRF, we similarly apply positional encoding to our inputs (x,ωl,ωo) and employ a hierarchical sampling procedure to recover higher quality appearance and geometry of learned objects.
During training, we assume a single point light source with radiance of (1, 1, 1). This simplifies Ls from Equation 2 to Ls(x,ωo) = L(x,ωl)fp(x,ωl,ωo) = fp(x,ωl,ωo). To learn per-object NeRFs independent of object rotation and translation, the inputs to FΘ must be in the object’s canonical coordinate frame. Given a object transformation Ti for object oi, we apply T−1i to (r,ωl,ωo) before feeding the inputs to the network.

heading: 4.2 RENDERING MULTIPLE OSFS
text: Once we have learned an OSF for each object, we aim at composing the learned objects into scenes. An overview of our procedure is visually depicted in Figure 3b.
Let O = {oi}Ni=1 be a set of N objects we wish to render. For simplicity, we first describe the rendering process for each object oi, then explain the process to combine results across all objects to render the final scene. Let oi ∈ O denote object i with transformation Ti ∈ R4×4 and bounding box dimensions Di ∈ R3. Further let r be a camera ray with origin c ∈ R3 and direction ωo ∈ R3, which we define with parameters γ = [c,ωo] ∈ R6. Our goal is to compute L(c,ωo) as described in Equation 3. We compute the ray-box intersection between the ray and the object to obtain near bound tin and far bound t i f such that r(t i n) and r(t i f ) each intersect a box plane, as shown in Figure 4. Note that rays that do not intersect with oi are excluded from our computation. We sampleM points between tin and t i f along ray r to obtain a sample X
i = {xim}Mm=1, where Xi ∈ RM×3. Given a light source l, we evaluate the object’s model FΘi(X
i,ωl,ωo) to obtain alpha valuesαi ∈ RM and phase function values ρi ∈ RM×3. It is not always possible for a light ray from light source l to reach the object oi. Any of the other objects in O′ = {oj ∈ O | j 6= i} in the scene may occlude the incoming light, casting a shadow on object oi. We compute shadows by sending a shadow ray rm from each of the M samples inXi
to the light source l. Evaluating the shadow ray enables us to determine the amount of light blocked along the ray by other objects. We define the parameters of the M shadow rays as Γ ∈ RM×6. For each object oj ∈ O′, we compute ray-box intersections between shadow rays Γ and oj’s bounding box. This allows us to compute the amount of light traveling towards oi that is blocked by oj . Similar to primary rays, we sample M points along each shadow ray to obtain a set of points Xj ∈ RM×M . We then evaluate the object model FΘj (Xj) to obtain alpha values Aj ∈ RM×M . For each shadow ray rm, we combine samples Ajm across the N − 1 objects in O′ by sorting according to sample distance to obtain alpha values Am ∈ RM(N−1). The fraction of unobstructed light traveling along the shadow ray rm is computed as the transmittance:
τ lm = M(N−1)∏ n=1 (1−Amn). (4)
Thus, the adjusted incoming radiance from light source l when accounting for occlusions is computed as Ll(xm,ωl) = τ lmLl(xm,ωl).
We follow the scattering equation in Equation 2 and now consider all incoming light directions over the unit sphere S. This accounts for secondary light rays traveling to an object oi indirectly from another object oj (indirect illumination). We approximate the integral over the unit sphere S by sampling K directions on the unit sphere uniformly at random. For each direction ωk randomly sampled for a point x, we send a secondary ray rk from x in direction ωk and evaluate the radiance L(x,ωk) traveling along the ray. To compute the radiance of the secondary ray L(x,ωk), we employ the same technique used to compute the radiance of a primary ray L(c,ωo) (described at the beginning of Section 4.2). The incoming radiance L(x,ωk) is multiplied with the phase function value ρ = fp(x,ωk,ωo) to determine the outgoing radiance L(x,ωo), where ρ is evaluated using FΘi . Note that this is possible due to the recursive nature of our formulation. Only secondary rays are described here (two bounces), but our method supports an arbitrary number of bounces. Rendering. We sample and evaluate all objects in O to obtain alpha values {αi}Ni=1 and phase function values {ρi}Ni=1 for a set of sampled points {Xi}Ni=1 along ray r. We sort the samples across all objects to produce a final set of P = M ·N samples {xm}Pm=1, {αm}Pm=1, and {ρm}Pm=1. Given light paths L containing both direct and indirect illumination, we render the final radiance of a ray with origin x0 and direction ωo with the following equation:
L(x0,ωo) = 1 |L| ∑ l∈L P∑ m=1 αmρ l mτmLl(xm,ωl), where τm = m−1∏ n=1 (1− αn), (5)
and Ll(xm,ωl) is the radiance from light path l arriving at point xm. Runtime. In total, the cost of rendering a single image with Npixel pixels and Nobject objects is O(P 2KNpixel). Note that P is an upper bound on number of samples that need to be evaluated. In practice, a single ray often only intersects with at most one object in the scene, which means that the proposed rendering procedure is not significantly more expensive than the single object setting. We also note that compared to NRF Bi et al. (2020a) or traditional volumetric path tracing methods, OSF crucially does not require running path tracing within each object to simulate intra-object light bounces. This is because OSF learns the object-level scattering function that directly predicts the effects after all light bounces (reflections) and occlusions (shadows) within an object have occurred. Thus OSF is significantly faster than NRF which relies on simulating intra-object light bounces while querying its learned BRDF model.
In our experiments, rendering a single image with a single OSF at a resolution of 256 × 256 takes roughly 3.7 seconds. While the computation cost is high, there are efforts to reduce the rendering speed of NeRF that are orthogonal to this work. For instance, KiloNeRF (Reiser et al., 2021) can easily adapted to this work by utilizing thousands of tiny MLPs instead of one single large MLP to represent each OSF to obtain 1-2 orders of magnitude speed up.

heading: 5 EXPERIMENTS
text: Datasets and evaluation metrics. We evaluate our approach on several image datasets:
• FURNITURE-SINGLE: 15 objects rendered with random object pose, point light, and viewpoint.
• FURNITURE-RANDOM: 25 dynamic scenes, each containing a random layout of multiple objects, point light, and viewpoint.
• FURNITURE-REALISTIC: Scenes containing realistic arrangements of objects in rooms. • REAL-NRF: Real-world objects from Bi et al. (2020a), captured in a dark room under varying
viewing and lighting directions. • REAL-OUTDOOR: Real-world outdoor scenes from Mildenhall et al. (2020).
For FURNITURE datasets, we use Blender’s Cycles path tracer (Blender Foundation, 1994) to render images at 256 × 256 resolution for different object arrangements, camera views, and lighting configurations. We report PSNR, SSIM (Wang et al., 2003), and LPIPS (Zhang et al., 2018) metrics. Baselines and ablations. We compare our method to the following baselines:
1. o-NeRF: A variant of the NeRF model, but with one NeRF trained per object. When o-NeRFs are composed into scenes, they are rendered separately. 2. o-NeRF + S: An extension of o-NeRF with inter-object shadows; reduces the light arriving at each o-NeRF by the cumulative opacity of shadowing objects along the ray from the light (§4.2).
These baselines represent what could be achieved by combining separately trained NeRFs into a scene. Of course, since o-NeRFs produce radiance fields (not scattering fields), we do not expect them to perform well in novel lighting environments or object placements.

heading: 5.1 NOVEL LIGHTING
text: In the first experiment, we investigate how OSF method handles novel lighting conditions.
We train one model per object in FURNITURESINGLE. For each object model, we train on 400 images with randomized viewpoint and lighting, and test on 20 images of novel viewpoint and lighting. As can be seen in Figure 5, our method produces more accurate appearance of the objects in comparison to oNeRF when tested on novel illumination conditions. In particular, o-NeRF fails to predict self-shadows for the couch and chair correctly. Additionally, o-NeRF fails to disentangle viewpoint versus lighting-dependent appearance, producing incorrect shadows for the couch and chair, and fails to capture the specular details of the ottoman. Quantitative results can be found in Table 1.

heading: 5.2 SCENE COMPOSITION
text: In a second experiment, we conduct a scene composition task on FURNITURE-RANDOM, where multiple object models are combined into scenes in random pose, lighting, and viewpoint configurations. For this task, we use the same object models trained in Section 5.1. Results are shown in Table 1 and Figure 6. While not shown in the main text, results for FURNITURE-REALISTIC can be found in Appendix B.
These results suggest that OSF outperforms all baselines and ablations, both quantitatively and qualitatively. As in the previous experiment (Section 5.1), we find that OSF reproduces object appearances and self-shadows more accurately than the baselines. The difference is especially apparent in the couches in scenes (a) and (b), where the couches predicted by o-NeRF are extremely dark. This is due to the fact that o-NeRF is unable to disentangle view-dependence appearance from lightdependent appearance, and simply interpolates the radiance field learned another different lighting configuration. Please note that OSF is able to model inter-object light transport effects by rendering shadows cast by one object onto another and on the ground plane. Plus, it is able to render indirect illumination of one object reflecting light onto another. For example, light reflected from the left wall causes the left of the couch and table in scenes (a) and (b) to be brighter. Neither of these lighting effects are present in the o-NeRF results.

heading: 5.3 REAL-WORLD SCENES
text: In this section we evaluate our method on real world objects and scenes from the REAL-NRF and REAL-OUTDOOR datasets. For these experiments, we train one OSF for each object in REAL-NRF and each scene in REALOUTDOOR.
Figure 7 shows a comparison between ground truth, our method (OSF), and Neural Reflectance Fields (NRF) (Bi et al., 2020a). We show that OSF recovers stronger, more accurate specular highlights compared to NRF. OSF also produces more detailed appearances (see pony logo). This comparison demonstrates the main advantage of OSF: the ability to handle complex scattering functions.
S
ur
f S
ho
p
For scene composition, the OSFs trained on each object are composed with a synthetic floor OSF in Figure 8 row (a). Our method is able to compute accurate shadows, such as the shadow cast by the pony onto the two other objects in the scene. The indirect reflections from the floor allow the shadowed objects to be slightly visible as shown in the “OSF” panel.
Figure 8 rows (b) and (c) show results on inserting REAL-NRF objects into real outdoor scenes (REAL-OUTDOOR). Shadows and reflections are rendered with randomized lighting directions to approximate the environment lighting. Our method accurately renders occlusions between the inserted objects and the vase in Figure 8 row (c). Due to the compositional nature of OSFs, we are able to insert the learned pinecone from Figure 8 (b) into (c).
In Figure 8, each column shows ablated versions of OSF to study the impact of computing shadows and indirect illumination with our path tracing algorithm. “No Shadows, No Indirect” represents a version of our model containing only direct illumination (without modeling inter-object lighting effects). We additionally show “No Indirect” and “Indirect Only” variants of our model which
represent computing shadows and indirect illumination, respectively. As illustrated by Figure 8, our full model containing both shadows and indirect illumination effects is the most realistic. Additional results on real-world scenes, including complex shadows, can be found in Appendix A.

heading: 6 DISCUSSION
text: We have proposed Object-Centric Neural Scattering Functions (OSFs), a method that enables composing objects captured only from photographs into photorealistic renderings of dynamic scenes. We demonstrated that decomposing a scene into implicit object functions that are view- and lightdependent enables reusabiliy of objects across scenes where objects, camera, and lighting can change. We presented a method for integrating our learned implicit functions with volumetric path tracing, and showed inter-object light transport effects such as shadow and indirect illumination for real-world objects where no computer graphics model is available. We believe our work is a step towards a graphics pipeline where real-world scenes are modeled by a composition of implicit functions to combine the flexibility of object-centric neural modeling with the photorealism of graphics rendering algorithms.
There are a few main limitations to OSF. First, the computational complexity of our method is high, but there are several works tackling the orthogonal issue of improving NeRF efficiency (as discussed in Section 4.2) that can easily be applied to OSFs. Second, while learning intra-object light transport means that intra-object path tracing is not needed, this formulation assumes that at test time, there are no occluders or light sources that intrude the object’s convex hull (Sloan et al., 2002) (e.g., a person sitting in a chair). However, OSFs can still be rendered even if their bounding boxes are intersecting, as long as this assumption is not violated. Finally, acquiring datasets of real world objects with varying point light sources and viewpoints is challenging, but we hope that in the future such acquisition of real world datasets will become easier to capture and more widely available.
REPRODUCIBILITY STATEMENT
We describe our method (Section 4) and experimental setup (Section 5) in detail to maximize reproducibility. We will release our code upon publication to facilitate future research.

heading: A REAL-WORLD SCENE COMPOSITION
text: Different scene configurations of composed objects from REAL-NRF are shown in Figure 9. We show the effect of moving the light, camera, or objects. Notice how the the appearance and shadows of the objects are updated across different scene configurations. Also notice that even when parts of the palm tree object and the cartoon object are cast under the pony’s shadow, they do not appear completely dark due to the indirect illumination from the floor.
Analyzing the effect of different numbers of indirect (secondary) rays per primary sample, Figure 10 shows the result. As can be seen from the figure, the noisiness of the indirect illumination render decreases as the number of samples increase. Results in this paper contain between one and five randomly sampled secondary ray for each primary ray sample.
Single-object renderings from REAL-NRF are shown in Figure 11. The objects were captured in a dark room with a one-light-at-a-time setup. After training OSF on each object in this dataset, we are able to render the objects from novel viewpoints and lighting directions.

heading: B ABLATION EXPERIMENTS
text: Figure 12 shows ablation results on FURNITURE-REALISTIC. We evaluate different variants of our model: “Direct Only” which considers only direct illumination; “Indirect Only” which considers only indirect illumination; “Direct + Shadows” which includes both direct illumination and shadows. Our full model (OSF) shows the most realistic rendering, with accurate shadows and indirect illumination effects such as the left side of the couches and tables appearing brighter due to indirect lighting from the left wall. Note that the white area on the right of the images represent rays with zero density that are composited onto a white background (and therefore do not contribute indirect illumination to the scene).

heading: C COMPLEX ILLUMINATION
text: In this experiment, we investigate how scenes composed of OSF objects can be rendered with complex illumination from an environment map.
Specifically, we apply the combination of a point light source and the environment map shown in the top-left corner of Figure 14 to light one of our scenes in FURNITURE-REALISTIC. This simulates the appearance of the scene as if the scene were inserted into a complex lighting environment, which stresses the benefits of the OSF path tracing framework.
For each OSF sample point, we project the equirectangular coordinates of the environment map into spherical coordinates, sample 20 directions on the unit sphere uniformly at random, evaluate the OSF function for each incoming direction, and integrate them outgoing radiance using Equation 5. Please note that a green-blue tint is slightly apparent in the scene rendering, due to the contribution of green and blue lighting from the environment map. Figure 14: Complex illumination results.
D IMPLEMENTATION DETAILS A flowchart of our method is shown in Figure 15.
We approximate our model FΘ with a multilayer perception (MLP) with rectified linear activations. The predicted density σ is view-invariant, while the scattering function value ρ is dependent on the incoming and outgoing light directions. We use an eight-layer MLP with 256 channels to predict σ, and a four-layer MLP with 128 channels to predict ρ. For positional encoding, we use W = 10 to encode the position x and W = 4 to encode the incoming and outgoing directions (ωl,ωo), where W is the highest frequency level. To avoid ρ from saturating in training, we adopt a scaled sigmoid (Brock et al., 2016) defined as S′(ρ) = δ(S(ρ)− 0.5) + 0.5 with δ = 1.2. We use a batch size of 4,096 rays.
For synthetic datasets, we sample Nc = 64 coarse samples and Nf = 128 fine samples per ray. For real world datasets, we sample Nc = 64 coarse samples and Nf = 64 fine samples per ray. We use the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.001, β1 = 0.9, β2 = 0.999, and = 10−7.

Review summaries:
This paper proposes a new neural implicit representation called object-centric scattering function for scene compositing application. The major extension is to add the lighting direction as an input so that the new representation can be used for relighting. To train this representation, the authors propose minimizing the rendering loss of images rendered from different views under different point lighting. Given this representation, the authors propose a standard volume path tracing framework to render different objects and scene structures together, with indirect illumination and shadow being modeled. Experiments show that compared with nerf, the proposed method achieves better accuracies in object compositing and can handle changes of illumination. 

The paper proposed a method to composite objects parameterized using implicit functions into realistic scenes. The idea is to first capture the representation for each object separately, and each object is represented using neural scattering function, which predicts the outgoing lighting transport conditioned on the input lighting direction, viewing direction and 3d location. The objects are composed into the full scene by doing volume rendering along the ray, and the radiance of each sampled point in the ray is calculated via integrating on a sphere to obtain the lighting radiance (including secondary (indirect) light effect, the shadow effect is obtained from iterating the ray from light position to the  sampled point). The results on both synthetic scene and real scene shows improvement over baselines (o-Nerf, o-Nerf S) that didn't consider the lighting transportation. 

This paper proposes a NeRF based method for composing photo-realistic scenes from captured images. The proposed method learns object-centric neural scattering functions (OSFs) to implicitly model per-object light transport using a lighting- and view-dependent neural network. Multiple objects can be rendered with volumetric path tracing. The proposed method has been evaluated on both synthetic and real-world datasets.

The paper proposes a decomposition of a 3D scene into object-level neural radiance fields. This allows to apply rigid transforms to each object independently and to rearrange the scene. Crucially, light transport is modelled, which turns the radiance fields into scattering functions. Direct lighting and shadowing, as well as indirect illumination from several light bounces are taken into account, such that the illumination of the modified scene looks correct. As long as the scattering fields are trained on a sufficient number of light positions, the lights can also be moved around at test time.