Learning Object-Centric Neural Scattering Functions for Free-viewpoint Relighting and Scene Composition
Authors: Hong-Xing Yu
Michelle Guo
Alireza Fathi
Yen-Yu Chang
Eric Ryan Chan
Ruohan Gao
Thomas Funkhouser
Jiajun Wu
Venue: Accepted by TMLR ()

Abstract: Photorealistic object appearance modeling from 2D images is a constant topic in vision and graphics. While neural implicit methods (such as Neural Radiance Fields) have shown high-fidelity view synthesis results, they cannot relight the captured objects. More recent neural inverse rendering approaches have enabled object relighting, but they represent surface properties as simple BRDFs, and therefore cannot handle translucent objects. We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct object appearance from only images. OSFs not only support free-viewpoint object relighting, but also can model both opaque and translucent objects. While accurately modeling subsurface light transport for translucent objects can be highly complex and even intractable for neural methods, OSFs learn to approximate the radiance transfer from a distant light to an outgoing direction at any spatial location. This approximation avoids explicitly modeling complex subsurface scattering, making learning a neural implicit model tractable. Experiments on real and synthetic data show that OSFs accurately reconstruct appearances for both opaque and translucent objects, allowing faithful free-viewpoint relighting as well as scene composition. In our supplementary material, we include a video for an overview. Project website with video results: https://kovenyu.com/OSF/

Published in Transactions on Machine Learning Research (05/2023)
Learning Object-Centric Neural Scattering Functions for Free-viewpoint Relighting and Scene Composition
Hong-Xing Yu Michelle Guo"1, Alireza Fathi?, Yen-Yu Chang', Eric Ryan Chan', Ruohan Gao Thomas Funkhouser", Jiajun Wu
Stanford University
Google Research
"Contributed equally.
Reviewed on OpenReview: https: openreview. net/ forun? id- WirSRR
Abstract
Photorealistic object appearance modeling from 2D images a constant topic vision and graphics. While neural implicit methods (such as Neural Radiance Fields) have shown high-fidelity view synthesis results, they cannot relight the captured objects. More recent neural inverse rendering approaches have enabled object relighting, but they represent face properties simple BRDFs, and therefore cannot handle translucent objects. We propose Object-Centric Neural Scattering Functions (OSFs) for learning to reconstruct ob- ject appearance from only images. OSFs not only support freo-viewpoint object relighting, but also can model both opaque and translucent objects. While accurately modeling sub- surface light transport. for translucent objects can be highly complex and even ractaale for neural methods, OSFs learn approximate the radiance transfer from distant light to an outgoing direction at any spatial location. This approximation avoids explicitly modeling complex subsurface scattering, making learning a neural implicit model tractable. Exper- iments real and synthetic data show that OSFs accurately reconstruct appearances for both opaque and translucent objects, allowing faithful free-viewpoint relighting as well as scene composition. Project website with video results: https: //to:///////....// com/OSF.
1 Introduction
Modeling the geometry and appearance of 3D objects from captured 2D images is central to many applications in computer vision, graphics, and robotics, such as shape reconstruction (Mescheder al., 2019; Park et al., 2019; Chang et al., 2015), view synthesis (Hedman et al., 2018; Mildenhall et al., 2020; Sitzmann et al., 2019a), relighting (Zhang et al., 2021c; Bi al., 2020b;a), and object manipulation (Simeonov et al., 2021). Traditional inverse rendering approaches (Zhou et al., 2013; Nam et al., 2018) focus on directly estimating objects' shape and material properties; they then compose them using the estimated surface properties. These explicitly reconstructed object representations often lead to visible artifacts or limited fidelity when rendering images using the recovered object assets.
Recently, neural implicit methods have attracted much attention in image-based appearance modeling. They implicitly represent objects and scenes using deep neural networks (Lombardi et al., 2019; Sitzmann et al., 2019a;b). Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) are one of the most representative methods due to their high-fidelity view synthesis results. NeRFs implicitly model scene's volumetric density and radiance via coordinate-basod deep neural networks, which can be learned from images via direct volume rendering techniques (Max, 1995). However, NeRE' only reconstruct the outgoing radiance fields under a fixed lighting condition. Thus, they cannot. relight the captured objects or compose multiple objects into new scenes.
To address this problem, few neural inverse rendering methods (Zhang al., 2021 Boss et al., 2021a; Zhang et al., 2021c) have been proposed to jointly estimate lighting, material properties, and geometry from

---

Published in Transactions on Machine Learning Research (05/2023)
Input images
Novel view synthesis
Relighting
Scene Composition
Figure 1: We propose a new neural object representation, Object-Centric Neural Scattering Functions (OSFs), to reconstruct object appearance from only images. OSBs can handle objects with complex materials shapes (e.g., the translucent bunny), and support both free-viewpoint relighting and scene composition.
images. The disentanglement of illumination enables relightable object appearance modeling. However, these methods all assume simple Bidircctional Reflectance Distribution Functions (BRDFs) when modeling surface properties, without taking subsurface light. transport into consideration". Therefore, they can only model the appearance of opaque objects and cannot handle translucent objects with complex material properties.
We observe that subsurface scattering effects are the key to relighting translueent objects. However, accu- rately modeling subsurface scattering can be complex and even intractable for neural implicit representations, because each sampling operation for numerical integration requires a complete forward pass of the network. This leads to a prohibitive computational cost both time and memory.
To address this challenge, we propose Object-Centric Neural Scattering Functions (OSFs) for amage-basee relightable neural appearance reconstruction. An OSF learns to approximate the cumulative radiance transfer function, which models radiance transfer from distant light to any outgoing direction at any spatial location for an object. This enables the use of volumetric rendering formulation to learn from images, while being able to approximate both opaque and translucent object appearances. Thus, OSFs allow free-viewpoint relighting from only 2D images (sce Figure 1).
Modeling object-level light transport. allows neural scene composition with trained OSF models, as light transport effects such as object shadows and indirect lighting can be naturally incorporated. To demonstrate this, present simple and computationally tractable method for scene composition. To further accelerate neural scene rendering, introduce two improvements on OSFs scene rendering that allow magnitudes of acceleration without compromising rendering quality.
Experiments on both real and synthetic objects demonstrate the offectiveness of our OSF formulation. We show that OSFs enable vrewvvienp relighting of both translucent and opaque objects by learning from only 2D images. We also show that in scene composition using the reconstructed objects, our approach significantly outperforms baseline methods.
In summary, contributions threefold. First, we introduce the concept of the cumulative radiance transfer function that models object-centric light transport. Second, we propose the Object-Centree Neural Scattering Functions (OSFs) that learn to approximate the cumulative radiance transfer function from only 2D images. OSEs allow free-viewpoint relighting and scene composition of both opaque and translucent objects. Third, we show experimental results on both real and synthetic objects, demonstrating the effectiveness of OSFs in free-viewpoint relighting and scene composition.
2 Related Work
Neural appearance reconstruction. Traditional methods use Srructure-From-Mere (Hartley & Zisser- man, 2003) and bundle adjustment. (Triggs et al., 1999) to reconstruct colored point clouds. More recently, a number of learning-based novel view synthesis methods have been proposed, but they require 3D geometry
We use "simple BRDPs" to refer BRDFs that do not consider subsurface scattering.
2

---

Published in Transactions on Machine Learning Research (05/2023)
inputs (Hedman et al., 2018; Thics et al., 2019; Meshry et al., 2019; Aliev et al., 2020; Martin- Brualla et al., 2018). Recently, neural volume rendering approaches have been used to reconstruct scenes from only images (Lombardi et al., 2019; Sitzmann et al., 2019a). However, the rendering resolution of these methods is limited by the time and computational complexity of the discretely sampled volumes. To address this issue; Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020) directly optimizes a continuous radiance field representation using a multi-layer perceptron. This allows for synthesizing novel views of realistic im- ages at an unprecedented level of fidelity. Various extensions of NeRF have also been proposed to improve efficiency (Liu et al., 2020; Yu et al., 2021; Garbin et al., 2021; Reiser et al., 2021), generalization (Niemeyer & Geiger, 2021; Yu et al., 2022; Yang et al., 2021), quality (Oechsle et al., 2021; Barron et al., 2021a;b; Verbin et al., 2021; Zhang et al., 2020), compositionality (Ost et al., 2021) etc. While these neural mots produce high-quality novel views of a scene, they do not support relighting, In contrast, our approach aims at relightable appearance reconstruction.
Learning- baased relighting. Learning beare methods for relighting without explicit geometry have been proposed (Sun ct al., 2019; Xu et al., 2018; Zhou et al., 2019), but they are not 3D-aware. Following the recent surge of neural 3D scene represeniations and neural differentiable rendering (Tewari et al., 2020; Mildenhall et al., 2020), neural relightable representations (Bi et al., 2020b;a; Boss et al., 2021a; Srinivasan et al., 2021; Zhang et al., 2021c; Baatz et al., 2021) and neural inverse rendering methods (Zhang et al., 2021b;a; Boss et al., 2021b) have been developed to address this limitation. However, most of them assume simple BRDFs which do not consider subsurface scattering in their formulations, and thus they cannot model translucent object appearances for relighting. In contrast, our method allows the reconstruction of both opaque and translucent object appearances. Some existing methods also approximate subsurface light transport for participating media (Kallweit et al., 2017; Zheng el al., 2021). While Kallweit et al. (2017) demonstrate high-quality scattering, they require 3D groundtruth data whereas our method learns from only images. Zheng et al. (2021) focus on relevant tasks as we do, using specific designs such as spherical harmonics for scattering and visibility modeling. In contrast, our model is simple and allows adopting recent advances in accelerating neural rendering. We demonstrate interactive frame rate in rendering by our KiloOSF.
Precomputed light transport. Our OSFs learn to approximate cumulative radiance transfer function, which is based on the classical idea from the precomputed light transport methods including the precomputed radiance transfer (Sloan et al., 2002) (also see (Lehtinen, 2007) and (Ramamoorthi, 2009) for overviews). These methods map incoming basis lighting to outgoing radiance, while we additionally learn to encapsulate the object-specific light transport at the visible surface. Related idcas include the classical reflectance field (Debevec et al., 2000; Garg et al., 2006). Our method is also related to classical lechniques for modeling aggregate scattering behavior (Moon et al., 2007: Lee & O'Sullivan, 2007; Meng et al., 2015; Blumer et al., 2016). These techniques consider modeling the full asset-level internal scattering, a more general form than our cumulative radiance transfer. Our cumulative radiance transfer function borrows the general idea from these classical methods and adapts to neural implicit volumetric rendering framework for relightable and compositional appearance modeling from images. This adaptation allows using the power of deep networks to approximate complex light transport. Recently a few neural methods have been proposed to model light transport of complex materials (Kuznelsov, 2021; Baalz el al., 2021), e.g.; Baatz et al. (2021) represent surface texture as neural reflectance fields which is technically relevant to ours. However, they need meshes as explicit object geometry, while ours learns object representations from only images.
3 Approach
We aim for relightable neural appearance reconstruction of both opaque and translucent objects from 2D images. To achieve this, we propose Object-centric neural Scattering Functions (OSTs) that learn to approx- imate subsurface scattering effects in a computationally tractable way. In the following, we first introduce our OSF formulation and discuss its relations to phase functions and BSSRDFs (Sec. 3.1). Then we show how we can model translucent objects (Sec. 3.2) and opaque objects (Sec. 3.3) with OSFs. Finally, we discuss how to learn OSFs from 2D images (Sec. 3.4) and compose multiple learned OSFs (Sec. 3.6).
3

---

Published in Transactions on Machine Learning Research (05/2023)
Distant light Light
out
@ light
W out
light
X out
X
(X, (n)
[.J, <(E, Fau dhio sat x
lou Jos p(x, w an bin n0 ,dw,
Lyse p(x, Orign 6 add bnghr
Phase function
BSSRDFs
Cumulative radiance transfer (Ours)
Figure Relations to classical graphics concepts for modeling subsurface scattering effects. The process of light transport from an unobstructed distant light to point x inside a translucent object is complex: 1) the light (radiance) hit a point on the visible surface area; 2) the surface transmits some amount of the light into the object interior; 3) the transmitted light scatters within the object; 4) some light finally arrives at from incoming directions Wini 5) the arrived light scatters to dircctions Wout and 6) this process happens at all points the visible surface, and the final outcome is the accumulated result. Phase functions only model 4) and 5), so one has to account for the other steps by multiple integrations. BSSRDFs encapsulate steps 2) 5), so they require two integrations for step 1) and step 6). In contrast, our OSF formulation encapsulates the entire process that takes all steps into account. This abstraction of modeling the whole process as cumulative radiance transfer eliminates the need of any integration, making it computationally tractable to use in neural volume rendering.
3.1 OSF Formulation
Our OSFs are based on Neural Radiance Fields (NeRFs) (Mildenhall et al., 2020), which learn view-dependent emitted radiance at any point in space. NeRF's can be formulated by NNe (x, Wout) + (Lout: o), where NN denotes a learnable deep neural network with parameters e, x denotes a 3D spatial location, Wout denotes an outgoing radiance direction, Touu denotes the outgoing radiance, and denotes the volumetric density. Inspired from the principles of volume rendering (Kajiya & Von Herzen, 1984), the computational model for the radiance L(o, Wout) arriving the camera. location from direction Wout (a.k.a. the radiance of camera ray r(t) twout) with near and far bounds and ty is given by:
=/(" (r)) (r(t), Wout
L(o, wout)
(1)
where T(t) ex:)==xp((-000 exp(- S",, (r(()))ds) denotes the transmittance along the ray from t, t.
While NeRF can render high-fidelity novel views of captured object, it does not support relighting because it is based on the absorption-pluu-emision model (Max, 1995), which itself is light nource-agnostic Namely, NcRFs model Lout as the emitted radiance after all light transport has occurred. To address this limitation, we propose OSFs that model Lout as the scallered radiance due to external light sources instead. More specifically, OSFs learn to approximate the cumulative radiance transfer from an unobstructed distant light, addition to the volume density:
NNe (x, Wlihtts Wont) + (p,o).
where p(x, Wlight, Wout) denoles the cunulative radionce transfer function, which is defined as the ratio of two quantities. The numerator is the outgoing radiance Lout (x, Wout) in the direction of Wout at location x that is caused by unobstructed distant light Light (Wight) from direction Wlight The denominator is Liight (Wigth)
It is cumulative about all paths that Lighh(WWight (Wight) scatters to x. See Figure 2 for an illustration.
Given the cumulative radiance transfer function the scattered outgoing radiance Lout is given by
Lout (x, Wout) / p(x, Wlight, Wout Liight (Wiihtt) ddwhtt
(3)
4

---

Published in Transactions on Machine Learning Research (05/2023)
We assume the object itself does not emit light. We argue that this distant light assumption in p is acceptable and can allow relighting with environment maps that are also distant light.
Phase Functions vs. BSSRDFs. vs. Cumulative Radiance Transfer. The cumulative radiance transfer function p approximates a complex radiance transfer process from an unobstructed distant light to an outgoing direction at a location, including those inside the object. is conceptually akin a phase function that describes the scattered radiance distribution at. a specific location in scattering process. It is also related to the Bidirectional Scattering Surface Reflectance Distribution Functions (BSSRDFs) (Jensen et al., 2001). See Figure/ 2 for detailed comparison of these concepts.
Below we list the major distinctions against phase functions:
Generally, a phase function is local, i.e., it depends only on the material/medil (and some cases also geometry) at the location. In contrast, the cumulative radiance transfer function p is object- specific and spatially varying even if the object is made from uniform homogeneous material. This is because P potentially needs to account for complex process, including the light transport for all surface points that are visible to the distant light, the subsurface scattering effects, and the radiance transfer at a specific location. Thus, this process depends on both the material and the object geometry at x, as well as the light direction.
Phase functions for some naturally occurring isotropic media only need 1 Degree of Freedom (DoF) for angular distribution, i.c., the cosine between incoming and outgoing radiance directions (sce Chapter 11 in Pharr et al. (2016)). However, generally needs 4 DoF' for angular distribution: 2 DoF of light direction because the light direction can interact with object geometry to affect the incoming radiance distribution at any given x, and 2 DoF of outgoing direction as we want to model view-dependent offects in NcRFs (Mildenhall ct al., 2020).
Compared to BSSRDFs which are surface models, p is used in a volumetric rendering process. OSFs do not explicitly model surface properties.
3.2 Modeling Translucent Objects with OSFs
Our OSF formulation can model the appearance of translucent objects in a computationally tractable way for volumetric neural implicit methods, while directly modeling the radiance transfer in volumetric scattering process intractable. In direct modeling, Lout Wout) = SSa p(x, w, Wout Lin w) where p denotes a phase function, Tin denotes the incoming radiance at location from the direction w. The recursive dependency of Lin over the unit sphere in the media makes the equation very difficult to solve (Novák al., 2018; Max & Chen, 2005). From the computational perspective, to numerically compute this integral to train the neural network, one needs to account for the whole sphere (due to subsurface scattering) through dense sampling. However, for neural implicit methods, each sample requires a forward pass of the deep network, leading to a high computation cost. Moreover, training deep networks often requires saving intermediate variables for auto-iifferentiitiinn (Paszke et al., 2017), which demands a massive amount of GPU memory for dense sampling. As a result, it can become computationally intractable to train the neural network in this way. In contrast, during training, we use lighting setups that match well the assumption of a single directional light source; the spherical integration in Eq. (3) thus reduces to single evaluation. The key feature of OSFs is that they approximate the complex cunulative radiance transfer process using the expressive power of deep neural networks.
3.3 Modeling Opaque Objects with OSFs
Apart from modeling translucent objects, OSF formulation is also able to model opaque objects. Theoreti- cally, the formulation may subsume surface light transport for opaque convex objects. To see this, consider a single distant light source from Wlight, the light transport at a visible surface point can be described by
Lout (Xourfs Wout) = f(Xaurf Wlight Wout Light [n (xXeuu))) Wighhl
(4)
5

---

Published in Transactions on Machine Learning Research (05/2023)
where f denotes BRDF, Xaurf denotes the first surface point hit by the camera ray r, and n(x) denotes the unit. normal vector at x. We have used the single distant light assumption to solve the integration. Now we consider an ideal case, where an OSF learns that oxx) = 5(x - -xsutt for all Xaurf- This solves the integration in Eq. (1) and gives
T(o, Wout) Trout (Xarrf, Waut) = pl Xsurrf Wlight Wout ) Lright.
(5)
We can see that if the learned OSF satisfies p(X Wlight: Wout) = f(x Wlight, Wout) |n(Xaurt) Wlight for all surface points, then it subsumes the surface light transport with the BRDF. Although a neural network cannot perfectly represent delta function for (x), in our experiments, learn to approximate the surface light transport for opaque objects of both convex and concave shapes.
observe that empirically OSFs
3.4 Learning OSFs from Images
We aim to learn OSFs from only 2D images, allowing easy capture of real objects. As discussed in Section 3.1, we prefer a tractable training setup. Thus, we propose capturing object images using only single distant light source, which leads to a light source radiance function (w) = L Wo), where Lu and wo are obtained from the capture setup. We can define the sampling Probabilistic Density Function (PDF) for lighting as PiFhighh (w) = 8(w - wo) and this gives an analytic solution of the integration in Eq. (3):
Llihht (w)
Lout (x, Wout)
=
E WnPDFiig (w)
p(x, w. Wout)
PDF light (w)
(6)
= p(x, Wo: Wont Lo.
(7)
We capture images of the object-to-roconstrucc under different we in a dark room. We describe our accossible real data capture setup using two iPhones Section 4.1.
Given Eq. (7), the camera ray radiance Wout) defined in Eq. (1) is trivially differentiable with respect to OSFs. We follow the learning strategy from NeRFs (Mildenhall et al., 2020), which modcls the pixel color of a ray in an analog to radiance instead of the raw radiometric quantity to simplify learning. This may be interpreted as allowing the neural network to implicitly deal with the camera response that maps camera ray radiance to pixel color (also see a recent discussion in Mildenhall et al. (2021)). We also follow NeRFs to use quadrature to compute Eq. (1), and use positional encoding and hierarchical volume sampling to facilitate training (i.e., we use a coarse network for coarse sampling a ray to give a rough estimation of transmittance distribution, and then use a fine network do additional informed sampling to further reduce variance). The loss function for learning an OSF is thus defined as
((0) = Err [1] Leonrss Wont) C(r)]1 + [Tfine (o, Wont) - C(r)
where C(r) denotes the groundtruth pixel color (which is up to a constant of the radiance assuming a linear camera response) for ray r = o twout, Leoarse (o, Wout) denotes the predicted radiance by the coarse network, and Tane (o, )out) denotes the predicted radiance by the fine network. During testing, we only use the predicted radiance of the fine network.
3.5 KiloOSF for Accelerating Rendering
Neural volumetric rendering is slow. We show a complexity analysis in the following subsection. To accelerate OSFs, in the same spirit as KiloNeRF (Reiser et al., 2021), we introduce a variant of our model called KiloOSF that represents the scene with a large number of independent and small MLPs. KiloOSF represents each object with thousands of small MLPs, cach responsible for a small portion of the object, making cach individual MLP sufficient. for high-quality rendering.
Specifically, we subdivide each object into uniform grid of resolution = (sx, Sy, s.). Each grid cell is of 3D index i = (iz, by iz). We define a mapping m from position x to index i through spatial binning as follows:
m(x) [x buiin) / (bmax
-
(9)
6

---

Published in Transactions on Machine Learning Research (05/2023)
where Dmin and 1max are the respective minimum and maximum bounds of the axis-aligned bounding box (AABB) enclosing the object. For each grid cell, a small MLP neural net with parameters (i)) is used to represent the corresponding portion of the object. Then, the radiance transfer and density values at a point x can be obtained by first determining the index m(x) responsible for the grid cell that contains this point, then querying the respective small MLP:
[N)((())) (x, Wlihtt Wout) - (p,
(10)
Following KiloNeRF, we use a "training with distillation" strategy. We first train an ordinary OSF model for each object and then distill the knowledge of the teacher model into the KiloOSF model. We also use empty space skipping and early ray termination to increase rendering efficiency (Reiser et al., 2021).
3.6 Composing Multiple Learned OSFs
We show how we use learned OSEs for visually plausible scene composition. Due to the complexity of the problem and the approximations made by OSFs, physical correctness may not be guaranteed in the following discussion. Below we present a simple and computationally tractable approximation for composing multiple learned OSFs. We leave the design of physically correct scene composition algorithms as future work.
We restrict our discussion to a scene with multiple objects represented by OSFs, and we assune a single distant light source to illuminate the scene. In this case, the light distribution in Eq. (3) can potentially contain two parts: one for direct lighting, and the other for indirect lighting. We discuss them separately below, and then provide an algorithm for rendering a composed scene.
Direct lighting. Direct lighting is assumed to be a distant light, so the major point to consider is solving visibility. Notice that although OSFs allow reconstructing translucent object appearances, advanced light transport effects including transmitted lights or caustics in this work, but only consider shadows caused by light visibility. While solving for hard visibility can be tricky for OSFs, because they do not have the concept of surface just like NeRFs, the transmittance T in Eq. (1) can be seen as a soft measure for visibility. Therefore, we use a shadow ray rshadow (t) = X - tWiight to solve for visibility at x. We volume-render this shadow ray for all other OSFs in the scene with compositional rendering (Niemeyer & Geiger, 2021), and use the obtained transmittance T(t) = exp(- Jr, (Fshadow (s))ds) to serve as visibility: Lairect = To, where abuse ty to denote the far bound of the shadow ray and Lo to denote the light radiance. Given Ldirect, we compute outgoing radiance using Eq. (7).
do not model
Notice that this is a biased estimation for translucent objects, because the visibility is solved by the ray cast from a single point x, while the cumulative transfer function & assumnes the full light visibility for the object surface (Figure 2). Therefore, if a translucent object is partially occluded from the distant light, the outgoing radiance is underestimated for those points casting occluded shadow rays, and overestimated for those points casting unobstructed shadow rays. This also happens for concave opaque objects due to global transport. But for ideally learned OSFs of convex opaque objects (i.e., they degenerate to BRDFs) as discussed in section 3.3, the estimation is unbiased.
Indirect lighting. Besides direct lighting, we also want to account for indirect lighting in evaluating Eq. (3). To do this, we use Monte Carlo integration with uniform sampling of solid angles. Specifically, we cast a ray from to the sampled direction, and volume-render the ray using Eq- (1) obtain the incoming radiance. Due to high computational cost, only consider one-bounce indirect lighting. Similar to direct lighting, the estimation is biased for translucent objects, because P assumes distant light, while inter-reficction from objects do not meet this assumption. Nonetheless, our experiments show that our approximation can create reasonably faithful scene composition results in Sec. 4.3.
Excluding non-intersecting rays to accelerating scene composition. The standard rendering proce- dure must be repeated per pixel to render an image. The total cost of rendering single image with Npixel pixels and Nobject objects is thus
((Npixel Nsample "kample-Nhoject ) (Night Nsampte Nobject) dd-1)
(11)
7

---

Published in Transactions on Machine Learning Research (05/2023)
(a) Primary ray sampling
(b) Shadow ray sampling
Figure 3: Sampling procedure. (a) Scene with a camera, light source, and object bounding boxes. Primary rays are sent from the camera inlo the scene. Rays that do not intersect with objects are pruned. Of the intersecting rays, we sample points within intersecting regions. (b) Shadow rays from each sample are sent to the light source, and samples within intersocting regions are evaluated.
where Naample denotes the number of samples per ray-object pair, Night denotes the number of light sources, and d denotes the number of light bounces we model. There are many samples for which we must query OSFs, which can result in significant rendering time. In addition KiloOSF, we further accelerate scene composition by excluding non-intersecting rays. Specifically, we implement sampling procedure that pre- cludes the evaluation of rays that do not intersect with objects. Our sampling procedure assumes access to (rough) bounding box dimensions for each object. In practice, such bounding box can be automatically computed from trained OSF by extracting the object's 's bounding volume (using the predicted alpha values). As shown Fig. 3, for each ray and each object of interest, we intersect the ray with the object's bounding box. If the ray does not intersect with the object's bounding box, then no computation required for the ray-object pair. If the ray does intersect with the object's bounding box, we adopt the intersection points the near and far sampling bounds for the ray-object pair. Thus the N (number of primary rays) and Niight (number of secondary rays) are upper bounds on the number of rays that need to be evaluated. In practice, single ray often only intersects with at most one object in the scene, which means that the proposed rendering procedure is not significantly more expensive than the single object setting.
Compared with traditional volumetric path tracing methods, our OSF model does not require running path tracing wilhvnn each object (by evaluating itself during secondary ray computations) to simulate intra-object light bounces, because it learns the object-level scattering function that directly predicts the effects after all light bounces (reflcctions) and occlusions (shadows) within an object have occurred. Thercfore, OSFs are faster than alternative methods such NRFs (Bi et al., 2020a), which relies on simulating intra-object light bounces while querying its learned BRDF model.
4 Experiments
We validate our approach by free-viewpoint relighting on both synthetic and real-world objects, including translucent and opaque ones. Furthermore, we also demonstrate that OSFs allow visually plausible scene composition by showcasing a scene with both synthetic and real objects. We leave implementation details and ablation studies in supplementary material.
4.1 Data and baselines
In our experiments, we use six synthetic objects, two captured real translucent objects and five real opaque objects from the Diligent-MV dataset (Li et al., 2020).
Diligent-MV dataset (Li et al., 2020). This datasct provides multi-view photometric images of five objects that feature different levels of shininess. For each object, there are 20 views forming a circle around the object. For each view, there are 96 calibrated light sources spatially fixed relative to the camera. We use
8

---

Published in Transactions on Machine Learning Research (05/2023)
COSLA
Ground truth OSF (ours)
O-NeRF
PhySG
NeRD
Ground truth OSF (ours)
O-NeRF
PhySG
NeRD
Figure 4: Results on free-viewpoint relighting for real translucent objects captured by a cellphone.
Ground truth OSF (ours)
O-NeRF
PhysG
IRON
Ground truth OSF (ours)
o-NeRF
PhySG
IRON
Figure 5: Froo-viewpoint relighting for real opaque objects from Diligent-MV (Li et al., 2020).
front view and back view (view 10 and view 20) as the testset, and the other 18 views the training set. Note that the light directions in the training set are completely disjoint from the testset since the light sources with the camera.
Real image capture setup. We use an easily accessible image-capturing setup, where we take photos with two iPhone 12 in dark room. We use the flash of a cellphone Pright as a distant light source. We position Plight such that is far away from the object (the distance to the object is about 10 times compared to the object diameter). We use the other cellphone Pimage as camera without using its flash. We capture images from random viewpoints for single random light direction, and we repeat this for 10 different light directions. This gives dataset of 200 images from different viewpoints. We split our dataset such that all 20 images of randomly chosen light direction are held out for testing and all other images for training, We use standard Structure from Motion (SIM) method, COLMAP (Schonberger & Frahm, 2016), to solve for camera poses for all images. To calibrate light position, we lake a photo using Pright for every light position, and solve the camera pose together with other captured images. Photos taken by Plght are only used for light direction calibration but not for training. In order for SIM work well in the dark room, we place highly-textured newspaper pieces surrounding the captured object to provide robust features and use foreground segmentation network (Qin al., 2020) to extract clean object images after SIM.
9

---

Published in Transactions on Machine Learning Research (05/2023)
Ground truth OSF (ours)
O-NeRF
PhySG
NeRD
Ground truth OSF (ours)
O-NeRF
PhySG
NeRD
Figure 6: Pree-viewpoint relighting for synthetic translucent objects from ObjectFolder.
PSNR1
SSIM1
LPIPS$
PSNRT
SSIM1
LPIPSL
PSNRT
SSIMf
LPIPS4
o-NeRF
31.36
0.941
0.038
o-NeRF
23.69
0.811
0.137
o-NeRF
17.63
0.725
0.331
PhySG
30.09
0.944
0.046
PhySG
27.19
0.923
0.043
PhySG
20.09
0.823
0.171
IRON
14.80
0,907
0.081
NeRD
24.39
0.892
0.125
NeRD
21.94
0.815
0.182
OSF (ours)
39.07
0.970
0.020
OSF (ours)
38.99
0.982
0.006
OSF (ours)
27.06
0.865
0.051
Table 3:
Freo-viewpoint relight- ing real opaque objects from Diligent-MV (Li et al., 2020).
Table 1: Pree-viewpoinl relighting on synthetic translucent objects.
Table 2: Free-viewpoint relighting on real translucent soaps.
Synthetic images. We use 5 synthetic objects from the ObjectFolder (Gao et al., 2021) dataset, well Stanford bunny. We assign translucent materials to half of them and opaque materials to the other half. For each synthetic object, we render 1,000 images, each of which is under a random light direction and a random viewpoint. We sample cameras uniformly on an upper hemisphere, and light directions uniformly on solid angles. We use 500 images for training and 500 for testing. We generate synthetic images in Blender 3.0, using the Cycles path tracer. We use the Principled BSDF (Burley, 2012) with Christensen-Buut Burley approximation to the physically-based subsurface volume scattering (Christensen, 2015). We also evaluate OSFs on different levels of translucency and two-light setting in our supplementary material.
Baselines. We consider the following baselines:
o-NeRF (Mildenhall et al., 2020): We train a NeRF for each object and refer to it as o-NeRF. Since o-NeRF is agnostic to lighting and cannot do relighting, we show view synthesis results.
PhySG (Zhang el al,, 2021b): A recent representative neural inverse rendering method that jointly estimates lighting, materials, and geometry from multi-vien images and masks of shiny objects.
NeRD (Boss et al., 2021a): A recent representative neural relightable represcntation that decomposes the appearance into a neural BRDF field and environment lighting.
IRON (Zhang et al,, 2022): A state-of-the-ane neural inverse rendering method. Note that IRON assumes collocated lighting for training images and assumes little sclf-shadow in training images. These assumptions do not hold for most of our data, we only test IRON on Diligent-MV dataset which has less self-shadows.
We use the following standard metrics in all our comparisons: peak signal-to-noise ratio (PSNR), structural similarity (SSIM) (Wang et al., 2003), and perceptual metric LPIPS (Zhang et al., 2018).
4.2 Free- Viee-Vie Relighting
OSFs take light directions as input in addition to the camera viewpoint. simultaneous relighting and novel view synthesis.
Thus, inherently supports
10

---

Published in Transactions on Machine Learning Research (05/2023)
Ground truth OSF (ours)
O-NeRF
PhySG
NeRD
Ground truth OSF (aurs)
O-NeRF
PhySG
NeRD
Figure 7: Free-viewpoint relighting for synthetic opaque objects from ObjectFolder (Gao et al., 2021).
Composed scene
PSNR
SSIM
LPIPSJ,
o-NeRF
20.93
0.817
0.181
PhySG
24.61
0.901
0.088
NeRD
21.12
0.866
0.106
OSF (ours)
34.26
0.923
0.034
New viewpoint
Table 4: opaque objects.
Free-viewpoint relighting on synthetic
PSNR
SSIM
LPIPS,
o-NeRF
17.21
0.678
0.244
OSF (ours)
35.57
0.927
0.031
Relighting
Free- viewpoit relighting results for scene composition.
Table 5:
FPS
SSIM
LPIPSL
OSF
0.27
0.923
0.034
KiloOSF
16.13
0.938
0.037
Ground truth
OSF (ours)
O-NeRF
Figure 8: Scene composition for translucent bowl, an opaque airplane, and an opaque floor.
Table 6: Evaluation of KiloOSF on relighting synthetic opaque objects.
Translucent objects. We show qualitative (Figure 6 and Figure 4) and quantitative (Table and Table 2) comparisons on both synthetic and real transhucent objects. Our approach significantly outperforms the baseline methods. It successfully models the material and shape of the objects, and accurately relights them from variod vicwpoints compared to the ground truth. In contrast, o-NeRF fails due to its assumption on fixed illumination. Methods that parameterize object materials by BRDFs, such as PhySG and NeRD, fail produce subsurface scattering effects. Moreover, since they assume opaque surfaces, subsurface scattering effects in the training images seem to be explained by geometry, producing inaccurate goometry reconstruc- tion (e.g., the bowl in Figure 6). Notably, for the real soaps, OSFs can produce correct highlights on the soap surface well as subsurface scattering effects (e.g., notice how the shading changes non-uniformly along the fracture surface of the orange soap in Figure 4). Furthermore, only OSI's can reproduce the text on the soaps.
Opaque objects. We show freo-viewpoint relighting results for real objects from Diligent-MV dataset in Figure (we drop NeRD it has convergence issue this dataset despite best efforts) and
11

---

Published in Transactions on Machine Learning Research (05/2023)
Reference
Ours
Ours (indirect only) Ours w/o indirect Ours w/o ind. or shadow
Analysis of light transport effects including shadow and indirect lighting. Note that here we generate the ray-traced reference image with only two light bounces as we only compute two light bounces in our composition. Images are tone-mapped sRGB space for display.
Figure 9:
synthetic opaque objects in Figure 7. We observe that OSFs produce faithful free-viewpoint relighting results, correctly synthesizing hard self-shadows (e.g., the airplane in Figure 7 and the bear in Figure 5) and highlight reflections (e.g., the buddha's belly and the cow in Figure 5). In contrast, baseline methods do not senthesize these effects well. PhySG produces some self-shadows, while it does not reconstruct geometries well on Di objects. IRON assumes collocated lighting and it assumes little self-shadows in training images. Thus it does not reconstruct or relight the objects correctly. We show quantitative results in Table 4 and Table 3, where OSFs significantly outperform all baselines.
4.3 Scene Composition
Since OSFs object-centric representations that model light transport for object, they allow trained models to be readily composed into new scenes. To showcase neural scene composition, use translucent and an opaque object with randomized poses, lighting directions, and viewpoints. We show relighting and view synthesis of this example scene in Figure 8. We compare with the o-NeRF baseline. Qualitatively, while o-NeRF produces incorrect shadows and artifacts, our method generates more realistic compositions that better match the ray-traced groundtruth images. We also show quantitative results Table 5, where we observe that OSFs significantly outperform o-NeRF.
Light transport analysis. To analyze the light transport effects including shadows and indirect lighting in our composition method, we showcase simple scene consisting of grey cube and red floor in Figure 9, where we show our scene composition and variants ablating shadows and indirect lighting. From Figure 9 we observe increased realism when we add shadow and indirect lighting. Specifically, we see a color bleeding effect from the red floor onto the cube by the indirect lighting. With these light transport effects, it corroctly synthesizes the scene up two light bounces (i.e., direct lighting and one-bounce indirect lighting). While here we showcase simplistic scene and compute bounce for indirect lighting due to computational constraints, our method may benefit from future advances in efficient neural rendering to scale to more complex scenes and more light bounces.
4.4 Evaluation on KiloOSF
We introduce KiloOSF for accelerating OSF rendering. Thus, in this section, we evaluate KiloOSF terms of rendering speed and relighting performance. To this end, we test KiloOSF on all synthetic translucent objects. We show the rendering speed (frames per second) and the rendering quality in Table 6. KiloOSF achieves 60x (16.13 vs. 0.27 FPS) speed up while maintaining comparable relighting performances, suggesting the benefits brought by the KiloOSF design.
5 Conclusion
We presented novel, learning- based object representation Object-Cennte Centric Neural Scattering Functions (OSFs). An OSF models a cunulative radiance transfer function, allowing both free-viewpoint relighting
12

---

Published in Transactions on Machine Learning Research (05/2023)
and scene composition. Moreover, OSFs model objects of complex shape and materials, and can relight both opaque and translucent objects. Our work offers new promising neural graphics method for modeling real-world scenes.
Limitations. The major limitation of our model comes from the assumption of unobstructed distant lighting, which leads biased estimation in, c.g., computing inter-reflcctinn especially considering shiny objects. real capture, distant light can be approximated by holding flashlight far away from small objects, but. for big objects this is difficult. We envision wider applications in future research that relaxes the assumption.
This work is in part supported by Google, NSF RI #2211258, ONR MURI N00014- 22-1-2740, Stanford Institute for Human- Centeree Artificial Intelligence (HAI), Amazon, Bosch, Ford, and Qualcomm.
Acknowledgments.
References
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmiiry Ulyanov, and Victor Lempitsky. Neural point- based graphics. Europeon Conference on Computer Vision, 2020.
Hendrik Baatz, Jonathan Granskog, Marios Papas, Fabrice Rousselle, and Jan Novák. Nerf-tex: Neural reflectance field textures. In Computer Graphics Forum. Wiley Online Library, 2021.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855 5864, 2021a.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360:
Unbounded anti-aliased neural radiance fields. arXiv preprint arXiv:2111.190 12077, 2021b.
Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold- Geoffroy, David Kriegman, and Ravi Ramamoorthi. Neural reflectance fields for appearance acquisition. arXiv preprint Xxxi:2008.0388. 2020a.
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi. Deep reflectance volumes: Relightable reconstructions from multi-view photometric images, European Conference on Computer Vision, 2020b.
Adrian Blumer, Jan Novák, Ralf Habel, Derek Nowrouzezahrai, and Wojciech Jarosz. Reduced aggregate scattcring operators for path tracing. In Computer Grophics Forum. Wiley Online Library, 2016.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectance decomposition from image collections. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 12684 12694,, 2021a.
Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan Barron, and Hendrik Lensch. Neural-pil: Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing Systems, 34, 2021b.
Brent Burley. Physically-based shading at disney. In Acm Siggraph, 2012.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint ar
2015.
Per H. Christensen. An approximate reflectance profile for efficient subsurface scattering. In ACM SIG- GRAPH 2015 Tolks. 2015.
Paul Debevec, Tim Hawkins, Chris Tehou, Haarm-Picter Duiker, Westley Sarokin, and Mark Sagar. Ac- quiring the reflectance field of human face. In Proceedings of the 27th onnual conference on Computer graphics and interactive techniques, pp. 145-156, 2000.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Ruohan Gao, Yu Chang, Shivani Mall, Li Fei-Fei, and Jiajun Wu. Objectfolder: A dataset of objects with implicit visual, auditory, and factile representations. In CoRL, 2021.
Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High- fidelity neural rendering at 200fps. arXiv:2103. 10380, 2021.
Gaurav Garg, Eino-Ville Talvala Marc Levoy, and Hendrik PA Lensch. Symmetric photography: Exploiting dala-sparseness in reflectance fields. In Rendering Techniques, pp. 251 262, 2006.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. Deep blending for free-viewpoint image-based rendering. ACM Transactions Graphics, 37(6):1-15 15, 2018.
Henrik Wann Jensen, Stephen R Marschner, Mare Levoy, and Pat Hanrahan. A practical model for subsur- face light transport. Proceedings of the 28th annual conference on Computer graphics and interactive techniques, 2001.
James Kajiya and Brian Von Herzen. Ray tracing volume densities. SIGGRAPH, 1984.
Simon Kallweit, Thomas Müller, Brian Mcwilliams, Markus Gross, and Jan Novák. Deep scattering: Render- ing atmospheric clouds with radiance-predictin neural networks. ACM Transactions on Grophics (TOG), 36(6):1-11, 2017.
Alexandr Kuznetsov. Neumip: Multi-resolution neural materials. ACM Transactions Graphics (TOG), 40(4), 2021.
Richard Lee and Carol O'Sullivan. Accelerated light propagation through participating media. In VG@ Eurographics, pp. 17-23, 2007.
Jaakko Lehtinen. A framework for precomputed and captured light transport. ACM Transactions on Graphics (TOG), 26(4):13-cs, 2007.
Min Li, Zhenglong Zhou, Zhe Wu, Boxin Shi, Changyu Diao, and Ping Tan. Multi-view photometric stereo: A robust solution and benchmark dataset for spatially varying isotropic materials. IEFF Transactions Image Processing, 2020.
Lingjic Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Thcobalt. Neural sparse voxel fields. Advances in Newral Information Processing Systems, 2020.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics, 38 (4):1-14, 2019.
Ricardo Martin Brraal,, Rohit Pandey, Shuoran Yang, Pavel Pidlypenskyi, Jonathan Taylor, Julien Valentin, Sameh Khamis, Philip Davidson, Anastasia Tkach, Peter Lincoln, et al. Tookingood: Enhancing perfor- mance capture with real-time neural re-rendering. ACM Transactions Grophics, 2018.
Nclson Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108, 1995.
Nelson Max and Min Chen. Local and global illumination in the volume rendering integral. Technical report, Lawrence Livermore National (LLN(L.. Livermore, CA (United States), 2005.
Johannes Meng, Marios Papas, Ralf Habel, Carsten Dachsbacher, Steve Marschner, Markus H Gross, and Wojeiech Jarosz. Multi-scale modeling and rendering of granular materials. ACM Trans. Graph., 34(4): 49-1 2015.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Conference on Compuler Vision and Pattern Recognition, pp. 4460 4470, 2019.
Moustafa. Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla, Neural rerendering in the wild. In Conference on Compuler Vision and Pattern Recognition, pp.
2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view syuthesis. In European Conference on Computer Vision, 2020.
Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan, and Jonathan T Barron. Nerf in the dark: High dynamic range view synthesis from noisy raw images. arXiv preprint arXiv:2111.19679, 2021.
Jonathan T Moon, Bruce Walter, and Stephen R. Marschner. Rendering discrete random media using precom- puted scattering solutions. In Proceedings of the 18th Eurographics conference on Rendering Techniques, pp. 231-242, 2007.
Giljoo Nam, Joo Ho Lee, Diego Gutierrez, and Min H Kim. Practical svbrdf acquisition of 3d objects with unstructured flash photography. ACM Transactions on Grophics (TOG), 2018
Michael Niemeyer and Andreas Geiger. Girafle: Representing scenes compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11453-11444 11464, 2021.
Jan Novák, Ilyyan Georgiev, Johannes Hanika, and Wojeiech Jarosz. Monte carlo methods for volumetric light transport simulation. In Computer Graphics Forum. Wiley Online Library, 2018.
Michael Oechsle, Songyou Peng, and Andreas Geiger. Unisurf: Unilying neural implicit surfaces and radiance fields for multi-view reconstruction, 2021.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 2856-2865, 2021.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Matt Pharr, Wenzel Jakob, and Greg Humphreys. Physically based rendering: From theory to implementa- lion. Morgan Kaufmann, 2016.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pallern Recognilion, 106: 107404, 2020.
Ravi Ramamoorthi. Precomputation- rendering. NOW Publishers Inc, 2009.
Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14335 14345, October 2021.
Johannes L Schonberger and Michacl Frahm. Structure-frmmmmotte revisited. Proceedings of the IEFE conference computer vision and pattern recognition, pp. 4104-4113, 2016.
15

---

Published in Transactions on Machine Learning Research (05/2023)
Anthony Simeonov, Yihun Du, Andrea Tagliasacchi, Joshua B Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann. Neural descriptor fields: Se (3)-equivariant object representations for manipulation. In CoRL, 2021.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. Deepvoxels: Learning persistent 3d feature embeddings. In Conference on Computer Vision and Pattern Recognition, pp. 2437-2446, 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-aware neural scene representations. In Advances in Neural Information Processing Systems, pp. 1121-1132, 2019b.
Peter-Pike Sloan, Jan Kautz, and John Snyder. Precomputed radiance transfer for real-time rendering in dynamic, low-froquency lighting environments. In Conference on Computer Graphics and Interactive Techniques, pp. 527 536, 2002.
Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron. Nerv: Neural reflectance and visibility fields for relighting and view syuthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognittio,, pp. 7495-7504, 2021.
Tiancheng Sun, Jonathan Barron, Yun-Ta Tsai, Zexiang Xu, Xueming Yu, Graham Fylle, Christoph Rhemann, Jay Busch, Paul E Debevec, and Ravi Ramamoorthi. Single image portrait relighting. ACM Transactions on Graphics, 38(4):79 2019.
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, et al. State of the art on neural rendering. In Computer Graphics Forum. Wiley Online Library, 2020.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Transactions on Graphics, (0(4):1-12, 2019.
Bill Triggs, Philip F McLauchlan, Richard I Hartley, and Andrew W Fitzgibbon. Bundle adjustment modern synthesis. In International Workshop on Vision Algorithmms, pp.
Springer, 1999.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan. Ref- nerf: Structured view-dependent appearance for neural radiance fields. ar preprint arXiv:2112.03907, 2021.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assess- ment. In Asilomar Conference on Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402. IEEE, 2003.
Zexiang Xu, Kalyan Sunkavalli, Sunil Hadap, and Ravi Ramamoorthi. Deep image- optimal sparse samples. ACM Transactions Graphics, 37(4):1-13, 2018.
image-based based relighting from
Bangbang Yang, Yinda Zhang, Yinghao Xu, Yijin Li, Han Zhou, Hujun Bao, Guofeng Zhang, and Zhaopeng Cui. Learning object-compositional neural radiance field for editable scene rendering. In International Conference on Computer Vision (ICCV), October 2021.
Alex Yu, Ruilong Li, Matthow Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrocs for real-time rendering of neural radiance fields. In ICCV, 2021.
Hong-Xing Yu, Lconidas J Guibas, and Jiajun Wu. Unsupervised discovery of object radiance fields. In ICLR, 2022.
Jason Zhang, Gengshan Yang, Shubham Tulsiani, and Deva Ramanan. Ners: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. Advances in Newral Information Processing Systems, 34, 2021a.
Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. NERF++: Analyzing and improving neural radiance fields. http://wrrii.og/bb/////////
2020.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition, 2021b.
Kai Zhang, Fujun Luan, Zhengqi Ti, and Noah Snavely. Tron: Inverse rendering by optimizing neural sdfs and materials from photometric images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pallern Recognilion, pp. 5565 3565-5574, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as percept metric. In Conference Computer Vision and Pollern recognilion, pp- 586-595, 2018.
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor:
Neural factorization of shape and reflectance under an unknown illumination. ACM Transaclions on Graphics (TOG), 40(6):1 18, 2021c.
Quan Zheng, Gurprit Singh, and Hans-Peter Seidel. Neural relightable participating media rendering. Ad- vonces in Neural Information Processing Systems, 34:15203-1522 2021.
Hao Zhou, Sunil Hadap, Kalyan Sunkavalli, and David W Jacobs. Deep single-image portrail relighting. In International Conference on Computer Vision, pp. 7194-7202, 2019.
Zhenglong Zhou, Zhe Wu, and Ping Tan. Multi-view photometric stereo with spatially varying isotropic materials. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2013.
17


Reviews:
[{"review_id": "afQSAMHutH", "reviewer": "Reviewer_sjM8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper Strengths\n\nI appreciate the effort to tackle the challenging problem of translucent object simulation. \n\nThe introduction section provides a clear motivation for the method.\n\nAn efficient computational model based on KiloNerf is incorporated as well.\n\nPaper Weaknesses\n\nThere should be an effort to make the formulation in the method section clearer. \nSome examples:\nWhat does {} stands for in equation (2)? \n\nIn equation (1), r is a function of t, but t is also the integrand variable. So, what is L(r) in the LHS?\n\nIn eq (8), what are the loss parameters to be minimized?\n\n\nMore Evaluation is needed.\n\nThe evaluation could be more extensive. It would be beneficial to evaluate the method on popular benchmarks such as the DTU MVS repository (Jensen et al. ).\nMoreover, it seems that the baselines chosen cannot overfit specific details in the input images (iike the captions in figure 5).  It is unclear to me why NerF would fail to overfit these details, as I can imagine it can “encode” these details in the color network for specific views and only fail in generalization to novel views. It would be beneficial to provide rendering results for training images as well.\n\nMissing discussion about the method\n\nIt seems that the main idea suggested is to replace the computationally intractable integral in equation (6) by evaluating it on one predefined light view. What are some immediate limitations that such an approximation yields? It would be beneficial to discuss and provide some examples.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the challenge of scene reconstruction from 2D image inputs. Following the success of NerF-based models, the paper suggests extending the underlying rendering procedure of NerF to take into account also the rendering of translucent objects. To do so, subsurface scattering effects are modeled by predicting the cumulative radiance in a specific view as an integration over all possible directions. \nThe method is evaluated on both synthetic and real captured scenes.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the above weakness, they list some comments to be considered in the next revision. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4HuJCVjWf0", "reviewer": "Reviewer_J5zP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is overall well written and straightforward to follow The draft is in good shape and the supplementary video is an excellent summary of the proposed work.\n- The approach is well motivated. Explicitly modeling subsurface scattering is difficult and existing neural field formulations do not propose ways of handling this, making them unsuitable for translucent objects. Using a neural network to model the complete process of how an object responds to a known light source makes sense.\n- In addition to proposing how to estimate cumulative radiance transfer with a neural network, the method contains additional improvements: modeling indirect lighting, an efficient ray sampling procedure, and a KiloNeRF-based extension that allows for fast inference.\n\nWeaknesses:\n- The proposed approach requires both images with known viewpoints and known light source directions. In the context of prior neural field works this limitation is OK given how difficult it is to model translucency. This requirement requires a more complex capture process and limits the method to indoor scenes of objects.\n- The real world evaluation is limited: this is done only on two simple, mostly convex candy-like objects. While it is clear that the proposed approach outperforms prior work, real world evaluation on objects with more complex geometry would help us better understand the limitations of the current approach.\n- Synthetic evaluation can be improved: how were the objects used for evaluation chosen, and how exactly were the translucent objects made to be translucent? What specific material parameters were chosen and why? The power of synthetic data is the fine grained control over aspects of the visual world that affect how a model learns to represent it. It would be very useful to vary the translucency parameters of objects in the synthetic domain and understand how well the proposed approach can model them and potentially identify some failure modes. This would be extremely useful for future work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a new neural object representation that models translucency and allows for novel view synthesis, relighting, object composition. Modeling translucency is the main distinction, which is possible due to a cumulative radiance transfer function formulation to model the appearance of objects. This formulation uses the neural network itself to implicitly learn the complex subsurface scattering properties of an object by modeling output radiance at a spatial location as a function of the radiance and direction of a distant light source and the outgoing light direction. This makes it possible to learn this representation from calibrated 2D images and light sources with known positions, without additional information about the shape of the object or its material properties.\n\nQualitative and quantitative evaluation is done on 5 synthetic objects from ObjectFolder and 2 real objects. The proposed method outperforms the standard NeRF and additional baseline methods PhySG and NeRD.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n- Real world evaluation on a 2-3 more objects with more complex geometry. This is important for understanding the limitations of the proposed method.\n- A detailed description of the synthetic data generation process (could go in the supplement) and an explanation behind the design decisions for this process (e.g. how were objects chosen, how were camera and lighting positions chosen, how were the translucent object materials implemented)\n\nChanges that would strengthen the work:\n- Carefully using the synthetic data to understand the connection between object geometry, translucency and how well the proposed approach can model them. \n", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rituZECybK", "reviewer": "Reviewer_QtbL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed framework is novel, and addresses an interesting problem. The different components of the system are reasonably motivated, and explained clearly.\n\nPre-integrating the radiance is not a new idea (as the paper notes); however I am not aware of any work that applies it in the exact setting here. It enables much faster rendering than \"more local\" representations such as (neural or traditional) BSDFs, which require tracing many light-paths when rendering.\n\nThe text is generally balanced, and explicitly acknowledges various limitations and their impact in the main text (e.g. strong assumptions on the incident lighting). I particularly appreciate this given the tendency of many works to 'bury' such information at the end!\n\nThe work of Zheng et al. (2021) is mentioned in sec. 2, but it is not very clear why exactly the proposed work is better than theirs (which is quite similar in task/setting). Please add either a theoretical justification (e.g. inability of theirs to model certain effects) or an empirical one (e.g. proposed is faster / more accurate).\n\n\"IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images\" [Zhang, CVPR '22] is very relevant, and I think SOTA for the multi-view multi-light setting considered in the paper (but it doesn't support subsurface scattering). It should at least be cited, and maybe compared against.\n\nThe quantitative evaluation appears to use appropriate protocols and metrics. However, only five synthetic objects are used, and it's not clear how these were selected (the underlying dataset is larger). Similarly, there are very few qualitative examples shown -- particularly for scene composition. Those which are present, demonstrate good performance, but it is difficult for the reader to get an idea of how well the method works in general on diverse objects/scenes.\n\nFor the real data captured with two phones, the train/evaluation split is done by holding out all viewpoints with particular light directions. This is clearly sufficient when measuring performance on relighting. However, when considering novel view synthesis, it is important that the camera poses of the held-out images are not too close to those from the training split -- there is no indication of whether this is the case, save for the statement that the camera was moved randomly.\n\nA benchmark dataset such as Diligent-MV (typically used for multi-view photometric stereo) should be used to give a more extensive (and standardised) evaluation of performance on real-world images, at least in the case of opaque surfaces.\n\nIt is mentioned in sec 3.5 that when composing multiple objects in close proximity, indirect illumination effects are not accurately captured by assuming infinite distance sources of radiance. This would be particularly true if the objects were more glossy, and hence sharper reflections of nearby objects should be seen. This seems like a significant limitation of the proposed approach. I'd be happier if the method were claimed simply to model individual objects under certain lighting conditions; however the fact that the composability is called out as an explicit benefit makes this more problematic (since the chosen representation is not so great for this).\n\nThe qualitative results on composition in fig. 8 are not particularly impressive, in terms of level of detail and realism of the composition (however, this is difficult to evaluate rigorously given the highly-compressed / low-resolution images). More problematically, the \"indirect only\" results appear very wrong. For example on the top surface of the highest object, there should be zero indirect contribution; similarly on all surfaces that are entirely (self-)occluded from other objects, there should also be zero indirect illumination. However, the figure seems to show a fairly uniform indirect component over most surfaces, regardless of orientation. It would be much more convincing if a scene were shown where some stronger indirect illumination effects should be seen -- c.f. the \"Cornell box\" with some strongly-colored object adjacent to a white one, showing clear scatter onto the latter.\n\nThe PSNR values in sec. 4.4 / tab. 5 seem to be wrong -- they are not in the correct range (by a large factor).\n\nA small weakness of the overall approach is that it assumes incident light directions are known. In the real-world experiments authors solve this by using a phone for the light, and taking a photo; however in practical capture scenarios the light-source may not have a camera attached. It would therefore be valuable to measure how sensitive the method is to errors in estimation of the lighting angles (e.g. by adding increasing levels of gaussian noise to the angle, and measuring how PSNR is affected). Moreover, the lights are assumed to be infinitely-distant (i.e. purely directional) sources. Again this is not satisfied in real-world imaging conditions, and it would be valuable to determine how sensitive the model is to this (rather than merely observing that it works 'well enough' with a phone flash at a certain distance).\n\nThe paper is well-written and pleasant to read throughout. The structure is fine, and figures/etc. are legible and appropriate.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a neural-field approach to capturing relightable 3D representations of objects. The proposed approach uses an MLP to map position, (incident) light direction, and camera/view direction to a radiance and density; the radiance measures the total (cumulative) radiance emitted at the relevant point/angle due to a distant light-source in the given direction, including any light transported to the observed point via subsurface scattering and/or intra-object reflections. This 'pre-integration' of radiance contrasts with a traditional (or indeed neural) BSDF, and allows efficient rendering of the object (including subsurface effects) without integrating over light-paths. The proposed representation also differs from NeRF in that it includes a dependence on light direction, hence supports relighting. Additionally, the paper discusses an approximate method for composing multiple pre-captured objects together in a scene, accounting for (limited) global/indirect illumination effects. The method is demonstrated on several synthetic and real captures, where it out-performs a number of baselines quantitatively and qualitatively.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Critical\n\n- please provide higher-resolution / less-compressed versions of the renderings in supplementary, since it is currently difficult to make detailed judgements of quality/fidelity.\n\n- discuss and/or evaluate against the references mentioned above\n\n- evaluate on a standard dataset, e.g. Diligent-MV\n\n- provide some convincing examples of accurate compositions including indirect illumination\n\n- fix tab. 5\n\n### Non-critical\n\n- evaluate on more objects, or at least say how/why the specific synthetic objects for evaluation were chosen\n\n- I assume objects were pre-segmented / masked to remove background; however this doesn't seem to be stated. Please clarify somewhere\n\n- sec 3.6 seems to apply exclusively to the compositional setting discussed in sec 3.5 -- it would therefore be clearer if 3.6 were a subsubsection of 3.5. Alternatively 3.5 & 3.6 could be moved to a separate top-level section.\n\n- tab. 3-5 are very squeezed, above fig. 7 which has a lot of whitespace. Perhaps this page could be arranged more efficiently", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3uJRery7KI", "reviewer": "Reviewer_XktD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This work studies a novel, challenging setting beyond traditional NeRF that requires the model to support relighting. By considering the direction of the light source, OFS allows synthesis with both novel viewpoints and novel light sources.\n\n- The proposed OSF is a novel method that natively supports light transport effects like shadows.\n\n- This work shows successful modeling of translucent objects, which is a major challenge in previous work like NeRF and other neural inverse rendering methods.\n\n- By combining with the idea of KiloNeRF, an efficient version of OSF can be implemented to accelerate rendering, achieving 60x speed up with only slight loss of rendering quality.\n\nWeaknesses:\n\n- One limitation of the proposed setting which OSF studies is that it only allows **single distant light source**. In practice, there can be multiple light sources contributing to a more realistic scene, and they may be ambient or point light sources. Currently, OSF cannot model such light sources.\n\n- Another challenge lies in the real-world data collection process. To generate a training dataset for OSF, one has to capture images of objects in a dark room, and also calibrate the light position with SfM methods. This data collection pipeline is significantly more complicated than standard NeRF and requires more expertise. Synthetic data can be generated as easily as traditional NeRF though.\n\n- In Table 1, OSF performs slightly worse than the baseline PhySG in terms of the SSIM metric, which contradicts the claim (in the text) that “our approach significantly outperforms the baseline methods”. Though, OSF indeed outperforms other methods with a higher PSNR. It would be better if authors explain why the trend on SSIM is different from other metrics, or check if the computation of SSIM is inconsistent.\n\n- The dataset for evaluation is too simple. It does not include challenging visual effects like mirror reflections or light spots, even few view-dependent effects. I wonder if OSFs can still perform well with these effects.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes Object-Centric Neural Scattering Functions (OSFs), a novel formulation to model photorealistic object appearance from collections of 2D images with neural implicit models. By taking an incoming light direction and subsurface scattering effects into consideration, OSFs support modeling both opaque and translucent objects, and can relight object appearance using a novel light source. In addition, multiple learned OSFs can be composed together and render a complex scene of multiple objects. Photorealistic scenes are visualized and quantitative results are included to demonstrate the efficacy of OSFs in modeling object appearance and composing new scenes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In addition to the weaknesses described in the section above, the following changes can be made to strengthen this work:\n\n- The baseline o-NeRF does not take the light source into account. It would be better to explain how to relight with this method in the experiments.\n\n- It would be more convincing to generate more scene dataset with more challenging visual effects and evaluate them.\n\n- In Table 5, the PSNR of two methods are abnormally small compared to other tables (the MSE before calculating PSNR will be ~ 0.99). I wonder if there is a mistake in this table.\n", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 reviewers who have submitted their final recommendations have all agreed on accepting the paper. The 4th reviewer who has not submitted the final recommendation had similar initial concerns with the other 3. Based on the rebuttal and discussion, AE believes those concerns were addressed as well. Hence, AE recommends acceptance of this paper.
Decision: Accept (poster)