VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis
Authors: Angtian Wang
Peng Wang
Jian Sun
Adam Kortylewski
Alan Yuille
Venue: ICLR 2023 poster (2023)
Keywords: Differentiable Rendering
Analysis-by-Synthesis
Pose Estimation

Abstract: Differentiable rendering allows the application of computer graphics on vision tasks, e.g. object pose and shape fitting, via analysis-by-synthesis, where gradients at occluded regions are important when inverting the rendering process.To obtain those gradients, state-of-the-art (SoTA) differentiable renderers use rasterization to collect a set of nearest components for each pixel and aggregate them based on the viewing distance. In this paper, we propose VoGE, which uses ray tracing to capture nearest components with their volume density distributions on the rays and aggregates via integral of the volume densities based on Gaussian ellipsoids, which brings more efficient and stable gradients. To efficiently render via VoGE, we propose an approximate close-form solution for the volume density aggregation and a coarse-to-fine rendering strategy. Finally, we provide a CUDA implementation of VoGE, which gives a competitive rendering speed in comparison to PyTorch3D. Quantitative and qualitative experiment results show VoGE outperforms SoTA counterparts when applied to various vision tasks, e.g., object pose estimation, shape/texture fitting, and occlusion reasoning. The VoGE code is available at: https://github.com/Angtian/VoGE.

Published as a conference paper at ICLR 2023
VoGE:A DIFFERENTIABLE VOLUME RENDERER USING GAUSSIAN ELLIPSOIDS FOR ANALYSIS-BY- SYNTHESIS
Angtian Wang'. Peng Wang?, Jian Sun', Adam Kortylewski", and Alan Yuille!
Johns Hopkins University
ByteDDance Inc. 3 Max Planck Institute for Informatics
ABSTRACT
The Gaussian reconstruction kernels have been proposed by Westover (1990) and studied by the computer graphics community back in the 90s, which gives an alternative representation of object 3D geometry from meshes and point clouds. On the other hand, current state-of-the-art (SoTA) differentiable renderers, Liu rasterization to collect triangles or points each image pixel
et al. (2019), and blend them based on the viewing distance. In this paper, we propose VoGE, which utilizes the volumetric Gaussian reconstruction kernels geometric prim- itives. The VoGE rendering pipeline uses ray tracing to capture the nearest prim- itives and blends them as mixtures based on their volume density distributions along the rays. To efficiently render via VoGE, we propose an approximate close- form solution for the volume density aggregation and a coarse-to-fine rendering strategy. Finally, we provide CUDA implementation of VoGE, which enables real-time level rendering with competitive rendering speed in comparison PyTorch3D. Quantitative and qualitative experiment results show VoGE outper- forms SoTA counterparts when applied to various vision tasks, e,g., object pose estimation, shape/texture fitting, and occlusion reasoning. The code is available: https://gthub..com/ wtt.ggghhhb.com//giiiiiaaa//aa/ hatpr/nnnntttttttiee VoGE.
INTRODUCTIONS
Recently, the integration of deep learning and computer graphics has achieved significant advances lots computer vision tasks, egg. pose estimation Wang et al. (2020a), 3D reconstruction Zhang et al. (2021), and texture estimation Bhattad et al. (2021). Although the rendering quality of has significant improved over decades of development computer graphics. the differentiabiiity of the rendering process still remains be explored and improved. Specifically, differentiable renderers compute the gradients w.r.t. the image formation process, and hence enable to broadcast cues from 2D images towards the parameters of computer graphics models, such the camera parameters, and object geometries and textures. Such an ability is also essential when combining graphics models with deep neural networks. In this work, we focus on developing differentiable renderer using ex- plicit object representations, i.e. Gaussian reconstruction kernels, which can be either used separately for image generation or for serving as 3D aware neural network layers.
The traditional rendering process typically involves a naive rasterization Kato et al. (2018), which projects geometric primitives onto image plane and only captures the nearest primitive for each pixel. However, this process eliminates the cues from the occluded primitives and blocks gradients toward them. Also the rasterization process introduces a limitation for differentiable rendering, that rasterization assumes primitives do not overlap with each other and are ordered front to back along the viewing direction Zwicker et al. (2001). Such assumption raise a paradox that during gradient based optimization, the primitives order along viewing direction. Liu et al. (2019) provide a naive solution that tracks a set of nearest primitives for each image pixel, and blending them based the viewing distance. However, such
necessary to overlap with each other when they change the

---

Published as a conference paper at ICLR 2023
Rey Vertor
near
Kemel--n to Pixel Weight
(ree)-TO
Camen
Gavasien Ellipsoids
Syulhesized Imsge
Weigttof Kermess onr())
Attibots
Figure 1: VoGE conducts ray tracing volume densities. Given the Gaussian Ellipsoids, iee. a set of ellipsoidal 3D Gaussian reconstruction kernels, VoGE first samples rays r(t). And along each ray, VoGE traces the density distribution of each ellipsoid ps (r(t)) respectively. Then occupancy T(r(t)) is accumulated via density aggregation along the ray. The observation of each Gaussian ellipsoid kernels W₂ is computed via integral of reweighted per-kernel volume density W& (r(t)). Finally, VoGE synthesizes the image using the computed Wi on each pixel to interpolate per kernel attributes. In practice, the density aggregation is bootstrapped via approximate close-form solutions.
approach introduces an ambiguity that, for example, there is a red object floating in front of large blue object laying background. Using the distance based blending method, the cues of the second object will change when moving the red one from far to near, especially when the red object are near the blue one, which will give unrealistic purple blending color. In order resolve the ambiguity, we record the volume density distributions instead of simply recording the viewing distance, since such distributions provide cues on occlusion and interaction of primitives when they overlapped.
Recently, Mildenhall et al. (2020); Schwarz et al. (2020) show the power of volume rendering with high-quality occlusion reasoning and diffcrentiability, which benefits from the usage of ray tracing volume densities introduced by Kajiya & Von Herzen (1984). However, the rendering process in those works relies on implicit object representations which limits the modifiability and interpretabil- ity. Back in the 90s, Westover (1990); Zwicker et al. (2001) develop the splatting method, which reconstruct objects using volumetric Gaussian kernels and renders based on a simplification of the ray tracing volume densities method. Unfortunately, splatting methods were designed for graphics rendering without considering the differentiability and approximate the ray tracing volume densi- ties using rasterization. Inspired by both approaches, Zwicker et al. (2001); Liu et al. (2019), we propose VoGE using 3D Gaussians kernels to represent objects, which give soften boundary of prim- itives. Specifically, VoGE traces primitives along viewing rays density function, which gives probability of observation along the viewing direction for the blending process.
In VoGE rendering pipeline, the ray tracing method is designed to replace rasterization, and better blending function is developed based on integral of traced volume densities functions. As Figure shows, VoGE uses set of Gaussian ellipsoids to reconstruct the object in 3D space. Each Gaussian ellipsoid is indicated with a center location M, and a spatial variance Σ. During rendering, we first sample viewing rays by the camera configuration. We trace the volume density of each ellipsoiid as a function of distance along the ray respectively on each ray, and compute the occupancy along the ray via an integral of the volume density and reweight the contribution of each ellipsoid. Finally, interpolate the attribute of each reconstruction kernel with the kernel-to-pin cernel-to-pieel weights into an image. In practice, we propose an approximate close-form solution, which avoids computational heavy operation in the density aggregation without, e.g. integral, cumulative sum. Benefited from advanced differentiability, VoGE obtains both high performance and speed on various vision tasks.
In summary, the contribution of VoGE includes:
1. A ray tracing method that traces each component along viewing ray as density functions. VoGE ray tracer is replacement for rasterization.
2. A blending function based on integral of the density functions along viewing rays that reasons occlusion between primitives, which provides differentiability toward both visible and invisible primitives.
3. A differentiable CUDA implementation with real-time level rendering speed. VoGE can be easily inserted into neural networks via our PyTorch APIs.
4. Exceptional performance various vision tasks. Quantitative results demonstrate that VoGE significantly outperforms concurrent of differentiable renderer on in- wild object pose estimation tasks.
2

---

Published as a conference paper at ICLR 2023
Table I: Comparison with state-of-the-art differentiable renderers. Similar to NeRF but different from previous graphics renderers, VoGE uses ray tracing to record volume densities on each ray for each ellipsoid, and blends them with transmittance computed via volume densities.
Method
Representation
Primitives
Visibility Algorithm
Blending
Kato et al. (2018)
explicit
mesh
rasterization
none
Liu et al. (2019)
explicit explicit
mesh
rasterization
distance
Ravi al. (2020)
mesh/points
rasterization
distance
Yifan et al. (2019)
explicit
Gaussian
rasterization
distance
Lassner & Zollhofer (2021)
explicit
sphere
rasterization
distance
VoGE (ours)
explicit
3D Gaussian
ray tracing
transmittance
Mildenhall et al. (2020)
implicit
ray tracing
transmittance
2 RELATED WORKS
Volume Rendering. In the 1980s, Blinn (1982) introduces the volume density representation, which simulates the physical process of light interacting with matter. Kajiya & Von Herzen (1984) develop the ray tracing volume density aggregation algorithm, which renders the volume density via light scattering equations. However, obtaining the contiguous volume density function is infeasible in practice. Currently Mildenhall et al. (2020); Niemeyer et al. (2020); Mescheder et al. (2019); Genova et al. (2020); Zhang et al. (2019); Vicini et al. (2022). use implicit functions, e.g., neural networks, object representations. Though those implicit representations give satisfying performance, such representations are lacking interpretability and modifiability, which may limit their usage in analysis tasks. In this work, provide solution that utilizes explicit representation while rendering with the ray tracing volume density aggregation.
Kernel Reconstruction of 3D Volume. Westover (1990) introduces the volume reconstruction ker- nel, which decomposes a 3D volume into a sum of homogeneous primitives. Zwicker et al. (2001) introduces the elliptical Gaussian kernel and show such reconstruction gives satisfied shape approx- imation. However, both approaches conduct nonfdifferentiable rendering and use rasterization to approximate the ray tracing process.
Differentiable Renderer using Graphics. Graphics renderers use explicit object representations, which represent objects as set of geometry primitives. Table shows, concurrent differen- tiable graphics renderers use rasterization to determine visibility of primitives. In order to compute gradients across primitives boundaries, Loper & Black (2014); Kato et al. (2018), Liu et al. (2017) manually create the gradients while Liu et al. (2019); Yifan et al. (2019); Li et al. (2018); Nimier- David et al. (2019); Laine et al. (2020) use primitives with soft boundaries to allow gradients flow. Whereas to differentiate toward those occluded primitives, current differentiable renders, Liu et al. (2019); Yifan et al. (2019); Lassner & Zollhofer (2021), aggregate tracked primitives via viewing distance. However, all existing graphics renderers ignore the density distributions when conducting aggregation, which will introduce confusion while limiting differentiaailityy
Renderer for Deep Neural Features. Recent works demonstrate exceptional performance for ren- dering deep neural features. Specifically, for object pose estimation, Wang et al. (2020a); Iwase al. (2021), demonstrate rendering on deep neural features benefits the optimization process in render-and aadder-an--oopare Niemeyer & Geiger (2021) show rendering deep neural features also helps image generation tasks. our work, we show VoGE benefits rendering using deep neural features via a better reconstruction of the spatial distribution of deep neural features.
VOLUME RENDERER FOR GAUSSIAN ELLIPSOIDS
In this section, we describe VoGE rendering pipeline that renders 3D Gaussians Ellipsoids into images under certain camera configuration. Section 3.1 introduces the volume rendering. Section 3.2 describes the kernel reconstruction the 3D volume using Gaussian ellipsoids. In Section 3.3, propose the rendering pipeline for Gaussian ellipsoids via approximate closed-form solution of ray tracing volume densities. Section 3.4 discusses the integration of VoGE with deep neural networks.
3

---

Published as a conference paper at ICLR 2023
wk
k
W
Target
Cx-23385
0p-1619
1
[Q
0
[1.
(3
Gx-12259
(d)
Gx=000004
Qy- 135.4
13313
10 Q
(a) K-
(b) K=3
(c) K
[1.00)
Figure 2: Rendering with increasing numbers of Gaussian Ellipsoids. Top: the kernel-to-pixcl weight along the me- dian row the image, the colors demonstrate each corre- sponded Gaussian ellipsoids. Bottom: the rendered RGB image. Note VoGE resolves occlusion naturally in con- tiguous way.
Figure 3: Computing gradient of
M when rendering two ellipsoids. The colored numbers below indi- cate the of each ellipsoids. The red arrow and Gz Gy show the a(I-))² 8M
3.1
VOLUME RENDERING
Different from the surface-based shape representations, in volume rendering, objects are represented using continuous volume density functions. Specifically, for each point in the volume, we have a corresponded density p(z, y, with emitted color c(z, y. 2) (r. 9, b), where (I, y. denotes the location of the point in the 3D space. Kajiya & Von Herzen (1984) propose using the light scattering equation during volume density, which provides mechanism compute the observed color C(r)
along a ray r(t) = (z(()), =(t),((t))
" ((r) /' ((t))r(())c((((ddd where T(t) exp ( t..
p(r(s))ds
(1)
where T is a coefficient that determines the rate absorption, f, and ty denotes the near and far bound alone the ray, [tt) is the transmittance.
3.2 GAUSSIAN ELLIPSOID RECONSTRUCTION KERNEL
Due to the difficulty of obtaining contiguous function of the volume density and enormous computa- tion cost when calculating the integral, Westover (1990) introduces kernel reconstruction to conduct volume rendering in computationally efficient way. The reconstruction decomposes the contigu- ous volume into set of homogeneous kernels, while each kernel can be described with a simple density function. We use volume ellipsoidal Gaussians as the reconstruction kernels. Specifically, we reconstruct the volume with sum of ellipsoidal Gaussians:
((X) Σ
(X- EX
2 E
where K is the total number of Gaussian kernels, X y, 2) is arbitrary location in the 3D space. The MA, a 3 x 1 vector, is the center of k-th ellipsoidal Gaussians kernel. Whereas the Σn a 3 spatial variance matrix, which controls the direction, size and shape of k-th kernel. Also, following Zwicker et al. (2001), we assume that the emitted color is approximately constant inside each reconstruction kernel c(r(t)) Cj:-
The VoGE mesh converter creates Gaussian ellipsoids from mesh. Specifically, we create Gaus- sians centered all vertices' locations of the mesh. First, compute sphere-type Gaussians with same σą on each direction, via average distance from the center vertex to connected neighbors, 0 4-log(1/[) where ( is parameter controls the Gaussians size. Then, we flatten the sphere- type Gaussians into ellipsoids with flatten rate. Finally, for each Gaussian, compute rotation matrix via the mesh surface normal direction of the corresponded mesh vertex. We dot the rotation matrix the Σg to make the Gaussians flatten along the surface.

---

Published as a conference paper at ICLR 2023
Camera
Differentible forward
Non- Differentiable forward
Viewing Rays
Transfoms
Caussian Ellipsoick
Ray Tracing
Volume Density Blending
M
Coarse Rasterize (Optional)
M
The forward process for VoGE rendering. The camera is described with the extrinsic matrix E composed with R and well the intrinsic matrix I composed with F and Oz. Og- Given Gaussian Ellipsoids, VoGE renderer synthesizes an image O.
Figure 4:
3.3 RENDER GAUSSIAN ELLIPSOIDS
Figure 4 shows the rendering process for VoGE. VoGE takes inputs of perspective camera and Gaussian ellipsoids to render images, while computing gradient towards both camera and Gaussian ellipsoids (shows Figure 2 and 3).
Viewing transformation utilizes the extrinsic configuration E of the camera to transfer the Gaussian ellipsoids from the object coordinate to the camera coordinate. Let M2 denote centers of ellipsoids in the object coordinate. Following the standard approach, we compute the centers in the camera coordinate:
M R M +
where R and T are the rotation and translation matrix included in E. Since consider 3D Gaus- sian Kernels are ellipsoidal, observations of the variance matrices are also changed upon camera rotations:
Dz' = (
(1 -1.
Perspective rays indicate the viewing direction in the camera coordinate. For each pixel, we com- pute the viewing under the assumption that the camera is fully perspective:
r(t) = D * -
i-0= F 1) *
(5)
where p= (i, )) is the pixel location on the image, Oz, Oy is the principal point of the camera, is the focal length, D is the ray direction vector.
Ray tracing observes the volume densities of each ellipsoid along the ray r respectively. Note the observation of each ellipsoid is ID Gaussian function along the viewing ray (for detailed mathematics, refer to Appendix A.1):
(s-Lm) 1m)
Pm (r(s)) = exp(çm 2 o2
(6)
where Im M. x_1 D+D" E. M.. is the length along the ray that gives peak activation for m-th
2D -
kernel. Om !VT m
V.. = Mm - Im D computes peak density of m--th kernel alone the ray. The 1D variance is computed via x² = DT. Σ 1-1.1. D. Thus, when tracing
Ex1 x-1.
along each ray, only need to record lm. fm and om for each ellipsoid respectively.
Blending via Volume Densities computes the observation along the ray r. As Figure shows, dif- ferent from other generic renderers, which only consider the viewing distance for blending, VoGE blends all observations based on the integral volume densities along the ray. However, comput- ing the integral using brute force is computationally inefficient that even infeasible for coneur- rent computation power. To resolve this, we propose an approximate closed-form solution, which conducts the computation in both accurate and effective way. We the Error Function erf to compute the integral of Gaussian, since it can be computed via a numerical approach directly. Specifically, with Equation 2 and Equation 5, we can calculate the transmittance (()) (for proof about this approximation, refer to Appendix A.2):
T(t) = exp(- S p(r(s))ds) = exp(-7 Σ cm erf((t lm) om) +1
K
2
m=1

---

Published as a conference paper at ICLR 2023
Rendering Speed Jimagges per second)
Remderinss Sprzed (Imaggss pet secooddo
1.0
00-
50
20c
1.5k
Py Pylooch3D
300
Vodc
64
126 256
512
1024
Image width height (ginels)
Number of Components, per Poeels
Figure 5: Comparison for rendering speeds of VoGE and Py MyTrhh3D, reported images per second (higher better). We evaluate the rendering speed using cuboids with different number of primitives (vertices, ellipsoids), which illustrated using different colors, also different image sizes and number of primitives per pixel.
Now, to compute closed-form solution of the outer integral in Equation I, we use the T(t), f 15 at the peak of p(r(t)) alone the rays. Here provide the closed-fon form solution for C(r):
K
C(r) = L T(b))((()))(((()))
T(
Note based on the assumption that distances from camera to ellipsoids are significantly larger than ellipsoid sizes, thus it is equivalent to set Coarse-to-fine rendering. In order to improve the rendering efficiency, we implement VoGE ren- dering with coarse-to-fine strategy. Specifically, VoGE renderer has optional coarse rasterizing stage that, for each ray, selects only around 10% of all ellipsoids (details Appendix A.3). Besides, the ray tracing volume densities also works in a coarse-to-fine manner. VoGE blends K' nearest ellipsoids among all traced kernels that gives e thr 0.01. Using CUDA from NVIDIA al. (2022), we implement VoGE with both forward and backward function. The CUDA-VoGB is packed as an easy-to-use "autogradable" PyTorch API.
and t, cc.
3.4 VoGE IN NEURAL NETWORKS
VoGE be easily embedded into neural networks by serving neural sampler and renderer: As a sampler, VoGE extracts attributes (e.g., deep neural features, textures) from images or feature maps into kemeelccorrespond attributes, which conducted via reconstructing their spatial distribution in the screen coordinates. When serving as renderer, VoGE converts kernel-correspond attributes into images or feature maps. Since both sampling and rendering give the same spatial distribution of feature/texture, it is possible for VoGE to conduct geometry-based image-to-image transformation.
Here discuss how VoGE samples deep neural features. Let & denotes observed features, where &p is the value location p. Let A kK___ (ax) denotes the per kemel attribute, which want to discover during sampling. With a given object geometry == - (M,, Σイ} and viewing rays r(p). The the observation formulated with conditional probability regarding og:
K
8' (p) Σ ((()
Since & is a discrete observation of a continuous distribution Φ(p) on the screen, the synthesis can only be evaluated at discrete positions, the pixel centers. As the goal is to make 4' similar +b on all observable locations, we resolve via an inverse reconstruction:
Σp
W p.k
Σ PloW)),, p=1
(10)
Σ/
W p.k
where W P.A T( is the kernel-to-pixel weight described in 3.1.

---

Published as a conference paper at ICLR 2023
Table 2: Pose estimation results on the PASCAL3D+ and the Occluded PASCAL3D+ dataset. Oc- clusion level LO is the original images from PASCAL3D+, while Occlusion Level L1 to L3 are the occluded PASCAL3D+ images with increasing occlusion ratios. NeMo is an object pose estimation pipeline via neural feature level render-and-compare We compare the object pose estimation per- formance using different renderers, i.e. VoGE, Soft Rasterizer, DSS, Py Torch3D (which is used in NeMo originally).
Evaluation Metric
ACCa
ACC
MedErr
Occlusion Level
L0
L1
L2
L3
L0
L1
L2
L3
L0
L1
L2
L3
Res50-General
88.1
70.4
52.8
37.8
44.6
25.3
14.5
6.7
11.7
17.9
30.4
46.4
ResSO-Specific
87.6
73.2
58.4
43.1
43.9
18.6
9.9
11.8
17.3
26.1
44.0
StarMap
89.4
71.1
47.2
22.9
59.5
34.4
13.9
3.7
9.0
17.6
34.1
63.0
NcMo+SoftRas
85.3
75.2
63.0
44.3
59.7
46.7
32.1
16.8
9.1
14.8
24.0
39.3
NeMo+DSS
81.1
71.9
56.8
38.7
33.5
30.4
23.0
14.1
16.1
19.8
25.8
40.4
NcMo+PyTorcc3D
86.1
76.0
63.9
46.8
61.0
46.3
32.0
17.1
8.8
13.6
20.9
36.5
NeMo+ VoGE(Ouur)
90.1
83.1
72.5
56.0
69.2
56.1
41.5
24.8
6.9
9.9
15.0
26.3
Input Image
NeMo-VaGE(ours)
NeMo-PyTorchID
Image
NeMo VoGB(ours)
NeMo-PyTorahSDD
Figure 6: Qualitative object pose estimation results on PASCAL3D+ dataset. We visualize the predicted object poses from NeMo+VoGE and standard NeMo. Specifically, we use standard mesh renderer render the original CAD model under the predicted pose and superimpose onto the input image.
EXPERIMENT
We explore several applications of VoGE. In section 4.1, we study the object pose estimation using VoGE in feature level render-and-compare pose estimator. In section 4.2, we explore texture ex- traction ability of VoGE. In section 4,4, we demonstrate VoGE can optimize the shape representation via multi-viewed images. Visualizations of VoGE rendering are included in Appendix B.
Rendering Speed. As Figure shows, CUDA-Vode provides a competitive rendering speed com- pare to state-of-the the-hht- art differentiable generic renderer when rendering single cuboidal object.
4.1 OBJECT POSE ESTIMATION IN WILD
We evaluate the ability of VoGE when serving as a feature sampler and renderer an object pose estimation pipeline, NeMo Wang al. (2020a), in-wild category-level object 3D posc estimator that conducts render-and-compe (2020) as the feature sampler and renderer, which converts the feature maps vertex corresponded feature vectors and conducts the inverse process. In our NeMo+ VoGE experiment, we use VoGE to replace the Py Torch3D sampler and renderer via the approach described in Section 3.4.
on neural feature level. NeMo utilizes PyTorch3D Ravi et al.
Dataset. Following NeMo, we evaluate pose estimation performance on the PASCAL3D+ dataset Xiang al. (2014), the Occluded PASCAL3D+ dataset Wang al. (2020b) and the ObjectNet3D dataset Xiang et al. (2016). The PASCAL3D+ dataset contains objects in 12 man-made categories with 1045 training images and 10812 testing images. The Occluded PASCAL33+ contains the occluded version of same images, which is obtained via superimposing occluder cropped from MS- COCO dataset Lin et al. (2014). The dataset includes three levels of occlusion with increasing occlusion rates. In the experiment ObjectNet3D, we follow NeMo to test on 18 categories.
Evaluation Metric. We measure the pose estimation performance via accuracy of rotation er- under given thresholds and median of per image rotation errors. The rotation error is de- fined as the difference between the predicted rotation matrix and the ground truth rotation matrix: loo m (n Rgt)
A (Rpred, Rge))
/2

---

Published as a conference paper at ICLR 2023
Table Pose estimation results on the ObjectNet3D dataset. Evaluated via pose estimation accuracy for error under 6 (higher better).
ACC₃ 1
bed shelf calculator cellphone computer
cabinet
guitar
iron
knife
StarMap
40.0 72.9
21.1
41.9
62.1
79.9
38.7
2.0
6.1
NeMo+PyTorch3D
56.1 53.7
57.1
28.2
78.8
83.6
38.8
32.3
9.8
NeMo+ VoGE(OU))
76.8 83.2
77.8
50.7
78.8
83.6
54.6
45,4
12.1
ACC= 1
oven
pen
pot
rifle
slipper
stove
toilet
tub
wheelchair
StarMap
86.9
12.4
45.1
3.0
13.3
79.7
35.6
46.4
17.7
NeMo+PyTorch3D
90.3
3.7
66.7
13.7
6.1
85.2
74.5
61.6
71.7
NeMo+ VoG)(OUrs)
94.9
13.5
77.8
30.8
22.2
89.8
81.9
68.9
68.4
Synthesizud Image on Novel
Ingut Image
Goussian Ellipenids
-30
-10P
+10*
430
(a)
(b)
Figure 7: Sampling texture and rerendering on novel view. The inputs include a single RGB image and the Gaussian Ellipsoids with corresponded pose. Note the result is produced without any train- ing or symmetrical information.
Baselines. We compare our VoGE for object pose estimation with other ifate-of-thh-art differ- entiable renderers, i.e. Soft Rasterizer, DSS, and PyTorch3D. For comparison. we use the same training and inference pipeline, and same hyper-parameters for all 4 experiments. Our baselines also includes Res50- Seeer//eeeea which converts object pose estimation into a bin classification problem, and StarMap Zhou et al. (2018) which first detect keypoints and conduct pose estimation via the PnP method.
Experiment Details. Following the experiment setup in NeMo, we train the feature extractor 800 epochs with a progressive learning rate. During inference, for each image, we sample 144 starting poses and optimizer 300 steps via ADAM optimizer. We convert the meshes provided by NeMo using the method described Section 3.2.
Results. Figure 6 and Table 2 show the qualitative and quantitative results of object pose estimation on PASCAL3D+ and the Occluded PASCAL33+ dataset. Results in Table 2 demonstrate significant performance improvement using VoGE compared to Soft RasterizerLiu et al. (2019), DSSYifan al. (2019) and PyTorch3DRavi et al. (2020). Moreover, both qualitative and quantitative results show our method a significant robustness under partial occlusion and out distributed cases. Also, Figure 6 demonstrates our approach can generalize to those out distributed cases, e.g., a car without front bumper, while infeasible for baseline renderers. Table shows the results on ObjectNet3D, which demonstrates significant performance gain compared to the baseline approaches. The ablation study is included in Appendix C.I.
4.2 TEXTURE EXTRACTION AND RERENDERING
As Figure shows, we conduct the texture extraction on real images and rerender the extracted textures under novel viewpoints. The qualitative results is produced on PASCAL3D+ dataset. The experiment conducted on each image independently that there is no training included. Specifically, for each image, we have only three inputs, i.e. the image, the camera configuration, the Gaussian ellipsoids converted from the CAD models provided by the dataset. Using the method proposed in 3.4, we extract the RGB value for each kernel on the Gaussian ellipsoids using the given groundtruth camera configuration. Then we rerender Gaussian ellipsoids with the extracted texture under a novel view, that increase decrease the azimuth of the viewpoint (horizontal rotation). The qualitative
8

---

Published as a conference paper at ICLR 2023
Tages
Lige mmgge (witt) Opanbee Image (WKKR)
Rooth (VKC
X0
Figure 8: Reasoning multi-object occlusions for single view op- timization of object location. Left: initialization and target loca- tions. Middle: target images, note the target image generate via each rendering method. Right: results. Videos: VoGE, SoftRas.
Figure 9: Shape fitting results with 20 multi-viewed images following the PyTorch3I) Ravi et al. (2020) official tutorial.
results demonstrate a satisfying texture extraction ability of VoGE, even with only single image. Also, the details (e.g., numbers the second car) retained in high quality under the novel views.
4.3 OCCLUSION REASONING OF MULTIPLE OBJECTS
Figure shows differentiating the occlusion reasoning process between two objects. Specifically, target image, and the colored cuboid models and initialization locations, are given the method. Then we render and optimize the 3D locations of both the cuboids. In this experiment, we find both SoftRas and VoGE can successfully optimize the locations when the occludee (blue cuboid) is near the occluder (red cuboid), which is 1.5 scales behind the occluder the thickness of the occluder 0.6 scales. However, when the the occludee is far behind the occluder (5 scales), SoftRas fails to produce correct gradient to optimize the locations, whereas VoGE still successfully optimize the locations. We think such advantage benefits from the better volume density blending compared to the distance based blender used in SoftRas.
4.4 SHAPE FITTING VIA INVERSE RENDERING
Figure shows the qualitative results of multi-viewed shape fitting. In this experiment, we follows the setup fir a mesh with texture via rendering from PyTorch3D official tutorial Ravi et al. (2022a). First, a standard graphic renderer is used to render the cow CAD model in 20 different viewpoints under fixed light condition, which are used as the optimization targets. For both baseline and ours, we give sphere object geometry with 2562 vertices and optimize toward target images using the same configuration, e.g. iterations, learning rate, optimizer, loss function. During the shape optimization process, we compute MSE loss both silhouettes and RGB values between the syn- thesized images and the targets. The vertices locations and colors are gradiently updated with an ADAM optimizer Kingma & Ba (2014). We conduct the optimization for 2000 iterations, while in each iteration, we randomly select 5 out of 20 images to conduct the optimization. In Figure 9 (e) and (f), the object geometry, while in (d) no additional loss is used. From the results, we can see that VoGE has competitive ability regarding shape fit via deformation. Specifically, VoGE gives better color prediction and smoother object boundary. Also, we observe the current geometry constrain losses do not significantly contribute to our final prediction. We argue those losses are designed for surface triangular meshes, that not suitable for Gaussian ellipsoids. The design of geometry constraints that are suitable for Gaussian ellipsoids an interesting topic but beyond scope of this paper.
the normal consistency, edge and Laplacian loss Nealen et al. (2006) constrain
CONCLUSION
this work, we propose VoGE, a differentiable volume renderer using Gaussian Ellipsoids. Ex- periments in-wild object pose estimation and neural view matching show VoGE extraordinary ability when applied on neural features compare to the concurrent famous differential generic ren- derers. Texture extraction and rerendering experiment shows VoGE the ability feature and texture sampling. which potentially benefits downstream tasks. Overall, VoGE demonstrates better differ- entiability, which benefits vision tasks, while retains competitive rendering speed.

---

Published as a conference paper at ICLR 2023
REFERENCES
Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, and Bryan Catanzaro. View general- ization for single image textured 3d models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6081-6090, 2021.
James F Blinn. Light reflection functions for simulation of clouds and dusty surfaces, Acm Siggraph Computer Graphics, 16(3):21-25 29, 1982.
Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference Computer graphics and interactive techniques, pp. 303-312, 1996.
Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Shun Iwase, Xingyu Liu, Rawal Khirodkar, Rio Yokota, and Kris M Kitani. Repose: Fast 6d ob- ject pose refinement via deep texture rendering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
James T Kajiya and Brian Von Herzen. Ray tracing volume densities. ACM SIGGRAPH computer graphics, 18(3):165-11))) 1984.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference computer vision and pattern recognition, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint UrXiv:1412.66980 2014.
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high- performance differentiable rendering. ACM Transactions on Graphics (TOG), 39(4):1-14, 2020.
Christoph Lassner and Michael Zollhofer. Pulsar: Efficient sphere-based neural rendering. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions Graphics (TOG), 37(,):1-11,,00
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitncck. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 2014.
Guilin Liu, Duygu Ceylan, Ersin Yumer, Jimei Yang, and Jyh-Ming Lien. Material editing using a physically based rendering network. In Proceedings of the IEEE International Conference on Computer Vision, 2017.
Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision, 2019.
Matthew M Loper and Michael Black. Opendr: An approximate differentiable renderer. In Euro- pean Conference Computer Vision. Springer, 2014.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision. Springer, 2020.
10

---

Published as a conference paper at ICLR 2023
Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Mare Alexa. Laplacian mesh optimization. In Proceedings of the 4th international conference on Computer graphics and interactive techniques in Australasia and Southeast Asia, 2006.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumet- ric rendering: Learning implicit 3d representations without 3d supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG), 38(6):1-17, 2019.
NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. Cuda, release: 11.2.89, 2022. URL https: //developer nnviddid aovidia.. www catpo//ccaatttooottcce
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. AXiv:2007.00801, 2020.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin John- son, and Georgia Gkioxari. Pytorch3d official tutorials: Fit a mesh via rendering. https: stt.riaaeeettttttttteeeeeeeeeeee 2022a. Accessed:
2021-11-10.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Lo, Justin Johnson, and Georgia Gkioxari. Pytorch3d official tutorials: Render colored point cloud. https: www.ooorooo...ggtttooroeoooooeee wwaerriialseeddde_e_eeeee wwwdereendoeeeeee ctaendee__coooore_poitttttt 2022b. Accessed: 2021- 11-10.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33, 2020.
Delio Vicini, Sébastien Speierer, and Wenzel Jakob. Differentiable signed distance function render- ing. ACM Transactions on Graphics (TOG), 1-88) 2022.
Angtian Wang, Adam Kortylewski, and Alan Yuille. Nemo: Neural mesh models of contrastive features for robust 3d pose estimation. In International Conference on Learning Reprexentations, 2020a.
Angtian Wang, Yihong Sun, Adam Kortylewski, and Alan L. Yuille. Robust object detection under occlusion with context-aware compositionalnets. In Proceedings the the IEEE/CVF Conference Computer Vision and Pattern Recognition, June 2020b.
Lee Westover. Footprint evaluation for volume rendering. Proceedings of the 17th annual con- ference on Computer graphics and interactive techniques, pp. 367-376, 1990.
Yu Xiang, Roozbeh Mottaghi. and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection the wild. In IEEE Winter Conference on Applications of Computer Vision, 2014.
Yu Xiang, Wonhui Kim, Wei Chen, Jingwei Ji, Christopher Choy, Hao Su, Roozbeh Mottaghi, Leonidas Guibas, and Silvio Savarese. Objectnet3d: A large scale database for 3d object recog- nition. European Conference Computer Vision, 2016.
Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Horuung Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics, 38(6); 1-14, 2019.
Cheng Zhang, Lifan Wu, Changxi Zheng, loannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. A differential theory of radiative transfer. ACM Transactions on Graphics (TOG), 38(6): 1-16, 2019.
Jason Zhang, Gengshan Yang. Shubham Tulsiani, and Deva Ramanan. Ners: Neural reflectance sur- faces for sparse-view 3d reconstruction the wild. Advances in Neural Information Processing Systems, 34, 2021.
11

---

Published as a conference paper at ICLR 2023
Xingyi Zhou, Arjun Karpur, Linjie Luo, and Qixing Huang. Starmap for category- keypoint and viewpoint estimation. In Proceedings of the European Conference on Computer Vision, 2018.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. In Proceedings Visualization, 2001. VIS'01. IEEE, 2001.
12

---

Published as a conference paper at ICLR 2023
A ADDITIONAL DETAILS OF VOGE RENDERER
In this section we provide more detailed discussion for the math of ray tracing volume densities in VoGE (section A.1 and A.2), coarse-to-fine rendering strategy (section A.3), and the converters (section A.4).
A.I RAY TRACING
In this section, we provide the detailed deduction process for Equations 6 in the main text. First, let's recall the formula of Ray tracing volume densities Kajiya Von Herzen (1984):
= f/ C(r)
T(t))(((())(rr(ttddd
(11)
where T(t) = exp (-r p(r(s))ds) ((r(s)))
where T(t is the occupancy function alone viewing ray r(t), as we describe in Equation 5 in main text:
r(1)
(12)
where is the normalized direction vector of the viewing ray.
Also, as we describe in Section 3.2, we reconstruct the volume density function p(r(t)) via the sum of set of ellipsoidal Gaussians:
K
p(X)
2(--M,)* E. (X--M.)
(13)
2T
EE
k=1
where K is the total number of Gaussian kernels, X (I, 2; is an arbitrary location in the 3D
volume. Mg is the center k-th ellipsoidal Gaussians kernel:
M = (Hk,er Mk.ges (Ak,a)
(14)
whereas the Eg the spatial variance matrix:
Tkka
dk,zy
0kizz
Σx-
Ok.z
Okiya
(15)
Ok,
Ok,ay
Note that Σg is a symmetry matrix, e.g., covariance dkiay =
0k.yz-
Occupancy Function. Based on Equation 13 and 11, T(t) can be computed via:
T(t) exp (- (-r[ p(r(s))ds)
(16)
/' K k=
=exp(-7
(aD M,)" >g' (sD-M:)ds
2T
Σg
Now, let M 1D VE, where l is length along the viewing ray, VA M I,D is the vector from location 4 on the ray to the vertex M2 (we will discuss solution for V₃ and 4 later). Equation 16 can be simplified as:
K
T(t) = exp(
V 2T Ex
(17)
( (VV Ex D+D" EF V.) - tv
Vvds)
In order to further simplify T(t), we take VA that makes:
Vf. E D+DD+ D E- V =0
(18)
13

---

Published as a conference paper at ICLR 2023
D
WG
(P(())
69- 7'00
WC2) (+
6.22
02
(a)
(b)
(c)
Figure 10: Approximate computation of integral along the viewing ray for single kernel. (a) T(t)
the real occupancy function along the ray, T (t) means use the occupancy 4 of p(r(t)).
since p(r(t)) are mainly concentrate near lg- (b) shows W(t) = f T(s) p(r(s))ds, W (t) S- T' (s) p(r(s))ds and the difference ((1) W (t). Since we use the infinite integral with only error end of & axis need be consider. (c) shows the accumulative ( W (t) using different ø = D D D. Interestingly, the final error gives a fix value which is independent from σ. Note that the final error 0.0256 which can be ignored when compared to the integral result W =1 1.
which can be solve using V = M2 - D:
(M, DD) D3 D D Σx (M, (DD) =
M Σx 1 D + D Σ M2 24 Σx"
D 0
(19)
4. M. Σ D DT. Σ M
2. D EF D
Note that 4 is also the length that gives the maximum density Pa: (r())) along the ray for k-th kernel. To proof this, we compute:
at P(((((())) Öt 22 Ex
θ
[ID X, ((D-Ma)
(ID M T Σ (ID M₂)
4 Exlla
(20)
(M Dx D + DJ M 2ID '
'D)
e ((D -. (DDMMc)
Obviously, the solve for (r((tt) 0s:
M Σ D+D E M 12
(21)
2. E
Now the density function of the k-th ellipsoid along the viewing ray r(s) gives an 1D Gaussian function:
V Σx
Vv
px(r(s))
PD I.D
2m
Ex
(22)
(s- 1
exp(gg
/
2n
Ex
2 o
Σz D. Thus, when tracing along each ray, we only need record 4t, Oh and dg for each ellipsoid respectively.
where = - Σg V oo
= D
14

---

Published as a conference paper at ICLR 2023
(a) K 5
(b) K = 10
(c) 20
(d) K
1)) = 0.1
( 0.01 (g) = 0.001
(h) 0.0001
Figure 11: Rendering cuboid using different K and 17- (a) (d) shows the rendering result using
different K", the threshold fix as 7 0.01. (e) (h) shows the rendered cuboid with different 7), while fixing K" 20.
A.2 BLENDING VIA VOLUME DENSITY
Since yn is independent from I, the Equation 16 can be further simplified:
T()) exp(
Th
1
(s- (s-12) 1o-1t)//oo /ot
/2
Ex
(23)
K ah
crf((t 4))oo)) + +
= exp(-
2
where erf is the error function, that concurrent computation platforms, egg., PyTorch, Seipy, have already implemented.
Scattering Equation. Now we compute the final color observation C(r). As we describe in Section 3.2, we assume each kernel has homogeneous C₁. Thus, here we compute:
Wa(1) = (())(((())dt
(24)
T(t)
1 (X TT -E, (-)--x))
MM) dt
2
k=1
where DD. Similar previous simplifications, use qu and replace M2 in Equation 24:
K
Wx(t)
= 9E
T(t)
(25)
2T
Ex
Due to the error function is already complex function, it is infeasible to compute the integral of T(t). We propose an approximate solution that we use T(kx) to replace T(t) inside the integral. Now the final closed-form solution for W W is computed by:
Wa(1) Σ
T(l)e
1
(t- (x)² 2 /
/2xT .
2
(26)
K Σ
T(
A=1
15

---

Published as a conference paper at ICLR 2023
Because of the complexity when computing integral of the erf function, here we prove that in prac- tice such approximate gives high enough accuracy. To simplify the problem, we study the case that the volume only contains single Gaussian ellipsoid kernel. We further suggest that in the multi- kernel cases, the errors between different kernels introduced by the approximation will be lower. Because m-th kernel has low Pm (r(t)) 4g, which makes the corresponded T(t) more flatten, thus the approximation fits better. As Figure 10 (a) shows, we plot density function along the ray. Specifically, we sample 10k points on the ray, and for each point, we plot its density, the real occu- pancy, and the approximate occupancy. Figure 10 (b) shows the real weight W which is computed via the cumulative sum along the ray, and the approximate weight W which is computed via our proposed approximate closed-from solution. We also show the difference between W and W with the green line, which is significantly smaller compare to W. Interestingly, as Figure 10 (c) shows, we find the error (t) W' (t) is independent from Σ and D, that always converge to a same value: 0.0256. Though we cannot give mathematical explanation regarding this phenomenon, argue the result is already enough to draw the conclusion that such approximation gives satisfying accuracy.
A.3 COARSE-TO-FINE FINE RENDERING WITH KERNEL SELECTION
As we discussed in Section 3.3 in the main text, in order to efficiently render Gaussian ellipsoids, we design the coarse-to- fine rendering strategy. Specifically, we gradually reduce the number of ellip- soids that interact with viewing rays. Following Py Torch3D, we develop a optional coarse rasteriza- tion stage, which select 10% of all ellipsoids and feed them into the ray tracing stage. Specifically, we project the center of each ellipsoid onto the screen coordinate via standard object-to-camera transformation, then for each ellipsoids, we compute the height b, and width by of maximum bounding box the ellipsoids in 2D screen coordinate. The height and width are computed via:
.] = log(-1 de
Pb
bw
Ω 1..-1.1
(27)
where dy is the distance from camera to the center of ellipsoid, r) is the threshold for maximum volume density, 52 is the projection matrix from camera coordinate to screen:
0
Ω
0
(28)
1
Then we rasterize the bounding boxes to produce a tixel-to-kernels assignment in low resolution (8 times smaller compared to the image size), which indicates the set of ellipsoid kernels for each pixel to trace.
Similarly, the ray tracing stage is also select only part of all Gaussian ellipsoids to feed into the blending stage. When conducting ray tracing, we only trace K' nearest kernels that has non-trivial contributions regarding its final weight WR- Specifically, we first record all ellipsoids that gives a maximum density e9k > y)- For all the recorded kernels, we sort them via the length to the ID Gaussian center la and select K' nearest ellipsoids. In the experiment, we find K' has significant impact on the quality of rendered images, while the threshold r) has relatively low impact, but needs be fit with K", Here we provide default settings that give satisfying quality with low computation cost: K' 20, r7 = 0.01.
Figure/ 11 shows the rendered cuboids using different K' and r7. Here the results demonstrate that inadequate K" will lead some dark region around the boundary of kernels, which we think is caused by the hard cutoff of the boundary. On the other hand, decreasing the threshold r could make the object denser (less transparent), but need more kernels (higher K') to avoid the artifacts.
A.4 MESH POINT CLOUD CONVERTER
We develop simple mesh converter, which converts triangular meshes into isotropic Gaussian ellipsoids, and a point cloud converter. In the mesh converter, we retain all original vertices on the mesh and compute the Σ₁ using the distance between each vertex and its connected neighbors.
16

---

Published as a conference paper at ICLR 2023
+0" azimuut
+48" aztmuth
490 aztmouth
wimuts
0.31
2.0.3
+0.3
0.8
(et(E)) 0.55
0.7
0.61
11.7
0.9
0.0
1.0.6
der(E) 0.28
Lo.9
1.2 1.6
20.79
#0.79
0.8
0.9
1
det(3) 0.0159
Figure 12: Rendering single anisotropic ellipsoidal Gaussian. Left column shows the Σ for the kernel. We render the kernel under 3 different viewpoint: 0 azimuth, 45 azimuth, and 90 azimuth.
Figure 13: Rendering flattened Gaussian ellipsoids to approximate the 2D Gaussian ellipses. For each row, we show two Gaussian ellipsoids viewed in different viewpoints. From top to bottom: decrease det to flatten Gaussian ellipsoids.
Specifically, for each vertex, we compute the average length d, of edges connected that vertex. Then Σg is computed via:
T
Σ
0
Ok
0
(29)
0
where is computed via the coverage rate C and dA.
(dg/2)2
o log(1/C)
(30)
Similarly, in the point cloud converter, the Eg is controlled with the same function, but the d2 is determined by the distance to m nearest points of the target points.
Since the concurrent mesh converter does not consider the shape of the triangles, admittedly think this could be improved via converting each triangle into an anisotropic Gaussian ellipsoid, which we still working
B ADDITIONAL RENDERING RESULTS
B.1 RENDERING ANISOTROPIC GAUSSIAN ELLIPSOIDS
As Figure shows, VoGE rendering pipeline natively supports anisotropic ellipsoidal Gaussian kernels, where for each kernel the spatial variance is represented via the x 3 symmetric matrix Σg. Note that, the spatial covariances, e.g., k, cannot exceed square root of dot product the two variances, e.g., 0 otherwise, the kernel will become hyperbola instead of ellipsoids (as the last row Figure shows).
On the other hand, suggest that ellipsoidal Gaussian kernels can also approximate the 2D Gaus- sian ellipses (the representation used DSS Yifan al. (2019)), which can simply done by set det( + O, where det is the determinant of matrix. Figure 13 shows the rendering result using flattened Gaussian ellipsoids. As we demonstrated in the third row, VoGE rendering pipeline allows rendering the surface-likes representations in stable manner.
B.2 RENDERING SURFACE NORMAL
As Figure 14 shows, we render CAD models provided by The Stanford 3D Scanning Repository Curless & Levoy (1996). Specifically, mesh converter to convert the meshes provide by
17

---

Published as a conference paper at ICLR 2023
(a) Bunny
(b) Dragon
(c) Armadillo
Rendering the surface normal using VoGE under 8 different viewpoints. The Gaussian ellipsoids are converted from meshes provided by The Stanford 3D Scanning Repository.
Figure 14:
the dataset into Gaussian ellipsoids. In detail, the Bunny contains 8171 vertices, the Dragon contains 22998 vertices, and the Armadillo contains 33792 vertices. During rendering, we compute surface normals via PyTorch3D and use the normals the RGB value of each vertex. Then we render the Gaussian Ellipsoids in 8 different viewpoints, and interpolate the RGB value into images.
B.3 RENDERING QUALITY VS NUMBER OF GAUSSIANS
Figure 16 shows surface normal rendering quality of VoGE using different number of Gaussians. We also include comparison of rendering quality of VoGE vs Py Torch3D mesh renderer. In each image, control same number of Gaussians mesh vertices, which gives similar number of parameters that 9 NGans VS 3 Neerts - 3 * N Maces- Here we observe that increasing number of Gaussians will significant improve rendering quality. Admittedly, VoGE renderer gives slight fuzzier boundary compare to mesh renderer.
B.4 LIGHTING WITH EXTERNAL NORMALS
Although Gaussian ellipsoids do not contain surface normal information (since they are represented as volume), VoGE still can utilize surface normal via processing them as an extra attribute in an external channel as we describe in section B.2. Once the surface normals are rendered. the light diffusion method in the traditional shader can be used to integrate lighting information into VoGE rendering pipeline. Figure shows the results that integrate lighting information when rendering the Stanford bunny mesh using VoGE. Specifically, we first render the surface normals computed via PyTorch3D into an image-liked map (same as the process in section B.2). Then we use the diffuse function www.rrch....ndddee.....iigg
compute the brightness of the rendered bunny under point light. In the visualization, we place the light source at variant locations, while using a fully white texture the bunny.
B.5 RENDERING POINT CLOUDS
Figure 17 shows the point clouds rendering results using VoGE and PyTorch3D. We follow the Render colored point cloud from PyTorch3D official tutorial Ravi et al. (2022b). Specifically, we use the PittsburghBridge point cloud provided by PyTorch3D, which contains 438544 points with RGB color for each point respectively. We first convert the point cloud into Gaussian ellipsoids using the method described in A.4. Then we render the Gaussian ellipsoids using the same configuration (Except the camera. As the tutorial uses orthogonal camera, which concurrently don't support,
18

---

Published as a conference paper at (CLK 2023
(a) 30, 45
(b) 30, 45
(c) 30, a 135
ce) 0..=0,4.0
(d) 0,a -45
(l) = 0. a
1.5
@g = 30, 45
(h) -30, 45
300 135
Figure 15: Lighting rendered mesh using external normals. We first render the surface normat of mesh using VoGE. Then light the render surface normal. For each image, we place poim light source in the object space using different elevations (e) and azimuth (a). The distance from the light source the object center diffferi is fixed as 1.
the bumny
the light diffusion functions provided by ToTorchs to
Table- 4: (Leff)) Ablation study for object pose estimation on PASCAL3D+. We control the coverage rate ( when Aomputiin E. higher gives larger values in Σ. w/o grad 7(r) means we block the pradieet from T (rr), while w/o grad p(r) means gradient on e in Equation 8 is blocked.
ACC 1
ACC11 i
MedErr
Exp. Setup
0.2
89.9
68.7
7.0
0.5 (standard)
90.1
69.2
6.9
Ç - 0.8
90.3
64.7
8.5
w/o grade (rr)
47.1
18.9
39.1
w/o grid p(r)
31.7
8.0
48.1
we altermate the camera using a C with similar viewing scope). The qualiitative muilss cooonsstttte VoGE better quality with smoother boundaries.
assulta
19

---

Published as a conference paper at ICLR 2023
Gaussions 33515
Gaussians: 8207
3335
Guussians: 135
x1
x2
x2
(a) VoGE
Verts:33515, F6ces:67116
Verts:8207, Faces:1 6508
Verts:33515, 666
Verts:331185 F044s:270448
x1
X2
PyTorh33
Figure 16: Comparison of rendering quality with number of primitives using VoGE vs PYTorc33D hard mesh renderer.
(a) PyTorch3D 1x
PyTorch3D 4x
(c) PyTorch3D 16x
(d) VoGE 1x
VoGE
(1) VoGE
Figure 17: Rendering point clouds using VoGE and PyTorc3DD. In the 1x visualization, we render the image with camera with standard focal length. In 4x and 16x we zoom-in the camera by increase the focal length by 4x and 16x.
20

---

Published as a conference paper at ICLR 2023
Table 5: Per category result for in-wild object pose estimation results on PASCAL3D+. Results reported in Accuracy (percentage, higher better) and Median Error (degree, lower better).
J MedErr
+ ACC
ACCx
NeMo+ NeMo+ VoGiE (ours)
NeMo+ DSS
NeMo+ SofiRas
StarMap
Res Res:00-Specfft Specific
Res50-GGeneral
NeMo+ NeMo+VoGEE VoGiE(ours)
NeMo+Py CoMo-PPyTorcha Torch3D
NeMo+DSS
NeMo+SoftRaa
StarMap
Res50- Res50-Speciin Specific
Res50 General
NeMo+ VoGE
NeMo+Py PeMMo+PyTorchin TD
NeMo+DSS
NeMo+SoftPee
StarMap
Res50-S Speccifc
PeMo+PyTorcha3
-ennerll
7.5
10.1
16.6
10.6
10.0
14.2
13.3
61.4
49.7
22.8
47.5
49,8
29.1
31.3
89.7
82.2
77.2
80.8
85.5
79.5
83.0 79,6
aero
12.8
16.3
23.0
17.3
14.0
17.3
15.9
40.3
29.5
10.2
26,2
34,2
22.9
25.7
82.6
78.4
69.3
79.2
84.4
75.8
bike boat
9.8
14.9
19.5
15.1
19.7
15.4
15.6
51.2
37.7
23.7
36.2
25.4
25.3
23.9
77.7
68.1
65.4
70.3
65.0
73.5
73.1
9.1
10.2
13.1
10.0
8.8
11.7
12.1
53.9
49.3
37,8
49,9
56.8
39.0
35.9
83.2
88.0
83.7
88.0
93,0
90.3
87.9
2.6
3.2
9.3
3.3
3.2
9.0
89
93.8
89.3
52.8
85.5
90.3
62.7
67.2
98.1
91.7
91.4
89. 98.4
98.0
93.5 95.6
96.8 95.5
2.9
3.2
11.7
3.4
8.8
8.8
96.7
94.7
38.
94.5
81.9
62.9
99
98.2
96.5 80.9
97.8
car
8.6
10.1
16.2
10.4
1200
11.5
58.6
49.5
23.1
46.7
67.1
37.
37.0
90.5
87.0
85.6 74.9
94.4
89
91.1
5.8
9.3
18.7
9.9
8.5
11.0
11.4
70.8
52.9
15.9
50.7
57.5
42.0
40
84.8
76.9
67.8
82.7
82.4
82.0
table
12.5
14.1
21.9
14.9
14.5
17.1
16.6
39,6
29.0
12.1
29.8
27.7
19.5
18.9
87.5
85.0
71.0
82.0
85.3
79.7
80.7
mbike
8.6
14.2
8.4
9.2
8.7
63.8
58.5
31.7
59.5
70.3
57.5
62.5
94.9
95.0
89,9
95.7
97.5
96.3
97.0
4.3
5,4
20.5
6.1
10.0
9.9
79,3
70.1
18.7
63.9
69.7
50.2
51
89,2
830 82.2
76.3 77.5
76.2
93.8
96.0
94.9
10.5
12.2
18.2
12.3
12.1
14.9
15.8
47,9
25.7
42.6
40.0
25.4
24.9
83.9
82.3
89,4
84.6
83.3
6.9
8.8
16.1
9.1
9,0
11.8
11.7
69.2
61.0
27.8
59.7
59.5
43.9
44,6
90.1
86.1
81.1
85.3
89.4
87.6
88.
Mean
21

---

Published as a conference paper at ICLR 2023
Toyat Irage
NeMo- VoCTlpong)
Ny TowhdD
Irpec Image
NeMae (oGF(in))
NcMo-PyTewckDD
Figure 18: Additional qualitative in-wild object pose estimation results for NeMo+VoGE and NeMo+PyTone Torchh....
iterO
iter250
iter 500
iter 1000
iter 2000
target
PyTorchDD
VoGE
Figure 19: The shape fitting process regarding optimization iterations. visualize both VoGE and PyTorch3D with all constraints.
C ADDITIONAL EXPERIMENT RESULTS
C.I IN-WILD OBJECT POSE ESTIMATION
Ablation Study. As Table shows, conduct controlled experiments validate the effects of different geometric primitives. Using the method we described in 3.2, we develop tools that convert triangle meshes Gaussian ellipsoids, where a tunable parameter, coverage rate, is used control the intersection rate between nearby Gaussian ellipsoids. Specifically, the higher coverage rate gives the large Σ. which makes the feature more smooth but also fuzzy, vice versa. As the results demon- strate, increasing Σ can increase the rough performance under E. while reducing it can improve the performance under the more accurate evaluation threshold. We also ablate the affect regarding block part of the gradient in Equation 8. Specifically, conduct two experiments all kernels, we block the gradient on T T(4x) and e respectively. The results show blocking either term leads significant negative impact the final performance.
Additional Results. Table 5 shows the per-category object pose estimation results on PASCAL3D+ dataset (LO). All NeMo Wang et al. (2020a) baseline results and ours conducted using the single cuboid setting described NeMo. Specifically, Gaussian ellipsoids used VoGE is converted from the same single cuboid mesh models provided by NeMo (coverage rate 0.5).
22

---

Published as a conference paper at ICLR 2023
Loss vs iterations
rgb loss
sil houette loss edge loss
0.25
0.70
normal loss
leplacian loss
Loss
0.35
250
500
750
1000
1150
1560
1750
2000
Iteration
(a) VoGE with all constraints.
Loss iterations
rgb loss
silhouette loss
0.0
edge
0.5
normal loss
laplacian loss
250
300
150
1000
1250
1300
1750
2000
Iteraion
(b) VoGE without constraints.
Loss vs iterations
rgb loss
0.755
silhouette loss
0150
edge loss
normal
laplacian loss
Loss
0.75
0.50
Iteration
(c) PyTorc33 all constraint.
Figure 20: Losses in the shape fitting experiment. Note that for the VoGE without constraint, only calculate the geometry loss but not compute gradient with those losses.
Figure 18 shows the additional qualitative results of the object pose estimation. In the visualization,
we use standard graphic renderer to render the original CAD models provide by PASCAL3D+ dataset under the predicted pose, and superimpose the rendered object onto the input image.
23

---

Published as a conference paper at ICLR 2023
C.2 TEXTURE EXTRACTION AND RERENDERING
Figure 21 shows the additional texture extraction and rerendering results on car, bus and boat images from PASCAL3D+ dataset. Interestingly, Figure 21 (g) shows the texture extraction using VoGE demonstrate stratifying generation ability on those out distributed cases.
C.3 SHAPE FITTING VIA INVERSE RENDERING
Figure shows the losses in the multi-viewed shape fitting experiment. Specifically, we plot losses regarding optimization iterations using the method provided by fir mesh with texture via rendering from PyTorch3D official tutorial Ravi et al. (2022a). Note the geometry constraint losses except normal remain relatively low in VoGE without constraints experiment. We think such results demonstrate the optimization process using VoGE can give correct gradient toward the optimal so- lution effectively, that even without geometry constraint the tightness of the Gaussian ellipsoids is still retained. As for the normal consistency loss, since we use the volume Gaussian ellipsoids, the surface normal directions are longer informative.
24

---

Published as a conference paper at ICLR 2023
Synthesized Image on Novel View
Inpul Image
Gaussaa Elligooids
+10
(b)
(h)
(k)
Figure 21: Additional results for texture extraction experiment on car, bus and boat category in PASCAL3D+ dataset. We extract texture using in-wild image and Gaussian ellipsoids with corre- sponded viewpoint, and render under novel viewpoint.
25


Reviews:
[{"review_id": "srV5kSIHQf", "reviewer": "Reviewer_LqFK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel differentiable renderer. It represents the scene \nusing a set of Gaussian ellipsoids that determines the density of each point. \nDuring rendering, it traces each ray and accumulates the contribution of \neach ellipsoid using volume rendering. The paper demonstrates the effectiveness \nof the renderer on tasks such as pose estimation and shape fitting, and shows\nthat it achieves better performance that rasterization-based differentiable\nrenderer such as Pytorch3D.", "summary_of_the_review": "Overall, I believe that the combination of 3D Gaussian and volume renderings is \nnovel and technically sound. The paper shows the effectiveness of the proposed\nmethod on multiple tasks. Therefore, I hold a positive attitude toward the\npaper.", "strength_and_weaknesses": "Strengths\n1. Compared to rasterization-based differentiable renderers, the proposed method \nutilizes volume density and can better reason about occlusions and\noverlapped components. \n\n2. Compared to implicit volumetric representation, the usage of Gaussian\nellipsoid is more intuitive and can be easily converted from other\nrepresentations such as meshes and point clouds.\n\n3. The paper derives an efficient approximation for the aggregation function  \nthat is differentiable with respect to both visible and invisible components.\n\n4. The paper shows that the proposed renderer achieves better performance than\nbaseline methods on multiple downstream tasks. \n\nWeakness\n1. When talking about implicit representations such as Mildenhall2020, the paper\nsays that it lacks modifiability and interpretability. It's not clear to me why \nthe proposed Gaussian ellipsoids representation can be better in these aspects,\nand it's worth adding more discussions here. Also recently there are new\nvolumetric representations such as Plenoxels and Direct Voxel Grid Optimization\nthat apply feature grids + MLPs. It will also be better to discuss these related works, \nand how the proposed method will be better. \n\n2. It's also worth discussing relations to physically-based differentiable\nrendering methods, such as \n* Differentiable Monte Carlo Ray Tracing through Edge Sampling\n* Differentiable Signed Distance Function Rendering\n* A Differential Theory of Radiative Transfer\n\n3. In the inverse rendering tasks, the produced results are blurry. What are the\nlimiting factors for the proposed method to generate sharper renderings? Can it\nbe improved by increasing the number of Gaussian ellipsoids?\nAlso, the paper says a silhouette is used. I am wondering whether it's really\nnecessary, considering that volume-based methods such as Mildenhall2020 do not\nrequire mask supervision. Similarly, it's also not clear to me why additional\nconstraints are needed.  \n\n4. In Figures 14 and 16, it seems that the method produced normal maps with\nobvious artifacts, especially on the Armadillo and the dragon scene. What are\nthe reasons for those artifacts?\n\n5. I think one advantage of the proposed method over rasterization-based\nmethod is that it's using volume rendering and therefore can handle translucent\nobjects. It will be interesting to add a comparison on tasks like fitting a\ntranslucent object.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WLt-jeWKdeh", "reviewer": "Reviewer_ppJY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new differentiable volumetric rendering method based on ray casting through a discrete number of Gaussian shaped occlusion density functions. In practice, assumptions are made to simplify the volumetric rendering equation in terms of peak/max densities in order to reduce the rendering equation to something that can be computed efficiently in closed form. The authors show competitive speed to PyTorch3D, with applications to pose estimation, novel view synthesis, and inverse rendering.", "summary_of_the_review": "I like this work but do not like this paper. I think the work should be published and would be of considerate interest to the community. However, I think the paper is not well-organized and the ideas are not well-presented. Part of this problem is due to simple typographical issues, but part of this problem is due to hard to understand and haphazard exposition.", "strength_and_weaknesses": "Strengths: The idea is good and the background is well established and covered. As a differentiable rendering method, the idea to decompose the scene into \"ellipsoidal blobs\" allows much better occlusion/de-occlusion reasoning for the applications covered in the paper, and therefore, better performance than image space differentiable rendering techniques.\n\nWeaknesses: The paper is fairly confusing, and the exposition needs to be reworked. I had trouble understanding it during the first read-through, in part due to Equation references that do not appear in the main text (is the main paper referencing equations from Appendices?). There is also a lot of typographical errors, including simple citation errors that could have been easily spotted and fixed with a single read-through. This tells me that, despite its length and the richness of the idea, this paper was surely rushed. \n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F-fer8jqAL", "reviewer": "Reviewer_cWiy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a differentiable rendering algoithm that converts 3D meshes into a collection of 3D gaussians and ray traces the resulting scene. The conversions to volumetric gaussians smoothes the visibility function of the scene, similar to the smooth visibility and blending functions of soft differentiable rasterizers (SoftRasterizer, PyTorch3D).\n\nThe paper describes how to project a set of 3D gaussians along a 1D camera ray, and how to solve the resulting volumetric rendering integral in closed form. \n\nTwo approximations are mentioned: One sub-samples the geometry by only rendering 10% of all gaussians in a coarse-to-fine approach. The other optimization clusters nearby gaussians, which further reduces the number of 3D density samples.\n\nThe paper describes how arbitrary attributes can be associated with the 3D gaussians, which allows for the rendering of RGB textures or neural features.\n\nThe paper presents several experiments, including pose estimation of CAD shapes, texturing, shape fitting, and a synthetic occlusion experiment.\n", "summary_of_the_review": "The closed form solution for rendering a set of gaussians is a worthwhile contribution, but overall the exposition in the paper is quite difficult to follow. Experiment 4.1 appears to be a strong result, but Experiment 4.2 is in my opinion not very relevant. Experiments 4.3 and 4.4 are quite basic and would benefit from a more thorough discussion of how mesh conversion and coarse-to-fine rendering is handled.\n", "strength_and_weaknesses": "With 3D gaussians, the paper introduces an interesting representation for differentiable volumetric rendering. The algorithm for rendering a set of gaussians is sound and the experiments demonstrate strong results on the pose estimation task and plausible behavior on the inverse rendering experiments.\n\n---\n\nThe paper is unfortunately quite difficult to follow. The formatting suggests that not much time was spent on proof-reading the paper, as the citation format expected by the authors is inconsistent with the ICLR format. Some sections are tiring to read from the frequent duplication of author names in the citations (see Section 2, first paragraph).\n\nThe authors use non-standard terminology for common concepts, which does not help the readability. For example, in Table 1, it would be better to use conventional graphics terms like these:\n\n - \"Component\" -> \"Geometric primitive\"\n - \"Component Tracking\" -> \"Visibility algorithm\"\n - \"Aggregation\" -> \"Blending\"\n\nCertain statements are confusing and likely incorrect, for example the statement that \"Graphics renderers use explicit object representations, which represent objects as a set of isotropic components.\"  (Sec. 2) (What components are referred to? What is \"isotropic\" about conventional graphics rendering?)\n\n---\n\nWhile the paper is quite clear in how a set of gaussians is rendered, the way in which shapes are converted to sets of gaussians is not sufficiently discussed. There is one paragraph on the bottom of page 4 that mentions how to pick the covariance matrix, but it is not mentioned in the main text how the number or position of the gaussians is chosen. From the appendix, it becomes clear that there is a 1-to-1 mapping with variance based on the neighboring edge length. \n\nThere are potential issues with this approach, in that the softness of the rendering (both spatially and along the ray) will depend on the mesh tesselation. Coarse meshes will be very soft, while finely tesselated meshes will converge to point-based splatting. The paper should include an analysis of this effect with an ablation and/or failure cases.\n\n---\n\nThe mentioned coarse-to-fine approximations are not well motivated and their impact not thoroughly evaluated.\n\n---\n\nSection 4.2 / Figure 7:\n\nIf the CAD model and pose is provided, this result is expected of any renderer that supports RGB textures. This result does not require a differentiable renderer, as each vertex (or gaussian) can simply be projected into the image, where its RGB value can be looked up without back propagation or optimization. It is not clear why training or symmetry information (Figure caption) would help.\n\n---\n\nFigure 8: How many gaussians are rendered for each cuboid? It appears that there are more than 8 gaussians, but the appendix mentions a 1-to-1 vertex to gaussian mapping. Can each gaussian vary its position independently, or are the gaussian positions constrained by the cuboid shape?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1rRqq7Jw5a", "reviewer": "Reviewer_qZYG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a method for differentiable volume rendering using a set of Gaussian ellipsoid primitives. The key idea is to use a large set of Gaussian ellipsoids to parameterize the density and color within a volume and then to render using ray marching to solve the volume rendering equation. Conveniently, integrals along rays through the ellipsoids can be computed in closed form, leading to computationally efficient rendering. The method is demonstrated on a pose estimation task, where a ground truth mesh is aligned to a model, as well as for rendering textures, point clouds, and for shape fitting from multiview imagery. Overall, the method appears to outperform other techniques for the pose estimation task, and qualitative results suggest good performance for other tasks.\n", "summary_of_the_review": "Overall, while I think the idea of using Gaussian ellipsoids as a primitive for differentiable rendering is interesting and potentially very reasonable (as it builds on a long legacy of using Gaussian primitives for volume rendering), I have some reservations about the evaluation and lack of quantitative comparisons for 3/4 of the presented tasks. I find it difficult to get a sense of how well the method performs. There are also some technical details which appear to be missing from the paper. As it stands, I lean slightly negative on the paper.", "strength_and_weaknesses": "The idea of using Gaussian ellipsoids in the context of volumetric rendering, while not new, is nevertheless interesting in the context of differentiable rendering because one can compute the rendering integrals analytically. The paper does a good job of conveying the efficiency of the approach, as even megapixel images can be rendered in real time for a scene with thousands of Gaussian ellipsoids.\n\nThe quantitative results also shows that the method performs well for the pose estimation task (which involves aligning ground truth geometry to an input image).\n\nStill, the paper has a few weaknesses: there are a large number of typos and grammatical errors that hinder the clarity of the paper, and there are several details lacking from the technical description and evaluation of the method that seem important.\n- There are no quantitative results that evaluate the texture extraction and rerendering, occlusion reasoning, and shape fitting via inverse rendering tasks.\n- It's not always clear how the method is initialized, which is one potential drawback. How many Gaussian ellipsoids are required in the experiments and how does the rendering quality depend on this? This seems like a critical parameter to evaluate.\n- Given the connection to volumetric rendering methods and the interest in multiview reconstruction, I was hoping to see more reconstruction results and how this compares to methods that use a neural network to parameterize the volume density and color. Is there an advantage here? Or does the requirement of initializing a constant number of ellipsoids a priori limit the performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster