Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new differentiable volumetric rendering method based on ray casting through a discrete number of Gaussian shaped occlusion density functions. In practice, assumptions are made to simplify the volumetric rendering equation in terms of peak/max densities in order to reduce the rendering equation to something that can be computed efficiently in closed form. The authors show competitive speed to PyTorch3D, with applications to pose estimation, novel view synthesis, and inverse rendering.
strength_and_weaknesses: Strengths: The idea is good and the background is well established and covered. As a differentiable rendering method, the idea to decompose the scene into "ellipsoidal blobs" allows much better occlusion/de-occlusion reasoning for the applications covered in the paper, and therefore, better performance than image space differentiable rendering techniques.

Weaknesses: The paper is fairly confusing, and the exposition needs to be reworked. I had trouble understanding it during the first read-through, in part due to Equation references that do not appear in the main text (is the main paper referencing equations from Appendices?). There is also a lot of typographical errors, including simple citation errors that could have been easily spotted and fixed with a single read-through. This tells me that, despite its length and the richness of the idea, this paper was surely rushed. 

 
clarity,_quality,_novelty_and_reproducibility: Quality: I think this is good work, and does contribute a nice idea to the field of differentiable/inverse rendering. I do take small issues with some of the results and figures. For example, I don't find Figure 5 very illuminating without also comparing rendering fidelity. That is, I don't care if one method is faster than the other if has worse fidelity and vice versa. Also, I am a little curious about the novel view synthesis results of Figure 7-- without training, the network cannot predict any new information-- so is this result simply due to the bilateral symmetry of a car and ellipsoidal coloring that can be rotated geometrically? In general, I would want more explanation given for this surprising result.

Clarity: The clarity, at least for me (and I do research in this are), was very bad. Yet, I believe there is a much more straightforward exposition possible, I just don't think the authors took the time to structure their exposition for comprehension. I wish the paper was better written because I really do like the idea and execution, but I don't think it is publication-ready at the current time. For example, I would put the method for turning a mesh into Gaussians in the main text, and I would also explain how the K nearest ellipsoids are selected for ray tracing (as well as compare compute times & fidelity with and without this technique). Other time I think a simple proofread could have helped clarity-- for example, in Section 3.4, the word "donates" is used several times, but I think the word should have been "denotes"? 

Originality: I think the idea is original and I think the author's well-covered earlier work that has similar ideas. I also appreciate Table 1 for contextualizing the method.
summary_of_the_review: I like this work but do not like this paper. I think the work should be published and would be of considerate interest to the community. However, I think the paper is not well-organized and the ideas are not well-presented. Part of this problem is due to simple typographical issues, but part of this problem is due to hard to understand and haphazard exposition.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces a differentiable rendering algoithm that converts 3D meshes into a collection of 3D gaussians and ray traces the resulting scene. The conversions to volumetric gaussians smoothes the visibility function of the scene, similar to the smooth visibility and blending functions of soft differentiable rasterizers (SoftRasterizer, PyTorch3D).

The paper describes how to project a set of 3D gaussians along a 1D camera ray, and how to solve the resulting volumetric rendering integral in closed form. 

Two approximations are mentioned: One sub-samples the geometry by only rendering 10% of all gaussians in a coarse-to-fine approach. The other optimization clusters nearby gaussians, which further reduces the number of 3D density samples.

The paper describes how arbitrary attributes can be associated with the 3D gaussians, which allows for the rendering of RGB textures or neural features.

The paper presents several experiments, including pose estimation of CAD shapes, texturing, shape fitting, and a synthetic occlusion experiment.

strength_and_weaknesses: With 3D gaussians, the paper introduces an interesting representation for differentiable volumetric rendering. The algorithm for rendering a set of gaussians is sound and the experiments demonstrate strong results on the pose estimation task and plausible behavior on the inverse rendering experiments.

---

The paper is unfortunately quite difficult to follow. The formatting suggests that not much time was spent on proof-reading the paper, as the citation format expected by the authors is inconsistent with the ICLR format. Some sections are tiring to read from the frequent duplication of author names in the citations (see Section 2, first paragraph).

The authors use non-standard terminology for common concepts, which does not help the readability. For example, in Table 1, it would be better to use conventional graphics terms like these:

 - "Component" -> "Geometric primitive"
 - "Component Tracking" -> "Visibility algorithm"
 - "Aggregation" -> "Blending"

Certain statements are confusing and likely incorrect, for example the statement that "Graphics renderers use explicit object representations, which represent objects as a set of isotropic components."  (Sec. 2) (What components are referred to? What is "isotropic" about conventional graphics rendering?)

---

While the paper is quite clear in how a set of gaussians is rendered, the way in which shapes are converted to sets of gaussians is not sufficiently discussed. There is one paragraph on the bottom of page 4 that mentions how to pick the covariance matrix, but it is not mentioned in the main text how the number or position of the gaussians is chosen. From the appendix, it becomes clear that there is a 1-to-1 mapping with variance based on the neighboring edge length. 

There are potential issues with this approach, in that the softness of the rendering (both spatially and along the ray) will depend on the mesh tesselation. Coarse meshes will be very soft, while finely tesselated meshes will converge to point-based splatting. The paper should include an analysis of this effect with an ablation and/or failure cases.

---

The mentioned coarse-to-fine approximations are not well motivated and their impact not thoroughly evaluated.

---

Section 4.2 / Figure 7:

If the CAD model and pose is provided, this result is expected of any renderer that supports RGB textures. This result does not require a differentiable renderer, as each vertex (or gaussian) can simply be projected into the image, where its RGB value can be looked up without back propagation or optimization. It is not clear why training or symmetry information (Figure caption) would help.

---

Figure 8: How many gaussians are rendered for each cuboid? It appears that there are more than 8 gaussians, but the appendix mentions a 1-to-1 vertex to gaussian mapping. Can each gaussian vary its position independently, or are the gaussian positions constrained by the cuboid shape?

clarity,_quality,_novelty_and_reproducibility: There are issues with the clarity of the writing, see my comments in the previous section. 

A few additional references should be added,

"Learning Deformable Tetrahedral Meshes for 3D Reconstruction"
NeurIPS 2020

"Differentiable Monte Carlo ray tracing through edge sampling"
ACM Transactions on Graphics 2 TOG 2018

"Mitsuba 2: A Retargetable Forward and Inverse Renderer"
SIGGRAPH Asia 2019

"Modular Primitives for High-Performance Differentiable Rendering"
ACM Transactions on Graphics 2020

summary_of_the_review: The closed form solution for rendering a set of gaussians is a worthwhile contribution, but overall the exposition in the paper is quite difficult to follow. Experiment 4.1 appears to be a strong result, but Experiment 4.2 is in my opinion not very relevant. Experiments 4.3 and 4.4 are quite basic and would benefit from a more thorough discussion of how mesh conversion and coarse-to-fine rendering is handled.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel differentiable renderer. It represents the scene 
using a set of Gaussian ellipsoids that determines the density of each point. 
During rendering, it traces each ray and accumulates the contribution of 
each ellipsoid using volume rendering. The paper demonstrates the effectiveness 
of the renderer on tasks such as pose estimation and shape fitting, and shows
that it achieves better performance that rasterization-based differentiable
renderer such as Pytorch3D.
strength_and_weaknesses: Strengths
1. Compared to rasterization-based differentiable renderers, the proposed method 
utilizes volume density and can better reason about occlusions and
overlapped components. 

2. Compared to implicit volumetric representation, the usage of Gaussian
ellipsoid is more intuitive and can be easily converted from other
representations such as meshes and point clouds.

3. The paper derives an efficient approximation for the aggregation function  
that is differentiable with respect to both visible and invisible components.

4. The paper shows that the proposed renderer achieves better performance than
baseline methods on multiple downstream tasks. 

Weakness
1. When talking about implicit representations such as Mildenhall2020, the paper
says that it lacks modifiability and interpretability. It's not clear to me why 
the proposed Gaussian ellipsoids representation can be better in these aspects,
and it's worth adding more discussions here. Also recently there are new
volumetric representations such as Plenoxels and Direct Voxel Grid Optimization
that apply feature grids + MLPs. It will also be better to discuss these related works, 
and how the proposed method will be better. 

2. It's also worth discussing relations to physically-based differentiable
rendering methods, such as 
* Differentiable Monte Carlo Ray Tracing through Edge Sampling
* Differentiable Signed Distance Function Rendering
* A Differential Theory of Radiative Transfer

3. In the inverse rendering tasks, the produced results are blurry. What are the
limiting factors for the proposed method to generate sharper renderings? Can it
be improved by increasing the number of Gaussian ellipsoids?
Also, the paper says a silhouette is used. I am wondering whether it's really
necessary, considering that volume-based methods such as Mildenhall2020 do not
require mask supervision. Similarly, it's also not clear to me why additional
constraints are needed.  

4. In Figures 14 and 16, it seems that the method produced normal maps with
obvious artifacts, especially on the Armadillo and the dragon scene. What are
the reasons for those artifacts?

5. I think one advantage of the proposed method over rasterization-based
method is that it's using volume rendering and therefore can handle translucent
objects. It will be interesting to add a comparison on tasks like fitting a
translucent object.
clarity,_quality,_novelty_and_reproducibility: 1. The paper is math-intensive, especially in the derivation of the aggregation 
function. I would suggest the authors add an illustration to the paper with 
each symbol, and also it will be better to start with a high-level idea of the 
approximation. It's difficult to follow in the current format.

2. The quality and novelty of the paper look good to me. The combination of
3D Gaussians and volume rendering is interesting and technically sound.

3. The reproducibility remains a concern, considering that there are many
components in the paper with subtle details. It will be great if the authors can
release the code.


summary_of_the_review: Overall, I believe that the combination of 3D Gaussian and volume renderings is 
novel and technically sound. The paper shows the effectiveness of the proposed
method on multiple tasks. Therefore, I hold a positive attitude toward the
paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method for differentiable volume rendering using a set of Gaussian ellipsoid primitives. The key idea is to use a large set of Gaussian ellipsoids to parameterize the density and color within a volume and then to render using ray marching to solve the volume rendering equation. Conveniently, integrals along rays through the ellipsoids can be computed in closed form, leading to computationally efficient rendering. The method is demonstrated on a pose estimation task, where a ground truth mesh is aligned to a model, as well as for rendering textures, point clouds, and for shape fitting from multiview imagery. Overall, the method appears to outperform other techniques for the pose estimation task, and qualitative results suggest good performance for other tasks.

strength_and_weaknesses: The idea of using Gaussian ellipsoids in the context of volumetric rendering, while not new, is nevertheless interesting in the context of differentiable rendering because one can compute the rendering integrals analytically. The paper does a good job of conveying the efficiency of the approach, as even megapixel images can be rendered in real time for a scene with thousands of Gaussian ellipsoids.

The quantitative results also shows that the method performs well for the pose estimation task (which involves aligning ground truth geometry to an input image).

Still, the paper has a few weaknesses: there are a large number of typos and grammatical errors that hinder the clarity of the paper, and there are several details lacking from the technical description and evaluation of the method that seem important.
- There are no quantitative results that evaluate the texture extraction and rerendering, occlusion reasoning, and shape fitting via inverse rendering tasks.
- It's not always clear how the method is initialized, which is one potential drawback. How many Gaussian ellipsoids are required in the experiments and how does the rendering quality depend on this? This seems like a critical parameter to evaluate.
- Given the connection to volumetric rendering methods and the interest in multiview reconstruction, I was hoping to see more reconstruction results and how this compares to methods that use a neural network to parameterize the volume density and color. Is there an advantage here? Or does the requirement of initializing a constant number of ellipsoids a priori limit the performance?
clarity,_quality,_novelty_and_reproducibility: With respect to clarity, there are a large number of typos and grammatical errors that would need to be corrected. I list a few below:
- Intro: "Liu et al. Liu et al. (2019)"
- Related works: "Blinn Blinn" and other typos for the references. There are similar typos with the references throughout the paper that should be corrected.
- Volume density aggregation: This sentence has many grammatical errors: "However, computing the integral using brute force is so [computationally] inefficient that [it is] infeasible for [current devices]"
- Section 4.4 "official tutorial pyt"

There are a number of other technical questions I had, which I did not see addressed:
- For the texture extraction and rerendering task is the CAD model already aligned to the image? If so, how does this compare to other simple baselines? For example, couldn't one simply map each vertex to a color and render using, e.g., rasterization and barycentric interpolation of the colors on the mesh faces? 
- When ray tracing how does one know which ellipsoids contribute to the ray? Does this require a hit test against all ellipsoids?
- How are the ellipsoids initialized when doing shape reconstruction?

As for novelty, there is a long history of using "Gaussian transfer functions" for volume visualizations. Some of the references in this line of work appear in the paper. So, the main novelty here is the incorporation of Gaussian primitives into the differentiable rendering framework.   

With respect to reproducibility, I think there is no problem as the authors also promise to release the code.

Additional comment:
- Fig. 2: indicate the row of the image for which the plot is given (e.g., with a dashed line across the image).
summary_of_the_review: Overall, while I think the idea of using Gaussian ellipsoids as a primitive for differentiable rendering is interesting and potentially very reasonable (as it builds on a long legacy of using Gaussian primitives for volume rendering), I have some reservations about the evaluation and lack of quantitative comparisons for 3/4 of the presented tasks. I find it difficult to get a sense of how well the method performs. There are also some technical details which appear to be missing from the paper. As it stands, I lean slightly negative on the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work focuses on a novel proposed method for differentiable volume rendering.

The reviewers pointed out, the main strengths of this work are the performance of the proposed method, especially its performance on some downstream tasks compared to existing baselines. They also commanded the general proposed idea, and how well the literature is covered. They found that the main weakness was the exposition, both in terms of clarity and use of unconventional terms.

These have been appropriately addressed by the authors during the rebuttal process.


justification_for_why_not_higher_score: There is not a wide consensus among the reviewers in favour of this work, and there were some flaws in the initial version of this work.
justification_for_why_not_lower_score: See meta review above.
Decision: Accept: poster