Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for converting a standard non-differentiable rasterization-based renderer into a differentiable counterpart using recent advances in the perturbed optimizer. The main contribution lies in the generic nature provided by the theoretical formulation, where some specifically chosen priors in the proposed framework can retrieve recent state-of-the-art differentiable renderers, e.g. SoftRas. 
As the proposed approach mainly relies on a Monte-Carlo estimator for generating the smoothing parameters. Variance-reduction mechanism and sensitivity analysis are also incorporated into the proposed framework to ensure the robustness of the approximate renderer. Experiments on pose estimation and single-view reconstruction are conducted for comparing the performance of the proposed method and previous state-of-the-art approaches.
main_review: Pros:

- By introducing the perturbed optimizer to analyzing the non-differentiability of standard rasterization-based rendering pipeline provides new insight into constructing a generic differentiable renderer. The perturbed differentiable rendering can also be implemented on top of existing renderers without major modifications, which makes this approach versatile and a potential performance booster for existing DR techniques.

Cons:

- In terms of technical novelty, it seems to be a direct application of perturbed optimizer to differentiable rendering. Though it requires some efforts (such as the variance reduction and sensitivity analysis) to adapt the existing theory to this specific application, it is not very challenging as there are existing protocols to follow.

- Evaluation-wise, in the task of single-view reconstruction, though the proposed method can achieve comparable performance compared to the state-of-the-art approaches, it still fails to outperform DIB-R in mean accuracy and also outperformed by SoftRas in many categories, such as the Dresser, Car, etc. The 3D pose estimation is only evaluated on a simple cube which cannot provide a very comprehensive measurement of the proposed approach. 
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper proposes a differentiable rendering method that builds connection between the 3D model and 2D perspective appearance. Different from previous methods, the differentiable rendering is formulated into an optimization problem. Two computer vision tasks, including pose estimation and 3D shape from color image, are evaluated with the proposed renderer and compared to SoftRas [22].
main_review: Strength:
- The performance for pose estimation outperforms existing methods.
- The formulation seems novel.

Weakness:
- Could the proposed formulation be applied to other 3D shape representations, e.g. point cloud, implicit? If so, an experiment should be added. If not, then better to mention this is a mesh-specific technic.

- The algorithm seems to be computational costy, e.g. 1 min per image. Would this heavily affect the training efficiency? While comparing to other methods, e.g. SoftRas, the run-time efficiency and hyper-parameters, e.g. number of iteration, training epoch, should be mentioned. Otherwise the comparison may not be on a fair basis.

- Keeping the boundary differentiable is one of the main issues in existing differentiable rendering technics, which is also argued mainly in this paper. However it is not clear how this is improved by the proposed method. From Fig 2 and 4, it seems the boundary area has obvious artifacts. Also some experiments should be performed to highlight the improvements over occlusion boundary.

- The performance on 3D shape prediction does not outperform existing approaches. 

Overall, despite a novel formulation, the improvements over existing approaches are not clear. As shown in the experiment, the propose method does not necessarily produce more accurate rendering or support better for vision tasks. Considering that the proposed tech is also quite slow, the overall advantage over existing approaches is not very clear.
limitations_and_societal_impact: No comment.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new differentiable renderer by introducing perturbed optimizers. The paper re-writes the non-differentiable rasterization and aggregation steps to heaven side functions and relaxes them to be differentiable by adding noise. 
The author shows that previous work, especially softras, can be treated as a special case of the perturbed renderer with specific prior selections. The authors also demonstrated in several experiments they achieve better or comparable results than the state of the art.
main_review: Strengths

1) Differentiable reformulation
This paper treats rasterization as a linear programing problem.  It extends the [4] to the rendering pipeline and relaxes the non-differentiable rendering to be differentiable by introducing perturbed optimizers. It provides a new differentiable renderer from a new perspective.

2) Better performance
The authors show AS and VR are very useful in optimization, for both perturbed renderer and softras. In the 6D pose comparison, the results are better than softras.

 

limitations_and_societal_impact: There are several weak points of this paper.

1) The paper is not self-contained.  I have a hard time understanding each equation. While the authors largely rely on [4], it makes readers hard to follow without knowing [4]. Moreover, the notations in this paper are very vague.   E.g. line 66 it should be j \in [1,.., m]?  line 76 z should be depth order?  If it is depth, how do you compute <z, y>?  line 88, what's v?  The paper writing definitely needs to be improved.


2) Relation of softras and noise types.  The authors show that softras can be treated as the special case of the perturbed renderer. Softras has some hyperparameters which are hard to explain but in the new renderer, all the parameters have clear noise intensity meaning.  While this is very good, but the paper lacks further analysis of which noise should we use. E.g. in table 1, it seems Gaussian noise is the best. But could it be generalized to other cases? 


3) Comparison.  In line 214 the authors said, "Unless stated, the results for the other renderers are extracted from their original publications, as we were not able to get access to their implementations."  I am a little confused about this.  Clearly, N3mr, softras, dibr released their renderers. Thus, why not simply replace the renderer from perturbed renderer to new renderers and compare the score?  Directly copying scores from others' papers may have the wrong comparison. While all the methods use shapenet, the rendered images could vary from lighting to camera views. I would highly recommend the authors redo the comparison.


4)  The authors assume each pixel will be influenced by all the triangles. For each pixel, it renders the influences of all the triangles and combines them together(Eq 3).  I wonder what's the rendering speed of the perturbed renderer? Theoretically, such design plus the MC sampling will make the renderer very time-consuming.  The authors should show the rendering speed comparison.


5) Following the last concern.  Generally speaking, rasterization accelerates the rendering speed by assuming each pixel is influenced by the frontest triangle.  On the other side, MC is always used in ray tracing to compute multiple bounces. The authors introduce MC in rasterization, which seems to adopt the weak points of two rendering methods.   I wonder did the authors consider extending it to RT?

Post rebuttal:

The authors addressed my concerns and I will raise the score to be marginally above the threshold and be happy to see this paper get accepted.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a new differentiable rendering method, differentiable perturbed renderers,  wherein the gradients of the rasterization step are approximated using perturbed optimizers. They implement this change on top of the SoftRas method and demonstrate that it leads to better pose estimation, and on par performance with state of the art methods in unsupervised single image 3d object reconstruction. 
main_review: This is a strong paper. It is written clearly, and the method is explained properly. Perturbed optimizers seem to be a new way for tackling optimization over discrete operations in deep learning, and their application to rasterization is well motivated and appropriate. The theory required for applying them to this new setting seems right, and is well explained. The results are also adequate for the claims made. It would have been nice to see higher performance on the reconstruction task, though diff rendering is a relatively new and so highlighting new avenues to explore , such as your proposed perturbed optimizers,  is important, and so I do not believe it is necessary.

 I would say one nice set of experiments to see would be optimization over the same set of parameters as in the DIB-R  paper Figure 2. ( camera position, texture , vertex positions, etc. ), with qualitative and quantitative comparisons to DIB-R and SoftRas. Having this would go a long was to increasing my score, and I believe would sell you paper quite a bit better. The pose estimation seems quite a bit easier than optimizing for texture and vertex position for example, and it would be nice to see how you method fairs, even if the performance is only on par or worse. 
limitations_and_societal_impact: Yes.
review_title: null

Meta-review: This paper has mixed reviewers. Most of reviewers agree with the novelty and contribution of proposing a new differentiable rendering method, differentiable perturbed renderers. The whole paper is solid and experiments are good enough to validate the contribution. Reviewer yvH4 had some serious concerns about the practical issues of the proposed methods, including generalization ability to other 3D shape representations, computational expensive, not competitive to existing approaches. The authours clarified some of these points. In the final version, the authours are advisable to futher integrate the rebuttal and improve the paper quality.
Decision: Accept (Poster)