Differentiable rendering with perturbed optimizers
Authors: Quentin Le Lidec
Ivan Laptev
Cordelia Schmid
Justin Carpentier
Venue: NeurIPS 2021 Poster (2021)
Keywords: Differentiable rendering
Computer Vision
Optimization
Randomized smoothing

Abstract: Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision. Solutions to this inverse and ill-posed problem typically involve a search for models that best explain observed image data. Notably, images depend both on the properties of observed scenes and on the process of image formation. Hence, if optimization techniques should be used to explain images, it is crucial to design differentable functions for the projection of 3D scenes into images, also known as differentiable rendering. Previous approaches to differentiable rendering typically replace non-differentiable operations by smooth approximations, impacting the subsequent 3D estimation. In this paper, we take a more general approach and study differentiable renderers through the prism of randomized optimization and the related notion of perturbed optimizers. In particular, our work highlights the link between some well-known differentiable renderer formulations and randomly smoothed optimizers, and introduces differentiable perturbed renderers. We also propose a variance reduction mechanism to alleviate the computational burden inherent to perturbed optimizers and introduce an adaptive scheme to automatically adjust the smoothing parameters of the rendering process. We apply our method to 3D scene reconstruction and demonstrate its advantages on the tasks of 6D pose estimation and 3D mesh reconstruction.  By providing informative gradients that can be used as a strong supervisory signal, we demonstrate the benefits of perturbed renderers to obtain more accurate solutions when compared to the state-of-the-art alternatives using smooth gradient approximations.

Differentiable Rendering with Perturbed Optimizers
Quentin Le Lidec, Ivan Laptev, Cordelia Schmid and Justin Carpentier
Inria Départemmet d' d'Informatique de l'École normale supérieure, PSL Research Un iversity (quentin le- lidee,dee,, ivan. ivan.laptev, hut.://aa...ccm......un.aooooooo
Abstract
Reasoning about 3D scenes from their 2D image projections is one of the core problems in computer vision. Solutions to this inverse and ill-posed problem typi- cally involve search for models that best explain observed image data. Notably, images depend both on the properties of observed scenes and on the process of image formation. Hence, if optimization techniques should be used to explain images, it is crucial to design differentiable functions for the projection of 3D scenes into images, also known as differentiable rendering. Previous approaches differentiable rendering typically replace non-differentiable operations by smooth approximations, impacting the subsequent 3D estimation. In this paper, we take more general approach and study differentiable renderers through the prism of randomized optimization and the related notion of perturbed optimizers. particu- lar, our work highlights the link between some well-known differentiable renderer formulations and randomly smoothed optimizers, and introduces differentiable per- turbed renderers. We also propose variance reduction mechanism alleviate the computational burden inherent to perturbed optimizers and introduce an adaptive scheme automatically adjust the smoothing parameters of the rendering process. We apply our method to 3D scene reconstruction and demonstrate its advantages the tasks of 6D pose estimation and 3D mesh reconstruction. By providing informative gradients that can be used as a strong supervisory signal, we demon- strate the benefits of perturbed renderers to obtain compared to the state-of-the-arr alternatives using smooth gradient approximations.
accurate solutions when
Introduction
Many common tasks in computer vision such as 3D shape modelling [5, 15, 34, 36, 37] or 6D pose estimation [18, 22, 25, 30, 32] aim at inferring 3D information directly from 2D images. Most of the recent approaches rely on (deep) neural networks and thus require large training datasets with 3D shapes along with well-chosen priors on these shapes. Render compare methods [18, 22] circumvent the non-differentiabiiiyy of the rendering process by learning gradients steps from large datasets. Using more structured strategy would allow to alleviate the need for such a strong supervision.Usiee Using a more structured strategy would allow alleviate the need for such strong supervision. In this respect, differentiable rendering intends to model the effective image generation process to compute gradient related the task to solve. This approach has the benefit of containing the prior knowledge of the rendering process while being interpretable. This makes it possible to provide a supervision for neural networks in challenge of differentiable rendering lies in the non-smoothness of the classical rendering process. In renderer, both the rasterization steps, which consist in evaluating the pixel values by discretizing the 2D projected color-maps, and the aggregation steps, which merge the color-maps of several objects along the depth dimension by using Z-buffering operation, are con-differentiable operations (Fig. 1). Intuitively, these steps imply discontinuities in the final rendered image with respect to the 3D positions of the scene objects. For example, if object moves on plane parallel to the
manner [10, 16, 23). The main
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Projection
Rasterization
Aggregation
3D mesh
plane
Pixel maps
Final pixel
Perlurbalion elong depth axis
Aggregation
Average
Pixel maps
Pixel maps with perturbed depth
Perturbed
Smooth aggregated map
maps
Figure 1: Top: Overview of the rendering process: both rasterization and aggregation steps induce non-smoothness in the computational flow. Bottom: Illustration of the differentiable perturbed aggregation process. The rasterization step is made differentiable in a similar way.
camera, some pixels will immediately change color at the moment the object enters the camera view or becomes unocluded by another object.
In this paper, we propose to exploit randomized smoothing techniques within the context of dif- ferentiable rendering automatically soften the non-smooth rendering operations, making them naturally differentiable. The generality of our approach offers a theoretical understanding of some of the existing differentiable renderers while its flexibility leads to competitive results in practice. We make the following contributions:
state-of-the-ard
We formulate the non-smooth operations occurring the rendering process solutions of optimization problems and, based on recent work [4], we propose natural way of smoothing them using random perturbations. We highlight the versatility of this smoothing formulation and show how it offers theoretical understanding of several existing differentiable renderers.
We propose general way to use control variate methods to reduce variance when estimating the gradients the perturbed optimizers, which allows for sharp gradients even the case of weaker perturbations.
We introduce an adaptive scheme to automatically adjust the smoothing parameters by relying sensitivity analysis of differentiable perturbed renderers, leading robust and adaptive behavior during the optimization process.
We demonstrate through experiments on pose optimization and 3D mesh reconstruction that resulting gradients combined to the adaptive smoothing provide strong signal, leading to more accurate solutions when compared to ofaee-of-the-at alternatives based on smooth gradient approximations [23].
Background
In this section, we review the fundamental aspects behind image rendering and recall the notion of perturbed optimizers which are at the core of the proposed approach.

---

Rasterizer and "aggregater" as optimizers, We consider the rendering process of an RGB image of height h, and width wy, from scene composed with meshes represented by a set of m triangles. We assume that every triangle has already been projected onto the camera plane and their associated color maps C₁ E RgArxwyX X3 j E .. vi] have been computed using a chosen illumination model (Phong [29], Gouraud [13] etc.) and interpolating local properties of the mesh. We denote by / the occupancy map such that 13 is equal to if the center of the 7th pixel is inside the 2D projection of the jth triangle on the camera plane and 0 otherwise. By denoting d(i. j) the Euclidean distance from the center of the jth pixel to the projection of the jth triangle, the rasterization step (which actually consists in computing the occupancy map I) can be written as:
rj H (d(i,)))
where H corresponds to the Heaviside function defined by:
((x) - {
if x 0
1 otherwise 0šv51
argmax yr,
We call R E XgrxxuxX x3 the final rendered image which obtained by aggregating the color map of each triangle. In classical renderers, this step is done by using a Z-buffer so that only foreground objects are visible, This corresponds to:
R = Σ (=)C, with w²(2)
argmax
(z, v):
(3)
SSL 11 =y=0 15=0
where w, R'+! and for S S m, 2j is the inverse depth of the jth triangle and the (m + 1)th coordinate of and w account for the background. The inverse depth of the background is fixed to #m+11 Zmin- From Eq. (2) and (3), it appears that argmax operations play a central role in renderers and are typically non-differentiable functions with respect to their input arguments, as discussed next.
Perturbed optimizers. In [4], Berthet et al. introduce a generic approach to handle non-smooth problems of the form y²( argmax (0, v) with C a convex polytope, and make these problems
yEC
differentiable by randomly perturbing them. More precisely, y (0) necessarily lies on a vertex of the convex set C. Thus, when 0 is only slightly modified, y" remains on the same vertex, but when the perturbation grows up to a certain level, y" jumps onto another vertex. Concretely, this means that (0) is piece-wise constant with respect to O: the Jacobian Joy is null almost everywhere and undefined otherwise. This is why the rasterization (2) and aggregation (3) steps make renderers non-differentiable. Following [4], y² (0) can be approximated by the perturbed optimizer:
y. (0) Ez Ey (0 +EZ)
(4)
Ez argmax (0 eZ, u)
(5)
VEC
where Z is a random noise following distribution of the form (lz) x exp(-v(2)). Then, we have ve (0) +0,, y (0), y (0) is differentiable and its gradients are non-null everywhere. Intuitively, one can see from (5) that perturbed optimizers y2 are actually convolved version of the rigid optimizer y". Moreover, their Jacobian with respect to 0 is given by:
Joy. (0) = Ez y" (0 7 Vu(ZYT
worth noticing at this stage that, when = O, one recovers the standard formulation of the rigid optimizer. In general, v. (0) and Joy, (0) do not admit closed-form expressions. To overcome this issue, Monte-Carlo estimators are exploited to compute an estimate of these quantities, as recalled in Sec. 4.
is
3 Related work
Our work builds on results in differentiable rendering, differentiable optimization and randomized smoothing.
3

---

Differentiable rendering. Some of the earliest differentiable renderers rely on the rigid rasterivation-basee rendering process recalled in the previous section, while exploiting some gradient approximations of the non-smooth operations in the backward. OpenDR [24] uses first-order Taylor expansion to estimate the gradients of a classical renderer, resulting in gradients concentrated around the edges of the rendered images. In the same vein, NMR [16] proposes avoid the issue of local gradients by doing manual interpolation during the backward pass. For both OpenDR and NMR, the discrepancy between the function evaluated in the forward pass and the gradients computed in the backward pass may lead to unknown or undesired behaviour when optimizing over these approximated gradients. More closely related our work, SoftRas [23] directly approximates the forward pass of usual renderers to make it naturally differentiable. Similarly. DIB-R [8] introduces the analytical derivatives of the faces' barycentric coordinates to get a smooth rendering of foreground pixels during the forward pass, and thus, gets gradients naturally relying on local properties of meshes. Additionally, soft aggregation step is required to backpropagate gradients towards background pixels similar way to SoftRas. In parallel line of work, physically realistic renderers are made differentiable by introducing a stochastic estimation of the derivatives of the ray tracing integral [21, 28]. This research direction seems promising as it allows to generate images with global illumination effects. However, this class of renderers remains computationally expensive when compared to rasterization- based algorithms, which makes them currently difficult to exploit within classic computer vision or robotics contexts, where low computation timings matter.
Differentiable optimization and randomized smoothing. More generally, providing ways to dif- ferentiate solutions of constrained optimization problems has been part of the recent growing effort to introduce more structured operations in the differentiable programming paradigm [33], going beyond classic neural networks. A first approach consists in automatically unrolling the optimization process used solve the problem [11, 26]. However, because the computational graph grows with the number of optimization steps. implicit differentiation, which relies on the differentiation of optimality conditions, should be preferred when available [2, 3, 20]. These methods compute exact gradients whenever the solution is differentiable with respect to the parameters of the problem. In this paper, we show how differentiating through a classic renderer relates to computing derivatives of Linear Programming (LP) problem. In this particular case and as recalled in the Sec. 2, solutions of the optimization problem vary in a heavily non-smooth way as gradients are null Limost-everywher and non-definite otherwise, thus, making them hard to exploit within classic optimization algorithms. To overcome this issue, a first approach consists in introducing a gradient proxy by approximating the solution with piecewise- approach leverages random perturbations to replace the original problem by a smoother approx- this later method it requires only little effort to transform non-smooth optimizers into differentiable ones, while guaranteeing non-null gradients everywhere. Moreover, randomized smoothing has been shown to facilitate optimization of non-smooth problems [12]. Solving a smooth approximation leads to improved performance when compared to methods dealing with the original rigid problem. In addition to its smoothing effect, the addition of noise also acts as an implicit regularization during the training process which is valuable for globalization aspects [4, 9], i.e. converging towards better local optima.
interpolation [35] which can also lead to null gradient. Another
imation [1, 4]. For our differentiable renderer,
Approximate differentiable rendering via perturbed optimizers
In this section, we detail the main contributions of the paper. We first introduce the reformulation of the rasterization and aggregation steps as perturbed optimizers, and propose a variance reduction mechanism to alleviate the computational burden inherent to these kinds of stochastic estimators. Additionally, we introduce an adaptive scheme to automatically adjust the smoothing parameters inherent to the rendering process.
4.1 Perturbed renderer: general approximate and differentiable renderer
As detailed in Sec. 2, the rasterization step (2) can be directly written as the argmax of an LP. Similarly, the aggregation (3) step can be slightly modified to reformulate it as argmax of an LP.
Indeed, when y 2 0. the hard constraint yj 0 if T3- = 0 is equivalent to Ti > 0, which can be

---

dm0, 0-5e-4, 5-5e-3 M-d 0-2e-3, 2-2e22
0-5e-3,-22e44 y-2e-4, 1-1228
Figure 2: Examples of images obtained from perturbed differentiable rendering process with Gaussian noise. With smoothing parameters set to O, we retricve the rigid renderer, while adding noise makes pixels from the background appear the foreground and vice versa.
approximated by adding logarithmic barrier Vj In I in the objective function as done in classical interior point methods [6]:
1 + In(('). s.t. ||
W (2)
argmax
because In 1; 15-+0
1:70
0. - +oo approximates the hard constraint and allows to retrieve the classical formulation (3) of w. Using this formulation, is possible introduce differentiable approximation of the rasterization and aggregation steps:
enforcing w
1 = (d((, with Ho Ex H oX)]
Ex [argmax (y, 1 0SvS1
+ +o))
(8)
ît' Σ wan,(1 (=) with wa,(3) (=) = E2 [wa( + 72)]
Ezl
argmax
(y In In(',, + (10)
s.t. 1= a
By proceeding this way, get a rendering process for which every pixel is influenced by every triangle of meshes (Fig. 1,2), similarly to SoftRas [23]. More concretely, as shown in [4), the gradients obtained with the randomized smoothing are guaranteed to be non-null everywhere unlike some existing methods [8, 16, 24]. This makes it possible to use the resulting gradients as a strong supervision signal for optimizing directly at the pixel level, as shown through the experiments in Sec. 5. At this stage, it is worth noting that the prior distribution on the noise Z used for smoothing is not fixed. One can choose various distributions as prior thus leading to different smoothing patterns, which makes the approach versatile. Indeed, as a different noise prior induces different smoothing, some specific choices allow retrieve particular, using a Logistic and a Gumbel prior respectively on X and leads to SoftRas[23]. Further details included in Appendix A.
of the existing differentiable renderers [16, 23, 24). In
More practically, the smoothing parameters a, Y appearing in differentiable renderers such as SoftRas [23] or DIB-R [8] can be hard to set. In contrast, in the proposed approach, the smoothing naturally interpretable as a noise acting directly on positions of mesh triangles. Thus, the smoothing parameters representing the noise intensity can be scaled automatically according the object dimensions. In Sec. 4.3, we notably propose a generic approach to automatically adapt them along the optimization process.
4.2 Exploiting the noise inside perturbed renderer
In practice, (5) and (6) are useful even in the cases when the choice of prior on does not induce any analytical expression for y and Jev.- Indeed, Monte-Carlo estimators of these two quantities can be directly obtained from these two expressions:
1 M
M (x)
Σ
v +(0+)
(11)
M
i=1
1
M
(0 + eZ!0) - Vu(Z())
Jo vol (z)
Σ
(12)
M
1=1
5

---

Table 1: Time and memory complexity of our perturbed renderer for the forward and backward computation during the pose optimization task (5).
# of samples
2
8
32
64
SoftRas
Hard renderer
Forward (ms)
30 (=:3)
30 (+:3)
31 (4:3)
31 (+3)
31 (+:3)
29 (+:3)
29 (+-3)
Backward (ms)
(+1)
19 (+1)
19 (+2)
20 (+1)
22 (+1)
18 (+1)
N/A
Max memory (Mb)
217.6
217.6
217.6
303.1
440.0
180.3
178.5
190
244
230
637
120
160
Figure 3: Control variates methods [19] allow reduce the variance of the gradient of the smooth argmax. Left: Gaaussan-ppeturbeer argmax operator, Middle: estimator of gradient Right: variance- reduced estimator of gradient.
where M is the total number of samples used to compute the approximations.
Approximately, we have Var [Jey. x Varfy" (+V)
When decreasing e, the variance of the Monte-Carlo estimator increases, which can make the gradients very noisy and difficult to exploit in practice. For this reason, we use the control variates method [19] in order reduce the variance of our estimators. Because the quantity y (0)Vu(Z) has null expectation when Z has symmetric distribution and is positively correlated with y0 (0 + <Z) we can rewrite (6) as:
Med
Jøy (0) = Ez (O + <2) - y" (0)) vz(Z)
(13)
which naturally leads to variance-reduced Monte-Carlo estimator of the Jacobian:
M
Joy, (0)
M Σ (0(0+22()))0000 Vu(ZO)T.
(14)
The computation of yr" requires the solution of M perturbed problems instead only one in the case of classical rigid optimizer. Fortunately, this computation is naturally parallelizable, leading to constant computation time at the cost of an increased memory footprint. Using our variance- reduced estimator alleviates the need for high number of Monte Carlo samples, hence reducing the computational burden inherent to perturbed optimizers (Fig. 3). As shown in Sec. 5 (Fig. 5, left), M samples is already sufficient get stable and accurate results. In addition, computations from the forward pass can be reused in the backward pass and time and memory complexities to evaluate these passes comparable classical differentiable renderers (Tab. i). Consequently, evaluating gradients does not require any extra computation. is worth mentioning at this stage that this variance reduction mechanism in fact applies any perturbed optimizer. This variance reduction technique can be interpreted as estimating a sub- bub-gradient with the finite differences in random direction and is inspired by the field of random optimization [27].
4.3 Making the smoothing adaptive with sensitivity analysis
In a way similar to (6), it is possible to formulate the sensitivity of the perturbed optimizer with respect to the smoothing parameter
J.y (0) = Ez (0" +2+ (Vv(Z) -
-
(15)
and, as previously done, build variance-reduced Monte-Care Carlo estimator of this quantity:
M
- v0())) Vu(Z()TZ() -
Jay. (0) = M Σ
(16)

---

Target
Init
iter
100 Iter
Figure 4: Perturbed differentiable renderer and adaptive smoothing lead to precise 6D pose estimation.
In the case of differentiable rendering, notable property the positivity of the sensitivity of any locally convex loss L (which is valid for the RGB loss) with respect to the 7 smoothing parameter when approaching the solution. Indeed, in the neighborhood the solution, have az 0g and first-order Taylor expansion gives:
OL By (0) = (2) C" (I w,) 2 0,
(17)
because for locally convex loss, V²L((((0,) is positive definite local optimum. We exploit this property to gradually reduce the smoothing when needed. To do so, we track an exponential moving average of the sensitivity during the optimization process:
Oc
By By + (1 -3,
θ7 (0:)
(18)
where B, is a scalar parameter. Whenever ef, positive, the smoothing (a, 7) decreased constant rate (Fig. 4). Note that the adaptive algorithm does not require any further computation as sensitivity can be obtained from backpropagation and also beneficially applies to other differentiable renderers [23] (see 5).
Results
In this section, we explore applications of the proposed differentiable rendering pipeline standard computer vision tasks: single-view 3D pose estimation and 3D mesh reconstruction. Our implementation is based on Pytorch3d [31] and will be publicly released upon publication. corresponds to modular differentiable renderer where switching between different type of noise distributions (Gaussian, Cauchy, etc.) is possible. We notably compare our renderer against the open source SoftRas implementation available in Pytorch3d and DIB-R from Kaolin library. The results for these renderers are obtained by running the experiments our setup as we were not able to get access to the original implementations of the pose optimization and shape reconstruction problems.
Single-view 3D pose estimation is concerned with the retrieving the 3D rotation e SO(3) of an object from reference image. Similarly to [23], we first consider the of colored cube and fit the rendered image of this cube to single view of the true pose R(O,) (Fig. 4). The problem can be formulated regression problem of the form:
min LRGN (R(O.) R((0)
(19)
where it is the smooth differentiable rendered image. We aim analysing the sensitivity of perturbed differentiable renderer with respect to random initial guesses of various amplitudes from the real pose. Thus, the initial randomly perturbed from the true pose with various intensity for the perturbation (exploiting an angle-axis representation of the rotation, with randomized rotation axis). The intensity of the angular perturbation taken between 20 and 80 degrees. Consequently, the value of the average final error has an increased variance and is often perturbed by local minima (Tab. 2). To avoid these caveats, we rely on another metric: the percentage of solved tasks (see Tab. 2 and Fig. 5 and 6), accounting for the amount of final errors which are below given threshold (10 degrees for Tab. and bottom row of Fig. 6). For this task, we compare our method with two different distributions for the smoothing noise (Gaussian and Cauchy priors) to SoftRas. We use Adam [17] with parameters 31 0.9 and B2 0.999 and operate with 128 128 RGB images, each optimization problem taking about minute to solve Nvidia 6TX0000 GPU.

---

Table 2: Results of pose optimization from singlc-view image using perturbed differentiable renderer with variance reduction and adaptive smoothing.
Initial error
20
50"
80
Dift. renderer
SoftRas
Cauchy smoothing
Gaussian smoothing
SoftRas
Cauchy smoothing
Gaussian smoothing
SoftRas
Cauchy smoothing
Gaussion smoothing
Avg. error (*)
5.5
6.1
3.8
14.6
20.9
11.6
33.5
41.8
25.3
Std. error
4,9
5.1
4.4
26.6
21.0
22.5
35,3
34.0
34,9
Task solved (5)
84
+-4.0
82 +-2.3
93 +0.4
75 +5.33
71 +2.5
84 +1.3
55 +4.88
55 +5.5
67 +-266
20.07 initial perturbation
80 incial perturbation
2.0
0.0
6.6
Look
mediner,
Seftias
Aettiite rad mossow a IR. int ong Pertu readow cample
Pert rettuttdd resdene UR
Pertulleed te sieee Vi
(1)
Trreshold (*]
Figure 5: Left: Variance reduction (VR) drastically reduces the number of Monte-Carlo samples required to retrieve the true pose from a 20 perturbation. Right: Adaptive smoothing (AS) improves resolution of precise pose optimization for 80 perturbation.
We provide results of 100 random perturbations for various magnitude of initial perturbations in Tab. 2 and Fig. 5,6. Error bars provide standard deviations while running experiments with different random seeds. Fig. 5 (left) demonstrates the advantage of our variance reduction method to estimate perturbed optimizers and precisely retrieve the true pose at a lower computational cost. Indeed, the reeducdd perturbed renderer is able to perform precise optimization with only samples (80% of final errors are under 5" while it is less than 50% without the use of control variates). Additionally, our adaptive smoothing significantly improves the number of solved tasks for both SoftRas and our perturbed renderer (Fig. 5, right).
Additional results in Tab. 2, Fig. 6 and Fig. 11 in Appendix demonstrate that our perturbed renderers with a Gaussian smoothing, outperforms SoftRas and can achieve state-of-the-ate art results for pose optimization tasks on various objects. As shown in Appendix (Prop. 4), SoftRas is equivalent to using a Gumbel smoothing which is an asymmetric prior. However, no direction should be preferred when smoothing the rendering process which explain the better performances of the Gaussian prior.
20.0" Initial perturaation
50.0" initial parturaation
80.0 0" intiial parturcation
Nottrbed
Salfan
Soffun 15
vadeer 22
Thresvats ei
Threchelte")
20.0" Fitiall perturaation
50.0" iitial parturaation
80.0" iittial parturiaation
Nerubed weder: +1
Figure 6: Top row: perturbed renderer combined with variance reduction and adaptive scheme improves SoftRas results on pose optimization. Bottom row: The method is robust w.r.t. initial smoothing values. Higher is better.
8

---

Serdaned
coleroed
Resorehuctoo
Ortorentiable Rondbrer
Astheat ARGBLUGH
Figure 7: Left: A neural network is trained with self-supervision signal from a differentiable rendering process. Right: Qualitative results from self-supervised 3D mesh reconstruction using a perturbed differentiable renderer. 1st column: Input image. 2nd and 3rd columns: reconstructed mesh viewed from different angles.
Table 3: Results of mesh reconstruction on the ShapeNet dataset [7] reported with 3D IoU (%)
Aiplime
Bench
Droeur
Chuir
Displa
Lamp
Speaker
Riñe
Sofa
Table
Phane
Visuel
More
NMR [16]
58.5
45.7
74.1
71.3
41.4
55.5
36.7
67.4
55.7
6112
39.1
76.2
59.4
$7.0
SottRaa 22)
62.0
61.4
60.4
59.9
58.5
50,9
61.0
Ous
(10.1)
(20.20
(10.00
(20.12)
(10.07)
(20.11)
Bottom row Fig. shows that our method is robust to the choice of initial values of smoothing parameters. During optimization, the stochasticity of the gradient due the noise from the perturbed renderer acts as an implicit regularization minima saddle points and converge more stable minimal regions, leading to better optima. As mentioned earlier, results are limited by the inherent non-convexity of the rendering process and differentiable renderers not able recover from bad initial guess. Increasing the noise intensity to further smoothen the objective function would not help in this case, as it would also lead to critical loss of information on the rendering process.
the objective function. This helps to avoid sharp local
Self-supervised 3D mesh reconstruction from a single image. In our second experiment, we demonstrate the ability of our perturbed differentiable renderer to provide supervisory signal for training a neural network. We use the renderer to self-supervist the reconstruction of a 3D mesh and the corresponding colors from single image (see Fig. 7). To do so, we use the network architecture proposed in [23] and the subset of the Shapenet dataset [7] from [16]. The network is composed of convolutional encoder followed by two decoders: one for the mesh reconstruction and another one for color retrieving (more details in Appendix B).
Using the same setup as described in [8. 16. 23]. the training is done by minimizing the following loss:
L Coil ARGBLRGB + Atap
(20)
where Cat is the negative IoU between silhouettes I and i, CROB is the é RGB loss between R and R, Ltep is Laplacian regularization term penalizing the relative change in position between neighbour vertices (detailed description in Appendix B). For training, we use Adam algorithm with a learning rate of 10 and parameters 31 0.9, B2 0.999. Even if the memory consumption of the perturbed renderer may limit the maximum size of batches, its flexibility makes it possible to switch larger batches in order finetune the neural
to deterministic approximation such [23] network, needed.
We analyze the quality of obtained 3D reconstruction through rendering pipeline by providing IoU measured on 13 different classes the test set (see Tab. 3). We observe that our perturbed
'These numbers were obtained by running the renderer in our own setup (cameras, lightning. training parameters etc.) in order to have comparable results. This may explain the slight difference with the numbers from the original publications.

---

renderer obtains state-of-the-arr art accuracies on this task. Error bars (representing the standard deviation obtained by retraining the model with different random seeds) confirm the stability of our method. Additionally, we illustrate qualitative results for colorized 3D mesh reconstruction from a single image in Fig. 7(right) confirming the ability of perturbed renderers to provide strong visual supervision signals.
6 Conclusion
In this paper, we introduced a novel approach to differentiable rendering leveraging recent con- tributions perturbed optimizers. We demonstrated the flexibility of our approach grounded in its underlying theoretical formulation. The combination of our proposed perturbed renderers with suriance-reduction mechanisms and sensitivity analysis enables for robust application of our rendering approach in practice. Our results notably show that perturbed differentiable rendering can reach sfate-of-the-art performance on pose optimization and is also competitive for self-supervised 3D mesh reconstruction. Additionally, perturbed differentiable rendering can be easily implemented on top of existing implementations without major modifications, while generalizing existing renderers such as SoftRas. As future work, we plan to embed our generic differentiable renderer within a differentiable simulation pipeline in order learn physical models, by using visual data as strong supervision signals to learn physics.
Acknowledgments and Disclosure of Funding
We warmly thank Francis Bach for useful discussions. This work was supported in part by the HPC resources from GENCI-IDRISSGrral AD01 1012215), the French government under management of Agence Nationale de Recherche as part of the "Investissements d'avenir" program, reference ANR-19-P3IA-000 (PRAIRIE 31A Institute), and Louis Vuitton ENS Chair on Artificial Intelligence.
References
[1] J. Abernethy. 1 perturbation techniques in online learning and optimization. 2016.
[2] A. Agrawal, S. Barratt, S. Boyd, E. Busseti, and W. Moursi. Differentiating through a cone program. Journal of Applied and Numerical Optimization, 1(2), 2019.
[3] B. Amos and J. Z. Kolter. Optnet: Differentiable optimization as layer in neural networks. In International Conference on Machine Learning. PMLR, 2017.
[4] Q Berthet, M. Blondel, O. Teboul, M. Cuturi, J.-P. Vert, and F. Bach. Learning with differentiable perturbed optimizers, 2020.
[5] V. Blanz and T. Vetter. A morphable model for the synthesis of 3d faces. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 187-194, 1999.
[6] S. Boyd, S. Boyd, and L. Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] A. X. Chang, T. Funkhouser, I.. Guihas, P. Hanrahan, Q. Huang, 7. Li, S. Savarese, M. Savva, S. Song, Su, et al. Shapenet: An information-ricl 3d model repository. arXiv preprint arXiv:1512.03012. 2015.
[8] W. Chen, H. Ling, J. Gao, E. Smith, J. Lehtinen, Jacobson, and S. Fidler. Learning to predict 3d objects with an interpolation- bbasod differentiable renderer. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
[9] J. Cohen, E. Rosenfeld, and Z. Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning, pages 1310-1320. PMLR, 2019.
[10] X. Deng, Y. Xiang, A. Mousavian, C. Eppner, T. Bretl, and D. Fox. Self-supervised 6d object pose estimation for robot manipulation. In International Conference on Robotics and Automation (ICRA), 2020.
[11] Domke. Generic methods for optimization- bassed modeling.
D. Lawrence and M. Girolami. editors. Proceedings of the Fificenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 318 326,, La Palma, Canary Islands. 21-23 Apr 2012. PMLR.
10

---

[12] J. C. Duchi, P. L. Bartlett, and M. J. Wainwright. Randomized smoothing for stochastic optimization.
SIAM Journal Optimization, 22(2):674-701 2012.
[13] H. Gouraud. Computer Display of Curved Surfaces. PhD thesis, 1971. AA17127878.
[14] E. J. Gumbel. Statistical theory of extreme values and some practical applications: series of lectures. volume 33. US Government Printing Office, 1954.
[15] D. Hoiem, A. Efros, and Hebert. Recovering surface layout from an image. International Journal of Computer Vision, 75(1):151 (51))1511172, 2007.
[16] H. Kato, Y. Ushiku, and T. Harada. Neural mesh renderer, 2017.
[17] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. 2015.
[18] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic. Cosypose: Consistent multi-view multi-object 6d pose estimation. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
[19] A. M. Law, D. Kelton, W. Kelton, Simulation modeling and analysis, volume McGraw-Hii New York, 2000.
[20] Q. Le Lidec, I. Kalevatykh, I Laptev, C. Schmid, and J. Carpentier, Differentiable simulation for physical system identification. IEEE Robotics and Automation Letters, Feb. 2021.
[21] T.-M. Li, M. Aittala, Durand, and J. Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM Trans. Graph. (Proc. SIGGRAPH Asia), 3766):22: 2018.
[22] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox. Deepim: Deep iterative matching for 6d pose estimation. In Proceedings of the European Conference Computer Vision (ECCV), pages 683-698, 2018.
[23] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: differentiable renderer for image-based 3d reasoning. 2019.
[24] Loper and M. Black, Opendr: An approximate differentiable renderer. 09 2014.
[25] D. G. Lowe. Three-dimensional object recognition from single two-dimensional images. Artificial intelligence, 31(3):355 (135:355-395 1987.
[26] D. Maclaurin, D. Duvenaud, and R. Adams. Gradient- Gradient-based hyperparameter optimization through reversible learning. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2113-2122, Lille, France, 07-09 Jul 2015. PMLR.
[27] Y. Nesterov and Spokoiny, Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17(2):527-566, 2017.
[28] Nivier-David, Vicini, T. Zeltner, and Jakoh. Mitsuba A retargetable forward and inverse renderer. ACM Transactions on Graphics (TOG), 38(6):1-17, 2019.
[29] B. T. Phong. Illumination for computer generated pictures. Commun. ACM, 1113)) June 1975.
[30] M. Rad and V. Lepetit. Bh8: scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth. In Proceedings of the IEFE International Conference on Computer Vision, pages 3828-3836. 2017.
[31] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo. Johnson, and G. Gkioxari. Accelerating 3d deep learning with pytorch3d. arXiv preprint Ttt:::000000800011 2020.
[32] L. Roberts. Machine perception of three-dimensional solids. PhD thesis, Massachusetts Institute Technology, 1963.
[33] V. Roulet and Z. Harchaoui. Differentiable programming à la moreau. arXiv preprint 15545 2020.
[34] A. Saxena, M. Sun, and A. Y. Ng. Learning 3-d scene structure from a single still image. In 2007 IEEE 1tth intermational conference computer vision, pages 1-8. IERF, 2007.

---

[35] M. Vlastelica, A. Paulus, V. Musil, G. Martius, and M. Rolínek. Differentiation of blackbox combinatorial solvers, 2020.
[36] N. Wang, Y. Zhang, Z. Y. W. Liu, and Y.G. Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018.
[37] X. Yan, J. Yang. E. Yumer, Y. Guo, and H. Lee. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision. In Lee, Sugiyama, U. Luxburg, Guyon, and R. Gareet,, editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
12


Reviews:
[{"review_id": "rEod3tflXz", "reviewer": "Reviewer_wBDa", "summary": "This paper proposes a new differentiable renderer by introducing perturbed optimizers. The paper re-writes the non-differentiable rasterization and aggregation steps to heaven side functions and relaxes them to be differentiable by adding noise. \nThe author shows that previous work, especially softras, can be treated as a special case of the perturbed renderer with specific prior selections. The authors also demonstrated in several experiments they achieve better or comparable results than the state of the art.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1) Differentiable reformulation\nThis paper treats rasterization as a linear programing problem.  It extends the [4] to the rendering pipeline and relaxes the non-differentiable rendering to be differentiable by introducing perturbed optimizers. It provides a new differentiable renderer from a new perspective.\n\n2) Better performance\nThe authors show AS and VR are very useful in optimization, for both perturbed renderer and softras. In the 6D pose comparison, the results are better than softras.\n\n \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There are several weak points of this paper.\n\n1) The paper is not self-contained.  I have a hard time understanding each equation. While the authors largely rely on [4], it makes readers hard to follow without knowing [4]. Moreover, the notations in this paper are very vague.   E.g. line 66 it should be j \\in [1,.., m]?  line 76 z should be depth order?  If it is depth, how do you compute <z, y>?  line 88, what's v?  The paper writing definitely needs to be improved.\n\n\n2) Relation of softras and noise types.  The authors show that softras can be treated as the special case of the perturbed renderer. Softras has some hyperparameters which are hard to explain but in the new renderer, all the parameters have clear noise intensity meaning.  While this is very good, but the paper lacks further analysis of which noise should we use. E.g. in table 1, it seems Gaussian noise is the best. But could it be generalized to other cases? \n\n\n3) Comparison.  In line 214 the authors said, \"Unless stated, the results for the other renderers are extracted from their original publications, as we were not able to get access to their implementations.\"  I am a little confused about this.  Clearly, N3mr, softras, dibr released their renderers. Thus, why not simply replace the renderer from perturbed renderer to new renderers and compare the score?  Directly copying scores from others' papers may have the wrong comparison. While all the methods use shapenet, the rendered images could vary from lighting to camera views. I would highly recommend the authors redo the comparison.\n\n\n4)  The authors assume each pixel will be influenced by all the triangles. For each pixel, it renders the influences of all the triangles and combines them together(Eq 3).  I wonder what's the rendering speed of the perturbed renderer? Theoretically, such design plus the MC sampling will make the renderer very time-consuming.  The authors should show the rendering speed comparison.\n\n\n5) Following the last concern.  Generally speaking, rasterization accelerates the rendering speed by assuming each pixel is influenced by the frontest triangle.  On the other side, MC is always used in ray tracing to compute multiple bounces. The authors introduce MC in rasterization, which seems to adopt the weak points of two rendering methods.   I wonder did the authors consider extending it to RT?\n\nPost rebuttal:\n\nThe authors addressed my concerns and I will raise the score to be marginally above the threshold and be happy to see this paper get accepted.\n\n\n\n\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "c7JpE38n9Sd", "reviewer": "Reviewer_zsMR", "summary": "This paper proposes a new differentiable rendering method, differentiable perturbed renderers,  wherein the gradients of the rasterization step are approximated using perturbed optimizers. They implement this change on top of the SoftRas method and demonstrate that it leads to better pose estimation, and on par performance with state of the art methods in unsupervised single image 3d object reconstruction. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a strong paper. It is written clearly, and the method is explained properly. Perturbed optimizers seem to be a new way for tackling optimization over discrete operations in deep learning, and their application to rasterization is well motivated and appropriate. The theory required for applying them to this new setting seems right, and is well explained. The results are also adequate for the claims made. It would have been nice to see higher performance on the reconstruction task, though diff rendering is a relatively new and so highlighting new avenues to explore , such as your proposed perturbed optimizers,  is important, and so I do not believe it is necessary.\n\n I would say one nice set of experiments to see would be optimization over the same set of parameters as in the DIB-R  paper Figure 2. ( camera position, texture , vertex positions, etc. ), with qualitative and quantitative comparisons to DIB-R and SoftRas. Having this would go a long was to increasing my score, and I believe would sell you paper quite a bit better. The pose estimation seems quite a bit easier than optimizing for texture and vertex position for example, and it would be nice to see how you method fairs, even if the performance is only on par or worse. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 "}, "extra_text": {}}, {"review_id": "S4P_PUKyAE0", "reviewer": "Reviewer_qAqf", "summary": "This paper presents a method for converting a standard non-differentiable rasterization-based renderer into a differentiable counterpart using recent advances in the perturbed optimizer. The main contribution lies in the generic nature provided by the theoretical formulation, where some specifically chosen priors in the proposed framework can retrieve recent state-of-the-art differentiable renderers, e.g. SoftRas. \nAs the proposed approach mainly relies on a Monte-Carlo estimator for generating the smoothing parameters. Variance-reduction mechanism and sensitivity analysis are also incorporated into the proposed framework to ensure the robustness of the approximate renderer. Experiments on pose estimation and single-view reconstruction are conducted for comparing the performance of the proposed method and previous state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- By introducing the perturbed optimizer to analyzing the non-differentiability of standard rasterization-based rendering pipeline provides new insight into constructing a generic differentiable renderer. The perturbed differentiable rendering can also be implemented on top of existing renderers without major modifications, which makes this approach versatile and a potential performance booster for existing DR techniques.\n\nCons:\n\n- In terms of technical novelty, it seems to be a direct application of perturbed optimizer to differentiable rendering. Though it requires some efforts (such as the variance reduction and sensitivity analysis) to adapt the existing theory to this specific application, it is not very challenging as there are existing protocols to follow.\n\n- Evaluation-wise, in the task of single-view reconstruction, though the proposed method can achieve comparable performance compared to the state-of-the-art approaches, it still fails to outperform DIB-R in mean accuracy and also outperformed by SoftRas in many categories, such as the Dresser, Car, etc. The 3D pose estimation is only evaluated on a simple cube which cannot provide a very comprehensive measurement of the proposed approach. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "4oJ4Wt8zKQk", "reviewer": "Reviewer_yvH4", "summary": "This paper proposes a differentiable rendering method that builds connection between the 3D model and 2D perspective appearance. Different from previous methods, the differentiable rendering is formulated into an optimization problem. Two computer vision tasks, including pose estimation and 3D shape from color image, are evaluated with the proposed renderer and compared to SoftRas [22].", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The performance for pose estimation outperforms existing methods.\n- The formulation seems novel.\n\nWeakness:\n- Could the proposed formulation be applied to other 3D shape representations, e.g. point cloud, implicit? If so, an experiment should be added. If not, then better to mention this is a mesh-specific technic.\n\n- The algorithm seems to be computational costy, e.g. 1 min per image. Would this heavily affect the training efficiency? While comparing to other methods, e.g. SoftRas, the run-time efficiency and hyper-parameters, e.g. number of iteration, training epoch, should be mentioned. Otherwise the comparison may not be on a fair basis.\n\n- Keeping the boundary differentiable is one of the main issues in existing differentiable rendering technics, which is also argued mainly in this paper. However it is not clear how this is improved by the proposed method. From Fig 2 and 4, it seems the boundary area has obvious artifacts. Also some experiments should be performed to highlight the improvements over occlusion boundary.\n\n- The performance on 3D shape prediction does not outperform existing approaches. \n\nOverall, despite a novel formulation, the improvements over existing approaches are not clear. As shown in the experiment, the propose method does not necessarily produce more accurate rendering or support better for vision tasks. Considering that the proposed tech is also quite slow, the overall advantage over existing approaches is not very clear.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No comment.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has mixed reviewers. Most of reviewers agree with the novelty and contribution of proposing a new differentiable rendering method, differentiable perturbed renderers. The whole paper is solid and experiments are good enough to validate the contribution. Reviewer yvH4 had some serious concerns about the practical issues of the proposed methods, including generalization ability to other 3D shape representations, computational expensive, not competitive to existing approaches. The authours clarified some of these points. In the final version, the authours are advisable to futher integrate the rebuttal and improve the paper quality.
Decision: Accept (Poster)