Differentiable rendering with perturbed optimizers

1 Introduction . Many common tasks in computer vision such as 3D shape modelling [ 5 , 15 , 34 , 36 , 37 ] or 6D pose estimation [ 18 , 22 , 25 , 30 , 32 ] aim at inferring 3D information directly from 2D images . Most of the recent approaches rely on ( deep ) neural networks and thus require large training datasets with 3D shapes along with well-chosen priors on these shapes . Render & compare methods [ 18 , 22 ] circumvent the non-differentiability of the rendering process by learning gradients steps from large datasets . Using a more structured strategy would allow to alleviate the need for such a strong supervision.Using a more structured strategy would allow to alleviate the need for such a strong supervision . In this respect , differentiable rendering intends to model the effective image generation process to compute a gradient related to the task to solve . This approach has the benefit of containing the prior knowledge of the rendering process while being interpretable . This makes it possible to provide a supervision for neural networks in a weakly-supervised manner [ 10 , 16 , 23 ] . The main challenge of differentiable rendering lies in the non-smoothness of the classical rendering process . In a renderer , both the rasterization steps , which consist in evaluating the pixel values by discretizing the 2D projected color-maps , and the aggregation steps , which merge the color-maps of several objects along the depth dimension by using a Z-buffering operation , are non-differentiable operations ( Fig . 1 ) . Intuitively , these steps imply discontinuities in the final rendered image with respect to the 3D positions of the scene objects . For example , if an object moves on a plane parallel to the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . camera , some pixels will immediately change color at the moment the object enters the camera view or becomes unocluded by another object . In this paper , we propose to exploit randomized smoothing techniques within the context of differentiable rendering to automatically soften the non-smooth rendering operations , making them naturally differentiable . The generality of our approach offers a theoretical understanding of some of the existing differentiable renderers while its flexibility leads to competitive or even state-of-the-art results in practice . We make the following contributions : �→ We formulate the non-smooth operations occurring in the rendering process as solutions of optimization problems and , based on recent work [ 4 ] , we propose a natural way of smoothing them using random perturbations . We highlight the versatility of this smoothing formulation and show how it offers a theoretical understanding of several existing differentiable renderers . �→ We propose a general way to use control variate methods to reduce variance when estimating the gradients of the perturbed optimizers , which allows for sharp gradients even in the case of weaker perturbations . �→ We introduce an adaptive scheme to automatically adjust the smoothing parameters by relying on sensitivity analysis of differentiable perturbed renderers , leading to a robust and adaptive behavior during the optimization process . �→ We demonstrate through experiments on pose optimization and 3D mesh reconstruction that the resulting gradients combined to the adaptive smoothing provide a strong signal , leading to more accurate solutions when compared to state-of-the-art alternatives based on smooth gradient approximations [ 23 ] . 2 Background . In this section , we review the fundamental aspects behind image rendering and recall the notion of perturbed optimizers which are at the core of the proposed approach . Rasterizer and `` aggregater '' as optimizers . We consider the rendering process of an RGB image of height hI and width wI , from a scene composed with meshes represented by a set of m triangles . We assume that every triangle has already been projected onto the camera plane and their associated color maps Cj ∈ RhI×wI×3 , j ∈ [ 1 . . m ] , have been computed using a chosen illumination model ( Phong [ 29 ] , Gouraud [ 13 ] etc . ) and interpolating local properties of the mesh . We denote by I the occupancy map such that Iij is equal to 1 if the center of the i th pixel is inside the 2D projection of the jth triangle on the camera plane and 0 otherwise . By denoting d ( i , j ) the Euclidean distance from the center of the ith pixel to the projection of the jth triangle , the rasterization step ( which actually consists in computing the occupancy map I ) can be written as : Iij = H ( d ( i , j ) ) , ( 1 ) where H corresponds to the Heaviside function defined by : H ( x ) = � 0 if x ≤ 0 1 otherwise = argmax0≤y≤1 y x . ( 2 ) We call R ∈ RhI×wI×3 the final rendered image which is obtained by aggregating the color map of each triangle . In classical renderers , this step is done by using a Z-buffer so that only foreground objects are visible . This corresponds to : Ri = � j wij ( z ) C i j , with w i ( z ) = argmax y s.t . �y�1=1 , y≥0 , yj=0 if Iij=0 �z , y� , ( 3 ) where w , z ∈ Rm+1 and for 1 ≤ j ≤ m , zj is the inverse depth of the jth triangle and the ( m+ 1 ) th coordinate of z and w account for the background . The inverse depth of the background is fixed to zm+1 = zmin . From Eq . ( 2 ) and ( 3 ) , it appears that argmax operations play a central role in renderers and are typically non-differentiable functions with respect to their input arguments , as discussed next . Perturbed optimizers . In [ 4 ] , Berthet et al . introduce a generic approach to handle non-smooth problems of the form y∗ ( θ ) = argmax y∈C �θ , y� with C a convex polytope , and make these problems differentiable by randomly perturbing them . More precisely , y∗ ( θ ) necessarily lies on a vertex of the convex set C. Thus , when θ is only slightly modified , y∗ remains on the same vertex , but when the perturbation grows up to a certain level , y∗ jumps onto another vertex . Concretely , this means that y∗ ( θ ) is piece-wise constant with respect to θ : the Jacobian Jθy∗ is null almost everywhere and undefined otherwise . This is why the rasterization ( 2 ) and aggregation ( 3 ) steps make renderers non-differentiable . Following [ 4 ] , y∗ ( θ ) can be approximated by the perturbed optimizer : y∗� ( θ ) = EZ [ y∗ ( θ + �Z ) ] ( 4 ) = EZ � argmax y∈C �θ + �Z , y� � , ( 5 ) where Z is a random noise following a distribution of the form µ ( z ) ∝ exp ( −ν ( z ) ) . Then , we have y∗� ( θ ) �−→0−−−→ y∗ ( θ ) , y∗� ( θ ) is differentiable and its gradients are non-null everywhere . Intuitively , one can see from ( 5 ) that perturbed optimizers y∗� are actually a convolved version of the rigid optimizer y∗ . Moreover , their Jacobian with respect to θ is given by : Jθy ∗ � ( θ ) = EZ � y∗ ( θ + �Z ) � ∇ν ( Z ) � � . ( 6 ) It is worth noticing at this stage that , when � = 0 , one recovers the standard formulation of the rigid optimizer . In general , y∗� ( θ ) and Jθy ∗ � ( θ ) do not admit closed-form expressions . To overcome this issue , Monte-Carlo estimators are exploited to compute an estimate of these quantities , as recalled in Sec . 4 . 3 Related work . Our work builds on results in differentiable rendering , differentiable optimization and randomized smoothing . Differentiable rendering . Some of the earliest differentiable renderers rely on the rigid rasterization-based rendering process recalled in the previous section , while exploiting some gradient approximations of the non-smooth operations in the backward . OpenDR [ 24 ] uses a first-order Taylor expansion to estimate the gradients of a classical renderer , resulting in gradients concentrated around the edges of the rendered images . In the same vein , NMR [ 16 ] proposes to avoid the issue of local gradients by doing manual interpolation during the backward pass . For both OpenDR and NMR , the discrepancy between the function evaluated in the forward pass and the gradients computed in the backward pass may lead to unknown or undesired behaviour when optimizing over these approximated gradients . More closely related to our work , SoftRas [ 23 ] directly approximates the forward pass of usual renderers to make it naturally differentiable . Similarly , DIB-R [ 8 ] introduces the analytical derivatives of the faces ’ barycentric coordinates to get a smooth rendering of foreground pixels during the forward pass , and thus , gets gradients naturally relying on local properties of meshes . Additionally , a soft aggregation step is required to backpropagate gradients towards background pixels in a similar way to SoftRas . In a parallel line of work , physically realistic renderers are made differentiable by introducing a stochastic estimation of the derivatives of the ray tracing integral [ 21 , 28 ] . This research direction seems promising as it allows to generate images with global illumination effects . However , this class of renderers remains computationally expensive when compared to rasterization-based algorithms , which makes them currently difficult to exploit within classic computer vision or robotics contexts , where low computation timings matter . Differentiable optimization and randomized smoothing . More generally , providing ways to differentiate solutions of constrained optimization problems has been part of the recent growing effort to introduce more structured operations in the differentiable programming paradigm [ 33 ] , going beyond classic neural networks . A first approach consists in automatically unrolling the optimization process used to solve the problem [ 11 , 26 ] . However , because the computational graph grows with the number of optimization steps , implicit differentiation , which relies on the differentiation of optimality conditions , should be preferred when available [ 2 , 3 , 20 ] . These methods compute exact gradients whenever the solution is differentiable with respect to the parameters of the problem . In this paper , we show how differentiating through a classic renderer relates to computing derivatives of a Linear Programming ( LP ) problem . In this particular case and as recalled in the Sec . 2 , solutions of the optimization problem vary in a heavily non-smooth way as gradients are null almost-everywhere and non-definite otherwise , thus , making them hard to exploit within classic optimization algorithms . To overcome this issue , a first approach consists in introducing a gradient proxy by approximating the solution with a piecewise-linear interpolation [ 35 ] which can also lead to null gradient . Another approach leverages random perturbations to replace the original problem by a smoother approximation [ 1 , 4 ] . For our differentiable renderer , we use this later method as it requires only little effort to transform non-smooth optimizers into differentiable ones , while guaranteeing non-null gradients everywhere . Moreover , randomized smoothing has been shown to facilitate optimization of non-smooth problems [ 12 ] . Solving a smooth approximation leads to improved performance when compared to methods dealing with the original rigid problem . In addition to its smoothing effect , the addition of noise also acts as an implicit regularization during the training process which is valuable for globalization aspects [ 4 , 9 ] , i.e . converging towards better local optima .

Summary: This paper proposes a differentiable rendering method that builds connection between the 3D model and 2D perspective appearance. Different from previous methods, the differentiable rendering is formulated into an optimization problem. Two computer vision tasks, including pose estimation and 3D shape from color image, are evaluated with the proposed renderer and compared to SoftRas [22].