Learning with Algorithmic Supervision via Continuous Relaxations

1 Introduction . Artificial Neural Networks have shown their ability to solve various problems , ranging from classical tasks in computer science such as machine translation [ 1 ] and object detection [ 2 ] to many other topics in science such as , e.g. , protein folding [ 3 ] . Simultaneously , classical algorithms exist , which typically solve predefined tasks based on a given input and a predefined control structure such as , e.g. , sorting , shortest-path computation and many more . Recently , research has started to combine both elements by integrating algorithmic concepts into neural network architectures . Those approaches allow training neural networks with alternative supervision strategies , such as learning 3D representations via a differentiable renderer [ 4 ] , [ 5 ] or training neural networks with ordering information [ 6 ] , [ 7 ] . We unify these alternative supervision strategies , which integrate algorithms into the training objective , as algorithmic supervision : Definition 1 ( Algorithmic Supervision ) . In algorithmic supervision , an algorithm is applied to the predictions of a model and the outputs of the algorithm are supervised . In contrast , in direct supervision , the predictions of a model are directly supervised . This is illustrated in Figure 1 . In general , to allow for end-to-end training of neural architectures with integrated algorithms , the challenge is to estimate the gradients of a respective algorithm , e.g. , by a differentiable approximation . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Here , most solutions are tailored to specific problems like , e.g. , differentiable sorting or rendering . But also more general frameworks have recently been proposed , which estimate gradients for combinatorial optimizers . Examples for such approaches are the differentiation of black box combinatorial solvers as proposed by Vlastelica et al . [ 8 ] and the differentiation of optimizers by stochastically perturbing their input as proposed by Berthet et al . [ 9 ] . Both approaches focus on the problem that it is necessary to estimate the gradients of an algorithm to allow for an end-to-end training of neural architectures including it . To address this issue , Vlastelica et al . [ 8 ] estimate the gradients of an optimizer by a one-step linearization method that takes the gradients with respect to the optimizer ’ s output into account , which allows to integrate any off-the-shelf combinatorial optimizer . Berthet et al . [ 9 ] estimate the gradients by perturbing the input to a discrete solver by random noise . In this context , we propose an approach for making algorithms differentiable and estimating their gradients . Specifically , we propose continuous relaxations of different algorithmic concepts such as comparators , conditional statements , bounded and unbounded loops , and indexing . For this , we perturb those variables in a discrete algorithm by logistic distributions , for which we want to compute a gradient . This allows us to estimate the expected value of an algorithm ’ s output sampling-free and in closed form , e.g. , compared to methods approximating the distributions via Monte-Carlo sampling methods ( e.g. , [ 9 ] ) . To keep the computation feasible , we approximate the expectation value by merging computation paths after each conditional block in sequences of conditional blocks . For nested conditional blocks , we compute the exact expectation value without merging conditional cases . This trade-off allows merging paths in regular intervals , and thus alleviates the need to keep track of all possible path combinations . As we model perturbations of variables when they are accessed , all distributions are independent , which contrasts the case of modeling input perturbations , where all computation paths would have to be handled separately . To demonstrate the practical aspects , we apply the proposed approach in the context of four tasks , that make use of algorithmic supervision to train a neural network , namely sorting supervision [ 6 ] , [ 7 ] , [ 10 ] , shortest-path supervision [ 8 ] , [ 9 ] , silhouette supervision ( differentiable rendering ) [ 4 ] , [ 5 ] , [ 11 ] , and finally Levenshtein distance supervision . While the first three setups are based on existing benchmarks in the field , we introduce the fourth experiment to show the application of this idea to a new algorithmic task in the field of differentiable dynamic programming . In the latter , the Levenshtein distance between two concatenated character sequences from the EMNIST data set [ 12 ] is supervised while the individual letters remain unsupervised . We show that the proposed system is able to outperform current state-of-the-art methods on sorting supervision , while it performs comparable on shortest path supervision and silhouette supervision . 2 Related Work . We cover the field of training networks with algorithmic supervision . While some of these works cover single algorithmic concepts , e.g. , sorting [ 6 ] , [ 7 ] , [ 13 ] or rendering [ 4 ] , [ 5 ] , [ 14 ] , others have covered wider areas such as dynamic programming [ 15 ] , [ 16 ] or gradient estimation for general optimizers [ 8 ] , [ 9 ] , [ 17 ] , [ 18 ] . Paulus et al . [ 19 ] employ the Gumbel-Softmax [ 20 ] distribution to estimate gradients for concepts such as subset selection . Xie et al . [ 21 ] and Blondel et al . [ 13 ] employ differentiable sorting methods for top-k classification . Cuturi et al . [ 22 ] and Blondel et al . [ 23 ] present approaches for differentiable dynamic time warping . In machine translation , Bahdanau et al . [ 24 ] investigate soft alignments , while Collobert et al . [ 25 ] present a fully differentiable beam sorting algorithm . In speech recognition , Bahdanau et al . [ 26 ] propose differentiable surrogate task losses for sequence prediction tasks . Shen et al . [ 27 ] propose a differentiable neural parsing networks using the stick breaking process . Dasgupta et al . [ 28 ] model box parameters with Gumbel distributions to obtain soft differentiable box embeddings . In the following , we focus on those works that provide relaxations for tasks considered in this work . We close the overview with a review of some early works on smooth program interpretation . Sorting Supervision The task of training neural networks with sorting supervision has been first proposed by Grover et al . [ 6 ] . In this case , a set four-digit numbers based on concatenated MNIST digits [ 29 ] is given , and the task is to find an order-preserving mapping from the images to scalars . Here , a CNN should predict a scalar for each of n four-digit numbers such that their order is preserved among the predicted scalars . For training , only sorting supervision in the form of the ground truth order of input images is given , while their absolute values remain unsupervised . Grover et al . [ 6 ] address this task by relaxing the permutation matrices to double stochastic matrices . Cuturi et al . [ 7 ] pick up this on benchmark and propose a differentiable proxy by approximating the sorting problem with a regularizing optimal transport algorithm . Silhouette Supervision A line of work in computer vision is differentiable renderers for 3Dunsupervised 3D-reconstruction . Here , recent approaches have proposed differentiable renderers for 3D mesh reconstruction with only 2D silhouette supervision [ 4 ] , [ 5 ] on 13 classes of the ShapeNet data set [ 30 ] . Kato et al . [ 5 ] propose a renderer where surrogate gradients of rasterization are approximated to perform 3D mesh reconstruction via silhouette supervision as well as 3D style transfer . Liu et al . [ 4 ] propose a differentiable renderer without surrogate gradients by using a differentiable aggregating process and apply it to 3D mesh reconstruction as well as pose / shape optimization . Shortest-Path Supervision Two works , proposed by Vlastelica et al . [ 8 ] and Berthet et al . [ 9 ] presented methods for estimating gradients for off-the-shelf optimizers . In this context , Vlastelica et al . [ 8 ] proposed an experiment of shortest-path supervision , where an image of a terrain is given with different costs for each type of terrain , and the task is to predict the shortest path from the top-left to the opposite corner . Integrating a shortest-path algorithm into a neural architecture lets the neural network produce a cost embedding of the terrain which the shortest-path algorithm uses for predicting the shortest path . Vlastelica et al . [ 8 ] tackle this task by finding a linearization of the Dijkstra algorithm [ 31 ] , which they can differentiate . Berthet et al . [ 9 ] take up this problem and produce gradient estimates for the Dijkstra algorithm by stochastically perturbing the inputs to the shortest-path optimization problem . Smooth Interpretation In another line of work , in the field of computer-aided verification , Chaudhuri et al . [ 32 ] , [ 33 ] propose a program smoothing method based on randomly perturbing the inputs of a program by a Gaussian distribution . Here , an initial Gaussian perturbation is propagated through program transformations , and a final distribution over perturbed outputs is approximated via a mixture of Gaussians . The smooth function is then optimized using the gradient-free Nelder-Mead optimization method . The main differences to our method are that we perturb all relevant variables ( and not the inputs ) with logistic distributions and use this for gradient based optimization . 3 Method . To continuously relax algorithms and , thus , make them differentiable , we relax all values with respect to which we want to differentiate into logistic distributions . We choose the logistic distribution as it provides two distinctive properties that make it especially suited for the task at hand : ( 1 ) logistic distributions have heavier tails than normal distribution , which allows for larger probability mass and thus larger gradients when two compared values are further away from each other . ( 2 ) the cumulative density function ( CDF ) of the logistic distribution is the logistic sigmoid function , which can be computed analytically , and its gradient is easily computable . This contrasts the CDF of the normal distribution , which has no closed form and is commonly approximated via a polynomial [ 34 ] . Specifically , we relax any value x , for which we want to compute gradients , by perturbing it into a logistic random variable x̃ ∼ Logistic ( x , 1/β ) , where β is the inverse temperature parameter such that for β → ∞ : x̃ = x . Based on this , we can relax a discrete conditional statement , e.g. , based on the condition x < c with constant c ∈ R , as follows : [ y if x̃ < c else z ] ≡ ∫ c −∞ flog ( t ; x , 1/β ) · y dt + ∫ ∞ c flog ( t ; x , 1/β ) · z dt ( 1 ) = Flog ( c ; x , 1/β ) · y + ( 1− Flog ( c ; x , 1/β ) ) · z ( 2 ) = σ ( c− x ) · y + ( 1− σ ( c− x ) ) · z ( 3 ) where σ is the logistic ( sigmoid ) function σ ( x ) = 1/ ( 1 + e−xβ ) . In this example , as x increases , the result smoothly transitions from y to z . Thus , the derivative of the result wrt . x is defined as ∂ ∂x [ y if x̃ < c else z ] = ∂ ∂x ( y · σ ( c− x ) + z · ( 1− σ ( c− x ) ) ) ( 4 ) = ( z − y ) · σ ( c− x ) · ( 1− σ ( c− x ) ) ( 5 ) Hence , the gradient descent method can influence the condition ( x̃ < c ) to hold if the if case reduces the loss , or influence the condition to fail if the else case reduces the loss function . In this example , y and z may not only be scalar values but also results of algorithms or parts of an algorithm themselves . This introduces a recursive formalism of relaxed program flow , where parts of an algorithm are combined via a convex combination : [ f ( s ) if a < b else g ( s ) ] ≡ σ ( b− a ) · f ( s ) + ( 1− σ ( b− a ) ) · g ( s ) ( 6 ) where f and g denote functions , algorithms , or sequences of statements that operate on the set of all variables s via call-by-value and return the set of all variables s. The result of this may either overwrite the set of all variables ( s : = [ ... ] ) or be used in a nested conditional statement . After introducing if-else statements above , we extend the idea to loops , which extends the formalism of relaxed program flow to Turing-completeness . In fixed loops , i.e. , loops with a predefined number of iterations , since there is only a single computation path , no relaxation is necessary , and , thus , fixed loops can be handled by unrolling . The more complex case is conditional unbounded loops , i.e. , While loops , which are executed as long as a condition holds . For that , let ( si ) i∈N be the sequence of all variables after applying i times the content of a loop . That is , s0 = s for an initial set of all variables s , and si = f ( si−1 ) , where f is the content of a loop , i.e. , a function , an algorithm , or sequence of statements . Let a , b denote accessing variables of s , i.e. , s [ a ] , s [ b ] , respectively . By recursively applying the rule for if-else statements , we obtain the following rule for unbounded loops : [ while a < b do s : = f ( s ) ] ≡ ∞∑ i=0 ∏i j=0 ( σ ( bj − aj ) ) ︸ ︷︷ ︸ ( a ) · ( 1− σ ( bi+1 − ai+1 ) ) ︸ ︷︷ ︸ ( b ) · si ( 7 ) Here , ( a ) is the probability that the ith iteration is reached and ( b ) is the probability that there are no more than i iterations . Together , ( a ) and ( b ) is the probability that there are exactly i iterations weighing the state of all variables after applying i times f , which is si . Computationally , we evaluate the infinite series until the probability of execution ( a ) becomes numerically negligible or a predefined maximum number of iterations has been reached . Again , the result may either overwrite the set of all variables ( s : = [ ... ] ) or be used in a nested conditional statement . Complexity and Merging of Paths To compute the exact expectation value of an algorithm under logistic perturbation of its variables , all computation paths would have to be evaluated separately to account for dependencies . However , this would result in an exponential complexity . Therefore , we compute the exact expectation value for nested conditional blocks , but for sequential conditional blocks we merge the computation paths at the end of each block . This allows for a linear complexity in the number of sequential conditional blocks and an exponential complexity only in the largest depth of nested conditional blocks . Note that the number of sequential conditional blocks is usually much larger than the depth of conditional blocks , e.g. , hundreds or thousands of sequential blocks and a maximum depth of just 2− 5 in our experiments . An example for a dependency is the expression [ a : = ( f ( x ) if i < 0 else g ( x ) ) ; b : = ( 0 if a < 0 else a2 ) ] , which contains a dependence between the two sequential conditional blocks , which introduces the error in our approximation . In general , our formalism also supports modeling dependencies between sequential conditional blocks , however practically , it might become intractable for entire algorithms . Also , it is possible to consider relevant dependencies explicitly if an algorithm relies on specific dependencies . Perturbations of Variables vs . Perturbations of Inputs Note that modeling the perturbation of variables is different from modeling the perturbation of the inputs . A condition , where the difference becomes clear , is , e.g. , ( x̃ < x ) . When modeling input perturbations , the condition would have a strong implicit conditional dependency and evaluate to a 0 % probability . However , in this work , we do not model perturbations of the inputs , but instead model perturbations of variables each time they are accessed such that accessing a variable twice is independently identically distributed ( iid ) . Therefore , ( x̃ < x ) evaluates to a 50 % probability . To minimize the approximation error of the relaxation , only those variables should be relaxed for which a gradient is required .

Summary: The paper proposes an approach for continuous relaxations of algorithms that allows their integration into end-to-end trainable neural network architectures. It shows that the general approach can compete with SOTA continuous relaxations and could relaxes for complex algorithms, e.g., shortest-path algorithm or a renderer. Some experiments show the effectiveness.