Learning with Algorithmic Supervision via Continuous Relaxations

1 Introduction . Artificial Neural Networks have shown their ability to solve various problems , ranging from classical tasks in computer science such as machine translation [ 1 ] and object detection [ 2 ] to many other topics in science such as , e.g. , protein folding [ 3 ] . Simultaneously , classical algorithms exist , which typically solve predefined tasks based on a given input and a predefined control structure such as , e.g. , sorting , shortest-path computation and many more . Recently , research has started to combine both elements by integrating algorithmic concepts into neural network architectures . Those approaches allow training neural networks with alternative supervision strategies , such as learning 3D representations via a differentiable renderer [ 4 ] , [ 5 ] or training neural networks with ordering information [ 6 ] , [ 7 ] . We unify these alternative supervision strategies , which integrate algorithms into the training objective , as algorithmic supervision : Definition 1 ( Algorithmic Supervision ) . In algorithmic supervision , an algorithm is applied to the predictions of a model and the outputs of the algorithm are supervised . In contrast , in direct supervision , the predictions of a model are directly supervised . This is illustrated in Figure 1 . In general , to allow for end-to-end training of neural architectures with integrated algorithms , the challenge is to estimate the gradients of a respective algorithm , e.g. , by a differentiable approximation . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Here , most solutions are tailored to specific problems like , e.g. , differentiable sorting or rendering . But also more general frameworks have recently been proposed , which estimate gradients for combinatorial optimizers . Examples for such approaches are the differentiation of black box combinatorial solvers as proposed by Vlastelica et al . [ 8 ] and the differentiation of optimizers by stochastically perturbing their input as proposed by Berthet et al . [ 9 ] . Both approaches focus on the problem that it is necessary to estimate the gradients of an algorithm to allow for an end-to-end training of neural architectures including it . To address this issue , Vlastelica et al . [ 8 ] estimate the gradients of an optimizer by a one-step linearization method that takes the gradients with respect to the optimizer ’ s output into account , which allows to integrate any off-the-shelf combinatorial optimizer . Berthet et al . [ 9 ] estimate the gradients by perturbing the input to a discrete solver by random noise . In this context , we propose an approach for making algorithms differentiable and estimating their gradients . Specifically , we propose continuous relaxations of different algorithmic concepts such as comparators , conditional statements , bounded and unbounded loops , and indexing . For this , we perturb those variables in a discrete algorithm by logistic distributions , for which we want to compute a gradient . This allows us to estimate the expected value of an algorithm ’ s output sampling-free and in closed form , e.g. , compared to methods approximating the distributions via Monte-Carlo sampling methods ( e.g. , [ 9 ] ) . To keep the computation feasible , we approximate the expectation value by merging computation paths after each conditional block in sequences of conditional blocks . For nested conditional blocks , we compute the exact expectation value without merging conditional cases . This trade-off allows merging paths in regular intervals , and thus alleviates the need to keep track of all possible path combinations . As we model perturbations of variables when they are accessed , all distributions are independent , which contrasts the case of modeling input perturbations , where all computation paths would have to be handled separately . To demonstrate the practical aspects , we apply the proposed approach in the context of four tasks , that make use of algorithmic supervision to train a neural network , namely sorting supervision [ 6 ] , [ 7 ] , [ 10 ] , shortest-path supervision [ 8 ] , [ 9 ] , silhouette supervision ( differentiable rendering ) [ 4 ] , [ 5 ] , [ 11 ] , and finally Levenshtein distance supervision . While the first three setups are based on existing benchmarks in the field , we introduce the fourth experiment to show the application of this idea to a new algorithmic task in the field of differentiable dynamic programming . In the latter , the Levenshtein distance between two concatenated character sequences from the EMNIST data set [ 12 ] is supervised while the individual letters remain unsupervised . We show that the proposed system is able to outperform current state-of-the-art methods on sorting supervision , while it performs comparable on shortest path supervision and silhouette supervision . 2 Related Work . We cover the field of training networks with algorithmic supervision . While some of these works cover single algorithmic concepts , e.g. , sorting [ 6 ] , [ 7 ] , [ 13 ] or rendering [ 4 ] , [ 5 ] , [ 14 ] , others have covered wider areas such as dynamic programming [ 15 ] , [ 16 ] or gradient estimation for general optimizers [ 8 ] , [ 9 ] , [ 17 ] , [ 18 ] . Paulus et al . [ 19 ] employ the Gumbel-Softmax [ 20 ] distribution to estimate gradients for concepts such as subset selection . Xie et al . [ 21 ] and Blondel et al . [ 13 ] employ differentiable sorting methods for top-k classification . Cuturi et al . [ 22 ] and Blondel et al . [ 23 ] present approaches for differentiable dynamic time warping . In machine translation , Bahdanau et al . [ 24 ] investigate soft alignments , while Collobert et al . [ 25 ] present a fully differentiable beam sorting algorithm . In speech recognition , Bahdanau et al . [ 26 ] propose differentiable surrogate task losses for sequence prediction tasks . Shen et al . [ 27 ] propose a differentiable neural parsing networks using the stick breaking process . Dasgupta et al . [ 28 ] model box parameters with Gumbel distributions to obtain soft differentiable box embeddings . In the following , we focus on those works that provide relaxations for tasks considered in this work . We close the overview with a review of some early works on smooth program interpretation . Sorting Supervision The task of training neural networks with sorting supervision has been first proposed by Grover et al . [ 6 ] . In this case , a set four-digit numbers based on concatenated MNIST digits [ 29 ] is given , and the task is to find an order-preserving mapping from the images to scalars . Here , a CNN should predict a scalar for each of n four-digit numbers such that their order is preserved among the predicted scalars . For training , only sorting supervision in the form of the ground truth order of input images is given , while their absolute values remain unsupervised . Grover et al . [ 6 ] address this task by relaxing the permutation matrices to double stochastic matrices . Cuturi et al . [ 7 ] pick up this on benchmark and propose a differentiable proxy by approximating the sorting problem with a regularizing optimal transport algorithm . Silhouette Supervision A line of work in computer vision is differentiable renderers for 3Dunsupervised 3D-reconstruction . Here , recent approaches have proposed differentiable renderers for 3D mesh reconstruction with only 2D silhouette supervision [ 4 ] , [ 5 ] on 13 classes of the ShapeNet data set [ 30 ] . Kato et al . [ 5 ] propose a renderer where surrogate gradients of rasterization are approximated to perform 3D mesh reconstruction via silhouette supervision as well as 3D style transfer . Liu et al . [ 4 ] propose a differentiable renderer without surrogate gradients by using a differentiable aggregating process and apply it to 3D mesh reconstruction as well as pose / shape optimization . Shortest-Path Supervision Two works , proposed by Vlastelica et al . [ 8 ] and Berthet et al . [ 9 ] presented methods for estimating gradients for off-the-shelf optimizers . In this context , Vlastelica et al . [ 8 ] proposed an experiment of shortest-path supervision , where an image of a terrain is given with different costs for each type of terrain , and the task is to predict the shortest path from the top-left to the opposite corner . Integrating a shortest-path algorithm into a neural architecture lets the neural network produce a cost embedding of the terrain which the shortest-path algorithm uses for predicting the shortest path . Vlastelica et al . [ 8 ] tackle this task by finding a linearization of the Dijkstra algorithm [ 31 ] , which they can differentiate . Berthet et al . [ 9 ] take up this problem and produce gradient estimates for the Dijkstra algorithm by stochastically perturbing the inputs to the shortest-path optimization problem . Smooth Interpretation In another line of work , in the field of computer-aided verification , Chaudhuri et al . [ 32 ] , [ 33 ] propose a program smoothing method based on randomly perturbing the inputs of a program by a Gaussian distribution . Here , an initial Gaussian perturbation is propagated through program transformations , and a final distribution over perturbed outputs is approximated via a mixture of Gaussians . The smooth function is then optimized using the gradient-free Nelder-Mead optimization method . The main differences to our method are that we perturb all relevant variables ( and not the inputs ) with logistic distributions and use this for gradient based optimization . 3 Method . To continuously relax algorithms and , thus , make them differentiable , we relax all values with respect to which we want to differentiate into logistic distributions . We choose the logistic distribution as it provides two distinctive properties that make it especially suited for the task at hand : ( 1 ) logistic distributions have heavier tails than normal distribution , which allows for larger probability mass and thus larger gradients when two compared values are further away from each other . ( 2 ) the cumulative density function ( CDF ) of the logistic distribution is the logistic sigmoid function , which can be computed analytically , and its gradient is easily computable . This contrasts the CDF of the normal distribution , which has no closed form and is commonly approximated via a polynomial [ 34 ] . Specifically , we relax any value x , for which we want to compute gradients , by perturbing it into a logistic random variable x̃ ∼ Logistic ( x , 1/β ) , where β is the inverse temperature parameter such that for β → ∞ : x̃ = x . Based on this , we can relax a discrete conditional statement , e.g. , based on the condition x < c with constant c ∈ R , as follows : [ y if x̃ < c else z ] ≡ ∫ c −∞ flog ( t ; x , 1/β ) · y dt + ∫ ∞ c flog ( t ; x , 1/β ) · z dt ( 1 ) = Flog ( c ; x , 1/β ) · y + ( 1− Flog ( c ; x , 1/β ) ) · z ( 2 ) = σ ( c− x ) · y + ( 1− σ ( c− x ) ) · z ( 3 ) where σ is the logistic ( sigmoid ) function σ ( x ) = 1/ ( 1 + e−xβ ) . In this example , as x increases , the result smoothly transitions from y to z . Thus , the derivative of the result wrt . x is defined as ∂ ∂x [ y if x̃ < c else z ] = ∂ ∂x ( y · σ ( c− x ) + z · ( 1− σ ( c− x ) ) ) ( 4 ) = ( z − y ) · σ ( c− x ) · ( 1− σ ( c− x ) ) ( 5 ) Hence , the gradient descent method can influence the condition ( x̃ < c ) to hold if the if case reduces the loss , or influence the condition to fail if the else case reduces the loss function . In this example , y and z may not only be scalar values but also results of algorithms or parts of an algorithm themselves . This introduces a recursive formalism of relaxed program flow , where parts of an algorithm are combined via a convex combination : [ f ( s ) if a < b else g ( s ) ] ≡ σ ( b− a ) · f ( s ) + ( 1− σ ( b− a ) ) · g ( s ) ( 6 ) where f and g denote functions , algorithms , or sequences of statements that operate on the set of all variables s via call-by-value and return the set of all variables s. The result of this may either overwrite the set of all variables ( s : = [ ... ] ) or be used in a nested conditional statement . After introducing if-else statements above , we extend the idea to loops , which extends the formalism of relaxed program flow to Turing-completeness . In fixed loops , i.e. , loops with a predefined number of iterations , since there is only a single computation path , no relaxation is necessary , and , thus , fixed loops can be handled by unrolling . The more complex case is conditional unbounded loops , i.e. , While loops , which are executed as long as a condition holds . For that , let ( si ) i∈N be the sequence of all variables after applying i times the content of a loop . That is , s0 = s for an initial set of all variables s , and si = f ( si−1 ) , where f is the content of a loop , i.e. , a function , an algorithm , or sequence of statements . Let a , b denote accessing variables of s , i.e. , s [ a ] , s [ b ] , respectively . By recursively applying the rule for if-else statements , we obtain the following rule for unbounded loops : [ while a < b do s : = f ( s ) ] ≡ ∞∑ i=0 ∏i j=0 ( σ ( bj − aj ) ) ︸ ︷︷ ︸ ( a ) · ( 1− σ ( bi+1 − ai+1 ) ) ︸ ︷︷ ︸ ( b ) · si ( 7 ) Here , ( a ) is the probability that the ith iteration is reached and ( b ) is the probability that there are no more than i iterations . Together , ( a ) and ( b ) is the probability that there are exactly i iterations weighing the state of all variables after applying i times f , which is si . Computationally , we evaluate the infinite series until the probability of execution ( a ) becomes numerically negligible or a predefined maximum number of iterations has been reached . Again , the result may either overwrite the set of all variables ( s : = [ ... ] ) or be used in a nested conditional statement . Complexity and Merging of Paths To compute the exact expectation value of an algorithm under logistic perturbation of its variables , all computation paths would have to be evaluated separately to account for dependencies . However , this would result in an exponential complexity . Therefore , we compute the exact expectation value for nested conditional blocks , but for sequential conditional blocks we merge the computation paths at the end of each block . This allows for a linear complexity in the number of sequential conditional blocks and an exponential complexity only in the largest depth of nested conditional blocks . Note that the number of sequential conditional blocks is usually much larger than the depth of conditional blocks , e.g. , hundreds or thousands of sequential blocks and a maximum depth of just 2− 5 in our experiments . An example for a dependency is the expression [ a : = ( f ( x ) if i < 0 else g ( x ) ) ; b : = ( 0 if a < 0 else a2 ) ] , which contains a dependence between the two sequential conditional blocks , which introduces the error in our approximation . In general , our formalism also supports modeling dependencies between sequential conditional blocks , however practically , it might become intractable for entire algorithms . Also , it is possible to consider relevant dependencies explicitly if an algorithm relies on specific dependencies . Perturbations of Variables vs . Perturbations of Inputs Note that modeling the perturbation of variables is different from modeling the perturbation of the inputs . A condition , where the difference becomes clear , is , e.g. , ( x̃ < x ) . When modeling input perturbations , the condition would have a strong implicit conditional dependency and evaluate to a 0 % probability . However , in this work , we do not model perturbations of the inputs , but instead model perturbations of variables each time they are accessed such that accessing a variable twice is independently identically distributed ( iid ) . Therefore , ( x̃ < x ) evaluates to a 50 % probability . To minimize the approximation error of the relaxation , only those variables should be relaxed for which a gradient is required .

Summary: The continuous relaxation of certain elements of certain programming primitives have been used in deep learning to incorporate constraints or logic into architectures. Attention, for example, has been viewed as a key-value dictionary lookup operation. This paper aims to formalise some of these operations in a more general framework.  The main aspect that is considered is the if-else construct in programming. Rewriting the if-else conditionals as an integral over the logistic, they arrive at the classic gating function we often use in deep learning (e.g. GRUs, LSTM). This is extended to while loops, which are repeated applications of the if-else conditional. This particular equation looks like an expectation over the stick-breaking process (https://en.wikipedia.org/wiki/Dirichlet_process#The_stick-breaking_process)  Similar relaxations are then proposed for the equals comparator, with interesting initial assumptions that lead to the use of sech. Relaxed indexing also accounts for cases where the indexing is ordinal.