AlgoNet: $C^\infty$ Smooth Algorithmic Neural Networks

Artificial neural networks have revolutionized many areas of computer science in recent years , providing solutions to a number of previously unsolved problems . On the other hand , for many problems , classic algorithms exist , which typically exceed the accuracy and stability of neural networks . To combine these two concepts , we present a new kind of neural networks—algorithmic neural networks ( AlgoNets ) . These networks integrate smooth versions of classic algorithms into the topology of neural networks . A forward AlgoNet includes algorithmic layers into existing architectures to enhance performance and explainability while a backward AlgoNet enables solving inverse problems without or with only weak supervision . In addition , we present the algonet package , a PyTorch based library that includes , inter alia , a smoothly evaluated programming language , a smooth 3D mesh renderer , and smooth sorting algorithms . 1 INTRODUCTION . Artificial Neural Networks are employed to solve numerous problems , not only in computer science but also in all other natural sciences . Yet , the reasoning for the topologies of neural networks seldom reaches beyond empirically-based decisions . In this work , we present a novel approach to designing neural networks—algorithmic neural networks ( short : AlgoNet ) . Such networks integrate algorithms as algorithmic layers into the topology of neural networks . However , propagating gradients through such algorithms is problematic , because crisp decisions ( conditions , maximum , etc . ) introduce discontinuities into the loss function . If one passes from one side of a crisp decision to the other , the loss function may change in a non-smooth fashion—it may “ jump. ” That is , the loss function suddenly improves ( or worsens , depending on the direction ) without these changes being locally noticeable anywhere but exactly at these “ jumps. ” Hence , a gradient descent based training , regardless of the concrete optimizer , can not approach these “ jumps ” in a systematic fashion , since neither the loss function nor the gradient provides any information about these “ jumps ” in any place other than exactly the location at which they occur . Therefore , a smoothing is necessary , such that information about the direction of improvement becomes exploitable by gradient descent also in the area surrounding the “ jump. ” That is , by smoothing , e.g. , an if , one can smoothly , by gradient descent , undergo a transition between the two crisp cases using only local gradient information . Generally , for end-to-end trainable neural network systems , all components should at least be C0 smooth , i.e. , continuous , to avoid “ jumps. ” However , having Ck smooth , i.e. , k times differentiable and then still continuous components with k ≥ 1 is favorable . This property of higher smoothness allows for higher-order derivatives and thus prevents unexpected behavior of the gradients . Hence , we designed smooth approximations to basic algorithms where the functions representing the algorithms are ideally C∞ smooth . That is , we designed pre-programmed neural networks ( restricted to smooth components ) with the structure of given algorithms . Algorithmic layers can solve sub-problems of the given problem , act as a custom algorithmic loss , or assist in finding an appropriate solution for ( ill-posed ) inverse problems . Such algorithmic losses can impose constraints on predicted solutions through optimization with respect to the algorithmic loss . Ill-posed problems are a natural application for algorithmic losses and algorithmic layers . For that , we introduce the Reconstructive Adversarial Network ( RAN ) in Sec . 3.3 . In this work , we describe the basic concept of algorithmic layers and present several applications . In Sec . 3.1.1 , we start by proving that any algorithm , which can be emulated by a Turing machine , can be approximated by a C∞ smooth function . In Sec . 4 , we present some algorithmic layers that we designed to solve underlying problems . In the appendix , we present a case study on 3D geometry reconstruction that demonstrates the applicability of RANs . All described algorithmic layers and models are provided in the algonet package , a PyTorch ( Paszke et al . ( 2017 ) ) based library for AlgoNets . 2 RELATED WORK . Related work ( Mart ’ in Abadi et al . ( 2015 ) ; Che et al . ( 2018 ) ; Henderson & Ferrari ( 2018 ) ) in neural networks focused on dealing with crisp decisions by passing through gradients for the alternatives of the decisions . There is no smooth transition between the alternatives , which introduces discontinuities in the loss function that hinder learning , which of the alternatives should be chosen . TensorFlow contains a sorting layer ( tf.sort ) as well as a while loop construct ( tf.while_loop ) . Since the sorting layer only performs a crisp relocation of the gradients and the while loop has a crisp exit condition , there is no gradient with respect to the conditions in these layers . Concurrently , we present a smooth sorting layer in Sec . 4.1 and a smooth while loop in Sec . 3.1.1 . Theoretical work by DeMillo et al . ( DeMillo & Lipton ( 1993 ) ) proved that any program could be modeled by a smooth function . Consecutive works ( Nesterov ( 2005 ) ; Chaudhuri & Solar-Lezama ( 2011 ) ; Yang & Barnes ( 2017 ) ) provided approaches for smoothing programs using , i.a. , Gaussian smoothing ( Chaudhuri & Solar-Lezama ( 2011 ) ; Yang & Barnes ( 2017 ) . ) 3 ALGONET . To introduce algorithmic layers , we prove that smooth approximations for any Turing computable algorithm exist and explain two flavors of AlgoNets : forward and backward AlgoNets . 3.1 SMOOTH ALGORITHMS . To design a smooth algorithm , all discrete cases ( e.g. , conditions of if statements or loops ) have to be replaced by continuous or smooth functions . The essential property is that the implementation is differentiable with respect to all internal choices and does not—as in previous work—only carry the gradients through the algorithm . For example , an if statement can be replaced by a sigmoid-weighted sum of both cases . By using a smooth sigmoid function , the statement is smoothly interpreted . Hence , the gradient descent method can influence the condition to hold if the content of the then case reduces the loss and influence the condition to fail if the loss is lower when the else case is executed . Thus , the partial derivative with respect to a neuron is computed because the neuron is used in the if statement . In contrast , when propagating back the gradient of the then or the else case depending on the value of the condition , there is a discontinuity at the points where the value of the condition changes and the partial derivative of the neuron in the condition equals zero . s1 ( x , s ) = 1 1 + e−x·s with s = 1 ( 1 ) s2 ( x ) = { 0 if x < 0 1 else ( 2 ) Here , the logistic sigmoid function ( Eq . 1 ) is a C∞ smooth replacement for the Heaviside sigmoid function ( Eq . 2 ) , which is equivalent to the if statement . Alternatively , one could use other sigmoid functions , e.g. , the C1 smooth step function x2 − 2 · x3 for x ∈ [ 0 , 1 ] , and 0 and 1 for all values before and after the given range , respectively . Another example is the max-operator , which , in neural networks , is commonly replaced by the SoftMax operator ( SoftMax ( x ) i = exp ( xi ) ∑ j exp ( xj ) ) . After designing an algorithmic layer , we can use it to enhance a neural network and to solve for its inverse by using the reconstructive adversarial neural network ( RAN ) as shall be described in Sec . 3.2 , and 3.3 . 3.1.1 SMOOTH WHILE-PROGRAMS . In this section , we prove that for any algorithm , C∞ smooth approximations exist . For that , we present smooth and differentiable approximations to an elementary programming language based on the WHILE-language by Uwe Schöning ( Schöning ( 2008 ) . ) We do not want to imply that such a direct translation of any algorithm into the WHILE-language to make it smooth is the best or even the only option . It is rather meant as a fallback solution if no better solution can be found for a sub-problem . We will present several more efficient and canonical approximations in Sec . 4 . The WHILE-language is Turing-complete , and for variables ( var ) its grammar can be defined as follows : prog = WHILE var ! = 0 DO prog END | prog prog | var : = var // left and right var unequal | var : = var + 1 // left and right var equal | var : = var - 1 // left and right var equal Here , var ∈ { xn | n ∈ N0 } and while x0 is the output , xn where n ∈ N0 are variables initialized to 0 if not set to an input value . Thus , { xn | n ∈ N+ } are inputs and/or local variables used in the computations . Although this interpretation allows only for a single scalar output value ( x0 ) , an arbitrary number of output values can be reached , either by interpreting additional variables as output or by using multiple WHILE-programs ( one for each output value ) . Provided Church ’ s thesis holds , this language covers all effectively calculable functions . The WHILE-language is equivalent to register machines which have no WHILE , but instead IF and GOTO statements . We generate the approximation to this language by executing all statements only to the extent of their probability . That is , we keep track of a probability p , indicating whether the current statement is still executed . Initially , p = 1 . In the body of a while loop , the probability for an execution is p ( new ) k ( x ) = p ( old ) k · φk ( x ) where k defines the smoothness of the probability function for exiting the loop φk . To obtain C∞ smooth WHILE-programs , we used φ∞ : R → [ 0 , 1 ] : x 7→ ( esx−1 ) 2 e2sx+1 = 1− sech ( sx ) . For C 0 smoothness , we used the shouldered fuzzy set φ0 : R → [ 0 , 1 ] : x 7→ 1 −max ( 0 , 1 − |x| ) = min ( 1 , |x| ) . For x0 ... xn initialized as integers , using φ0 , the result for the C0 WHILE-program always equals the result for the discrete WHILE-program since the probability is always either 1 or 0 . For all k ∈ N0 , φk exists . Fig . 1 shows how these exit probability functions behave . Because of the symmetry of φ , w.l.o.g. , we can assume that ∀x ∈ R≥0 . If the loop ( in the discrete version ) increases x by 1 , x will diverge . If x decreases by 1 , p converges to 0 . Since φ0 ( x ) ≥ φ∞ ( x ) , it suffices to show that p converges to zero for φ0 . Since p ( new ) ( x ) = p ( old ) ( x ) · φ0 ( x ) ≤ φ0 ( x ) ≤ |x| , x : = x − p ( new ) ( x ) ≥ x − |x| = 0 . Thus , x and φ0 ( x ) monotonically decrease and x ≥ 0 . Eventually , φ0 ( x ) < 1 . Thus , p ( x ) ≤ ( φ0 ( x ) ) n n→∞ −−−−→ 0 . Here , x is the value of the current variable , s is the steepness , and p is the probability of the execution . To apply the probabilities on the assignment , increment and decrement operators , we redefine them as : x0 : = x1 −→ x0 : = p · x1+ ( 1− p ) · x0 ( 3 ) x0 : = x0+ 1 −→ x0 : = x0+ p ( 4 ) x0 : = x0− 1 −→ x0 : = x0− p ( 5 ) Since the hyperbolic secant is C∞ smooth , our version of the WHILE-language is C∞ smooth . While the probability converges to zero , it ( in most cases ) never reaches zero , and the loop would never exit . Thus , we introduce ǫ > 0 and exit the loop if p ≤ ǫ or a maximum number of iterations is reached . Although this introduces discontinuities , by choosing an ǫ of numerically negligible size , the discontinuities also become numerically negligible . As an experiment , we implemented the multiplication on positive integers , as shown on the left : WHILE x2 ! = 0 DO | WITH p1 : = 1 ; p ′ 1 : = p1 · φ ( x2 ) DO x3 : = x1 | x3 : = p1 · x1+ ( 1− p1 ) · x3 WHILE x3 ! = 0 DO | WITH p2 : = p1 ; p ′ 2 : = p2 · φ ( x3 ) DO x0 : = x0 + 1 | x0 : = x0 + p2 x3 : = x3 - 1 | x3 : = x3 - p2 END | WHILE p2 ≥ ǫ x2 : = x2 - 1 | x2 : = x2 - p1 END | WHILE p1 ≥ ǫ Contrary to the discrete implementation , the smooth interpretation ( as on the right ) can interpolate the result for arbitrary values x1 , x2 ∈ R+ . Since the WHILE-language is Turing-complete , any high-level program could , in principle , using an appropriate compiler , be translated into an equivalent program in WHILE-language . To this WHILE-program , automatic smoothing could be applied using the rules that are illustrated here . Of course , there are better ways of smoothing : manual smoothing using domain-specific knowledge and smoothing using a higher-level language outperform the low-level automatic smoothing . For example , in a higher-level language , the multiplication would be implemented since it is smooth itself . Using domain-specific knowledge , algorithms could be reformulated in such a way that smoothing is possible in a more canonical way . To translate a WHILE-program into a neural network , the WHILE loops are considered as recurrent sub-networks .

Summary: The paper conceptualizes a neural network architecture that integrates smoothed versions of traditional algorithms into the network topology. The AlgoNet concept employs smoothed versions of algorithms implemented in the WHILE language, with options for different levels of differentiability. The paper outlines a forward version of AlgoNet based on traditional, skip, and residual connections, and a backward version for solving inverse problems in a reconstructive, autoencoder-like fashion. As smoothing introduces a form of domain shift, the paper the reconstructive adversarial network that that employs "domain translators" as well as a discriminator that are trained in an adversarial fashion. The paper concludes by describing versions of AlgoNet for various different algorithms.