Differential Equation Networks

1 INTRODUCTION AND RELATED WORK . Driven in large part by advancements in storage , processing , and parallel computing , deep neural networks ( DNNs ) have become capable of outperforming other methods across a wide range of highly complex tasks . Although DNNs often produce better results than shallow methods from a performance perspective , one of the main drawbacks of DNNs in practice is computational expense . One could attribute much of the success of deep learning in recent years to cloud computing , and GPU processing . While deep learning based applications continue to be integrated into all aspects of modern life , future advancements will continue to be dependent on the ability to perform more operations , faster , and in parallel unless we make fundamental changes to the way these systems learn . State-of-the-art DNNs for computer vision , speech recognition , and healthcare applications require too much memory , computation , and power to be run on current mobile or wearable devices . To run such applications on mobile , or other resource-constrained devices , either we need to use these devices as terminals and rely on cloud resources to do the heavy lifting , or we have to find a way to make DNNs more compact . For example , ProjectionNet by Ravi ( Ravi , 2017 ) , and MobileNet by Howard et al . ( Howard et al. , 2017 ) are both examples of methods that use a compact DNN representations with the goal of on-device applications . In ProjectionNet , a compact ” projection ” network is trained in parallel to the primary network , and used for the on-device network tasks . MobileNet proposes a streamlined architecture in order to achieve network compactness . One drawback to these approaches is that network compactness is achieved at the expense of performance . In this paper , we propose a different method for learning compact , powerful , stand-alone networks ; we allow each neuron to learn its individual activation function . Up to this point , advancements in deep learning have been largely associated with one of four areas : optimization , regularization , activation , and network structure . Some of these advancements , such as network structure , address issues related to the type of problem at hand , and the nature of the data . Since the advent of the classic feed-forward neural network ( FFNN ) , researchers have introduced highly effective network structures such as convolutional neural networks ( CNN ( Krizhevsky et al. , 2012 ) , popular in computer vision , and Recurrent Neural Networks ( RNN ) for sequences . The advent of new activation functions such as rectified linear units ( ReLU ) ( Nair & Hinton , 2010 ) , exponential linear units ( ELU ) ( Clevert et al. , 2015 ) , and scaled exponential linear units ( SELU ) ( Klambauer et al. , 2017 ) address a networks ability to effectively learn complicated functions , thereby allowing them to perform more complicated tasks . The choice of an activation function is typically determined empirically by tuning , or due to necessity . For example , in modern deep networks , the ReLU activation function is often favored over the sigmoid function , which used to be a popular choice in the earlier days of neural networks . A reason for this preference is that the ReLU function is nonsaturating and does not have the vanishing gradient problem when used in deep structures ( Hochreiter , 1998 ) . Recently , searching algorithms have proven to be valuable for automatic construction of previously hand-designed network components and activation functions ( Zoph et al. , 2017 ; Bello et al. , 2017 ; Ramachandran et al. , 2017 ) . Zoph et al . ( 2017 ) ; Bello et al . ( 2017 ) ; Ramachandran et al . ( 2017 ) have proposed one such method for searching for more effective activation functions . At first glance , and from an applicability perspective , our approach is similar to max-out networks ( Goodfellow et al. , 2013 ) , adaptive piecewise linear units by Agostinelli et al . ( 2014 ) , and Ramachandran et al . ( 2017 ) . However , in Ramachandran et al . ( 2017 ) , their functions do not transform themselves during network training , and all neurons utilize the same activation function . We introduce differential equation networks ( DifENs ) , where the activation function of each neuron is the nonlinear , possibly periodic solution of a second order , linear , ordinary differential equation . While the number of parameters learned by max-out and PLU is proportional to the number of input weights to a neuron , and the number of linear units in that neuron , for each DifEN activation function we learn only five additional parameters . One feature of DifENs that is particularly interesting within the healthcare industry is the ability of DifENs to regress decaying and periodic functions . Although efforts have been made to explore the usefulness of periodic functions in neural networks since the 1990s , their applicability has not yet been appreciated ( Sopena et al. , 1999 ) . Thanks to the successes of deep learning in recent years , researchers have begun re-exploring the potential of periodic functions as activations ( Parascandolo et al. , 2016 ) . This capability makes DifENs particularly well suited to problems where the desired output signals display combinations of periodicity and decay . Some examples include disease management , modeling chronic conditions such as exacerbation of COPD , medication adherence , and acute conditions that are expected to get better over time such as cancer with a successful course of treatment . Inversely , healthiness indices that might decay over time , and that could drastically effect a patients health if effectively modelled ( e.g . cancer with an unsuccessful course of treatment ) . Our contributions in this paper include the following : We introduce differential equation networks , which can learn complex concepts in compact representations . We propose a learning process to learn the parameters of a differential equation for each neuron . Finally , we empirically show that a differential equation network is effective when applied to real-world problems , and that they are capable of changing activation functions during the learning process . 2 NETWORK COMPRESSION . The size of a neural network is delineated its number of hidden neurons and their interconnections , which together determine the network ’ s complexity . The ideal size of a network depends on the intricacy of the concept it is required to learn . A network that is too small can not entirely , and circumstantially learn a hard problem . The existing , successful structures of neural networks Sutskever et al . ( 2014 ) ; LeCun et al . ( 2015 ) ; Bahdanau et al . ( 2014 ) use hundreds of thousands to millions of parameters to learn common tasks , such as translation or object recognition . Although these tasks are complex to model , and still challenging for modern neural networks ( Sutskever et al. , 2014 ) , they are considered ” easy ” when performed by a qualified human . It follows that , if we want to create a neural network with abilities that are beyond average human level aptitude , the size of such a network would need to grow considerably . For example , we have seen accuracy improvements in state-of-the-art DNNs applied to massive image classification problems resulting in part from a large increase in the number of layers He et al . ( 2016 ) . If this trend continues to subsist , shortly , we will witness neural networks with billions of parameters that are designed for sophisticated tasks such as live speech understanding and translation , as well as controlling robots or humanoids , which would require an immense increase in storage and computational resources . For these reasons , we want to highlight the importance of efficient network design , and the opportunities presented by network contraction methodologies . We show that differential equation networks learn a given task with a smaller network when compared to a fixed activation DNNs . 3 METHOD . Inspired by functional analysis and calculus of variations , instead of using a fixed activation function for each layer , we propose a novel solution for learning an activation function for each neuron in the network . In the results section we show that by allowing each neuron to learn its own activation function , the network as a whole can perform on par with ( or even outperform ) much larger baseline networks . In the supplementary material , we have briefly outlined how in Calculus of Variations , the Euler-Lagrange formula can be used to derive a differential equation from an optimization problem with function variables ( Gelfand & Fomin , 1963 ; Gelfand et al. , 2000 ) . The idea of this paper is to find the parameters of an ordinary differential equation ( ODE ) for each neuron in the network , whose solution would be used as the activation function of the neuron . As a result , the differential equation network enables each neuron to learn a personalized activation function flexibly . We select ( learn ) the parameters of the differential equation from a low dimensional space ( i.e. , five ) . By minimizing the network loss function , our learning algorithm smoothly updates the parameters of the ODE , which would result in an uncountably 1 extensive range of possible activation functions . 3.1 DIFFERENTIAL EQUATION ACTIVATION . We parametrize the activation function of each neuron using a linear , second order ordinary differentiatial equation ay′′ ( t ) + by′ ( t ) + cy ( t ) = g ( t ) , parameterized by five coefficients ( a , b , c , c1 , c2 ) , which can be learned by the backpropagation algorithm . These coefficients are the only additional parameters that we learn for each neuron . a , b and c are the scalars that we use to parametrize the ODE , and c1 and c2 represent the initial conditions of the ODE ’ s solution . g ( t ) is a regulatory function that we have called the ” core activation function ” . Mainly to simplify the math , and because it is a standard practice in control theory , we have set g ( t ) to the Heaviside step function . Therefore , the ODE that we have chosen is the following : ay′′ ( t ) + by′ ( t ) + cy ( t ) = u ( t ) , where u ( t ) = { 0 x ≤ 0 1 x > 0 ( 1 ) In engineering and physics , such a model is often used to denote the exchange of energy between mass and stiffness elements in a mechanical system , or between capacitors and inductors in an electrical system ( Ogata & Yang , 2002 ) . Interestingly by using the solutions of this formulation as activation functions , we can gain a few key properties : approximation or reduction to some of the standard activation functions such as sigmoid or ReLU ; the ability to capture oscillatory forms ; and , exponential decay or growth . The latter commonly appear in health-care problems . 3.2 THE LEARNING ALGORITHM . For fixed a , b and c , the solution of the differential equation will be y = f ( t ; a , b , c ) +c1f1 ( t ; a , b , c ) + c2f2 ( t ; a , b , c ) for some functions f , f1 , f2 . y lies on an affine space parametrized by scalars c1 and c2 that represent the initial conditions of the solution . As described in the following two subsections , our learning algorithm has two main parts : solving the differential equations once , and using a backpropagation-based algorithm for jointly learning the network weights and the five parameters of each neuron .

Summary: This paper proposes an intriguing idea, that of using solutions to a differential equation as activation functions in a neural network. Coefficients of the differential equation (five parameter equations in implementations done) are trainable, realising different activation functions on different nodes. Back propagation is used with the ADMA optimiser to train the parameters of the DE as well as the remaining weights. What the network implements is shown to achieve universal approximation by considering a second order differential equation producing different sinusoidal functions which can add up to a desired function. The paper is written clearly and easy to follow. The idea is novel. The work is illustrated on three problems: (a) a toy dataset, (b) MNIST classification problem and (c) a regression task from a diabetes problem.