Learning Neuron Non-Linearities with Kernel-Based Deep Neural Networks

1 INTRODUCTION . By and large , the appropriate selection of the activation function in deep architectures is regarded as an important choice for achieving challenging performance . For example , the rectifier function ( Glorot et al. , 2011 ) has been playing an important role in the impressive scaling up of nowadays deep nets . Likewise , LSTM cells ( Hochreiter & Schmidhuber , 1997 ) are widely recognized as the most important ingredient to face long-term dependencies when learning by recurrent neural networks . Both choices come from insightful ideas on the actual non-linear process taking place in deep nets . At a first glance , one might wonder why such an optimal choice must be restricted to a single unit instead of extending it to the overall function to be learned . In addition , this general problem has been already been solved ; its solution ( Poggio & Girosi , 1990 ; Girosi et al. , 1995 ; 2000 ) is in fact at the basis of kernel machines whose limitations as shallow nets have been widely addressed ( see e.g . ( LeCun et al. , 2015 ; Mhaskar et al. , 2016 ) ) . However , the optimal formulation given for the neuron non-linearity enjoys the tremendous advantage of acting on 1-D spaces . This strongly motivates the reformulation of the problem of learning in deep neural network as a one where the weights and the activation functions are jointly determined by optimization in the framework of regularization operators ( Smola et al. , 1998 ) , that are used to enforce the smoothness of the solution . The idea of learning the activation function is not entirely new . In ( Turner & Miller , 2014 ) , activation functions are chosen from a pre-defined set and combine this strategy with a single scaling parameter that is learned during training . It has been argued that one can think of this function as a neural network itself , so the overall architecture is still characterized by a directed acyclic graph ( Castelli & Trentin , 2014 ) . Other approaches learn activation functions as piecewise linear ( Agostinelli et al. , 2014 ) , doubled truncated gaussian ( Su et al. , 2017 ) or Fourier series ( Eisenach et al. , 2016 ) . While working on this representational issues we have recently discovered the paper by ( Scardapane et al. , 2017 ) , which introduces a family of activation functions that are based on a kernel expansion at every neuron . The proposed approach is based on the nice intuition that a kernel-based representation at for the neuron function is computationally efficient , yet very effective in terms of representation . The authors provide strong support to their idea by significant experimental results . Interestingly , in this paper , where we provide additional independent support to the kernel-based representation of the neuron function given in ( Scardapane et al. , 2017 ) by reinforcing the idea in different ways . In particular , we prove that , like for kernel machines , the optimal solution of the variational problem that characterizes the process of supervised learning in the framework of regularization can be expressed by a kernel expansion , so as the overall optimization is reduced to the discovery of a finite set of parameters . The risk function to be minimized contains the weights of the network connections , as well as the parameters associated with the the points of the kernel expansion . Hence , the classic learning of the weights of the network takes place with the concurrent development of the optimal shape of the activation functions , one for each neuron . As a consequence , the machine architecture turns out to enjoy the strong representational issues of deep networks in high dimensional spaces that is conjugated with the elegant and effective setting of kernel machines for the learning of the activation functions . The powerful unified regularization framework is not the only feature that emerges from the proposed architecture . Interestingly , unlike most of the activation functions used in deep networks , those that are typically developed during learning , are not necessarily monotonic . This property has a crucial impact in their adoption in classic recurrent networks , since this properly addresses classic issues of gradient vanishing when capturing long-term dependencies . Throughout this paper , recurrent networks with activation functions based on kernel expansion , are referred to as Kernel-Based Recurrent Networks ( KBRN ) . The intuition is that the associated iterated map can either be contractive or expansive . Hence , while in some states the contraction yields gradient vanishing , in others the expansion results in to gradient pumping , which allows the neural network to propagate information back also in case of long time dependences . The possibility of implementing contractive and expanding maps during the processing of a given sequence comes from the capabilities of KBRN to develop different activation functions for different neurons that are not necessarily monotonic . This variety of units is somewhat related to the clever solution proposed in LSTM cells ( Hochreiter & Schmidhuber , 1997 ) , where the authors realized early that there was room for getting rid of the inherent limitation of the contractive maps deriving from sigmoidal units . This contribution to the representation and learning in recurrent neural network is another difference with respect to the related contribution given in ( Scardapane et al. , 2017 ) . The given experimental results provided below demonstrate this property on challenging benchmarks that are inspired from seminal paper ( Bengio et al. , 1993 ) , where the distinctive information for classification of long sequences is only located in the first positions , while the rest contains uniformly distributed noisy information . We get very promising results on these benchmarks when comparing KBRN with state of the art recurrent architectures based on LSTM cells . 2 REPRESENTATION AND LEARNING . The feedforward architecture that we consider is based on a directed graph D ∼ ( V , A ) , where V is the set of ordered vertices and A is the set of the oriented arcs . Given i , j ∈ V there is connection from i to j iff i ≺ j . Instead of assuming a uniform activation function for each vertex of D , a specific function f is attached to each vertex . We denote with I the set of input neurons , with O the set of the output neurons and with H = V \ ( I ∪ O ) the set of hidden neurons ; the cardinality of these sets will be denoted as |I| , |O| , |H| and |V | ≡ n. Without loss of generality we will also assume that : I = { 1 , 2 , . . . , |I| } , H = { |I|+ 1 , |I|+ 2 , . . . , |I|+ |H| } and O = { |I|+ |H|+ 1 , |I|+ |H|+ 2 , . . . |I|+ |H|+ |O| } . The learning process is based on the training set TN = { ( eκ , yκ ) ∈ R|I| × R|O| | κ = 1 , . . . N } . Given an input vector z = ( z1 , z2 , . . . z|I| ) , the output associated with the vertices of the graph is computed as follows1 : xi ( z ) = zi ( i ∈ I ) + fi ( ai ) ( i /∈ I ) , ( 1 ) with ai = ∑ j∈pa ( i ) wijxj + bi , where pa ( i ) are the parents of neuron i , and fi : ΩΛ → R are one dimensional real functions ; ΩΛ : = [ −Λ , Λ ] , with Λ chosen big enough , so that Eq . ( 1 ) is always well defined . Now let f = ( f1 , f2 , . . . , fn ) and define the output function of the network F ( · , w , b ; f ) : R|I| → R|O| by Fi ( z , w , b ; f ) : = xi+|I|+|H| ( z ) , i = 1 , . . . , |O| . The learning problem can then be formulated as a double optimization problem defined on both the weights w , b and on the activation functions fi . It is worth mentioning that while the optimization on the weights of the graph reflects all important issues connected with the powerful representational 1We use Iverson ’ s notation : Given a statement A , we set ( A ) to 1 if A is true and to 0 if A is false properties of deep nets , the optimal discovery of the activation functions are somewhat related to the framework of kernel machines . Such an optimization is defined with respect to the following objective function : E ( f ; w , b ) : = 1 2 n∑ i=1 ( Pfi , Pfi ) + N∑ κ=1 V ( eκ , yκ , F ( eκ , w , b ; f ) ) , ( 2 ) which accumulates the empirical risk and a regularization term ( Smola et al. , 1998 ) . Here , we indicate with ( · , · ) the standard inner product of L2 ( ΩΛ ) , with P a differential operator of degree p , while V is a suitable loss function . Clearly , one can optimize E by independently checking the stationarity with respect to the weights associated with the neural connections and the stationarity with respect to the activation functions . Now we show that the stationarity condition of E with respect to the functional variables f ( chosen in a functional space Xp that depends on the order of differential operator P ) yields a solution that is very related to classic case of kernel machines that is addressed in ( Smola et al. , 1998 ) . If we consider a variation vi ∈ C∞c ( ΩΛ ) with vanishing derivatives on the boundary 2 of ΩΛ up to order p− 1 and define ϕi ( t ) : = E ( f1 , . . . , fi + tvi , . . . , fn ; w , b ) . The first variation of the functional E along vi is therefore ϕ′i ( 0 ) . When using arguments already discussed in related papers ( Poggio & Girosi , 1990 ; Girosi et al. , 1995 ; Smola et al. , 1998 ) we can easily see that ϕ′i ( 0 ) = ∫ ΩΛ ( Lfi ( a ) + N∑ κ=1 ακi δaκi ( a ) ) vi ( a ) da , where ακi = ∇FV · ∂fiF and L = P ∗P , P ∗ being the adjoint operator of P . We notice in passing that the functional dependence of E on f is quite involved , since it depends on the compositions of linear combinations of the functions fi ( see Figure 1– ( a ) ) . Hence , the given expression of the coefficients ακi is rather a formal equation that , however , dictates the structure of the solution . The stationarity conditions ϕ′i ( 0 ) = 0 reduce to the following Euler-Lagrange ( E-L ) equations Lfi ( a ) + N∑ κ=1 ακi δaκi ( a ) = 0 , i = 1 . . . n , ( 3 ) where aκi is the value of the activation function on the κ-th example of the training set . Let g be the Green function of the operator L , and let k BE the solution of Lk = 0 . Then , we can promptly see that fi ( a ) = k ( a ) − N∑ κ=1 ακi g ( a− aκi ) ( 4 ) 2We are assuming here that the values of the functions in Xp at the boundaries together with the derivatives up to order p− 1 are fixed . is the general form of the solution of Eq . ( 3 ) . Whenever L has null kernel , then this solution is reduced to an expansion of the Green function over the points of the training set . For example , this happens in the case of the pseudo differential operator that originates the Gaussian as the Green function . If we choose P = d/dx , then L = −d2/dx2 . Interestingly , the Green function of the second derivative is the rectifier g ( x ) = − 12 ( |x|+ x ) and , moreover , we have k ( x ) = mx+ q . In this case fi ( a ) = θia+ νi + 1 2 N∑ κ=1 ακi |a− aκi | , ( 5 ) where θi = m+ 12 ∑N κ=1 α κ i , while νi = q− 12 ∑N κ=1 α κ i a κ i . Because of the representation structure expressed by Eq . ( 4 ) , the objective function of the original optimization problem collapses to a standard finite-dimensional optimization on3 Ê ( α , w , b ) : = E ( k ( a ) − ∑ κ ακg ( a− aκ ) ; w , b ) = R ( α ) + N∑ κ=1 V ( eκ , yκ , F̂ ( eκ , w , b ; α ) ) ; here R ( α ) is the regularization term and F̂ ( eκ , w , b ; α ) : = F ( eκ , w , b ; k ( a ) − ∑ κ α κ i g ( a − aκi ) ) . This collapse of dimensionality is the same which leads to the dramatic simplification that gives rise to the theory of kernel machines . Basically , in all cases in which the Green function can be interpreted as a kernel , this analysis suggests the neural architecture depicted in Figure 1 , where we can see the integration of graphical structures , typical of deep nets , with the representation in the dual space typical of kernel methods . We can promptly see that the idea behind kernel-based deep networks can be extended to cyclic graphs , that is to recurrent neural networks . In that case , the analogous of Eq . ( 1 ) is : ht+1i = fi ( a t+1 i ) ; a t+1 i = bi + ∑ j∈pat→t+1 ( i ) wijh t j + ∑ j∈pat+1 ( i ) uijx t+1 j . Here we denote with xti the input at step t and with h t i the state of the network . The set pat→t+1 ( i ) contains the vertices j that are parents of neuron i ; the corresponding arcs ( j , i ) are associated with a delay , while pat ( i ) vertices j with non-delayed arcs ( j , i ) . The extension of learning in KBDNN to the case of recurrent nets is a straightforward consequence of classic Backpropagation Through Time .

Summary: The paper investigates the problem of designing the activation functions of neural networks with focus on recurrent architectures. The authors frame such problem as learning the activation functions in the space of square integrable functions by adding a regularization term penalizing the differential properties of candidate functions. In particular, the authors observe that this strategy is related to some well-established approaches to select activation functions such as ReLUs.