Decoupling Gating from Linearity

1 INTRODUCTION . An artificial neuron with the ReLU activation function is the function fw ( x ) : Rd → R such that fw ( x ) = max { x > w , 0 } = ( 1x > w≥0 ) · ( x > w ) . The latter formulation demonstrates that the parameter vector w has a dual role ; it acts both as a filter or a gate that decides if the neuron is active or not , and as linear weights that control the value of the neuron if it is active . We introduce an alternative neuron , called Gated Linear Unit or GaLU for short , which decouples between those roles . A 0− 1 GaLU neuron is a function gw , u ( x ) : Rd → R such that gw , u ( x ) = ( 1x > u≥0 ) · ( x > w ) . ( 1 ) GaLU neurons , and therefore GaLU networks , are at least as expressive as their ReLU counterparts , since fw = gw , w . On the other hand , GaLU networks appear problematic from an optimization perspective , because the parameter u can not be trained using gradient based optimization ( since ∇ugw , u ( x ) is always zero ) . In other words , training GaLU networks with gradient based algorithms is equivalent to initializing the vector u and keeping it constant thereafter . A more general definition of a GaLU network is given in section 2 . The main claim of the paper is that GaLU networks are on one hand as effective as ReLU networks on real world datasets ( section 3 ) while on the other hand they are easier to analyze and understand ( section 4 ) . 1.1 RELATED WORK . Many recent works attempt to understand deep learning by considering simpler models , that would allow theoretical analysis while preserving some of the properties of networks of practical utility . Our model is most closely related to two such proposals : linear networks and non-linear networks in which only the readout layer is being trained . Deep linear networks is a popular model for analysis that lead to impressive theoretical results ( e.g . Saxe et al . ( 2013 ) ; Kawaguchi ( 2016 ) ; Lu & Kawaguchi ( 2017 ) ) . Linear networks are useful in order to understand how well gradient-based optimization algorithms work on non-convex problems . The weakness of linear network is that their expressive power is very limited : linear networks can only express linear functions . It means that their usefulness to understand the practical success of standard networks is somewhat limited . Training only the readout layer is an alternative attempt to understand deep learning through simpler models , that also gave theoretical interesting results ( e.g . Saxe et al . ( 2011 ) ; Mairal et al . ( 2014 ) ; Daniely et al . ( 2016 ) ) . The idea is that all the layers but the last one implement a non-linear constant transformation , and the last layer is learning a linear function on top of this transformation . The weakness of this model is that there is a big practical difference between training all the layers of a network and training only the last one . Our model is similar in certain aspects to both of those models , but it enjoys a much better practical utility than either one . See section 3 for an empirical comparison . 2 GALU NETWORKS . Recall the definition of a basic GaLU neuron given in equation 1 . We consider a more general GaLU neuron of the form gw , u , σ ( x ) = σ ( x > u ) · ( x > w ) for some non-linear scalar function σ : R → R. If σ is differentiable , we could train the vectors u with gradient based algorithms , but the focus of this paper is on untrained gates . That is , we assume that the vectors { u } are kept to their initial values throughout the optimization procedure and only the linear part of the GaLU neurons is being optimized . GaLU networks with a single hidden layer have the following property : for any given example , the values of the gates in the network remain constant . In networks with more than one hidden layer this not true . Consider a standard fully connected feed-forward network , let x ( 0 ) be the input to the network and let x ( 1 ) , x ( 2 ) , . . . be the inputs to intermediate layers of the network . The output of a GaLU neuron at layer i will be σ ( x ( i−1 ) > u ) · ( x ( i−1 ) > w ) . So while the filter parameter vector , u , is not optimized upon , the value of the gate , σ ( x ( i−1 ) > u ) , can change as x ( i−1 ) changes . This adds an additional complication to the dynamics of the optimization that we wish to avoid . An alternative way to define a GaLU neuron at layer i is σ ( x ( 0 ) > u ) · ( x ( i−1 ) > w ) . In that case , the value of the gate is determined by the original input , and only the linear part depends on the output of the previous layer of the network . We call such a neuron a GaLU0 neuron , and a GaLU0 network is a network where all the neurons are GaLU0 neurons . In GaLU0 networks the gate values remain constant along the training , producing simpler dynamics . 3 EMPIRICAL SUCCESS . In order to check the hypothesis that effectiveness of ReLU networks stems mostly from the ability to train the linear part of the neurons , and not the gate part , we tested both GaLU01 and ReLU networks on the standard MNIST ( LeCun & Cortes , 2010 ) and Fashion-MNIST ( Xiao et al. , 2017 ) datasets . For both , we used PCA to reduce the input dimension to 64 , and then trained a two hidden layers fully-conneted networks on them , with k hidden neurons at each hidden layer . Figure 1 summarizes the results , showing that GaLU0 and ReLU achieve similar results , both outperforming linear networks of the same size . Training only the readout layer of a ReLU network gave much poorer results ( which were omitted from the graphs for clarity ) . 4 THEORETICAL SIMPLICITY : SOLVING Rd → R1 PROBLEMS WITH ONE HIDDEN LAYER NETWORKS We now turn to some very basic theoretical analysis of GaLU networks with a single hidden layer . Our goal is to show that GaLU networks are simpler to analyze than standard networks . 1We used the logistic sigmoid function , as it gave better results on the test set than the binary gate function . Consider a GaLU network with a single hidden layer of k neurons : N ( x ) = ∑k j=1 αjgwj , uj ( x ) . A convenient property of a GaLU neuron is that it is linear in the weights wj , hence , αjgwj , uj ( x ) = gαjwj , uj ( x ) . It means that the network can be rewritten as N ( x ) = k∑ j=1 αjgwj , uj ( x ) = k∑ j=1 gαjwj , uj ( x ) = k∑ j=1 gw̃j , uj ( x ) with w̃j = αjwj . Because we want to optimize over the weights w1 , . . . , wk , α1 , . . . , αk , we might as well optimize over the reparameterization w̃1 , . . . , w̃k without losing expressive power . It means that in a GaLU network of this form , it is sufficient to train the first layer of the network , as the readout layer adds nothing to the expressiveness of the network . The previous term can be further simplified : N ( x ) = k∑ j=1 gwj , uj ( x ) = k∑ j=1 σ ( x > uj ) x > wj = [ σ ( x > u1 ) x > σ ( x > u2 ) x > . . . σ ( x > uk ) x > ] w1 w2 ... wk = Φu ( x ) > w where Φu ( x ) = σ ( x > u1 ) x σ ( x > u2 ) x ... σ ( x > uk ) x , w = w1 w2 ... wk , u = [ u1 u2 . . . uk ] . So it turns out that a GaLU network is nothing more than a random non-linear transformation Φu : Rd → Rkd and then a linear function . 4.1 EXPRESSIVITY . There are different notions for the expressivity of a model , and one of the simplest ones is the finitesample expressivity over a random sample . This notion fits well to our model , because we are not interested in the absolute expressivity of a GaLU network , but of the expressivity of a GaLU network with random filters . So the question is how well does a randomly-initialized network can fit a random sample . Note that given the constant filters , solving for the best weights is a convex problem . Hence , there is no “ expressivity – optimization gap ” in GaLU networks – every expressivity results is immediately also an optimization result . Let S = { ( x1 , y1 ) , ( x2 , y1 ) , . . . , ( xm , ym ) } be a random sample , such that x1 , . . . , xm ∼ N ( 0 , Id ) and y1 , . . . , ym ∼ N ( 0 , 1 ) , all of which are independent . Clearly , it is impossible to generalize from the sample to unseen examples ; the best possible test loss is 1 , and is achieved by the constant prediction 0 . However , it is an interesting problem because it allows us to measure the expressivity of GaLU networks , by showing how much overfit we can expect from the network for a non-adversarial sample . Equivalently , it tells us how well the network can perform memorization tasks , where the only solution is to memorize the entire sample . We train the network for the standard mean-squareerror regression loss . Because the network is simply linear function over a constant non-linear transformation , and because we use the MSE loss , there is a closed form solution to the optimization problem minw 1 m ∑m i=1 ( N ( xi ) − yi ) 2 which is X̄ = Φu ( x1 ) > Φu ( x2 ) > ... Φu ( xm ) > w∗ = X̄+ y1 y2 ... ym with X̄+ being a pseudo-inverse of X̄ . This gives us Theorem 1 Let x1 , . . . , xm ∈ Rd , u1 , . . . , uk ∈ Rd be arbitrary vectors . Define X̄ as above . Let y1 , . . . , ym ∼ N ( 0 , 1 ) be independent random normal variables . Define the expected squared loss on the training set , for weights w , as LS ( w ) . Then , E [ min w LS ( w ) ] = 1− rank ( X̄ ) m . Proof Every vector y = ( y1 , . . . , ym ) ∈ Rm can be decomposed to a sum y = a + b where a is in the span of the columns of X̄ and b is in the null space of X̄ . It follows that minw LS ( w ) = ‖b‖2/m . The claim follows because if y ∼ N ( 0 , Im ) then the expected value of ‖b‖2 is m− rank ( X̄ ) . It is always true that rank ( X̄ ) ≤ min { m , kd } . Empirical experimentation shows that if x1 , . . . , xm , u1 , . . . , uk ∼ N ( 0 , Id ) then with high probability rank ( X̄ ) = min { m , kd } .

Summary: The authors propose a modified ReLU, the GaLU, where the nonlinearity gating role is decoupled from the linear weights. Similar ideas have been previously proposed. For example Tsai et al: http://papers.nips.cc/paper/6516-tensor-switching-networks: "The TS network decouples a hidden unit’s decision to activate (as encoded by the activation weights) from the analysis performed on the input when the unit is active (as encoded by the analysis weights)" and Veness et al: Online learning with gated linear networks, https://arxiv.org/abs/1712.01897.