Training invariances and the low-rank phenomenon: beyond linear networks

1 INTRODUCTION . Recently , great progress has been made in understanding the trajectory of gradient flow ( GF ) ( Ji & Telgarsky , 2019 ; 2020 ; Lyu & Li , 2020 ) , gradient descent ( GD ) ( Ji & Telgarsky , 2019 ; Arora et al. , 2018 ) and stochastic gradient descent ( SGD ) ( Neyshabur et al. , 2015 ; 2017 ) in the training of neural networks . While good theory has been developed for deep linear networks ( Zhou & Liang , 2018 ; Arora et al. , 2018 ) , practical architectures such as ReLU fully-connected networks or ResNets are highly non-linear . This causes the underlying optimization problem ( usually empirical risk minimization ) to be highly non-smooth ( e.g . for ReLUs ) and non-convex , necessitating special tools such as the Clarke subdifferential ( Clarke , 1983 ) . One of the many exciting results of this body of literature is that gradient-based algorithms exhibit some form of implicit regularization : the optimization algorithm prefers some stationary points to others . In particular , a wide range of implicit biases has been shown in practice ( Huh et al. , 2021 ) and proven for deep linear networks ( Arora et al. , 2018 ; Ji & Telgarsky , 2019 ) , convolutional neural networks ( Gunasekar et al. , 2018 ) and homogeneous networks ( Ji & Telgarsky , 2020 ) . One well known result for linearly separable data is that in various practical settings , linear networks converge to the solution of the hard SVM problem , i.e. , a max-margin classifier ( Ji & Telgarsky , 2019 ) , while a relaxed version is true for CNNs ( Gunasekar et al. , 2018 ) . This holds even when the margin does not explicitly appear in the optimization objective - hence the name implicit regularization . Another strong form of implicit regularization relates to the structure of weight matrices of fully connected networks . In particular , Ji & Telgarsky ( 2019 ) prove that for deep linear networks for binary classification , weight matrices tend to rank-1 matrices in Frobenius norm as a result of GF/GD training , and that adjacent layers ’ singular vectors align . The max-margin phenomenon follows as a result . In practice , Huh et al . ( 2021 ) empirically document the low rank bias across different non-linear architectures . However , their results include ReLU fully-connected networks , CNNs and ReNet , which are not all covered by the existing theory . Beyond linear fully connected networks , Du et al . ( 2018 ) show vertex-wise invariances for fully-connected ReLU networks and invariances in Frobenius norm differences between layers for CNNs . Yet , despite the evidence in ( Huh et al. , 2021 ) , it has been an open theoretical question how more detailed structural relations between layers , which , e.g. , imply the low-rank result , generalize to other , structured or local nonlinear and possibly non-homogeneous architectures , and how to even characterize these . Hence , in this work , we take steps to addressing a wider set of architectures and invariances . First , we show a class of vertex and edge-wise quantities that remain invariant during gradient flow training . Applying these invariances to architectures containing fully connected , convolutional and residual blocks , arranged appropriately , we obtain invariances of the singular values in adjacent ( weight ) matrices . Second , we prove how the controlled singular values imply low-rank results for fully connected ReLU networks . Third , we obtain low-rank results for arbitrary non-homogeneous networks whose last few layers contain fully-connected and ResNet blocks . To the best of our knowledge , this is the first time a low-rank phenomenon is proven rigorously for these architectures . Our theoretical results offer explanations for empirical observations on more general architectures , and apply to the experiments in ( Huh et al. , 2021 ) for ResNet and CNNs . They also include the squared loss used there , in addition to the exponential or logistic loss used in most theoretical low-rank results . Moreover , our Theorem 2 gives an explanation for the “ reduced alignment ” phenomenon observed by Ji & Telgarsky ( 2019 ) , where in experiments on AlexNet over CIFAR-10 the ratio ‖W‖2/‖W‖F converges to a value strictly less than 1 for some fully-connected layer W towards the end of the network . One challenge in the analysis is the non-smoothness of the networks and ensuring an “ operational ” chain rule . To cope with this setting , we use a specific decomposition of an arbitrary ReLU architecture into a multilinear function and a ReLU network with +1/-1 weights . This reduction holds in the stable sign regime , a certain convergence setting of the parameters . This regime is different from stable activations , and is implied , e.g. , by directional convergence of the parameters to a vector with non-zero entries ( Lemma 2 ) . This construction may be of independent interest . In short , we make the following contributions to analyzing implicit biases of general architectures : • We show vertex and edge-wise weight invariances during training with gradient flow . Using these invariances , we prove that for architectures containing fully-connected layers , convolutional layers and residual blocks , when appropriately organized into matrices , adjacent matrices have the same singular value , up to initialization ( Theorem 1 ) . • We prove that in the stable sign regime , weight matrices in a fully-connected ReLU-activated network tend to rank-1 matrices ( Corollary 1 ) . We also show a low-rank bias for arbitrary nonhomogeneous feedforward networks whose last few layers are a composition of fully-connected and ResNet blocks ( Theorem 2 ) . In particular , if the Frobenius norms of these layers diverge , then the ratio between their operator norm and Frobenius norm is bounded non-trivially by an expression C fully specified by the architecture . To the best of our knowledge , this is the first time this type of bias is shown for nonlinear , nonhomogeneous networks . • We prove our results via a decomposition that reduces arbitrarily structured deep feedforward networks with positively-homogeneous activation ( e.g. , ReLU or linear ) to a multilinear structure ( Lemma 2 ) . 1.1 RELATED WORKS . Decomposition of fully-connected neural networks into a multilinear part and a 0-1 part has been utilized by Choromanska et al . ( 2015 ) ; Kawaguchi ( 2016 ) but their formulation does not apply for trajectory studies . In Section 3 , we give a detailed comparison between their approach and ours . Our decomposition makes use of the construction of a tree net that has been used by Khim & Loh ( 2019 ) in analyzing generalization of fully-connected networks . We describe their approach in 2 and show how we extend their construction to arbitrary feedforward neural nets . This construction makes up the first part of the proof of our decomposition lemma . The paper also makes use of path enumeration of neural nets , which overlaps with the path-norm literature of Neyshabur et al . ( 2015 ; 2017 ) . The distinction is that we are studying classical gradient flow , as opposed to SGD ( Neyshabur et al. , 2015 ) or its variants ( Neyshabur et al. , 2017 ) . For linear networks , low-rank bias is proven for separable data and exponential-tailed loss in Ji & Telgarsky ( 2019 ) . Du et al . ( 2018 ) give certain vertex-wise invariances for fully-connected ReLU networks and Frobenius norm difference invariances for CNNs . Compared to their results , ours are slightly stronger since we prove that our invariances hold for almost every time t on the gradient flow trajectory , thus allowing for the use of a Fundamental Theorem of Calculus and downstream analysis . Moreover , the set of invariances we show is strictly larger . Radhakrishnan et al . ( 2020 ) show negative results when generalizing the low-rank bias from ( Ji & Telgarsky , 2019 ) to vectorvalued neural networks . In our work , we only consider scalar-valued neural networks performing binary classification . 2 PRELIMINARIES AND NOTATION . For an integer k ∈ N , we write the set [ k ] : = { 1 , 2 , . . . , k } . For vectors , we extend the sign function sgn : R → { −1 , 1 } coordinatewise as sgn : ( xi ) i∈ [ d ] 7→ ( sgn ( xi ) ) i∈ [ d ] . For some ( usually the canonical ) basis ( ei ) i∈ [ n ] in some vector space Rn , for all x ∈ Rn we use the notation [ x ] i = 〈x , ei〉 to denote the i-th coordinate of x. Clarke subdifferential , definability and nonsmooth analysis . The analysis of non-smooth functions is central to our results . For a locally Lipschitz function f : D → R with open domain D , there exists a set Dc ⊆ D of full Lebesgue measure on which the derivative ∇f exists everywhere by Rademacher ’ s theorem . As a result , calculus can usually be done over the Clarke subdifferential defined as the set ∂f ( x ) : = CONV { lim i→∞ ∇f ( xi ) | xi ∈ Dc , xi → x } where CONV denotes the convex hull . The Clarke subdifferential generalizes both the smooth derivative when f ∈ C1 ( continuously differentiable ) and the convex subdifferential when f is convex . However , it only admits a chain rule with an inclusion and not equality , which is though necessary for backpropagation in deep learning . If one is willing to work with a slightly restricted set of functions - formally known as tame , Whitney stratifiable , path differentiable or definable in an o-minimal structure ( Davis et al. , 2020 ; Ji & Telgarsky , 2020 ; Bolte & Pauwels , 2020 ) , the Clarke subdifferential admits an operational chain rule . All architectures considered in this paper are definable ( Ji & Telgarsky , 2020 ) . Here , we use the form from Ji & Telgarsky ( 2020 ) . Lemma 1 ( Lemma B.9 in Ji & Telgarsky ( 2020 ) ) . If f : D open → R is locally Lipschitz and defineable , then for any interval I , arc z : I → D , a.e . t ∈ I , df ( z ( t ) ) dt = 〈 y , dztdt 〉 , ∀y ∈ ∂f ( z ( t ) ) . Neural networks . Consider a neural network ν : Rd → R. The computation graph of ν is a weighted directed graph G = ( V , E , w ) with weight function w : E → R. For each neuron v ∈ V , let INv : = { u ∈ V : uv ∈ E } and OUTv : = { w ∈ V : vw ∈ E } be the input and output neurons of v. Let { i1 , i2 , . . . , id } = : I ⊂ V and O : = { o } ⊂ V be the set of input and output neurons defined as IN ( i ) = ∅ = OUT ( o ) , ∀i ∈ I . Each neuron v ∈ V \I is equipped with a positively-1-homogeneous activation function σv ( such as the ReLU x 7→ max ( x , 0 ) , leaky ReLU x 7→ max ( x , αx ) for some small positive α , or the linear activation ) . To avoid unnecessary brackets , we write we : = w ( e ) for some e ∈ E. We will also abuse notation and write w ∈ RE where E is the set of learnable weights as the vector of learnable parameters . Let P be a path in G , i.e. , a set of edges in E that forms a path . We will abuse notation and write v ∈ P for some v ∈ V if there exists u ∈ V such that uv ∈ P or vu ∈ P . Let ρ be the number of distinct paths from any i ∈ I to o ∈ O . Let P : = { p1 , . . . , pρ } be the enumeration of these paths . For a path p ∈ P and an input x to the neural net , denote xp the coordinate of x used in p. Given a binary classification dataset { ( xi , yi ) } i∈ [ n ] with xi ∈ R d , ‖xi‖ ≤ 1 and yi ∈ { −1 , 1 } , we minimize the empirical risk R ( w ) = 1n ∑n i=1 ` ( yiν ( xi ) ) = 1 n ∑n i=1 ` ( ν ( yixi ) ) with loss ` : R → R , using gradient flow dw ( t ) dt ∈ −∂R ( w ( t ) ) . As we detail the architectures used in this paper , we recall that the activation of each neuron is still positively-homogeneous . The networks considered here are assumed to be bias-free unless stated otherwise . Definition 1 ( Feedforward networks ) . A neural net ν with graph G is a feedforward network if G is a directed acyclic graph ( DAG ) . Definition 2 ( Fully-connected networks ) . A feedforward network ν with graph G is a fullyconnected network if there exists partition of V into V = ( I ≡ V1 ) t V2 t . . . t ( VL+1 ≡ O ) such that for all u , v ∈ V , uv ∈ E iff there exists i ∈ [ L ] such that u ∈ Vi and v ∈ Vi+1 . Definition 3 ( Tree networks ) . A feedforward network ν with graph G is a tree network if the underlying undirected graph of G is a tree ( undirected acyclic graph ) . Examples of feedforwards networks include ResNet ( He et al. , 2016 ) , DenseNet ( Huang et al. , 2017 ) , CNNs ( Fukushima , 1980 ; LeCun et al. , 2015 ) and other fully-connected ReLU architectures . For a fully-connected network ν with layer partition V = : V1 t . . . t VL+1 where L is the number of ( hidden ) layers , let ni : = |Vi| be the number of neurons in the i-th layer and enumerate Vi = { vi , j } j∈ [ ni ] . Weights in this architecture can be organized into matricesW [ 1 ] , W [ 2 ] , . . . , W [ L ] where Rni+1×ni 3W [ i ] = ( ( wvi , jvi+1 , k ) ) j∈ [ ni ] , k∈ [ ni+1 ] , for all i ∈ [ L ] . Tree networks . Most practical architectures are not tree networks , but they have been used to prove generalization bounds for adversarial risks . In particular , for fullyconnected neural networks f whose activations are monotonically increasing and 1- Lipschitz , Khim & Loh ( 2019 ) defined the tree transform as the tree network Tf ( x ; w ) =∑nL pL=1 W [ L ] 1 , pL σ ( . . . ∑n2 p2=1 W [ 2 ] p3 , p2σ ( wp2 .. pL + ∑n1 p1=1 W [ 1 ] p2 , p1xp1 ) ) for vectors w with∏L j=2 nj entries , indexed by an L-tuple ( p2 , . . . , pL ) . We extend this idea in the next section .

Summary: The paper shows under certain assumptions, there exist invariants in DNNs with ReLU (or leaky ReLU and identity) activations - similar to the invariants observed in linear networks. To show this invariance a decomposition of the DNN as the composition of a multilinear function and a network with {+1,-1} weights. Using these invariants they show that for linearly separable data, the absolute value of the weight matrices converge to rank one matrices (assuming directional convergence).