Representation Costs of Linear Neural Networks: Analysis and Design

For different parameterizations ( mappings from parameters to predictors ) , we study the regularization cost in predictor space induced by l2 regularization on the parameters ( weights ) . We focus on linear neural networks as parameterizations of linear predictors . We identify the representation cost of certain sparse linear ConvNets and residual networks . In order to get a better understanding of how the architecture and parameterization affect the representation cost , we also study the reverse problem , identifying which regularizers on linear predictors ( e.g. , lp quasi-norms , group quasi-norms , the k-support-norm , elastic net ) can be the representation cost induced by simple l2 regularization , and designing the parameterizations that do so . 1 Introduction . In a class of parameterized models , penalizing the l2 norm of the parameters induces regularization on function space which can be interpreted as a complexity measure on the class of learned functions . In this paper , we study how different parameterizations induce different complexity measures . We consider parameterized mappings f : X × Rp −→ Rm , from input x ∈ X and parameters w ∈ Rp to predictions f ( x ; w ) . We denote the predictor implemented with parameters w by F ( w ) : X −→ Rm defined as F ( w ) ( x ) : = f ( x ; w ) . Then image ( F ) is the set of functions from X to Rm which can be obtained from this class of parameterized models . We will use F to denote a set of functions from X to Rm . The l2 regularization on parameters , either explicitly or implicitly , is a common phenomenon . As an example , in deep learning , explicit l2 regularization on parameters ( a.k.a . weight decay ) improves generalization ( [ 20 , 37 ] ) . Implicit regularization of l2 norm of parameters appears when we use gradient descent ( GD ) to train the model ( [ 2 , 35 , 33 , 19 , 23 , 32 , 14 , 25 , 9 , 18 ] ) . In particular , GD on homogeneous neural networks with logistic loss implicitly regularizes l2 norm on weights [ 23 , 32 ] . The representation cost ( [ 15 , 31 , 27 ] ) of a function g in image ( F ) under the parametrization F is RF ( g ) = min { ∥w∥22 : F ( w ) = g } . ( 1 ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Consider learning a predictor F ( w ) with some loss function L ( · ) while controlling the l2 norm of parameters w by minimizing min w∈Rp L ( F ( w ) ) + λ∥w∥22 . ( 2 ) This is clearly equivalent to learning a function g in image ( F ) by controlling RF ( g ) defined in Eq . ( 1 ) : min g∈image ( F ) L ( g ) + λRF ( g ) . ( 3 ) In other words , the representation cost under the parameterization F , RF ( · ) , captures the regularization on function space image ( F ) induced by l2 regularization on parameter space . In this paper , we are interested in understanding how different parameterizations , regularize the function space differently . Since GD on homogeneous neural networks with logistic loss implicitly regularizes l2 norm on weights [ 23 , 32 ] , representation cost induced by l2 regularization in weight space captures the implicit regularization in homogeneous models , but not necessarily in non-homogeneous models . Thus , representation costs of predictors parameterized by homogeneous models are arguably more related to implicit regularization , while representation costs of predictors parameterized by both homogeneous and non-homogeneous models are related to explicit regularization . In this paper , we first develop results for homogeneous neural networks . Then we reduce the non-homogeneous neural network to the homogeneous ones by arguing that the asymptotic behavior of its representation cost can be captured by the representation cost of some homogeneous subnetwork of the non-homogeneous network . One way to motivate the study of representation cost is by considering the popularity of the overparameterized models , in which the number of parameters is greater than the number of samples . Surprisingly , it has been observed that in the overparameterized regime , interpolative predictor generalizes well [ 7 , 39 , 16 , 3 ] . One way to explain this is that although there are many predictors which perfectly fit the training data , gradient based algorithms choose the one with the smallest representation cost ( [ 15 , 23 ] ) . In these cases , representation cost operates as a regularization in the function space which enables good generalization . Thus , understanding representation cost helps us understand the generalization of the model . In particular , representation cost of predictors induces an ordering on the space of predictors . Since representation cost is determined by the specific parametrization , each parameterization induces an ordering on the function space . This can be interpreted as an induced complexity measure of the predictor space , where penalizing the cost is the same as minimizing the complexity . Definition 1.1 ( Induced complexity measure ) . Let F be a set of functions from X to Rm and F∗ be the set of all functions from F to R. We define the equivalence relation in F∗ such that h1 and h2 ∈ F∗ ( i.e. , h1 , h2 : F → R ) are equivalent ( h1 ∼= h2 ) if there exists a strictly increasing function ψ : R −→ R h1 = ψ ◦ h2 . Let F∗/ ∼= be the set of equivalence classes of F∗ under ∼= . 1 Given a parameterization F : Rp → F , let F be image ( F ) . In this case , for each value of parameter w ∈ Rp , F ( w ) ∈ F is a mapping from X to Rm . Let the representation cost RF under parameterization F be as in Eq . ( 1 ) . We say that an equivalence class h̄ ∈ F∗/ ∼= is the induced complexity measure of the parameterization F if for any representative ( i.e. , element ) h in class h̄ , RF is a strictly increasing function of h. We study the dependence of induced complexity measures on parameterizations from two perspectives : First , given a parameterization F , we analyze its representation cost . Second , given a regularizer on function space , we study when and how it can be the induced complexity measure of some parametrizations . In this paper , we start answering these questions by focusing on linear predictors parameterized by linear neural networks . Note that for linear networks with single outputs , the function space being parameterized does not depend on the architecture . Thus , the change in architecture only changes the induced complexity measure . This makes it appealing for highlighting and understanding what the effect of changing the architecture is in changing the induced complexity measure . In the first part of the paper ( Section 3 ) , we identify the representation costs of various architectures . Specifically , we look into fully connected networks and convolutional networks with multiple outputs . In addition , we show how the representation cost of convolutional networks with restricted filter 1In other words , F∗/ ∼= is a partition of F∗ into classes with the following property : given a class h̄ ∈ F∗/ ∼= , any two elements h1 , h2 ∈ h̄ are equivalent ( i.e . h1 = ψ ◦ h2 ) . width changes as their filter width changes . Then , we show that the representation cost of residual networks interpolates between the representation costs of two of their component networks . Finally , we give two characterizations of the representation costs of depth-two neural networks . The results of this part are summarized in Table 1 . In the second part ( Section 4 ) , given a few regularizers , we study when and how they can be the induced complexity measure of some architectures . Specifically , we show that lp quasi-norm can be the induced complexity measure induced by l2 regularization on some linear neural networks if and only if 2/p is an integer . Moreover , when 2/p is an integer , we characterize all the architectures whose induced complexity measures are lp quasi-norms . In addition , we design architectures whose induced complexity measures are k-support-norm and lp , q group quasi-norms . Then , we show that elastic nets and lp , q quasi-norm with overlapping groups can not be the induced complexity measure of any linear neural network . On the contrary , we show that there exist homogeneous parametrizations whose induced induced complexity measures are elastic nets and lp , q quasi-norm with overlapping groups . The results of this part are summarized in Table 2 . Finally , in the conclusion , we discuss some interesting future directions . Further related works : Some previous work focuses on the expressive power image ( F ) of the model [ 22 , 29 , 38 , 21 ] . However , as discussed in [ 26 , 24 ] , some other capacity control , different from network size , plays a role in deep learning . This motivates the study of representation cost and its relation to parametrizations . Representation cost has been studied before under various models . [ 15 ] showed that the representation cost of a linear convolutional neural network of depth d is strictly increasing in the Schatten-2/d quasi-norm on the Fourier domain , whereas the representation cost of a linear fully connected network of depth d is strictly increasing in the l2 norm . [ 31 ] and [ 27 ] studied depth-two fully connected network with infinite width and ReLU activation . They show that the representation cost of any continuous function depends on the Laplacian of that function . Parallel to our work , [ 18 ] studies the representation cost of linear convolutional neural network with restricted filter width ( a.k.a . kernel size ) and multiple channels using anlaytical tools from semidefinite programming . In spite of different approaches between our work and [ 18 ] the results on CNN with restricted filter width are similar in two papers . Another line of work studies the relationship between neural networks with l2 regularization on weights and convex optimization problems [ 28 , 12 , 11 , 30 ] . In [ 28 , 12 , 11 , 30 ] , the authors showed that training a neural network with explicit l2 regularization on weights is equivalent to a convex regularized optimization problem in some higher dimensional space . In contrast , motivated by the literature on implicit regularization of gradient descent [ 2 , 35 , 33 , 19 , 23 , 32 , 14 , 25 , 9 , 18 ] , we looked into the induced regularization of weight decay on function space . Some of the results in [ 28 , 12 , 11 , 30 ] are similar to the results in our work . For instance , the results on linear convolutional neural network in [ 28 , 12 , 11 , 30 ] suggest that explicit l2 regularization on weight space is related to l1 regularization on the Fourier transform of the predictor . This result was also discovered in [ 15 ] and is generalized to multiple output and restricted filter width ( a.k.a . kernel size ) case in our work and in [ 18 ] . On the other hand , we considered other architectures beyond fully connected and convolutional neural networks . For instance , we studied architectures that induce k-support norms and architectures that induce lp , q group quasi-norms , which are not included in [ 28 , 12 , 11 , 30 ] . As another related work , [ 36 ] studied the equivalence between l2 regularization on weights and some sparsity-inducing regularization on the function space for various architectures . They considered the architecture which induces l2/d,2 group quasi-norms on the function space , for any d ∈ N. We also studied a similar question in section 4.1.2 . However , we found architectures that induce lp , q group quasi-norms for both the case p > q and the case p < q . In addition , we showed that in the case p < q , lp , q quasi-norm can be induced by some linear neural network as induced complexity measure if and only if 2/p , 2/q ∈ N and designed architectures that do so .

Summary: The paper studies representation costs of linear networks and their induced complexity measures. First the representation costs of some standard architecture are given (fullly-connected nets, diagonal nets, convolutional nets, and residual nets), then for shallow networks a general formula for the representation cost (and its dual norm, given that the representation cost is convex) of any network is given. The second part of the paper is dedicated to the characterization of which complexity measures can appear from the representation cost of a neural network. Though this question is not answered in general, the authors study a number of related questions: a characterization of which $l_p$ can be represented by a neural network is given, then some example architectures are given to obtain group $l_{p,q}$ norms or $k$-support norms, then it is shown that $l_{p,q}$ group norm with overlapping groups and elastic nets cannot be recovered by a neural network representation cost (though it is possible for the representation cost of a general homogeneous parametrization).