Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors study deep feedforward neural network training with $\ell^2$
regularization on the weight matrices. They reformulate this cost (via
an equivalent optimization problem) in two different ways, in order to get
insight into the features that are learned at optimality. The first
reformulation expresses the network weights in terms of the preactivations, via
an orthogonal projection trick; the authors interpret the resulting problem in
terms of the local interactions between neighboring preactivations across
layers of the network. The second reformulation builds off the first: it
replaces the preactivations with the corresponding covariance matrices, and the
authors interpret the resulting cost in terms of the rank constraints imposed
by this structure when the layer widths are sufficiently large relative to the
number of samples, which implies some sparsity of the optimal features. Small
experiments are presented to illustrate the interpretations.


strengths_and_weaknesses: ## Strengths

- The reformulation in terms of preactivations is insightful, and gives an
  interesting framework for understanding global minimizers of the deep network
  training cost.
- The authors focus on tools that apply to deep networks, which is important --
  many works in the theoretical literature are restricted to shallow networks.

## Weaknesses

- It is slightly unclear what to conclude from the authors' reformulations and
  analysis. The authors discuss the fact that the reformulated optimization
  problems are not computationally useful, given the undesirable scaling with
  the number of data samples relative to training the network weights (section
  3.3); for the second reformulation, there is no correspondence between the
  objective landscapes of the original and reformulated problem (and see a
  question below about the first reformulation). It could be helpful to see
  some experiments on toy data where the scaling is less of an issue to see
  whether or not the reformulated landscapes empirically have some similarities
  to the original landscape, and/or provide useful insights.
- The analysis is focused on understanding learned features in the network, but
  it does not involve any specific structural assumptions on $Y$ and $X$. It
  would be interesting to see if such assumptions can be made, and accordingly
  the conclusions of the second reformulation in Section 4 could be
  strengthened.


## Minor Issues

- line 78: is the 'plus one' in the weight matrix dimensions definition
  misplaced? (the weights multiply the 'superscript sigma' preactivations,
  which are the lifted ones)
- line 112: orthogonal complement
- Proposition 1 proof: there is a typo in the definition of the $\Psi$ map (the
  typo is repeated in other contexts in the proof).  The typo extends to the
  statement of the proposition in the main paper, unless I am missing
  something.  Also in the proof, there are missing $r$ superscripts on some
  losses. 
- line 137 (item 2): how are these fractional powers to be interpreted for the
  generally non-square features/input data?
- Proposition 3: the usage of the $\mathrm{cone}$ operator doesn't match the
  way the notation is used above (are the braces and parentheses accidentally
  placed in the wrong order at eqn below line 188 and similar?)

questions: I am not able to understand the claim in line 61 of the appendix that $\Psi$ is
norm-preserving. Can you justify this? It seems like if some features are
poorly conditioned, the norm of the image of $\Psi$ will blow up. I am not sure
about the correspondence between minimizers asserted in Proposition 1 without
this justification.

I was a bit confused about Figure 1 -- are these plots for a network trained on
the reformulated loss (i.e. not really a network -- just learning features), or
a standard optimization over the weights $W$? Are these figures showing the
different attractive/repulsive terms in the trained network, or the initial
network?

Quadratic overparameterization may be more than is necessary for many tasks,
e.g. as discussed in [1], or when data have low-dimensional structure [2].
Can the requirements be relaxed (in principle?) in the latter case? What can
be said theoretically in this framework about networks that are less
overparameterized than quadratic?

[1] https://arxiv.org/abs/2205.10217
[2] https://openreview.net/forum?id=O-6Pm_d_Q-

limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper provided two reformulations of the L2-regularized DNN using the framework of representation cost. The first reformulation characterizes feature learning process in terms of attraction and repulsion, and further established an interesting connection between training a L2-penalized DNN with a partial convex optimization over a translated convex cone. The second reformulation utilizes the covariance learning and provides a mechanism for learning the sparsity effect of the L2-regularization in training of DNN with homogeneous non-linearities.
strengths_and_weaknesses: Strengths:

1. The two reformulations are interesting and lead to insightful understanding on the effect of L2 regularization in DNN trainings.

2. The second reformulation establishes an interesting sparsity result for homogeneous DNN (Proposition 7), and the tightness of is further analyzed.

Weakness:

While I enjoy learning these insights and new understandings of the effect of L2 regularization in DNN, I would definitely find it helpful to have more discussions on their implications. Specifically,

1. the first reformulation leads to the attraction-repulsion effects, which could potentially lead to better algorithmic understanding / design. As mentioned in point 3 in the bottom of pp.4, a block (of three layers) cordinate descent type of algorithm could be developed to minimize the multiplicative distances. But how would one deal with the interpolation requirement in the representation costs? While it might be natural to consider a projected gradient descent, going in this direction seems to be a waste of the established decomposition.

2. Proposition 5 establishes the equivalence between training a L2-regularized DNN and a partially convex problem over a translated convex cone. What actual implication could we obtain? What actual benefit can the partial convexity offer?

questions: For the second reformulation, it is great to have the understanding of no later than when will we observe the plateau. Could we have any understanding of how early the plateau could occur? I think this might be interesting, especially since an early occurrence of the plateau is observed in many datasets.

limitations: Please see the comments in the previous 2 sections.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the minima of the loss of $L_2$ regularized fully-connected DNNs. They show that the loss in terms of the parameters can be reformulated into a loss in terms of the layerwise activations of the training set, which has an attraction/repulsion problem formulation. For positively homogeneous nonlinearities, they show that the loss can be further reformulated in terms of the covariances of the hidden representations, which take the form of a partially convex optimization over a convex cone. 
strengths_and_weaknesses: Strengths:
- The two reformulations of loss of $L_2$ regularized DNNs are very novel, which may be helpful for our understanding of DNNs.
- The second reformulation has some implications for the sparsity of the local minimum of homogeneous DNNs.

Weaknesses:
- All the theoretical results are limited to the minima of the loss of $L_2$ regularized NN, i.e. based on the assumption that the minima can be achieved. They also assume the minima can perfectly interpolate the dataset.
questions: Some typos:
- Line 124 $\mathcal{L}^r_{\lambda}$, should be $\mathcal{L}_{\lambda}$?
- Line 130, $\mathcal{L}_{\lambda}(Z_1, \dots, Z_L)$ should be $\mathcal{L}^r_{\lambda}(Z_1, \dots, Z_L)$?
limitations: Yes
review_title: null

Meta-review: metareview: This paper studies l_2 regularization on the norms of fully connected networks and discusses how that influences feature learning. It gives two reformulations of the loss (from parameters to the activations) which give intuition on attraction/repulsion effects and an "effective" number of neurons (that the minimum can always be achieved by a network of size N(N+1)). Overall the reviewers find the reformulations novel and interesting, while there are some concerns most are addressed in the author response.
confidence: Less certain
Decision: Accept