Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper theoretically analyzes the Feedback Alignment (FA, Eqn 2) algorithm in the optimization of deep linear networks.
* Under the assumption that the data matrices, FA matrices and initial weight matrices can be diagonalized simultaneously, the optimization can be divided into several one-dimensional problems, and the paper proves the convergence of both continuous and discrete dynamics.
* After the diagonalization, each eigenvalue corresponds to a one-dimensional dynamics. The authors construct two types of initialization to show that the convergence for the larger eigenvalues can be either faster (implicit regularization) or slower (implicit anti-regularization) than the smaller eigenvalues.
* Numerical comparison between FA and GD with 2- and 3-layer linear networks and random initialization. FA converges faster than GD and shares similar implicit regularization behavior.
main_review: This paper focuses on the theoretical analysis of FA. The authors find a setting (linear networks with diagonalizable matrices) to simplify the original problem to one-dimensional dynamics, then analyze the convergence rigorously. An interesting implicit anti-regularization is also observed under certain initialization. The numerical experiment, despite the simple setting, shows some advantages of FA compared to GD.

I am not familiar with the related work about FA, but my main concern is that the one-dimensional dynamics may oversimplify the original problem; and for $L > 2$, the convergence is proved only for a special initialization.
* Compared to GD, even for linear networks, the gap between (diagonalizable) one-dimensional scalar dynamics and high-dimensional matrix dynamics is non-trivial. May we get similar results as GD (given the FA dynamics seems to be simpler than GD)?
* I tend to think that the implicit anti-regularization phenomenon is more like an interesting math problem under the special initialization. Do we observe the behavior in numerical experiments with commonly-used initialization?

As discussed in the last section, the ultimate goal of the theoretical analysis of FA is to prove its potential advantage over GD. I am afraid the current one-dimensional analysis is still far from filling this gap.
summary_of_the_review: This paper proves some theoretical optimization results about FA under some special settings, but I think the one-dimensional setting may oversimplify the original problem, thus the novelty may not be significant enough.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors prove the linear convergence rate of the feedback alignment algorithm on fully connected linear networks. They also identify the implicit anti-regularization phenomenon for certain initialization of the algorithm and further propose initialization schemes that provide a form of implicit regularization and facilitate the learning process.
main_review: It provides convergence results on deep linear neural networks for feedback alignment for both discrete and continuous cases.
The authors show that the feedback alignment algorithm sequentially fits the solutions of a reduced-rank regression problem, and an initialization scheme is proposed to address the problem of anti-regularization.
They also show that the convergence results hold for networks that are not necessarily over-parametrized.

Feedback alignment method is named after one of its outstanding characteristics that the forward weights get aligned with the random backward weights. However, this paper does not seem to have even mentioned any attempts on such explorations.
The current results focus primarily on the linear networks, which leaves a gap between the theory and the empirical successes on networks with non-linear activation functions.
The definition of the loss function assumes a strong condition on infinite sample size.
One of the key steps of reducing the original system of ODEs into a scalar system relies on the assumption that the initialization of the weight matrix for the second layer is deterministic given the one for the first layer. However, people often initialize each layer randomly and independently before training, and the condition seems to be more artificial on deep networks.

The overall structure is clear and well-structured.
summary_of_the_review: The paper is nicely written, and the phenomenon of anti-regularization is interesting. However, the convergence results seem to assume strong conditions that only works for specific cases.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study feedback alignment (FA, Lillicrap '14), an alternative
algorithm to the standard backpropagation algorithm to train neural
networks. The key idea is to replace the weight matrices of the network with
fixed, random "feedback matrices" when computing the weight updates while
training the network. A precise understanding how such a learning rule can lead
to learning remains an open question, and has recently attracted some interest
(see related works)

Here, the authors study the dynamics of learning with FA in (deep) linear neural
networks. The dynamics of linear neural networks trained with backpropagation
has been studied extensively, thus providing an ideal test bed for this
study. The authors derive a set of continuous-time equations governing the
dynamics of FA for linear networks, discussing in detail their assumptions
etc. They use these equations to discover an interesting implicit bias of FA:
for certain initialisations, features of the data are learnt in the *inverse*
order of importance, quantified by the singular value associated with each mode
of the data. For other initialisations, features are learnt in the order of
importance, as expected. The sensitivity of this bias to the initialisation
doesn't appear in backprop. Finally, the authors also study linear auto-encoders
trained with FA, and find that not FA does not only recover (a rotation of) the
eigenvectors, but that it also speeds up training significantly. They also
study the discrete-step dynamics of FA.
main_review: This paper is a nice theoretical development, extending the theory of linear
neural networks to the case of DFA. The authors first study the shallow case
before extending their arguments to the deep case. Throughout, they are clear
about the assumptions they make (choice of initialisation, commutativity of the
weight matrices, etc.)

I found the results concerning the implicit bias interesting - it is surprising
that FA would exhibit this inverse learning order, and I am not aware of an
implicit bias that would depend on the initial conditions in such a
qualitatively different way. This observation should be interesting beyond FA.

The numerical experiments nicely illustrate the results, and show very good
agreement with the theory, even if some of the assumptions (e.g. on
initialisation) are relaxed in the experiments.

The paper is well-structured and clearly written.

Further comments / questions:

- In the analysis of the discrete update equations, what exactly is the
  motivation for the additional term $W_1^{t+1/2}$ in Eq. 25? Is this system of
  equations simply more convenient for the analysis? If so, how does it connect
  to the algorithm?

- In your conclusion, you state that you anticipate that FA will "remove" some
  stationary points from the landscape of non-linear networks. This connection
  didn't become clear to me, could you elaborate?
  
Some cosmetic comments:

- Could you clarify over which distribution the the average is taken in Prop. 1?
  Is it w.r.t. the finite dataset or over the distribution $\mathcal{D}$?
- I would increase the fontsize of some of the plots to match the captions, in
  particular Fig 1.

summary_of_the_review: Feedback alignment has attracted some interest in recent years after it was
shown that it can train a range of neural networks, and some groups have started
investigating the mechanism that allow the training to proceed with random
matrices. This work is thus timely and should be of interest to the ICLR
community. It provides several interesting contributions regarding (1) the
mechanism behind feedback alignment algorithms and (2) the implict bias of
different learning algorithms, through an analysis of linear neural networks, a
workhorse in neural network theory. I therefore recommend acceptance at the
conference.

PS: I am not sure I fully understood the questionnaire below regarding the
difference between "technical novelty..." and "empirical novelty". Since this a
theory paper, I only voted on the technical novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the Feedback Alignment (FA) algorithm on deep linear networks. In continuous time, FA algorithm is an alternative to Gradient Flow (GF) In particular, to compute the time derivative of the weights, FA algorithm starts with the gradient of the weights at each layer, and replace the part that depends on the weights of succeeding layers by a fixed matrix.

The FA algorithm on deep linear networks under spectral initialization is studied regarding both convergence and implicit bias, and the analysis are provided for both continuous-time and discrete-time FA algorithm. 
main_review: This paper is clearly written. I checked most of the proofs and they seem correct. Moreover, some results, e.g. “backward learning”, are novel and interesting. 

My major concern is the significance of the results since all of them are under spectral initialization. Specifically:

1. I believe not all linear regression problems admit spectral initialization. It is only possible when $\Sigma_{xx}$ and $\Sigma_{xy}$ are co-diagonalizable, i.e. the decomposition the author has should have both $\Lambda_{xx}$ and $\Lambda_{xy}$ diagonal. The introduction of spectral initialization in Section 2 is misleading to me because it is written in a way that suggests one can do spectral initialization for any linear regression problem. The author should clarify this. 

2. Spectral initialization alone is too restrictive to understand the training algorithm. In practice, such a spectral initialization scheme is rarely used to train a neural network (Note: an entirely different form of spectral initialization is used in matrix completion problem), and instead one uses mostly random initialization. In my opinion, spectral initialization allows us to somehow theoretically understand phenomena that happen during practical training. In that sense, analysis for spectral initialization alone is insufficient because we don’t know whether the phenomena observed in this paper, monotonicity of the product, backward learning, etc. also happen under non-spectral initialization.
The author discussed the relaxation of the spectral assumption at the end of page 4. but I find it unconvincing without empirical evidence.

3. Following the point above, I think the author may want to add more experiments under non-spectral initialization to illustrate how should we understand these phenomena studied for spectral initialization. For example, monotonicity of the product can only happen for scalar dynamics, is there any related notion in the non-spectral case? Can backward learning happen under non-spectral initialization? Remember that the forward learning in Saxes’14 is observed empirically when weights are randomly initialized with small scale (non-spectral).

Aside from the concern on significance, I think the scalar dynamics in this paper can be understood better using visual tools. For the scalar dynamics (6), one can plot: 1) all the equilibrium points lie on the hyperbola $\theta_1\theta_2=\lambda$; 2) the trajectory of the dynamics is constrained on the parabola $\theta_2=\frac{1}{2d}\theta_1^2+K$, where $K$ depends on initialization. This makes some statements in the paper much easier to understand. For example, the flow function in (13) has three real roots if the parabola, determined by $d,K$, intersects with the hyperbola at three points. I believe the non-monotonicity of the product can also be illustrated.

Reference:
Andrew M Saxe, James L Mcclelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In International Conference on Learning Representations, 2014.
summary_of_the_review: My initial rating is "weak reject" because:
Strength: 
1. clear writing
2. detailed analysis
3. interesting observations

Weakness:
1. Significance, need connections to initialization schemes used in practice
2. The results are hard to interpret in general non-spectral settings
3. little numerical example
4. presentation can be improved by visual tools
review_title: null

Meta-review: This paper provides a theoretical analysis for the Feedback Alignment (FA) algorithm, an alternative to backpropagation for training deep linear neural networks. The main drawback of the analysis is that it assumes that the initial weight matrices are diagonal, which makes the dynamics of the algorithm reduce to K independent one-dimensional dynamic. Most of the reviewers feel that this assumption is too strong. Note that in many existing papers on the implicit bias/regularization of gradient descent (GD) for optimizing deep linear networks, they do not assume the initial weight matrices are diagonal. The authors provide some additional proof in appendix B during the rebuttal to try to relax the assumption on the diagonal initialization, but it is not critical clear if the same results still hold. While this paper studies a very important problem, I suggest the authors take into account the reviewers’ comments and improve the presentation/results. In addition, a comparison with the implicit bias of GD would help better position this work, as one of the reviewers suggested.
Decision: Reject