Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper builds on recent insights in applying adaptive control theory, Gauss-Newton optimization, target propagation, and dynamical inversion methods to the issue of local and bio-plausible credit assignment in neural networks. The authors propose deep feedback control (DFC) as a principled approach to learning feedforward and feedback weights via local plasticity rules to approximate Gauss-Newton optimization of an output loss function for a neural network.
main_review: This is a masterful paper that lucidly combines several recent insights towards solving the issue of credit assignment via local plasticity rules in the brain and neuromorphic hardware. A number of theorems/propositions are proved along the way to showing the conditions and approximations needed. (I have not checked the proofs, but the theorems and the conditions appear plausible.) A bio-plausible three-compartment neural implementation is also presented. The method is tested on simple benchmarks and shows good performance compared to backpropagation. I expect this work to open new directions as an alternative to approximating backpropagation to obtain brain-like learning rules.
 
Minor:
lines 5 and 56: motives -> motifs.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a deep feedback control (DFC) method for deep networks, which is biological plausibility, and the control signal can be used for credit assignment. Experiments on MNIST and Fashion MNIST dataset demonstrate the effectiveness of DFC, and the performances is competitive to back-propagation based learning algorithms.
main_review: The proposed deep feedback control approach for network learning is interesting and conveys innovate ideas both theoretically and algorithmically. The DFC method can be implemented with local learning, which is biological plausibility. It is also interesting to ask what are the possible applications of DFC based networks? Can the proposed model help explain biological mechanisms in the brain?

=============================


Post-rebuttal: 

The authors have addressed most of my concerns. I have increased my score from 6 to 7.


limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce Deep Feedback Control, an algorithm for training deep neural networks where credit assignment is done with the help of a proportional-integral-derivative (PID or rather PI in this case) controller. 

Theoretically, the authors show that a learning rule similar to what they propose (but not the same), under certain assumptions, gives the Gauss-Newton update for an L2 loss, and under milder assumptions gives the minimum norm updates of W. The authors show that the PI controller helps with stability and perform some experiments to show that the assumptions remain more or less valid during training.

### After rebuttal

Because of the increased quality of the work stemming from more theoretical arguments and more experiments, I have increased my score from 5 to 8.
main_review: ## Originality
The novelty of the method is in the use of the PI controller which helps stabilize the learning dynamics. Otherwise, proving theorems in the limit of small nudges, similarly showing a connection with GN is not new. 

## Quality
Overall this is a good paper. It is grounded in theory, guaranteeing performance in certain limits. The assumptions (but not the limits) of the theorems are more or less well explored. However it suffers from some weaknesses that if addressed can drastically improve the quality and the impact of the work. Here are a number of shortcomings in no particular order.

* The exact influence of the PI controller is not clear. It seems like Theorems 2 and 3 would hold also for purely proportional control (setting $\tau_u=0$), therefore the only effect of the integrative part is in stabilizing dynamics in stabilizing the dynamics such that $Q$ can be learned (sections 4 and 5). Given that without the integral part - that is just having just proportional feedback - brings the submission much closer to prior work, the authors should further clarify exactly where the integral part is essential, and what parts of their arguments would also hold for prior work. (The section in the appendix on related works is an attempt in this direction but still does not clarify where the differences are coming from. Is it in the architecture? In the controller? In the update rules?)

* There are some recent developments missing from the related work section and should be included (e.g  arxiv: 2011.15031).

* In lines (350-351) is is said that forward weight updates are not exactly GN etc. Similarly in Fig. 3 it is said that DFC-SSA is the ideal _damped_ MN update. First, the damped MN or GN update is nowhere defined, but I assume it refers to the fact that theorems 2,3 apply for update in Eq. (9) not the proposed update of Eq. (5). The effect of this seemingly adhoc dampening is mentioned in passing in lines 189 and expanded a little for MN optimization in the appendix, but never verified experimentally. This can be fixed by included MN and GN in Fig 3 and Tab 1.

* Similar to above, theorems 2,3 apply in the limit of $\alpha,\lambda\to0$. Unfortunately this aspect of the algorithm is never explored. In fact, the actual value of the $\lambda$ (is it actually small?) in experiments is not even found in the 45 page appendix! In biological networks, neither of these parameters can be too small. Because of noise and the limited dynamic range of neurons the nudging would have to be finite. Overall, the authors spend more time examining whether or not the conditions of their theorems are satisfied, than convincing the reader that the theorems are still approximately valid in realistic situations. If the authors spend a little bit of time exploring this, the potential impact of the paper would increase dramatically.

* Comparison to prior work (TP, DTP,  the hybrid GN-SGD of [20] etc) and non-damped GN and MN update. The authors do not answer the question of 'Does DFV perform better than the many (or even any) prior incarnations of TP?'

## Clarity

Overall the paper is clear. There are some small points.should be clarified). 

* As mentioned above, the differences with prior work (especially [20]) should be further clarified. For experts actively working on this problem, the submission might look like a big departure from prior work, but for general computational neuroscience audience, it looks like a small evolution of [20,21], with many arguments seemingly applicable to it as well. A clear section, mentioning 1. what exactly is different (architecture, learning rule etc) and 2. what is the effect of each departure from previous work (whether it improves stability or is required for relationship to GN etc.).

* It seems the influence of the controller is a little exaggerated. For example in line 142-143, it is mentioned that the controller dynamically inverts the output error. Judging by Fig. 1, the controller (the box at the top of the circuit) just integrates the error. The dynamical inversion is done in the body of the circuit (even if the controller was just spitting out the error itself).


## Significance

The paper is an evolution of the TP framework. With added clarity and extra experiments it can have a not-insignificant impact.
limitations_and_societal_impact: Yes
review_title: null

Meta-review: The paper introduces a new framework for biological plausible credit assignment algorithms in neural networks. The main idea consists in sending a top-down 'feedback' signal that drives the network to output the desired values, and update weights to make the feedback correction unneeded. They show a connection between the resulting algorithm and Gauss-Newton optimization. The work has connection to previous biological CA literature, in particular work on target propagation and feedback alignment.
Reviewers thought the contribution was insightful, and well written. Initial reviews brought about some concerns regarding novelty and what the precise contributions of some components were (whether they were important or needed at all, like the PI controller), which were extensively discussed in the rebuttal period. Please adjust the final version of the paper by taking these discussions into account.
A final improvement could include a more formal, clear theorem connecting DFC to GN: at the moment, it feels that one needs to connect theorem 1 and 2 to fully get the connection between the two. The relation between DFC and GN would ideally be made clear and rigorous in a single theorem, without having to connect dots.

Decision: Accept (Spotlight)