Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes use of the least-control principle which minimizes the amount of optimal control needed at equilibrium as a local alternative to backpropagation in recurrent neural networks. The authors establish conditions that should be met at equilibria, relate what is optimized back to the standard optimization problem, and relate their framework to constrained energy minimization. Finally, the approach is validated to get similar performance to backpropagation on a supervised learning settings based on MNIST, CIFAR-10, and a natural image dataset as well as in meta-learning experiments on Omniglot. 
strengths_and_weaknesses: Strengths: 
- The authors provide a solid theoretical foundation for their approach in Theorems 1 and 2. 
- I really appreciate sections 2.3 and 2.4, which relate back to the implications for the original learning problem and alternative interpretations of the least-control principle. 
- It is nice to see application in both supervised learning and meta-learning settings, but it would be preferable to consider a larger breadth of datasets and architectures to provide a stronger validation of the proposed meta-learning approach. 

Weaknesses: 
- Authors match but do not improve on backpropagation in terms of performance and the authors did not clearly highlight what the additional advantages of this framework would be presumably related to scalability and biological plausibility. 
- The title should be refined to highlight the focus on biological plausibility and recurrent neural networks. I believe I bid on this paper thinking it would probably relate to multi-agent RL and/or game theory in RL based on the title. You probably want to give readers a better idea about exactly what topics this paper will focus on in your title. 

While I think this paper is strong in a number of ways, I lean towards rejection at the moment because I feel that I am missing some basic understanding about the underlying motivation for exploring this direction. However, I look forward to getting answers to my questions from the authors and seeing what the other reviewers have to say. 
questions: - Could you provide an intuitive explanation of what it means to minimize the "amount of optimal control needed at equilibrium"? (i.e. from Figure 1)
- Can you provide a clear set of advantage and disadvantages of the proposed framework with respect to the recurrent backpropagation baseline? 
- Is it possible to create an experiment that could empirically highlight those advantages or is it beyond the scope of available computation? 
- Can you provide more clarity about the comparison between you approach and iMAML? I was wondering why you only included iMAML for comparison rather than normal MAML as well?
limitations: The authors can make the contribution clearer by having a section that really addresses a direct comparison of advantages and limitations with respect to standard recurrent backpropagation. As someone who is not an expert in this specific proposed approach, it remained very unclear to me reading through the paper what precisely the benefits would be. The authors seems to be motivated by biological plausibility or scalability as the paper does not claim that backpropagation is ever surpassed in the experiments. In light of this, it is also disappointing that potential benefits in terms of scalability could not be validated empirically.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The submission introduces 'the least control principle' a novel paradigm for training neural networks with local learning rules. The learning algorithm is intimately related to predictive coding and energy based models. The authors show that the learning algorithm is flexible by applying it to RNNs and meta-learning tasks. 

___ 

Post rebuttal update: The authors have addressed most of my concerns. I have accordingly modified my score.
strengths_and_weaknesses: **Strengths**
* The submission is overall very interesting and well written. It brings new insight to a field under active investigation.
* The empirical demonstrations showcase the flexibility of the algorithm.
* The relationship to the energy based models and the alternative limit of beta -> infinity is very interesting. 

**Minor Weaknesses**
* The evaluation of the algorithm on a well studied and popular benchmark is missing. The experiments on RNN and meta learning are great but because they are less studied examples, it is harder for the average reader to compare performance of the algorithm. Would be good for the authors to demonstrate that they can use their algorithm to train a deep net on a classic vision task like ImageNet (or even CIFAR10 with a feedforward network).
* The discussion of the limitations of the work is very narrow and implies that the method has no limitations beyond 'out-of-equilibrium' networks and explicit parameter dependence. However, for example we know that a major limitation of predictive coding is its practical inefficiency because of its equilibration phase. Doesn't least-control also suffer from the same issue? An honest discussion of practical limitations and compute requirements is missing. The compute details of the experiments is also missing. (Hardware/compute time for the different methods etc.)
* What are the consequences of alpha not being zero?
* Related to above, overall empirical demonstrations of the fate of the theorems in regions that they do not apply is lacking. For example what happens to thm 2, prop 3, prop 4 when alpha is finite or f is not strongly convex as in (11). The experiments showing that learning take place are good but it would be great to empirically show that the theorems/propositions still approximately hold if their conditions are not fully satisfied. Prop. S8 somewhat addresses this but assumes that the approximate state and optimal state are close. Is there a guarantee that this is the case? Might be a good idea overall to show one or two empirical plots exploring these issues.


**Exposition** 
The following are suggestions to improve the paper. (in no particular order)
* (Line 114) 'y are at a target value'. This probably only holds when the objective is supervised (e.g. regression/classification). Might be good to clarify this to avoid confusion.
* Sometimes the presentation is a little confusing and requires multiple reads. For example in section 2.2, the paragraphs jump between describing eq. 6 and describing the generality of thm 2. I would suggest the authors reword this paragraph such that the reader does not have to go back and forth as many times. (Pehraps provide the theorem first and then give examples and expand on the specific example.) In general, one would expect the paragraph before or after a theorem or proposition to explain the theorem/proposition in more understandable terms.  
* Another example of above is after Eq. 4. The authors alternate between describing their own algorithm and contrasting with backprop. I would suggest first describing eq. 4 in terms of Hebbian learning and then moving on to contrasting it with backprop. 
* Might be good to mention 'predictive coding' when referring to the model of Whittington and Bogacz (line 189). 
* Would be good to explicitly show the model parameter updates (eq. 4) for the specific examples of section 3. This would make it easier for the reader to see that they are indeed Hebbian. 
questions: I will gladly increase my score if/when the authors address the weaknesses in the previous section.
limitations: n/a
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a theoretical framework for optimizing equilibrium neural networks by reformulating the original constrained optimization problem into a least-control problem. The least-control problem is further solved with a two-steps iterative procedure that first drives the networks dynamics with a control signal towards minimizing fixed points verifying the initial problem constraints, then taking a gradient step to minimize the norm of the control terminal cost. 

Motivations for such a principle are manifold: First, solutions are shown to coincide with the initial optimization problem under feasible conditions on the control that leave flexibility for its explicit construction. Second, credit assignement and activation dynamics are originally combined in a single dynamic that result in a spatially and temporally local explicit weight update rule. Third, this formulation can also be linked to models of free energy minimization, offering an original perspective on the learning objective operated by such models as a relaxation of the proposed least-control objective.

Several experiment are conducted to showcase the learnability of such system for supervised image classification, implicit neural representation and meta-learning. The technique is shown to have comparative performance to the current leading technique for training equilibrium model recurrent backpropagation (RBP) and can encompass meta-learning with good results compared to gradient-based methods.
strengths_and_weaknesses: First, I find this works is nicely written, with a clear and rational development which makes it highly accessible. The technical problem formulation as well as the different connection with previous models are well executed and contributes to appreciating the original contribution of this work. The experiments are diverse and show the general applicability of the method.

I would only note that a lot of the paper technicality is deferred to the supplementary, and while the mathematical development is clear, on the other hand, much of the choices for solving experimentally the least-control problems are not discussed in the main text. I understand that this is a choice for exhaustivity but I find that it undermines slightly the experimental section for demonstrating the value of the least-control formulation. Globally however, I have a very positive opinion on this work as it offers a valuable theoretical contribution to the field. 





questions: I have three questions, with one regarding the neuroscientific interpretation of the update rule, a second for optimization and a third for the experiments:

- In line 98 is stated that "the resulting weight update now represents a local Hebbian rule multiplying presynaptic input with the postsynaptic control signal." Could you elaborate on this? Form equation (4), the weight update consist in the Jacobian of f at equilibirum multiplying the instantaneous variation of activity driven by f. Hence, the interpretation given by the authors is not straightfoward and does not seem to be local. Could you help clarify this point?

 - Could you relax the assumption of convexity of the function $L$ in proposition 3 to accomodate for locally convex functions? For instance, the meta-learning problem proposed in section 3.2 does not consider a convex loss function with respect to model parameters and the conditions for proposition 3/4 do not hold.

 - Regarding the experiments, can you provide more details regarding the computation cost of training such a system for the considered problems? It seems to me that the fixed point scheme for finding the equilibrium of the controlled dynamics would require careful tuning for finding a trade-off between rapidity and precision of the control estimate, that is not discussed in the empirical section. A temporal comparison with RBP would be welcome to characterize the efficiency of the optimization process.  


limitations: The work does not raise particular ethical or societal question. Conversely, it constitute an interesting direction of work for optimization for deep learning. The framework is however restricted to deep equilibrium model and an important development would be to extend such principle to arbitrary dynamics. Another interesting aspect of this work is the connection drawn with neuroscience-inspired models (predictive-coding and free-energy) that calls for more in-depth comparison.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a principled generalization of “Minimizing Control for Credit Assignment with Strong Feedback” (Meulemans et al, 2022) rooted in control theory and applying to any implicit model whose forward pass is defined as an equilibrium state. Following the same route as the aforementioned paper, the proposed approach builds upon the idea that credit assignment can be achieved by minimizing the amount of control needed to reach an equilibrium which minimizes the loss of interest (Eq. 1). This is referred to as the “least control principle” (LCP).

- Subsection 2.1 is dedicated to how to train the recurrent feedforward weights. More precisely, the optimization problem at hand is defined (Eq. 2), as well as the subsequent notion of “optimal control” (Eq. 3). Theorem 1 shows that, assuming such an optimal control, the gradient of the amount of control reads as a relatively simple local learning rule.

- Subsection 2.2 describes how to train the feedback weights. Namely, the control is specified as the linear mapping of a low-pass filtered loss gradient (with respect to the output units) onto each neuron (Eq. 6). Theorem 2 states that the resulting steady state control is optimal if it satisfies a “column space” (CS) condition (which can be remotely regarded as a kind of “weight symmetry” requirement) and in the limit of infinitely slow leaky integral controller ($\alpha \to 0$). The authors consequently prescribe general dynamics for the controller such that, by construction, the steady state control satisfies the CS condition (Eq.8). These dynamics are referred to as “dynamic inversion” (DI).

- In Subsection 2.3, it is shown under which hypothesis least control solutions lead to solutions of the original learning problem (Eq. 1) in Prop. 4. Extra theoretical guarantees as to when the hypothesis of the theory do not hold exactly are provided in the Appendix. 
\item In Subsection 2.4, the least control principle is re-casted into a constrained energy minimization problem, and the connection between this reformulation and predictive coding when the implicit function F is implemented by a feedforward network is discussed in the Appendix. This leads to the definition of energy-based control dynamics (Eq. 10).

- In Subsection 3.1, the equilibrium RNN model is introduced (Eq. 11) along with the LCP feedforward weight updates. Different feedback circuitries to implement the control are proposed.

- Subsubsection 3.1.1 proposes a control implemented as a direct linear feedback (LCP-LF). The rule used to train the feedback weights to achieve the CS condition is introduced in Meulemans et al (2022). Subsubsection 3.1.2. proposes one circuitry implementing dynamic inversion (LCP-DI, Eq. 8), another one implementing energy minimization, with only the former being experimentally tested. A more bioplausible implementation of DI with distinct feedback weights learned by Kollen-Pollack (LCP-DI, KP) learning rule is also considered. The resulting three algorithms (LCP-LF, LCP-DI, LCP-DI-KP) are tested on MNIST and achieve comparable accuracy with recurrent backpropagation (RBP). Next, LCP-DI is applied to a more complex deep equilibrium model (described in the Appendix) on CIFAR-10 and implicit neural representation (INR), with again an accuracy comparable to RBP.

- In subsection 3.2, it is shown that LCP can be used as a single phase first-order metalearning technique. The resulting inner-loop parameter updates, as well as the outer-loop metaparameter updates read in a simple fashion and achieve superior performance to other first-order single phase metalearning techniques.

- A thick appendix ($\approx 40$ pages) is provided to support the main mathematical claims, as well as many extra mathematical results.
strengths_and_weaknesses: Strengths:
- The paper is very well written.
- The theory is solid and all the mathematical content of the paper is accurate. I spent hours checking every single proof of the Appendix.
- An experimental effort has been made to demonstrate that LCP applies to a large body of models and problems (MNIST, CIFAR-10, INR, Omniglot).
- The proposed theory achieves credit assignment in a single phase – the forward pass and the gradient computation occur simultaneously. Such theories are rare in this literature (Payeur et al 2020, Sacramento et al 2018, Meulemans et al 2022). 
- The Appendix is extremely detailed and rigorous. 

Weaknesses:
- 10 lines, deep in the Appendix of around 40 pages, to compare your approach with the work *closest to yours* is not enough to my eyes. More precisely, the present paper does not build enough explicit comparison with Strong Deep Feedback Control (SDFC) – “Minimizing Control for Credit Assignment with Strong Feedback”, Meulemans et al, 2022 – which is the closest to theirs. The only explicit comparison appears in S6.3.3, L.1094-1103 of the Appendix. In the light of SDFC, the least control principle idea itself no longer appears as much of an original idea – see Eq. 5 of that paper. Also this approach also already achieves locality in space and time. While the authors re-build from scratch all the maths from first principles of constrained optimization in their appendix, I am very surprised they chose not to include as well the detailed theoretical foundations of this past work.
- The scope of the theory is not clear to me. Most importantly, I am not sure whether the theory extends or not to the case where only a subset of the neurons of the dynamical system are controlled. I detail my concern below. However, I might be wrong and I might well change my opinion about this if the authors address this concern accordingly. 

- The details of the CIFAR-10 experiments are hidden in the Appendix. I am a bit skeptical about the model choices – I’m afraid there could be some hidden backprop here. I am also disappointed about the 80% performance – for instance, equilibrium propagation along with the Kollen Pollack algorithm can achieve almost 87% test accuracy when applied to a Hopfield network without weight symmetry (Laborieux et al, 2021). I’ll come back to this in details in the questions part.

- Finally, as per the NeurIPS 2020 reviewing guideline (https://nips.cc/Conferences/2020/CallForPapers): "*While theoretically grounded arguments are encouraged, it is counterproductive to add “decorative math” whose primary purpose is to make the submission look more substantial or even intimidating, without adding significant insight*". While I do appreciate a lot the degree of details and rigor of the appendix, I think the authors could have better selected the most important information to put in the Appendix. Also, I'm wondering whether it can be expected that reviewers go through 40 pages of appendix when reviewing multiple papers over a short timeframe. 
questions: Below, I shall refer to your approach as LCP.
- L. 43: I think it’s a bit misleading to say that SDFC deals with “feedforward” architectures while LCP might extend the idea to “equilibrium architectures”: the architectures are also recurrent and converge to equilibrium in this past work! Wouldn’t it be a better way to phrase the difference in this regard (rather than feedforward models VS equilibrium models) that LCP can deal with any recurrent connectivity pattern?
 
- L.93-99 (main): what is the difference between LCP applied to a implicit function with a feedforward connectivity pattern on the one hand, and SDFC on the other hand? My take is that LCP allows you to have Q to depend on $\theta$ and $\phi$ (as you point out in the appendix), and therefore use the general-purpose algorithm of Eq. 8, while you couldn’t in the SDFC setting since Q is assumed to be constant, but I’m not sure. It would help to have an extra section in the appendix comparing precisely SDFC and LCP in the case of a feedforward connectivity pattern.

- The following point is particularly important. My understanding of the proposed approach is that the control applies to every single neuron of the dynamical system. If this is correct, in some sense, each neuron of the dynamical system receives a direct feedback from the controller – in this regard, it would be no surprise that the learning rule is local in space. However, I fail to see clearly if your learning theory extends to the case of partial control, e.g. only a subset of the neurons of the system are controlled by $\psi$. For instance, imagine an implicit model defined as the steady state of a layered Hopfield network – this falls in the general category of implicit functions you are considering in this paper. Now imagine that only the last layer of the model is controlled. While the remaining neurons are not explicitly acted upon by the controller, they might well be influenced, through recurrent feedback connections, by the output neurons being controlled. The resulting displacement of the neurons might contain some error information (at the expense of time locality though), e.g.
$\hat{\nabla}_{\Phi} \ell \approx \Phi_{controlled} - \Phi_{not controlled}$.
However, your learning rule prescribes a zero weight update for such neurons. What is happening here? I could have missed something here. But if not, I think it should be clearly stated that your theory does not cover this case and that the theory can only handle *direct* control. 

- The constrained energy minimization view is very interesting, as well as the discussion about the two proposed models in Eqs. (14) and (15). However you say “we do not expect major differences in practice between the two, as they have the same equilibrium”. Still, it would have been interesting to see the results you obtain, for instance on MNIST, with the energy-based approach to motivate it beyond the theoretical connection. It seems that you have gone only halfway through the energy-based approach. 

- I am a bit worried about the CIFAR-10 experiments. L. 1281, you say that the model is made up of an encoder, an implicit module and a decoder. How do you train the encoder $g(x,\theta)$? The encoder is made up of conv layer and batchnorm layer. Unless you: 1/ explicitely augment the state \phi of your model dynamics with the output of the conv layer and that of the BN (the same strategy that you apply to train the decoder) 2/ apply control $\psi$ to these two, I cannot see how you proceed. Also, to run the controller dynamics (Eq. 147), you need to compute the Jacobians $\partial f/\partial \phi$ and $\partial h/\partial\phi$. However f reads as the complex composition of several convolutional and group normalization operations, and g as the composition of batchnorm, average pooling and linear operations. Don’t you need to backprop through f and g and therefore all these complex operations (batch norm, group normalization) to run the controller dynamics? If it is so, it should be clearly stated in the limitations of the paper. Beyond this, the batchnorm operation is not very bioplausible and other more simple (non parametric) normalization schemes have been proposed (like in this paper: https://arxiv.org/abs/2003.01513). If my understanding of the CIFAR-10 experiment is not correct, please provide more details in the Appendix. 

limitations: My overall understanding of this paper is that credit assignment through control minimization (as introduced in Strong Deep Feedback Control) can be casted into an principled control theory-based approach, applying to any implicit model (as opposed to SDFC which only applies to converging RNNs with a feedforward connectivity pattern), which allows for $\phi$ and $\theta$ dependent feedback control weights $Q(\theta, \phi)$. Crucially, this allows to use a novel algorithm to satisfy the column space condition beyond the linear constant feedback matrix $Q$, possibly for more complex models on more complex tasks, as motivated in the main. However, as per my understanding, this new algorithm comes at the expense of having to backprop through the implicit model f and the decoder h, as it seems to be the case in the CIFAR-10 experiments. It's a little bit frustrating because the precise motivation of this new algorithm to train the feedback controller is to train more complex models for which the linear feedback controller might not sufficient for, and while you take care of not using BP through Kollen Pollack kind of mechanisms for the MNIST experiments, it seems to be overlooked in the CIFAR-10 experiments. Moreover, unless I misunderstood something again, it seems to me that the scope of the theory is limited to the case of a *direct* feedback upon *all* the neurons of the dynamical system. If all these remarks are sound -- which again may not be the case if I misunderstood something -- if should be very clearly stated in the limitations section.
review_title: null

Meta-review: metareview: As summarized by reviewer 5Kjg, this work proposes a theoretical framework for optimizing equilibrium neural networks by reformulating the original constrained optimization problem into a least-control problem. The least-control problem is further solved with a two-steps iterative procedure that first drives the network dynamics with a control signal towards minimizing fixed points verifying the initial problem constraints, then taking a gradient step to minimize the norm of the control terminal cost.

Motivations for such a principle are manifold: First, solutions are shown to coincide with the initial optimization problem under feasible conditions on the control that leave flexibility for its explicit construction. Second, credit assignment and activation dynamics are originally combined in a single dynamic that result in a spatially and temporally local explicit weight update rule. Third, this formulation can also be linked to models of free energy minimization, offering an original perspective on the learning objective operated by such models as a relaxation of the proposed least-control objective.

Several experiments were conducted to showcase the learnability of such a system for supervised image classification, implicit neural representation and meta-learning. The technique is shown to have comparative performance to the current leading technique for training equilibrium model recurrent backpropagation (RBP) and can encompass meta-learning with good results compared to gradient-based methods.

Almost all reviewers, including myself, consider this work to have potentially a high impact, linking control theory with neural network learning, backed by convincing experiments. They introduce 'the least control principle' paradigm for training neural networks with local learning rules, and such learning algorithms, which are biologically inspired to be modular, are intimately related to predictive coding and energy based models. I believe this work will definitely encourage much discussion in the research community, and open up many avenues of further investigation, and for that, I strongly recommend acceptance (as a spotlight or oral presentation, or equivalent format, if applicable for this year).

confidence: Certain
Decision: Accept