Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel meta-learning algorithm, effectively applying Equilibrium Propagation (EP, Scellier and Bengio, 2017) to the dynamical system defined by a learning algorithm minimizing a loss function (rather than the dynamical system defined by a model minimizing an energy function as in the original EP work). The authors define an augmented loss function (analogous to the "total energy function" of EP) consisting of a learning loss and an evaluation loss, weighted by a nudging strength. To estimate the total derivative with respect to the meta-parameters, they take the difference of the partial derivatives of the nudged and non-nudged loss, thus making a finite-difference approximation of the total derivative. Finally, the authors validate the algorithm with two specific meta-learning models (synaptic and neuronal) applied to four tasks (hyperparameter tuning, few-shot learning, regression, and reinforcement learning), demonstrating competitive performance, and provide a biological interpretation of their models. 
strengths_and_weaknesses: Originality. The paper is an interesting application of Equilibrium Propagation to meta-learning, suggesting a biological implementation/approximation of backpropagation-through-learning (BPTL), analogous to how EP proposes a biological implementation of backpropagation-through-time. Although the contrastive rule itself is not novel, it is used in a different setting and its effectiveness there is thoroughly demonstrated. Furthermore, the authors extend the original theoretical analysis to include the approximation error incurred from the finite-difference step (i.e. \beta) as well as from non-convergence to the fixed point. 

Quality. Overall, the paper is very thorough and rigorous, both theoretically with numerical validation of the theoretical results, and empirically in comparison to previous meta-learning benchmarks. A potentiall weakness is that for some of the empirical results on the various meta-learning tasks, it is unclear how to make a fair comparison to other methods (see Questions), equalizing architecture, computational resources, etc. 

Clarity. Paper is very clearly written and well-organized. 

Significance. From a machine learning perspective, this work provides a useful meta-learning approach which avoids storing the computational graph of the learning algorithm, thus using additional memory proportional to the number of meta-parameters (rather than the number of learning steps). The time complexity remains the same as BPTL. From a neurobiological perspective, it is an important step towards a biologically plausible meta-learning algorithm. Furthermore, the success of the modulatory CML algorithm is particularly interesting as it suggests a much more important role for learning via neuronal (rather than synaptic) changes than is typically considered. 

questions: - Are there cases where the meta-learning fails (e.g. does not converge to a viable solution for some/all of the learning tasks) due to the approximation? It would be instructive to include an example where CML entirely fails where another algorithm succeeds and vice versa (if such examples exist). 
- In section 5.2, the authors compare two methods (RBP, CG) which incur numerical errors in calculating the meta-gradient with CML's approximation. Is it possible to include an "upper-bound" result using the exact meta-gradient (computed with BPTL, or analytically as in section S3.6)?
- In section 5.3, MAML perhaps plays the role of BPTL as an "upper bound" for the performance. However, it is difficult to compare these side-by-side as it is unclear whether the performance differences result from a superior meta-learning algorithm, or a superior learning algorithm, or differences in architectures (number of tuneable parameters, number of tuneable meta-parameters).
- In section 5.4, can you clarify how the inputs are presented to the network, and how the outputs are read out? (e.g. for a given sinusoid amplitude/phase, is x_i a spike with probability p_i? Is y_k averaged across all 20 time steps? Is there only one y_k?)
- In section 5.4, it is surprising that the BPTT+eprop performance is much worse than CML+eprop. If CML is an approximation of the exact meta-gradient from BPTT, why isn't BPTT strictly better?
- Potential reference to include for section 4.2 or discussion: Titley et al., Toward a neurocentric view of learning, Neuron 2017

limitations: Technical limitations partially addressed through analysis of estimation error (see also Questions). Societal impact not discussed (but does not seem necessary for this work).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed a biologically inspired approach to meta-learning that avoids backpropagation through time. They change the optimization process and reorganize the loss function such that it mixes problem-specific and meta-specific parameters. 
strengths_and_weaknesses: The paper provides a novel approach to meta-learning. The new approach is more biologically plausible than existing approaches: it is time local and does not require backpropagation through time. The paper is well written and the authors conducted an informative theoretical analysis of their approach. 

The authors conducted experiments on several different types of datasets. They obtained better results than SOTA on meta-learning CIFAR-10, but comparable results on other datasets. While I think that the paper would be stronger with more experiments on more challenging datasets, I believe it is still a solid contribution. 

questions: It would be helpful if the authors discuss how their approach would scale to more complex problems. 
limitations: I believe a discussion about the limitations of the experimental results would help to better understand the strengths and weaknesses of the proposed approach. For example, the authors could discuss the possibility of using this method on more challenging meta-learning RL tasks.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a contrastive meta-learning rule having several appealing properties: (1) biologically-plausibility, (2) locality, (3) causality, and (4) learning process agnostic. Specifically, the rule contrasts the outcome of two learning episodes with/without meta-objective to update meta-parameters. The proposed meta-learning rule estimates meta-gradient by simply running the learning algorithm twice compared with the existing meta-learning methods which require either second-order terms or inverse learning loss Hessian. The authors theoretically analyze the meta-gradient estimation error and confirm this through the experiment results. Also, the proposed meta-learning rule shows similar performance to the existing meta-learning methods on various tasks and architectures.
strengths_and_weaknesses: Originality: The authors propose a novel method for meta-learning and applying this rule to the synaptic and modulatory models. The proposed method is simpler than the existing meta-learning methods including MAML and iMAML because it estimates the meta-gradients using only partial derivatives rather than second-order derivatives or inverse Hessian.

Quality: The authors' claims are well supported in the paper. The authors theoretically prove the meta-gradient estimation error of the proposed meta-learning rule in terms of the nudging strength and demonstrate it via the experimental results. Also, the proposed meta-learning rule matches or outperforms the reference methods on various benchmarks even its simplicity. However, it is not clearly written whether the simplicity of the proposed rule leads to better computational and memory efficiencies.

Clarity: The paper is generally well-written and easy to follow.

Significance: The results in the paper are important because the proposed meta-learning rule is simple but its estimation error is theoretically proven and the performance matches the reference algorithms on various tasks. Therefore, I expect this paper is likely to be used in meta-learning literature.
questions: - One of the main contributions of this paper is its simplicity. However, it is not clearly written whether such simplicity leads to better computational and memory efficiencies than the existing meta-learning rules. Therefore, it would be great to add a comparison table for computational and memory complexity.
- In Fig. 2A, it looks like input and output are Poisson spike trains. However, since the input and output are continuous values, please redraw Fig. 2A.
- In Section 5.5, the modulatory model performs much worse than the synaptic model and MAML as increases a task-specific radius while it outperforms the synaptic model on visual few-shot learning. However, CAVIA [1], which uses a modulatory approach as cited in the paper, outperforms MAML in the RL Cheetah experiments. So, it would be great to see the performance of CAVIA on the wheel bandit problem.

References\
[1] Luisa Zintgraf et al. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Paper addresses the problem of performing meta-learning when facing different tasks by introducing a general-purpose rule of contrastive kind that aims on biological plausibility. The main idea of contrastive meta-learning here is to run the underlying algorithm twice: once by learning the original presented task  and once to learn what the authors call "augmented" version of the task that includes the meta-objective. From this perspective, authors see their approach as generalization of classical Hebbian contrastive learning to meta-learning frame. When arguing for biological plausibility, authors state that 
1. the rule runs forward in time, avoiding backtracking through synaptic modifications in reverse time order,
2. the rule does not evaluate second derivatives, avoiding using information non-local to an updated parameter
3. implementing the rule requires only temporarily storing one intermediate state

Authors also derive an expression for estimating meta-gradient approximation error that arises by applying the rule, and state that meta-gradients can be consequently approximated as accurately as need, by investing more resources spent in learning and by controlling the nudging strength $\beta$ that controls contribution of the meta-objective term in the total loss.

To show concrete implementations of their rule, the authors show two distinct versions using different meta-parameters. One is based on the complex synapse idea, where the synapse is allowed to have further internal variables apart from the effective synaptic weight. The other uses notion of fast top-down modulation, which affects two additional meta-parameters per neuron - its adaptive multiplicative gain $g$ and adaptive threshold $b$. 

The authors then benchmark their proposed contrastive meta-learning algorithm using the two proposed implementations against alternatives using a number of tasks: supervised meta-learning on CIFAR-10; supervised few-shot meta-learning on miniImageNet and Omniglot; supervised meta-learning in a recurrent spiking network using sinusiodal functions dataset; and meta-learning in reward-based setting using wheel bandit task.

On the basis of the benchmarks, authors state that their contrastive meta-learning rule shows comparable of better performance than the alternative counterparts. Authors conclude that their biologically plausible rule has thus competitive or better performance than meta-learning methods that have to rely on non-local information. Finally, in the discussion section, authors elaborate on how the two phases of learning and the augmented version of the tasks that are required by their rule may be instantiated in the brain networks, taking hippocampus as possible candidate structure responsible for coordinating this learning form.
strengths_and_weaknesses: ## Strengths

The paper introduces a novel take on biologically plausible meta-learning formulating a rule that attempts to use only locally available information for parameter updates. It uses the idea of contrastive learning in novel context by proposing to contrast between two different learning episodes based on the same task. I think this is an interesting idea to elaborate on in general - contrasting on the level of task instead of usual notion of contrasting on the level of data samples within a task. Authors motivate their rule in clear and conscise manner, emphasizing the point of biological plausibility, resulting locality and the differences to conventional meta-learning approaches. While there is some amount of work being done on deriving local plasticity rules that may replace backprop in standard learning, to my knowledge it is still rare to treat the same problem in meta-learning frame, and therefore significant for the community. Theoretical derivation of meta-gradient approximation error and the notion that approximation can be in principle made arbitrary accurate adds further value.

## Weaknesses
While the contrastive meta-learning rule aims on handling a setting with multiple various tasks streaming in, it is not clear for me after reading the manuscript how such a multi task setting should be handled in biologically plausible manner, which is one of central claims of the paper. It seems that rule cannot work when task do not have clear boundaries and needs some mechanism that identifies a task while it is learned. Further it is not clear what happens if there are concurrent tasks present at the same time. 

Another point is handling of catastrophical forgetting and non-shuffled data: the authors do not elaborate whether their rule works under i.i.d assumption with respect to the tasks or whether it can also handle non-shuffled data (which would be another pre-requisite for biological plausible setting).

In the benchmarking, there is in my opinion lack of scenarios that test meta-learning in proper multi-task frame, where tasks have indeed different structure. So, meta-learning on CIFAR-10 is actually dealing with one task, while miniImageNet and Omniglot are used in rather classical few-shot scenarios. where the task is again the same across episodes. Other benchmarking tasks - sinusoidal function regression and wheel bandit - can be arguably considered as toy tasks, and it is not clear whether benchmarking on those can give us generic statements about the power of the algorithm. 

My impression is also that networks used to benchmark the algorithm are very small compared to even small scale state-of-the-art. Recently derived scaling laws (e.g Scaling Laws for Neural Language Models, https://arxiv.org/abs/2001.08361) were pointing to the fact that increasing model and data scale during training improves generalization and transfer - I therefore wonder, whether testing algorithms that aim on strong generalization across tasks with very small scale networks and data may lead to distorted picture, where a more sophisticated algorithm may be superior on small scale, but substantially underperform against simpler algorithms on larger scale.

Finally, one major unclear point to me is what authors attempt to elaborate on in discussion: how to organize and coordinate the contrastive learning which requires creation of the augmented version of the original task and careful execution of both original and augmented task learning in proximity to each other. I think authors attempt to bring in hippocampus as responsible structure falls short of explanation how it can manage to do those tasks - it feels to me like postulating a homunculus in form of hippocampus that has to figure out how to drive contrastive learning for all other brain networks involved. The authors should present at least some coarse explanation why they think that specifically hippocampus (and not, say, basal ganglia, prefrontal cortex, etc) is the candidate structure and how it may fulfull the required functions of creating augmented task and executing learning of both in coordinated way. Referring to "transfer" of "additional data" from hippocampus to cortex during sleep or resting state only adds to confusion here, in my opinion, as it introduces a very long time lag between original and augmented tasks and additional requirenment of specificity, where hippocampus should exactly to transfer data from a certain task to, and whether the target that should receive this data was not already exposed to data from various other tasks, which is not clear to me how to resolve.
questions: 1. It seems that the rule relies on knoweldge of the task identity $\tau$. It would be good to discuss how learning algorithm may obtain this knoweldge in biological plausible way and still remain local.
2. Does the training assume i.i.d shuffled data? If yes, how is that executed in biological plausible way? If no, what makes the algorithm perform continual learning?
3. Why not combining both complex synapse and top-down modulation implementation in one rule? It seems both mechanisms are co-existing in the brain.
4. What about benchmarking the algorithm on more realistic multi-task datasets? For instance, the standard ATARI game suit or MetaDataset (https://github.com/google-research/meta-dataset) would give a proper multi-task setting and allow for more generalizable comparision to other approaches.
5. In the discussion, you point to hippocampus as a "control" structure that organizes contrastive learning - creating augmented task version and coordinating original and augmented task learning. It would be good to give a more grounded explanation - why necessary hippocampus (and not another structure, say basal ganglia or pre-frontal cortex, or insula, that also have access to most of cortical networks) and how it may subserve these functions? How sleep related activity may be relevant here, given that there will be a large time lag between learning episodes and interference with all possible tasks that happened during this period of time?
6. Paper mentions one intermediate state that has to be temporary buffered after original task learning was executed, until augmented task learning has been performed. Is this state local to the synapses? Where is it in the update equations (is it $\phi_{\theta,0,\tau}$)? It would be good to make this explicit for the reader. What is the biological plausible way to perform this short-term storage?
7. It is stated that meta-gradient approximation error can be made arbitrary small by either running further learning or decreasing nudging factor $\beta$. What may be a biologically plausible way to tune the nudging factor?
8. It is mentioned that in case of implementing the rule by top-down modulation, the partial derivatives of the augmented loss function will be computed by automatic differentiation using standard backprop. To rescue biological plausibility, further mechanisms are invoked to replace backprop : prediction error neural subpopulations, dendritic error representations, or usage of equilibrium propagation. Would the same mechanisms then act for replacing backpropagation in general also for the original task learning, or is the idea here that those will be acting only for learning on augmented task?
9. In general, how would usual approaches to replace backpropagation with local rules (again, for instance equilibrium propagation) relate to the proposed contrastive rule - are they complementary, or belong to the same class of methods, only applied for meta-learning in this case?

limitations: One limitation of the study that would be good to mention explicitly is the very small scale of conducted experiments (both network size and data size wise), compared to current state-of-the-art in deep learning.

Authors emphasize strongly benefits of top-down modulation based rule implementation. I think it would be good to cite corresponding classical works (eg Schmidhuber in 90s, Fast Weight Programmers, see for example discussion in Going Beyond Linear Transformers with Recurrent Fast Weight Programmers, https://proceedings.neurips.cc/paper/2021/hash/3f9e3767ef3b10a0de4c256d7ef9805d-Abstract.html ) that deal with fast slow learners, describing the same way of fast weight modulation by a slowly learning network (dual architecture as mentioned by authors), as alternative to slow weight change for task learning.
review_title: null

Meta-review: metareview: The Reviewers appreciated the novelty factor of the contrastive meta-learning algorithm proposed in the paper, the theoretical analysis establishing a formal connection with equilibrium propagation, and the appealing features of the resulting meta-learning procedure, which include memory and computation efficiency, as well as the fact that the algorithm affords a biologically-plausible implementation that only requires locally available information for parameter updates.
To concretely showcase these properties the paper demonstrates two instantiations of the proposed algorithm that are mechanistically realized through synaptic consolidation and top-down neuronal modulation, respectively.
Finally, the paper validates the algorithm on standard few-shot learning benchmarks.
The main weaknesses of the paper identified by the Reviewers are the empirical evaluation, which would benefit from a more extensive comparison between methods and more experiments on more challenging meta-learning datasets, and the discussion on the candidate neurobiological substrate for a brain implementation of contrastive meta-learning, which would benefit from a more detailed and systematic description.
These limitations however do not substantially detract from the overall quality, relevance and interest of the paper, which Reviewers unanimously recommend for acceptance.
confidence: Certain
Decision: Accept