Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a gradient-based learning scheme that deals with changing parameters during learning. For this, the authors give a context of their algorithm to the classical momentum learning strategy and a multi-timescale model from neuroscience. The algorithm is derived using a Bayesian inference framework, where the posterior mean recovers the newly proposed learning method. Here, a continuous-time linear state space model is used to model the time-varying parameters. For the case of a supervised learning problem, the authors consider a nonlinear model for the time-varying parameters.
Bayesian inference is carried out by the authors by a variational approximation of the extended Kalman filter algorithm. The learning algorithm is tested on two synthetic problems, with time-varying parameters. For both linear regression and classification problems, the proposed algorithm leads to a smaller average loss. 
strength_and_weaknesses: Overall, I think the paper presents an interesting new algorithm, for a problem that is probably very relevant in many applications. I am not very used to the style of writing, where the authors are probably coming a bit more from the direction of neuroscience. For me, the paper was pretty hard to follow. However, the results look reasonable even though the derivations are derived in a bit nonstandard way. The presented two empirical case studies are fair and are showing promising results. However, I would have hoped for an application where this strategy is tested for a real-world problem, to show its applicability.

A sore point for me is the derivation using the power spectral analysis. I think this paper would have been much easier to understand and interpret would it had been derived using standard Bayesian inference techniques. For example, the definition of the $1/f$ noise process is very strange to me. To illustrate my confusion: equation (13) is a linear stochastic differential equation, for which the time-point-wise marginal densities are distributed multivariate Gaussian, see, e.g., [1]. Since the sum of the components of a multivariate Gaussian random variable in equation (14) is still Gaussian, see, e.g., https://web.ipac.caltech.edu/staff/fmasci/home/astro_refs/SumOfCorrelatedRVs.pdf , it implies that $\xi(t)$ is a Gaussian random variable, where the mean and variance parameter can be computed in closed form. Maybe the authors can explain, why the $1/f$ definition is needed.

Strengths:
- Interesting Setup
- Relation to well-known algorithms is given
- A new learning algorithm, which is easy to implement
- Convincing empirical results for some synthetic problems under the modeling assumption

Weaknesses:
- The derivations are hard to follow
- The $1/f$ noise definition is not clear to me
- No real-world application given in the paper

[1] Särkkä, Simo, and Arno Solin. Applied stochastic differential equations. Vol. 10. Cambridge University Press, 2019.
clarity,_quality,_novelty_and_reproducibility: For me, the paper is hard to follow. The results are however interesting and relevant. The inference techniques used are well-known and standard. However, I think the work is probably interesting for the community for trying to solve a learning problem with time-varying parameters.
summary_of_the_review: The paper presents an interesting topic. The resulting algorithm is easy to implement and is probably useful for learning problems with time-varying parameters. The derivations are not very clear to me and the paper, but the results look reasonable. An application to a real-world learning problem would have been very helpful to show its applicability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors make the connection between the framework of multiscale learning and models of fast weights, momentum, and synaptic modelling. The authors then go on to formulate a model of multiscale learning in terms of Bayesian inference over 1/f noise, which has been reported in prior work to match many real-world distributions. The authors test out their model on synthetic online prediction and classification tasks while showing a variational approximation can retain most of the benefits of the full model. 
strength_and_weaknesses: Strengths: 
- It is nice that the paper contextualized prior approaches within a common framework.
- I appreciate the synthetic experiments as they help build an understanding of what the model is doing.
- It is nice to show the effectiveness of the variational approach, hinting that the model may showcase scalability.

Weaknesses: 
- I did not find the insight about these models being special cases of a common more general framework to be particularly surprising or enlightening. It really only matters, in my view, if it results in a better alternative to the specific models from prior work considered. 
- The experiments are extremely limited and the barriers for scaling are not even clearly discussed. The authors end with the comment "the next step will be to investigate how well these methods scale up to large networks, and whether they confer an advantage in real online-learning domains." It feels to me that this is such an obvious step based on the discourse that the current paper comes across as incomplete. If the authors could present experiments showcasing this, I think this paper could have significant impact. However, if this is not possible, I do not believe there will be much interest in the perspective highlighted here. The experiments are so synthetic and toy relative to the typical standards of this conference that I feel that the lack of any experiments on a meaningful scale requires far more justification than provided within the submitted draft. 
clarity,_quality,_novelty_and_reproducibility: The writing quality is pretty good throughout -- although it could be even better if it was more streamlined and to the point in the introduction and conclusion. The results are likely to be easily reproducible. I feel that the approach considered is novel, but it still remains largely a mystery how scalable it is based on the submitted draft. I do not feel that the theoretical results showing that this framework generalized past work is that novel or surprising on its own merit. However, if the approach itself was very successful on large scale domains, I would feel that this was a nice part of the discourse. 
summary_of_the_review: Ultimately, I am on the fence about this paper. I definitely see some good parts about it, but ultimately feel that each strength is somewhat undercut by a corresponding weakness in the current submission. I lean towards rejection at the current time because I feel that the current paper is somewhat incomplete. If large scale experiments were provided validating the proposed method, I would think it could make a very nice contribution to the conference and I would be firmly voting for accept. However, the current experiments are well below the standards of this conference, and not enough justification is provided to make readers understand why this is a logical ending point for the work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Temporally structured environments involve multiple learning processes operating at different time-scales. The paper provides a unifying view of learning in these temporally structured environments. The authors begin by formalizing the multi-scale learning setup, where the weights are decomposed into sub-weights, each characterized by a timescale and learning rate and decay. The authors first establish fast weights as an instantiation of the multiscale optimizer with 2 timescales. Next, the authors establish the equivalence between the multiscale optimizer (in the configuration of a fast-weight model) and the Benna-Fusi model for real synapses, as well as momentum learning. An interesting observation from the analysis is that momentum can be characterized as a fast weight with a negative learning rate. The authors then present a Bayesian inference view for a multiscale learner, adopting the $1/f$ model, and describe exact inference with a Kalman filter and approximate inference with extended Kalman filters and variational approximation for extended Kalman filter. Finally the authors validate the multiscale learning framework with some experiments on synthetic tasks. 
strength_and_weaknesses: **Strengths**:

- The paper tackles an important problem of learning in non-i.i.d settings, in the specific case where multiple learning processes operate at different timescales. The authors present a unifying framework of multiscale optimizers for learning in such settings. The framework encompasses well studied approaches of momentum learning and the Benna-Fusi model, providing novel insights for momentum learning.  
- The Bayesian formulation provides a simple and straightforward approach for implementing the multiscale optimizer. 
- The synthetic simulations provide _some_ evidence on the applicability of the multiscale optimizer in practical settings. (I discuss this a bit more in the weakness)

**Weaknesses**

- The paper studies a particular setting where the multi-scale weights are additive. This however seems like a somewhat limiting assumption to me. The authors do not discuss this choice in the paper and it is not clear whether this assumption holds true in realistic settings.
- While the simulations provide a sanity check for the approach, they leave quite a bit to be desired. As mentioned in the first point, the assumption of weights of the weights of the different timescales being additive might not hold in any realistic setting. Experiments on more realistic settings would be helpful to address this concern. 
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is quite well written and easy to follow. Except for a few instances referring to a concurrent submission, the paper is self-contained with the relevant details described in sufficient depth. Minor correction: In the last paragraph of Section 4.1 line 5, "manifestation shift" -> "Manifestation shift"


**Quality and Novelty**

While the quality of the contributions in the paper is high, the perspective is not particularly novel. Multiscale learning has been explored in the RNN literature in the past (which is not mentioned in the paper) - e.g. [1]. Nonetheless, the insights presented are a substantial contribution to the community.


**Reproducibility**
The authors did not provide code with their submission, but the experiments are small and straight-forward, accompanied with enough details for reproduction. The theoretical results also stated and derived with clarity, 


[1] Long Expressive Memory for Sequence Modeling, ICLR 2022
summary_of_the_review: To summarize, the paper presents a unifying perspective for multiscale optimization in some non-iid settings. The paper makes useful connections to existing ideas and proposes a novel Bayesian formulation for multiscale optimizers. The empirical evaluations seem lacking given certain assumptions made in the formalization. More realistic experiments would be helpful in addressing this shortcoming. Nonetheless, the conceptual and theoretical contributions of the paper are substantial, so I lean towards acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a generalized multiscale learning framework for learning in temporally structured environments. The basic idea is that each weight in a NN is decomposed into a sum of subweights, each updated with different learning and decay rates, with all components evolving independently as compared to previous models which involved complex coupling between timescales. They show that these previous models can be shown to be equivalent to the multiscale learner, with no need for these couplings across timescales. They also show that momentum learning can be characterized within the multiscale learner framework as well. Lastly they derive an equivalent model of bayesian inference over 1/f noise and show that that model is also a special case of the multiscale learner framework.
strength_and_weaknesses: Strengths:
- The paper is well written, thorough in theory and background information (it does require reading the appendix to fully grasp the derivations).
- Simplifying and unifying various timescale learning methods into a single framework is a significant contribution.
- I specially enjoyed figures 1 and 2. They do a great job of summarizing the whole paper in to two clear, concise and simple figures + captions.

Weaknesses:
- I'm hard pressed to find one. I understand this is a mostly theoretical paper but it would still have been great to see experiments on more than the simulated cases.
clarity,_quality,_novelty_and_reproducibility: - Quality and Clarity: the paper is very well written. It is high quality and very clear. To the best of my knowledge it is also original.
- Reproducibility: The authors have not provided code which would help reproducibility. The experiments are described in fair bit of detail.
summary_of_the_review: I believe this is a paper worth appearing at ICLR. It presents a well thought out and well presented unifying and simplifying framework for learning in temporally structured environments.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Summary:

The paper proposes a gradient-based learning scheme that deals with changing parameters during learning. For this, the authors give a context of their algorithm to the classical momentum learning strategy and a multi-timescale model from neuroscience. The algorithm is derived using a Bayesian inference framework, where the posterior mean recovers the newly proposed learning method. Here, a continuous-time linear state space model is used to model the time-varying parameters. For the case of a supervised learning problem, the authors consider a nonlinear model for the time-varying parameters. Bayesian inference is carried out by the authors by a variational approximation of the extended Kalman filter algorithm. The learning algorithm is tested on two synthetic problems, with time-varying parameters. For both linear regression and classification problems, the proposed algorithm leads to a smaller average loss.

Strengths:

- Relation to well-known algorithms is given
- A new learning algorithm, which is easy to implement
- Convincing empirical results for some synthetic problems under the modeling assumption
- It is nice that the paper contextualized prior approaches within a common framework.
- I appreciate the synthetic experiments as they help build an understanding of what the model is doing.
- It is nice to show the effectiveness of the variational approach, hinting that the model may showcase scalability.
- The writing quality is pretty good throughout 
- The paper tackles an important problem of learning in non-i.i.d settings, in the specific case where multiple learning processes operate at different timescales. 
- The authors present a unifying framework of multiscale optimizers for learning in such settings. The framework encompasses well studied approaches of momentum learning and the Benna-Fusi model, providing novel insights for momentum learning.
- The Bayesian formulation provides a simple and straightforward approach for implementing the multiscale optimizer.
- The synthetic simulations provide some evidence on the applicability of the multiscale optimizer in practical settings. (I discuss this a bit more in the weakness)

Weaknesses:

- The derivations are hard to follow
- The 1/f noise definition is not clear
- No real-world application given in the paper
- The experiments are extremely limited.
- it still remains largely a mystery how scalable it is based on the submitted draft.
- The paper studies a particular setting where the multi-scale weights are additive. This however seems like a somewhat limiting assumption to me. The authors do not discuss this choice in the paper and it is not clear whether this assumption holds true in realistic settings.
- While the simulations provide a sanity check for the approach, they leave quite a bit to be desired. The assumption of weights of the weights of the different timescales being additive might not hold in any realistic setting. Experiments on more realistic settings would be helpful to address this concern.

Recommendation:

A majority of reviewers vote for acceptance. The reviewer voting for rejection asked for large scale experiments. These were added in the rebuttal. I, therefore, decide to accept the paper. I encourage the authors to use the feedback provided to improve the paper for its camera ready version.

justification_for_why_not_higher_score: The overall ratings by reviewers were not that high.
justification_for_why_not_lower_score: A majority of reviewers voted for acceptance.
Decision: Accept: poster