Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a recurrent spiking network-based reinforcement learning algorithm, in which the network consists of two major parts (an agent and a model). The agent is trained to act using policy gradient, while the model is trained to predict the effect's of the agent's actions (both the reward and the state transition). The authors apply their approach to the game of Pong, showing that dreaming indeed improves performance. They also consider an alternative to dreaming (namely, planning), which showed similar performance.
strength_and_weaknesses: Strengths:
- The paper addresses a number of highly relevant and important problems
- Using spiking networks for reinforcement learning is fundamentally challenging, but the authors manage to successfully handle the task
- The authors consider an expansion of their method to image-based inputs

Weaknesses:
- The paper focuses on many targets at once, such as biological plausibility, applicability in robotics, suitability to neuromophic hardware, as well as general usefulness of the approach as a general-purpose reinforcement learning algorithm. I believe that while some of these goals are correlated, they are not the same. As a result, the paper produces an unfocused impression. I believe that narrowing down the range of claims and delving deeper into one of these topics would greatly benefit the paper.
- The range of applications considered is limited. The method works reasonably well when applied to problems that are fundamentally extremely low-dimensional, but it's not clear whether more complex systems can be managed in a similar manner. While the authors say that "The approach described above allows, in principle, to apply our method to any other Atari games from pixels", it remains unclear whether this suggestion is practical.
-  The novelty of the approach is limited. While the use of spiking neural networks in this specific context is novel, the idea of using model-based simulations or dreaming to enhance learning is very well known. Again, I believe that additional, more focused results might help to mitigate the issue.

I clarify and expand upon some of these points in the main body of my review.
clarity,_quality,_novelty_and_reproducibility: ** Clarity **

The paper is generally well written and is a pleasure to read.

I feel that the fact that the paper has many ambitious goals at once sometimes makes the paper harder to follow. For example, neither the title nor the abstract mention spiking networks, and create the impression that the idea of "dreaming" is the key invention of the paper. At the same time, arguably, it's the tricky adaptation of spiking networks to the task which is the main achievement of the paper.

This and similar issues complicate reading and result in an impression that the paper attempts to balance a number of related, but not equivalent goals.

** Quality **

My main issue with the experimental planning is that the authors list many ambitious goals, but none of them (in my opinion) receives commensurate depth of exploration and justification. For example, consider the following paragraph:

"To our knowledge, there are no previous works proposing biologically plausible model-based reinforcement learning in recurrent spiking networks. Our work is a step toward building efficient neuromorphic systems for autonomous robots, capable of learning new skills in real-world environments. Even when the environment is no longer accessible, the robot optimizes learning by reasoning in its own mind. These approaches are of great relevance when the acquisition from the environment is slow, expensive (robotics) or unsafe (autonomous driving)."

While I agree that using spiking networks is indeed more biologically plausible than traditional artificial neural networks, I found the discussion of biological plausibility insufficiently deep. For example, the authors stress the fact that memory replay is biologically implausible, but I can not fully agree with that. Humans have reasonable episodic memory. Even though it is not infinite capacity, I don't believe that one can discard all episodic replay as biologically implausible. A deeper look is warranted into how the effects of dreaming in spiking networks is more similar to human dreaming than using, say, exponentially decaying replay buffers in traditional reinforcement learning algorithms.

Additionally, since learning is not done online, the biological plausibility argument becomes even less convincing.

The claims about the usefulness of the approach in autonomous driving and robotics similarly don't receive enough confirmation. While I understand the difficulties associated with training spiking networks and I don't expect a state-of-the-art performance across a range of image-based RL tasks, it is still crucial to look at how the performance scales with the size of the network and the dimensionality of the problem. For example, a toy task, similar to what is used in human experiments with multiple cue learning tasks (see https://psycnet.apa.org/record/2009-24669-008 for a review) could be a good testbed to see how the performance scales with the complexity of the task (It could also show that, potentially, spiking neural networks produce more human-like behaviour, and be used to investigate biological plausibility).

Lastly, the main limitation of all model-based approaches is that what happens if the model can not easily fit the environment. In humans, we know that dreams do not copy our real experience, but are still apparently useful in learning. If the "dreaming" part of the contribution is key, a deeper look into what happens when the model can not properly fit the task is warranted. Additionally, perhaps a more systematic investigation (on simulated tasks) is warranted, to answer the question of what are the conditions when dreaming is useful. For example, are there task where even an important model is very useful? Are there cases where learning a model is easier/faster than learning a policy? And so on.

** Novelty/Impact **

While the approach is sufficiently novel, the novelty of the paper is not high enough to be its main selling point. The work is heavily building upon existing ideas, which puts more importance on the depth and thoroughness of experimental support.

** Reproducibility **

The approach is described clearly, but the code is not provided. I can imagine that some researchers might struggle with implementing the algorithms in an exactly the same way as the authors and reproducing the results.

** Typos/phrasing **

still, a clear and coherent understanding of the mechanisms that induce generalized beneficial effects is still missing - "still" is repeated.

Taking inspiration from biology, an intriguing idea is that one possible usage of a learned model, is during periods in which the neural network is offline. -- I'd suggest rephrasing. E.g. "Taking inspiration from biology, we explore an intriguing idea that a learned model can be used when the neural network is offline."
summary_of_the_review: Overall, the paper presents a number of fascinating ideas and has a huge potential. At present, unfortunately, I believe that the balance between breadth and depth was not optimal, which does not allow me to recommend the paper for acceptance.

I believe that re-structuring the paper, either around its main technical achievement - adapting spiking networks to work in new circumstances, or around its main conceptual claim (biological plausibility) would greatly benefit the paper. In both cases, however, additional experiments would be needed to increase the impact of the contribution, and to more fully understand the properties of the proposed system.

** UPD **

After reading other reviews and the authors' responses, I leave my assessment unchanged. While some concerns were addressed, I believe that a more thorough reworking of the paper is necessary for it to be up to the standards of the ICLR conference.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a biologically plausible model-based RL approach that uses dreaming and planning to to efficiently use the learnt world model. The model comprises two recurrent spiking network modules, (i) to compute the policy to behave in an environment, and (ii) to learn to predict the next state of the environment given the past state and the action. The authors derive biologically plausible learning rules (local in both space and time) to train these modules. In the awake phase, the agent optimizes its policy and learns to predict the effects of its actions on the environment (i.e., both spiking network modules are updated). In the dream phase, however, the agent does not have access to the environment. Instead it uses its current model of the world to simulate experiences and uses these to optimize its policy. Planning is another approach used while the agent is in the awake phase to increase the data use to optimize the policy. Finally, the two-module spiking network is trained on a simplified version of the Atari pong game and the roles of dreaming and planning are evaluated.
strength_and_weaknesses: **Strengths**:
- This work addresses the challenge of developing an efficient, biologically plausible model based RL method. This is of high relevance for applications such as building efficient neuromorphic systems for autonomous robots, autonomous driving, etc. 
- Local learning rules are derived than enable online training of the recurrent spiking networks
- A concrete instantiation of dreaming is provided, and its benefit in learning is clearly demonstrated

**Weaknesses**:
The authors clearly state the limitations of their study:
- The approach is tested only on a simple game with direct access to states and a short time horizon
Nevertheless, it is a concrete step towards achieving biologically plausible and efficient model-based reinforcement learning.
 
clarity,_quality,_novelty_and_reproducibility: The paper is generally well-written and easy to follow. 
A few minor comments:
- Fig 1 is referred to instead of Fig 2 in a couple of lines on pages 6 and 7
- The caption for fig 3B refers to orange lines; it should be gray instead?
summary_of_the_review: The paper addresses the important problem of achieving biologically plausible and efficient model-based reinforcement learning. While the proposed network is evaluated only on a very simple task, it is nevertheless a concrete step towards achieving this goal. Overall, the paper is well presented and the proposed model is promising and could potentially lead to interesting avenues of future work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work introduces model-based reinforcement learning to the domain of spiking recurrent neural networks. To this end, the authors used two subnetworks – an “agent” one for computing policy and a “model” one for predicting future rewards and states of the environment. The authors then formulated local learning rules for their network and tested it on the ATARI Pong game where they observed higher sample efficiency compared to model-free spiking RNN approaches.
strength_and_weaknesses: The paper introduces an interesting and timely advance to the field of spiking neural networks by extending the existing RL approaches with a model-based part in that domain. The work is technically sound; the theory and the experiments are well-explained.

The weaknesses here are mostly specific not to this work, but rather to the field of spiking neural networks.

-The biological plausibility here is conditioned on the locality of the learning rules. This locality, in turn, is conditioned on the network being composed of one recurrent layer and one linear readout layer. Indeed, a similar locality of the learning rules can be achieved in a similarly structured non-spiking neural network train with backpropagation (if BPTT is not used).

-The task at hand (ATARI Pong, mostly with non-pixel input) is relatively simple, thus it can be solved with a relatively simple neural network. It is unclear whether this approach would scale to more-complex tasks normally requiring deeper networks.

-The paper mentions the potential for deploying the spiking RNN architecture in robots or on neuromorphic chips but for now these applications for this model remain theoretical. In practice, the model was trained on a conventional Python package and took much longer to train than a similar non-spiking model would.

Similar papers accepted to ICLR typically present a finding which could either be readily deployed in practice outperforming the existing approaches or could deepen our theoretical understanding of deep-learning models and/or biological objects. While the current work is interesting, well-conducted, and timely for its field, it may not fall into the scope of what’s usually accepted to ICLR.

clarity,_quality,_novelty_and_reproducibility: The text is well-written including a comprehensive introduction to the model, relevant literature, and the methods. The experiments are performed in accordance with what’s standard in the field of spiking neural networks. This makes the experiments reproducible.
summary_of_the_review: The paper presents a well-conducted study on adding model-based functionality to spiking RNN-based RL. Whereas the study is timely and interesting in the field, it may not deliver general enough results which could either extend our understanding of biological systems or be used in practice, so, currently, it may not be a sufficient contribution to the general ICLR community.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposed two strategies for biologically plausible learning rules; dreaming and planning. In Dreaming-Awake, the agent builds the world model. In the awake phase, the agent interacts with the environment, and in the dreaming phase, the agent interacts with the world model to update the weight. In the “planning” strategy, the agent interacts with both the world model and the environment.

strength_and_weaknesses: **[Strength]**

1. The idea of applying dreaming-awake in reinforcement learning is interesting.

**[Weakness]**

1. In Section 2.3, the authors mention supplementary material for derivation, but it is not submitted.

2. The authors only conduct experiments on Pong which is one of the most simple games on Atari. Although the authors argue that the proposed method is applicable to any Atari games, the actual experiment should be conducted.

3. In Section 4.1, what is the difference between an image-based task and a state-based task? Do the authors use the ROM version of the Atari environment, not the image version?

4. It seems that it is impossible to use dreaming and planning at the same time.

5. In Figure 1.D, how the background and the ball shape are changed during the dreaming phase?

6. The authors should compare with [Ref 1].

**[minor]**

1. Please differentiate \citep and \citet. It seems that all citations are \citet.

2. Typo:

1) In Section 3.2, all Fig.1 should be changed to Fig.2

2) In Section 1, “Okada & Taniguchi(2021) the authors” => Okada & Taniguchi(2021), the authors”

3) In Section 2.2, “membrane potential.The” => “membrane potential. The”

4) In Section 3.3, since dreams produces => since dreams produce.

[Ref 1] Ellis, Kevin, et al. "Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning." PLDI. 2021.


clarity,_quality,_novelty_and_reproducibility: There are several concerns regarding the clarity of the paper.

1. In Section 3.4, the authors mentioned 33600 pixels. Could you please explain why 33600 pixels? The most common setting might be 84 x 84 x 4 (stack) x 3 (RGB) = 84672 pixels (or 28224 with gray image).

2. In Section 2.1, h in $w_{ih}^{A,in}$ is not defined and in Section 2.2, $R_{ki}^\xi$ is not defined.

3. In Figures 2 and 3, the 80th percentile makes confusing. I recommend removing the dashed line.

4. How to measure the “compounding-error” in Section 3.3? Is it related to Figure 3. B?

5. The range of the reward of Pong is [-22, 22]. Is it normalized to -2 to 1?

summary_of_the_review: In general, although the idea of dreaming, awake, and planning is interesting, the experiment is only conducted in Pong and has many questions about results and settings.

-- Post Discussion Period --

After reading the authors' responses and other reviews, I maintain my rating. Although some concerns are resolved, there is still an unsolved concern and further analysis is needed including other environments, not just boxing and pong.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a spiking neural network model for reinforcement learning that utilizes two distinct modules, one an actor, and the other a model, which is used to learn transition dynamics for training the actor and planning. The authors show that the network can learn to play Pong and Boxing more rapidly than a network without the model module.

The reviewers were concerned about the limited tests provided, and precisely what the contributions of this paper are, given that there are other models that do improved RL via dreaming/planning, but at higher levels of performance and with more rigorous testing (e.g. https://arxiv.org/abs/2010.02193).

Based on the scores alone (average of 3.25) this is a fairly clear reject case. Moreover, my judgement as an AC is that the contributions of this paper are extremely limited, and can largely be summed up as seeing whether an approach that others have taken before can work with spiking networks trained with e-prop. In my assessment, this is an insufficiently important contribution to warrant publication at ICLR.
justification_for_why_not_higher_score: This paper does not provide any new theoretical insights, nor any clear empirical advances over previous approaches. It is at best a limited increase in biological plausibility of these models. I do not think this is enough for acceptance at ICLR.
Decision: Reject