Towards biologically plausible Dreaming and Planning
Authors: Cristiano Capone
Pier Stanislao Paolucci
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
Model based
Biologically Plausible

Abstract: Humans and animals can learn new skills after practicing for a few hours, while current reinforcement learning algorithms require a large amount of data to achieve good performances. 
Recent model-based approaches show promising results by reducing the number of necessary interactions with the environment to learn a desirable policy. However, these methods require biological implausible ingredients, such as the detailed storage of older experiences, and long periods of offline learning. The optimal way to learn and exploit word-models is still an open question.
Taking inspiration from biology, we suggest that dreaming might be an efficient expedient to use an inner model. We propose a two-module (agent and model) neural network in which "dreaming" (living new experiences in a model-based simulated environment) significantly boosts learning. We also explore "planning", an online alternative to dreaming, that shows comparable performances. Importantly, our model does not require the detailed storage of experiences, and learns online the world-model. This is a key ingredient for biological plausibility and implementability (e.g., in neuromorphic hardware).

Under review as a conference paper at ICLR 2023
TOWARDS BIOLOGICALLY PLAUSIBLE DREAMING AND PLANNING in recurrent spiking networks
Anonymous authors Paper under double-blind review
ABSTRACT
Humans and animals can learn new skills after practicing for few hours, while current reinforcement learning algorithms require large amount of data achieve good performances. Recent model-based approaches show promising re- sults by reducing the number of necessary interactions with the environment to learn desirable policy. However, these methods require biological implausible ingredients, such as the detailed storage of older experiences, and long periods of offline learning. The optimal way to learn and exploit word-models is still an open question. Taking inspiration from biology, we suggest that dreaming might be an efficient expedient to an inner model. We propose two-module (agent and model) spiking neural network in which "dreaming" (living new ex- periences in a model-based simulated environment) significantly boosts learning. We also explore "planning", an online alternative to dreaming, that shows compa- rable performances. Importantly, our model does not require the detailed storage of experiences, and learns online the world-model and the policy. Moreover, we stress that our network is composed of spiking neurons, furhter increasing the biological plausibility and implementability in neuromorphic hardware.
INTRODUCTION
Humans can learn new ability after practicing a few hours (c.g., driving or playing a game), while to solve the same task artificial neural networks require millions of reinforcement learning trials in virtual environments. And even then, their performances might be not comparable to human's ' abil- ity. Humans and animals, have developed an understanding of the world that allow them to optimize reinforcement
learning. This relies on the building of an inner model of the world. Model-bare learning Ye et al. (202 1); Abbeel et al. (chriitwwrr et al. (2020); Ha & Schmidhuber| (2018) (201 8);; Kaiser et al. (201 9); Hafner et al. (2020) have shown to reduce the amount of data required for learn- ing. However, these approaches do not provide insights on biological intelligence since they require biologically implausible ingredients (storing detailed information of experiences to train models, long off-line learning periods, expensive Monte Carlo three search to correct the policy). Moreover, the storage of long sequences is highly problematic on neuromorphic and FPGA platforms, where memory resources are scarce, and the use of an external memory would imply large latencies. The optimal way to learn and exploit the inner-model of the world is still an open question. Taking inspiration from biology, we explore an intriguing idea that learned model can be used when the neural network is offline. In particular, during deep-sleep, dreaming, and dry-dreaming.
Sleep known to be essential for awake performances, but the mechanisms underlying its cognitive functions are still to be clarified. A few computational models have started to investigate the interac- tion between sleep (both REM and NREM) and plasticity (González-Rudda (206)) 2018); Koorsak--oorrr and reorganized memories, in the after sleep network. wake-sleep learning algorithm has shown the possibility to extend the acquired knowledge with new symbolic abstractions and to train the neural network on imagined and replayed problems Ellis et al. (2020). However, a clear and coherent understanding of the mechanisms that induce generalized beneficial effects is missing. The idea that dreams might be useful to refine learned skill is fascinating and requires to be explored experimentally and in theoretical and computational models.
al. 2018);; Wei et al.) et al. (2000); Golosio et al. (202)) showing improved performances,

---

Under review as a conference paper at ICLR 2023
Here, we define "dreaming" as learning phase of our model, in which it exploits offline the inner- model learned while "awake". During "dreaming" the world-model replaces the actual world, to allow an agent live new experiences and refine the behavior learned during awake periods, even when the world is not available. We show that this procedure significantly boosts learning speed. This choice is biologically plausible alternative to experience replay Wang al. (20T6); Munos et al. (2016). which requires storing detailed information of temporal sequences of previous experi- ences. Indeed, even though the brain is capable to store episodic memories, it is unlikely that it is capable to sample offline from ten or hundreds of past experiences.
Usually, dreaming approaches are inefficient because of the large compounding errors associated to simulate long sequences in model- based simulated environment. In Ha & Schmidhuber (2018); Hafner et al. (2020); Okada & Taniguchi (202T), the authors have been able to train neural net- work only in its dreams. In Ha & Schmidhuber (2018) the authors used evolutionary methods to optimize the policy, that are not biologically plausible option, if compared to the reward based policy gradient. In Hafner et al. (2220) the authors simulated short sequences (T' 15), from ini- tial condition belonging to past experiences. We show that we are able to successfully train the agent in long simulated sequences (T 50). starting from random initial conditions. Allso, de- fined "planning", an online alternative to "dreaming", that allows to reduce the compounding-co eerorr by simulating online shorter sequences. However, this requires additional computational resources while the network is already performing the task. As stated above, learning the world-model usually requires the storage of an agent's 's experiences Ye al. (2021): 1); Abbeel al. (2006); /Schittwieser it al. (2020); Ha & Schmidhuber (2018); Kaiser et al. (2019): Hafner et al. (2020), to offline learn the model. We circumvent this problem by formulating an efficient learning rule, local in space and time, that allows learning online the world-model. In this way, no information storage (except the network parameters) is required.
For the sake of biological plausibility, we consider the most prominent model for neurons in the brain: leaky integrate-and-Ff neurons. But it an open problem how recurrent networks of spiking neurons (RSNNs) can learn Bellec et al. (2020); Muratore et al. (021): Capone et al. (2022). i.e., how their synaptic weights can be modified by local rules for synaptic plasticity so that the compu- tational performance of the network improves. Despite recent ad advances in this direction Bellec et al. 2020); Göltz et al. (2021); y Kheradpisheh & Masquelier (2020). there are yet very few applications to relevant tasks due to the difficulty to analytically tackle the discontinuity of the nature of the spike.
To our knowledge, there are no previous works proposing biologically plausible model-based rein- forcement learning in recurrent spiking networks. Our work is a step toward building efficient neuro- morphic systems where memory and computation are integrated in the same support, network of neurons. This allows to avoid the problem of latency when accessing to external storages, e.g. to perform experience replay, Indeed, we argue that the use of (1) spiking neurons, (2) on- line learning rules, (3) memories stored in the network (and not in an external storage), make our model, almost straightforward to be efficiently implemented in neuromorphic hardware.
awake
dreaming
awake
dreaming
Agent
Apent
model
model
enveonment
andernment
Figure 1: The agent-model spiking network. (A-B) Network structure: the network is composed of two modules: the agent and the model sub-networks, During the "awake" phase the two networks interact with the environment, learning from it. During the dreaming phase the environment is not accessible, and the model replaces the role of the environment. environment perceived by the network. (D) Example of the reconstructed environment during the dreaming phase.
(C) Example frame of the
2

---

Under review as a conference paper at ICLR 2023
2
METHODS
We propose a two-modules recurrent spiking network (See 1ig, one computing the policy to behave in an environment, and the other learning predict the next state of the environment, given the past states, and the action selected by the agent.
2.1 THE SPIKING NETWORK
Each module a (a for the agent network and, M for the model one) is composed of 500 neurons. Neurons are described real-valued variable v E R, where the i E {I....,N] N label identifies the neuron and
(1....,T) T) is discrete time variable. Each neuron exposes an observable state s. {0, 13, which represents the occurrence spike from neuron of the module at time t. We then define the following dynamics for our model:
82, exp
0,
+
exp
-
a,
vf
exp
a,
cxp
(
-aas + vrest) Wres a,i a.
3+1
θ
th
A is the discrete time-integration step, while тı and Tm are respectively the spike-filternn time con- stant and the temporal membrane constant. Each neuron is leaky integrator with recurrent filtered input obtained via synaptic matrix way (from neuron j to neuron i) and an external input current I Whes 20 accounts for the reset of the membrane potential after the emission of a spike. th = 0 and Urest = -4 are the threshold and the rest membrane potential. The input to the agent network is random projection of the state describing the world I'A, En Winh
A.i" 4, (where with A,in
randomly extracted from Gaussian distribution with zero mean and variance A,in 5). The input to the model network is a projection of the action chosen by the agent and of variables describ- M,inn& La*=h + En Wihh Sh (wit Mina and With M,in. are randomly extracted from Gaussian distribution with zero mean and variance respectively M,ina 5 and a M,,nn = 5). where we used the one-hot encoded action 1/'= at time t, which assumes the value 1 if and only if a' & (else it has value 0).
ing the world state / = >, With M.n.a
Finally, we define the following quantities, that are relevant for the learning rules described in the following sections. p',, is the pseudo-derivative (similarly to Bellec al. (2020); Capone et al. <d//ou
2022 pf Bn(e"]/5=+p) +1)2
(it peaks at of = and 8e defines its width) and e'.
the spike
Zwe
response function that can be computed iteratively as
ea. + -1-capp eep
(-Ar)
exp
(2)
Tm
Tm
2.2 LEARNING THE WORLD MODEL
We train the model-network predict the following state of the world g and reward yd given the current state & and the agent action a². The state t and reward yt are encoded as linear readouts & = >; R MM,0 x" = E, R of the spiking activity of the network, where s is a temporal filtering of the spikes s where At is the temporal bin of the simulation and the timescale of the filtering:
exp
At Tx
S. + (1 (1- exp
_
At
S
(3)
In order train the network to reproduce each time the desired output vector (St r*)), it is necessary to minimize the loss function:
B
Σ
(5t)
(l or Σ (ntt-rr)) r2)²
(4)
3

---

Under review as a conference paper at ICLR 2023
where cg 1.0 and Cr- 0.1 arbitrary coefficient. It is possible to derive the learning rules by differentiating the previous error function (similarly to what was done Bellee et al. (2020); Capone et al. ):022)):
Awy dE
= Ee c Ex Rss (5t = + c, (1ett1 ( rett)] Pres
dwiy
AR AR
x
dE
E. c (E c 8ha
(5)
dR
de
E, cr (par+) -t++) S.
dR
The resulting learning rule local both in space (the synaptic update only depends on the presy- naptic and postsynaptic neurons) and in time (the update at time t does not depend on future times). This allows to train online the recurrent spiking model-network.
2.3 LEARNING THE AGENT'S POLICY
The policy of defines the probability to pick the action a' = k at time t, and is defined as:
exp (v
Σ, exp (xf)
where, v/ - R a.t Following policy gradient approach to maximize the total reward ob- tained during the episode, is possible write the following loss function (see |Bllle et al. (2020); Sutton & Barto (2018)):
E^=-> Σ (o)(
where Rt = Ee> "0"""ttti is the total future reward (or return), and y = 0.99 the discount factor. Using the one-hot encoded action 1a'=x time t, which assumes the value and only a, (else it has value 0), arrive the following synaptic plasticity rules (the derivation is the same as Bellec et al. (2020), to which we refer for a complete derivation):
Aud
dE dud
Ex RR
Evsrrte (1ar=>- m1() phi
ART
dEA
E. Eesept (1 (La*=>-TH) A
aR
Intuitively, given trial with high rewards, policy gradient changes the network output y to increase the probability of the actions a² that occurred during this trial. Even in this case, the plasticity rule local both in space and time, allowing for an online learning for the agent-network. All the weight updates are implemented using Adam Kingma & Ba (2014) with default parameters and learning rate 0.001.
2.4 RESOURCES
The code to run the experiments written in Python 3. Simulations were executed on a dual-socket server with eight-core Intel(R) Xeon(R) E5-2620 v4 CPU per socket. The cores are clocked at 201GG with HyperThreading enabled, so that each core can run processes, for total of 32 processes per server. To reproduce each training curve the computation time is 1 2hours per realizations (10 realizations).
RESULTS
We considered a classical benchmark task for learning intelligent behavior from rewards Mnih et al. 2016): winning Atari video games provided by the OpenAl Gym Brockman al. (006)) (The MIT
4

---

Under review as a conference paper at ICLR 2023
License). We considered the case of a limited temporal horizon, T 100. In this case, in the best scenario the agent scores point and in the worst one the opponent scores 2 points. The world variables ç represent the paddles and ball coordinates. To win such game, the agent needs to infer the value of specific actions even if rewards are obtained in a distant future.
Indeed, Atari games still require large amount of computational resources and interaction with the environment to be solved. Only recently, model-based learning have shown to achieve efficient solutions for such tasks, see e.g. Ye al. (2021). This is why we introduce the model-based component to outperform the state of the art of reinforcement learning in recurrent spiking networks Bellec al. 2020
3.1 AWAKE
We defined an "awake" phase (see D in which our network interacts with the environment and plays a match. The dynamics of the agents and of the environment are defined by the following equations:
++1 = E, R
A. c++l G(a ++1 5k Sk
(9)
r+1 Fla 111
where the functions G(a G(a EL). F(a . are evaluated by the Atari environment. In this phase, the agents optimizes online its policy, updating its weights in agreement with cq At the same time, the netwrk. learns online to predict the effects of agent's actions on the environment (reward and world update) following 5
3.2 DREAMING
At the end of this phase, it follows the "dreaming" phase, in which the network is disconnected from the environment. The agent plays a match in its own mind (for temporal horizon of T 50), interacting with the nedel-network which replaces the world. In this phase, the agent updates its policy following the same update it would have in the awake phase (see eq while the model is not updated. The initial condition of the variables describing the world 5/ r" are extracted randomly, and their dynamics is defined by the equations:
E, R s4
c G(a +1 ,ft)= = Σ, Ri.
(10)
1dt F(ak ++1 Eti E, R
where now the functions G(a . and F(a .46 are estimated by the model-network.. this way, it is possible simulate Pong game (see 2A for example of simulated consecutive frames) in which the agent is able to test its policy and to improve it, even when the environment is accessible. In Bleff) it reported an example of the spiking activity in the two network (agent and model top and bottom respectively). In Fig B(right) it is shown an example of the output of the networks during one single dream. is interesting position estimated by the model-network, follows the policy computed by the agent-network (e.g., the y coordinate of the paddle increases when the probability to go up is high, green line).
that the change of the paddle
We run an experiment in which we alternate one game in the environment and one dream (see algorithm D. We report Fig (orange lines) the performances of the network as the average total reward
function of the number of interactions with the environment (dashed line, solid line and shading are the average, the 80th percentile and the standard error respectively, evaluated over 10 independent realizations of the experiment). We compare such performances with the ones obtained in the case in which the model does not dream between one (real) game and the other (see 2C black lines). We stress that this condition is equivalent to the approach used in Bellee et al. (2020), the state of the art for reinforcement learning in recurrent spiking networks, that is shown to achieve performances comparable to M3Ch et al. (016).. We observe that model allows
5

---

Under review as a conference paper at ICLR 2023
A
1.00
up
dreaming
down
neuron
10
0.75
0.500
0.25
0.00
200
model
* neuron
20
time
100
50
40
60
dreaming not dreaming
dreaming only
E
dreaming stop model
1.0
trprop RL
1.0
1.0
0.5
dreaming
0.5
0.0
reward (pona)
reward
-05
-1.0
-1.5
prop
-1.5
-1.5
-2.
-20
-2.0
20
40
60
20
60
frames (X500))
frames (X5020)
frames (x5000)
Figure 2: Dreaming. (A) Three consecutive frames of the reconstructed environment during dreaming phase. (B) Example of the spiking activity in the two sub-networks during dream (left). (right) Example, of the read out of the two sub-networks, representing the policy and the predicted position of the y paddle.
(C) Average (dashed line), standard error (shading), and 80th percentile (solid line) over 10 independent realizations of the achieved reward. Reward as a function of the number of interactions with the environment, with (orange) and without (black) the dreaming phase. Thin lines represent the single realizations. (D) Average (dashed line), standard error (shading) and 80th percentile (solid line) of the reward when the policy gradient is active only during sleep. (E) Same as in (D), but the update of the model is interrupted after 40 (x5000) interactions with the environment (blue dashed vertical line).
for sensible improvement of the learning speed, significantly reducing the number of interactions
with the environment required to achieve desirable performances.
To prove that the model actually learning during the dreaming phase, we made an experiment in which the policy gradient plasticity rule is active only during the dreaming phase (see Fig (D), showing the capability successfully learn win the game.
However, we note the importance to regularly to compare the model with the reality and to update it. Otherwise, the agent-network cannot improve indefinitely its policy in the simulated environment. We show that when we stop the update of the model, the policy doesn't improve anymore (see F. vertical blue line). This is probably due to the fact that after policy improvement, the agent has access new states configurations of the environment. E.g., at the beginning it scores few points, and it rarely observes positive rewards. As a consequence, the model-network is not capable predict positive rewards when it starts scoring more points.
Another contribution to this point, is online learning. This a choice we made as a constraint for biological plausibility. However, doing so, the model-network tends to forget situations that not observed since a long time. As an example, if the agent-networs is always scoring points, the model-network might forget situations in which negative reward is received.

---

Under review as a conference paper at ICLR 2023
Algorithm Dreaming
while iteration Niter do
Awake phase
play one game T 100 the real env.
update parameters of the agent-network {wi). Rigst
policy gradient D supervised
update parameters of the model-network (w)) Dreaming phase
M R kss R
play one game 7 50 in the simulated env.
update parameters of the agent-network {wiz, R2)
policy gradient
end while
A
planning
B
start planning
start planning
agent
1000
planning
model
200
planning
150
environment
experience
60
100
120
10me
c
planning not planning
D
1.0
1.00
0.75
0.5
dreaming
0.50
(pong)
0.0
0.25
-0.5
reward
0.00
-1.0
-0.25
awake only
-1.5
-0.50
-0.75
-2.0
1.00
20
30
10
20
30
40
50
frames (x5000)
Figure 3: Planning. (A) Sketch of the planning phase, the environment is always available. (B) Example of online "planning". during the observation the world (gray lines) the model used to predict fut steps in the future. (C) Average (dashed line), standard error (shading) and 80th percentile (solid line) of the reward as a function of the number of interactions with the environment, with (green) and without planning (black). (D) Average final reward function /ut--
3.3 PLANNING
We considered an online alternative use of the world model, that is "planning". Contrarily to the "dreaming". this is not separate phase. but acts in the awake phase (see Fig 3A). There many possible ways to use planning in reinforcement learning Wang et al. (2019). In this work, we use it, similarly to the dreaming, increase the data perform the policy gradient.

---

Under review as a conference paper at ICLR 2023
In principle, this approach should be much more accurate than "dreaming", since dreams produce long simulated games, possibly producing a large compounding-erae simulates few steps fut (see BB) in the future every At Atpred 2n f fut time steps in the real world (in order to have comparable data augmentation in planning and dreaming), minimizing the ef- fect of the compounding-errot However, "planning" requires online computational resources, while "dreaming" doesn't it since the extra computational cost is required offline, when the world would not be available anyway. For this reason, we report the performance obtained with and without planning for fut (see Fig Fi, respectively green and black). In Fig we show the average reward at the end of the training (average over the last 250 games, and over 10 independent realizations), as a function of fu. In F we observe that the performances obtained with dreaming and planning are comparable. This would be enough to prefer dreaming to planning, however it is possible that in other conditions (more complex games, longer temporal horizon) planning would show visible advantages.
On the contrary, "planning"
Of course the simulation of short sequences, starting from real-world initial conditions, could be done offline, but that would require the storage of detailed information (actions, states, rewards) about the "awake" experience, to define the initial condition of such sequences. This is something that is not biologically plausible, and that cannot be done in a one-shot learning fashion with current learning rules. For these reasons, we discard this approach.
3.4
GENERALIZABILITY OF THE APPROACH
3.4.1 PONG FROM PIXELS
The task described above is a now-dimensional task from the relevant variables of the pong environ- ments (ball and paddles positions). To show that our framework can be extended to other conditions, we run an additional experiment. We evaluated the Atari Pong from pixels. In order to do so, we opted for simple approach: took the black/white frames of the pong game, and randomly pro- jected all the 33600 pixels (x",, i == 1, 1...30000 into a small number of variables (D = 4 in our case). As a result, the world variables 4h (as described above) are a random combination of the 33600 33600 FEATAA where Fsn Gaussian weights with zero mean and 0.1 variance, h=1
pixels, 5k c++1 and h: = D. In other words, we randomly defined a low dimensional latent space. We observed results comparable to what shown in Fig(p. The approach described above allows, in principle, to apply our method to any other Atari games from pixels.
3.4.2 BOXING FROM PIXELS
When considering Boxing from pixels, we observe that dreaming and planning does not im- prove agent performances in the random latent space, as it is defined in the previous sec- tion. For this reason we used simple autoencoder to define the world variables, such that <(* = G(E)) (where G represents the encoder part of the autoencoder, and vector whose components are <t 1) The decoder is only used for visual purposes, to reconstruct the pre- dicted future frames (see Fig 4, bottom). The autoencoder is simply defined as a 3 (encoder) +3 3 (decoder) layer (33600, (33600,128,64,36) 64,66,,,,,,,,,18, 33600) feed forward network, trained on 10000 frames collected from the environment using a random policy. In this case, the dimensionality of the latent space describing the world was 36. We show that planning, in this latent space, shows performances advantages (see Fig A)).
CONCLUSION
We presented a two-module spiking network, capable of performing two alternative versions of model-based biologically plausible reinforcement learning: "dreaming" and "planning". We demon- strate that the simulated world observation produced by the model-network, provide valuable data for the agent-network to improve its policy, significantly reducing the necessary number of interac- tions with the real environment. A major goal of our model, is to achieve a high level of biological plausibility. For this reason, propose plasticity learning rules that are local in space and time. This is a major requirement in biology, since synapses have only access to local information (c.g., the activities of the pre- and post-synaptic neuron and the reward at that time). Moreover, this al-
8

---

Under review as a conference paper at ICLR 2023
A
B
3.25
3.00
2,75
reRards (pong)
2.50
2.25
2.00
0.5
1.0
1.5
2.0
number planning steps
Figure 4: (A) Average reward over 10 realizations, as a function of number of planning steps. (B) Top. Three consecutive frames in the environment. Bottom. Three consecutive reconstruced predicted frames.
lows for an online training of our networks. Usually, training the world-model requires the storage of an agent's experiences Ye et (2021): Abbeel al. (2006); /Schrittwiserr al. (2020); ; Ha & Scchiihhubrr (2018); Kaiser et al. (2019; Hafner et al. 2020) to offline learn the model. Besides being implausible, storage of long sequences highly problematic on neuromorphic and FPGA platforms, where memory resources synapses describing the model. The storage of a temporal sequence would require an order O(T) memory (where T is the number of temporal steps), hence saturating the available memory. A typical solution to store externally the temporal sequences, but that would imply a latency of order O(T) since the available bandwidth for memory access is much lower than the used exchange sig- nals internal to the network. In our case, the possibility to learn online, makes it unnecessary to store any information and "planning". Dreaming, simulates long sequences in the model-basee simulated environment. Usually, this approach fortably performed offline. On the other hand, planning simulates shorter sequences, but requires computation online, while the network already performing the task. The comparability of per- formances for planning and dreaming (for the same amount of simulated generated data) underlines the importance of the "dreaming" learning phase in biological and artificial intelligence. Our model provides proof of concept that even small and cheap networks with online learning rule can learn and exploit world models to boost learning. Our work is a step forward, towards building effi- cient neuromorphic systems for autonomous robots, to efficiently learn in real world environment. "Dreaming" intervenes when the robot is no longer able to explore the environment (it not acces- sible anymore, the robot low energy). In this condition, the robot optimizes learning by reasoning in its own mind. These approaches are of great relevance when the acquisition from the environment slow, expensive (robotics) or unsafe (autonomous driving).
scarce, and in general dedicated to the storage of the
the agent's experience. We observe an interesting tradeoff between "dreaming"
efficient because the large compounding errors, but can be com-
4.1 LIMITATIONS OF THE STUDY
This study, shows a promising biologically plausible implementation efficient model-based re- inforcement learning, proving that it is possible to speed up learning in biological- -asee networks. However, we considered a specific task with limited temporal horizon (T = 100). We plan to test our approach more complex and long-horizon tasks. Even though we are capable to show the effectiveness of our model different task (atari Pong and Boxing) and conditions (from RAM and from pixels), it remains crucial to look at how the performance scales with the size of the network and the dimensionality of the problem. A possible generalized task can take inspiration from what is used in human experiments with multiple learning tasks Osman (2010). That would be a good testbed to see how the performance scales with the complexity of the task, and how they compare to human performances.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Pieter Abbeel, Morgan Quigley, and Andrew Y Ng. Using inaccurate models in reinforcement learning. In Proceedings of the 23rd international conference on Machine learning, pp. 1-8, 2006.
Guillaume Bellec, Franz Scherr, Anand Subramoney, Elias Hajek, Darjan Salaj, Robert Legenstein, and Wolfgang Maass. A solution to the learning dilemma for recurrent networks of spiking neu- rons. Nature communications, 1(1):1-15, 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Cristiano Capone, Paolo Muratore, and Pier Stanislao Paolucci. Error-based or target-based? unified framework for learning in recurrent spiking networks. PLoS computational biology, 18 (6):e100221,, 2022.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: Growing gen- eralizable, interpretable knowledge with wake-sleep bayesian program learning, arXiv preprint arXiv:2006.0888/ 2020.
Bruno Golosio, Chiara De Luca, Cristiano Capone, Elena Pastorelli, Giovanni Stegel, Gianmarco Tiddia, Giulia De Bonis, and Pier Stanislao Paolucci. Thalamo-cortical spiking model of incre- mental learning combining perception, context and nrem-sleep. PLoS Computational Biology, 17 16):e400 2021.
Julian Göltz, Laura Kriener, Andreas Baumbach, Sebastian Billaudelle, Oliver Breitwieser, Ben- jamin Cramer, Dominik Dold, Akos Ferene Kungl, Walter Senn, Johannes Schemmel, et al. Fast and energy-efficient neuromorphic deep learning with first-spike times. Nature machine intelli- gence, 3(9) 835, 2021.
Ana Boanzález-Rudda. Victor Pedrosa, Rachael C Feord, Claudia Clopath, and Ole Paulsen. Activity- dependent downscaling of subthreshold synaptic inputs during slow-wave-sleep-lik activity in vivo. Neuron, (7(6))1244-111 2018.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02 193, 2020.
Lukasz Kaiser, Mohammad Bahaeizadeh, Piotr Milos, Blažej Osiáski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2019.
Saeed Reza Kheradpisheh and Timothée Masquelier. Temporal backpropagation for spiking neural networks with one spike per neuron. International Journal of Neural Systems, 30(06):2050027, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14/2.6994 2014.
A Koresak-Gorzo, MG Müller, A Baumbach, I. Leng, OJ Breitwieser, SJ van Albada, W Senn, K Meier, Legenstein, and MA Petrovici. Cortical oscillations implement a backbone for sampling-based computation in spiking neural networks. arXiv preprint arXiv:2006.1109) 2020.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. International conference on machine learning, pp. 1928 1928-1937. PMLR, 2016.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Mare Bellemare. Safe and efficient off- policy reinforcement learning. arXiv preprint 160iv:1606.000 2016.
10

---

Under review as a conference paper at ICLR 2023
Paolo Muratore, Cristiano Capone, and Pier Stanislao Paolucci. Target spike patterns enable efficient and biologically plausible learning for complex temporal tasks. Plos one, 16(2):e0247014, 2021.
Masashi Okada and Tadahiro Taniguchi. Dreaming: Model-based reinforcement learning by latent imagination without reconstruction. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 4209-4215. IEEE, 2021.
Magda Osman. Controlling uncertainty: review of human behavior in complex dynamic environ- ments. Psychological bulletin, 136(1):65, 2010.
Julian Schrittiiser, loannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604 609, 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforce- ment learning. arXiv preprint urXiv:1907.0200 2019.
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv preprint
arX1::161..0122. 2016.
Yina Wei, Giri P Krishnan, and Maxim Bazhenov. Synaptic mechanisms of memory consolidation during sleep slow oscillations. Journal of Neuroscience, 36(15):4231 5):4211-4247,, 2016.
Yina Wei, Giri P Krishnan, Maxim Komarov, and Maxim Bazhenov. Differential roles of sleep spindles and sleep slow oscillations in memory consolidation. PLoS computational biology, 14 (7):11006322, 2018
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data, Advances in Neural Information Processing Systems, 34, 2021.


Reviews:
[{"review_id": "lrWddbcFLam", "reviewer": "Reviewer_w9LE", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed two strategies for biologically plausible learning rules; dreaming and planning. In Dreaming-Awake, the agent builds the world model. In the awake phase, the agent interacts with the environment, and in the dreaming phase, the agent interacts with the world model to update the weight. In the “planning” strategy, the agent interacts with both the world model and the environment.\n", "summary_of_the_review": "In general, although the idea of dreaming, awake, and planning is interesting, the experiment is only conducted in Pong and has many questions about results and settings.\n\n-- Post Discussion Period --\n\nAfter reading the authors' responses and other reviews, I maintain my rating. Although some concerns are resolved, there is still an unsolved concern and further analysis is needed including other environments, not just boxing and pong.", "strength_and_weaknesses": "**[Strength]**\n\n1. The idea of applying dreaming-awake in reinforcement learning is interesting.\n\n**[Weakness]**\n\n1. In Section 2.3, the authors mention supplementary material for derivation, but it is not submitted.\n\n2. The authors only conduct experiments on Pong which is one of the most simple games on Atari. Although the authors argue that the proposed method is applicable to any Atari games, the actual experiment should be conducted.\n\n3. In Section 4.1, what is the difference between an image-based task and a state-based task? Do the authors use the ROM version of the Atari environment, not the image version?\n\n4. It seems that it is impossible to use dreaming and planning at the same time.\n\n5. In Figure 1.D, how the background and the ball shape are changed during the dreaming phase?\n\n6. The authors should compare with [Ref 1].\n\n**[minor]**\n\n1. Please differentiate \\citep and \\citet. It seems that all citations are \\citet.\n\n2. Typo:\n\n1) In Section 3.2, all Fig.1 should be changed to Fig.2\n\n2) In Section 1, “Okada & Taniguchi(2021) the authors” => Okada & Taniguchi(2021), the authors”\n\n3) In Section 2.2, “membrane potential.The” => “membrane potential. The”\n\n4) In Section 3.3, since dreams produces => since dreams produce.\n\n[Ref 1] Ellis, Kevin, et al. \"Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning.\" PLDI. 2021.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "_mv011WWyVQ", "reviewer": "Reviewer_k79x", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces model-based reinforcement learning to the domain of spiking recurrent neural networks. To this end, the authors used two subnetworks – an “agent” one for computing policy and a “model” one for predicting future rewards and states of the environment. The authors then formulated local learning rules for their network and tested it on the ATARI Pong game where they observed higher sample efficiency compared to model-free spiking RNN approaches.", "summary_of_the_review": "The paper presents a well-conducted study on adding model-based functionality to spiking RNN-based RL. Whereas the study is timely and interesting in the field, it may not deliver general enough results which could either extend our understanding of biological systems or be used in practice, so, currently, it may not be a sufficient contribution to the general ICLR community.", "strength_and_weaknesses": "The paper introduces an interesting and timely advance to the field of spiking neural networks by extending the existing RL approaches with a model-based part in that domain. The work is technically sound; the theory and the experiments are well-explained.\n\nThe weaknesses here are mostly specific not to this work, but rather to the field of spiking neural networks.\n\n-The biological plausibility here is conditioned on the locality of the learning rules. This locality, in turn, is conditioned on the network being composed of one recurrent layer and one linear readout layer. Indeed, a similar locality of the learning rules can be achieved in a similarly structured non-spiking neural network train with backpropagation (if BPTT is not used).\n\n-The task at hand (ATARI Pong, mostly with non-pixel input) is relatively simple, thus it can be solved with a relatively simple neural network. It is unclear whether this approach would scale to more-complex tasks normally requiring deeper networks.\n\n-The paper mentions the potential for deploying the spiking RNN architecture in robots or on neuromorphic chips but for now these applications for this model remain theoretical. In practice, the model was trained on a conventional Python package and took much longer to train than a similar non-spiking model would.\n\nSimilar papers accepted to ICLR typically present a finding which could either be readily deployed in practice outperforming the existing approaches or could deepen our theoretical understanding of deep-learning models and/or biological objects. While the current work is interesting, well-conducted, and timely for its field, it may not fall into the scope of what’s usually accepted to ICLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "Z5l1Gj33hA_", "reviewer": "Reviewer_7i2g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a recurrent spiking network-based reinforcement learning algorithm, in which the network consists of two major parts (an agent and a model). The agent is trained to act using policy gradient, while the model is trained to predict the effect's of the agent's actions (both the reward and the state transition). The authors apply their approach to the game of Pong, showing that dreaming indeed improves performance. They also consider an alternative to dreaming (namely, planning), which showed similar performance.", "summary_of_the_review": "Overall, the paper presents a number of fascinating ideas and has a huge potential. At present, unfortunately, I believe that the balance between breadth and depth was not optimal, which does not allow me to recommend the paper for acceptance.\n\nI believe that re-structuring the paper, either around its main technical achievement - adapting spiking networks to work in new circumstances, or around its main conceptual claim (biological plausibility) would greatly benefit the paper. In both cases, however, additional experiments would be needed to increase the impact of the contribution, and to more fully understand the properties of the proposed system.\n\n** UPD **\n\nAfter reading other reviews and the authors' responses, I leave my assessment unchanged. While some concerns were addressed, I believe that a more thorough reworking of the paper is necessary for it to be up to the standards of the ICLR conference.", "strength_and_weaknesses": "Strengths:\n- The paper addresses a number of highly relevant and important problems\n- Using spiking networks for reinforcement learning is fundamentally challenging, but the authors manage to successfully handle the task\n- The authors consider an expansion of their method to image-based inputs\n\nWeaknesses:\n- The paper focuses on many targets at once, such as biological plausibility, applicability in robotics, suitability to neuromophic hardware, as well as general usefulness of the approach as a general-purpose reinforcement learning algorithm. I believe that while some of these goals are correlated, they are not the same. As a result, the paper produces an unfocused impression. I believe that narrowing down the range of claims and delving deeper into one of these topics would greatly benefit the paper.\n- The range of applications considered is limited. The method works reasonably well when applied to problems that are fundamentally extremely low-dimensional, but it's not clear whether more complex systems can be managed in a similar manner. While the authors say that \"The approach described above allows, in principle, to apply our method to any other Atari games from pixels\", it remains unclear whether this suggestion is practical.\n-  The novelty of the approach is limited. While the use of spiking neural networks in this specific context is novel, the idea of using model-based simulations or dreaming to enhance learning is very well known. Again, I believe that additional, more focused results might help to mitigate the issue.\n\nI clarify and expand upon some of these points in the main body of my review.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N64mtuUJd1", "reviewer": "Reviewer_MjQE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a biologically plausible model-based RL approach that uses dreaming and planning to to efficiently use the learnt world model. The model comprises two recurrent spiking network modules, (i) to compute the policy to behave in an environment, and (ii) to learn to predict the next state of the environment given the past state and the action. The authors derive biologically plausible learning rules (local in both space and time) to train these modules. In the awake phase, the agent optimizes its policy and learns to predict the effects of its actions on the environment (i.e., both spiking network modules are updated). In the dream phase, however, the agent does not have access to the environment. Instead it uses its current model of the world to simulate experiences and uses these to optimize its policy. Planning is another approach used while the agent is in the awake phase to increase the data use to optimize the policy. Finally, the two-module spiking network is trained on a simplified version of the Atari pong game and the roles of dreaming and planning are evaluated.", "summary_of_the_review": "The paper addresses the important problem of achieving biologically plausible and efficient model-based reinforcement learning. While the proposed network is evaluated only on a very simple task, it is nevertheless a concrete step towards achieving this goal. Overall, the paper is well presented and the proposed model is promising and could potentially lead to interesting avenues of future work.", "strength_and_weaknesses": "**Strengths**:\n- This work addresses the challenge of developing an efficient, biologically plausible model based RL method. This is of high relevance for applications such as building efficient neuromorphic systems for autonomous robots, autonomous driving, etc. \n- Local learning rules are derived than enable online training of the recurrent spiking networks\n- A concrete instantiation of dreaming is provided, and its benefit in learning is clearly demonstrated\n\n**Weaknesses**:\nThe authors clearly state the limitations of their study:\n- The approach is tested only on a simple game with direct access to states and a short time horizon\nNevertheless, it is a concrete step towards achieving biologically plausible and efficient model-based reinforcement learning.\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject