Controlled maximal variability along with reliable performance in recurrent neural networks
Authors: Chiara Mastrogiuseppe
Rubén Moreno-Bote
Venue: NeurIPS 2024 poster (2024)
Keywords: Reinforcement Learning
Computational Neuroscience
Neural Variability
Recurrent Neural Network
Maximum Occupancy Principle
Maximum Entropy Reinforcement Learning
Primary area: neuroscience_and_cognitive_science
TL;DR: Maximizing cumulative future action entropy allows recurrent neural networks to perform tasks while maximizing variability.

Abstract: Natural behaviors, even stereotyped ones, exhibit variability. Despite its role in exploring and learning, the function and neural basis of this variability is still not well understood. Given the coupling between neural activity and behavior, we ask what type of neural variability does not compromise behavioral performance. While previous studies typically curtail variability to allow for high task performance in neural networks, our approach takes the reversed perspective. We investigate how to generate maximal neural variability while at the same time having high network performance. 
To do so, we extend to neural activity the maximum occupancy principle (MOP) developed for behavior, and refer to this new neural principle as NeuroMOP. NeuroMOP posits that the goal of the nervous system is to maximize future action-state entropy, a reward-free, intrinsic motivation that entails creating all possible activity patterns while avoiding terminal or dangerous ones.
We show that this goal can be achieved through a neural network controller that injects currents (actions) into a recurrent neural network of fixed random weights to maximize future cumulative action-state entropy. 
High activity variability can be induced while adhering to an energy constraint or while avoiding terminal states defined by specific neurons' activities, also in a context-dependent manner. The network solves these tasks by flexibly switching between stochastic and deterministic modes as needed and projecting noise onto a null space. Based on future maximum entropy production, NeuroMOP contributes to a novel theory of neural variability that reconciles stochastic and deterministic behaviors within a single framework.

Controlled maximal variability along with reliable performance in recurrent neural networks
Chiara Mastrogiuseppe
Center for Brain and Cognition, Department of Engineering
Universitat Pompeu Fabra
Barcelona, Spain
chi thiara.oe www.aaaaaarrsuupppepappppa edu
Rubén Moreno-Bote
Center for Brain and Cognition, Department of Engineering,
Serra Húnter Fellow Programme
Universitat Pompeu Fabra
Barcelona, Spain
ruben. muben.morenooupee edu
Abstract
Natural behaviors, even stereotyped ones, exhibit variability. Despite its role in ex- ploring and learning, the function and neural basis of this variability is still not well understood. Given the coupling between neural activity and behavior, we ask what type of neural variability does not compromise behavioral performance. While previous studies typically curtail variability to allow for high task performance in neural networks, our approach takes the reversed perspective. We investigate how to generate maximal neural variability while the same time having high network performance. To do so, we extend to neural activity the maximum occupancy principle (MOP) developed for behavior, and refer this new neural principle NeuroMOP. NeuroMOP posits that the goal of the nervous system is to max- imize future action-state entropy, a reward-free, intrinsic motivation that entails creating all possible activity patterns while avoiding terminal or dangerous ones. We show that this goal can achieved through a neural network controller that injects currents (actions) into recurrent neural network of fixed random weights to maximize future cumulative action-state entropy. High activity variability can be induced while adhering defined by specific neurons' activities, also in context-dependent manner. The network solves these tasks by flexibly switching between stochastic and determin- istic modes as needed and projecting noise onto a null space. Based on future maximum entropy production, NeuroMOP contributes to novel theory of neural variability that reconciles stochastic and deterministic behaviors within a single framework.
energy constraint while avoiding terminal states
Introduction
From opening door to crossing the street, everyday life hinges reliably executing actions. Despite that, natural behaviors, including repetitive movements from expert athletes [ 2
3 4 exhibit
variability [5] 6 77. The mechanisms governing the emergence of this variability from the central and
peripheral nervous systems remain unclear. Variability of neural activity in motor cortex subcortical structures
g [ I0 and
[2] controlling muscle movements, well as state changes of the effectors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

due to, e.g,, fatigue [3] 14], might contribute to the observed behavioral variability, However, as behavioral variability is also observed at longer time scales during perception [15] [6], decision
making 17 6 5. 18 and planning [1], neural fluctuations in most parts of the brain [20 21 22
might be involved in the generation of variable behavioral repertoires as a whole.
Several mechanisms have been put forward for the generation of neural variability, including synaptic noise within neural circuits [23 24 or non-linear network interactions leading to variable activity
patterns 125 26
27 28 These proposed mechanisms are predominantly designed to describe
variability during spontaneous activity in the absence of sensory stimuli - [22 20 or in response to
simple stimuli [29 30) most frequently outside a complex task. In other theoretical studies where the
goal is to maximize network performance in a task, variability is typically suppressed after learning
31 32 33.. If anything, the initial noise or activity variability is used as means to regularize and
promote exploration during learning [4 35 but they are considered to be unnecessary thereafter. This approach is also the one taken in state-of-art reinforcement learning, where variability is added during learning. but during task execution policies are forced to be deterministic [36 37].
Given these two coexisting sides of natural behavior namely, high task performance in spite of large variability - we ask whether it is possible to have neural networks generating maximal variability while at the same time being able to flexibly switch to deterministic behavioral modes when needed. We surmise that generating neural variability is a fundamental goal of the nervous system, as it enables the exploration of its entire dynamical range. This idea parallels the one that neural activity should be variable to generate the vast behavioral repertoires (38
as the The sought neural variability needs to be highly structured in order to avoid non-adaptive behaviors. One relevant example comes from reaching tasks, where neural activity is indeed found to be confined in null spaces as to avoid undesired movements [43] To address the above question, we build on the maximum occupancy principle (MOP) developed for behavior [44 45). which posits that agents ought to occupy action-state space by generating all sorts of action-state paths compatible with the dynamical and environmental constraints. Applied to neural activity, we introduce NeuroMOP, which puts forward the hypothesis that the brain should generate maximum entropy in the neural activity paths. Importantly, this entails avoiding the terminal states where further entropy cannot be generated. This principle aligns with broad body of Reinforcement Learning (RL) literature on reward-free algorithms based on purely-entropp objectives (37 44) (4]. By optimizing the cumulative sum of future action-state entropy, NeuroMOP emphasizes seeking future variability to the extent that does not compromise performance. By properly defining terminal states as absorbing states where more entropy can be generated, this principle seeks variability while also generating behaviors that guarantee future 'survival'.
ones empirically observed [5, I8 42 I1.
In this paper, we employ random recurrent neural networks (RNNs) of fixed weights as a simplified representation of brain dynamies, and we let them interact with a stochastic input current generator following MOP (Fig. 1). The input current generator (the agent) is designed to maximize the entropy hence, the variability - of the series of currents (a function of the actions) it injects in the RNN (the environment). As expected, variable currents lead to variable neural activities, but this variability becomes structured in order to avoid dangerous (terminal) states. To bridge neural variability with functionality, we test our architecture in a series of different problems. First, show that the NeuroMOP network learns to satisfy energy constraints, while generating large neural variability. Second, a subset of RNN neuron activities can be confined within complex regions while remaining free within the region, thereby 'drawing' different symbols, also in a context dependent manner. Crucially, the NeuroMOP network not only learns effectively solve tasks, but it maintains a high dimensionality of action signals whenever possible, allowing the visitation of a wide range of activity patterns. By flexibly reducing its dimensionality when close to terminal states, we show that low dimensionality is an emergent property under constrained tasks where higher dimensionality is the default mode.
2 Methods
2.1 NeuroMOP architecture: controller and RNN
The NeuroMOP architecture consists of a controller (agent) injecting currents into an RNN of fixed random weights (environment) (Fig. I The state I E R of the RNN, where I; is the activity of
2

---

Interaction
Forward Prediction
A (al) (a."..... ................................ w....eet
π xx) 27 2o aoreeeooooo bnveroodd Pledlr
O O
V
log
Σ et Satteeaettss PEA
Votin.e)
O V(a,w)
Cost Crost
Figure I: Schematic of the NeuroMOP network. At each time step the controller reads the activity (state) of the RNN and samples action from the policy π (--12)). Using the optimal policy in Eq. requires predicting the effect that each of the possible actions a E A(x)) would have the state of the RNN by computing the successor state x' (1x, (*)) and then evaluating the corresponding value function V (a' () in that state. The value function is approximated by feedforward network (FFN). Once sampled, the now-dimensional action a is expanded and transformed into currents / via matrix K and fed into the RNN. Next, the RNN state evolves one time step and the loop is repeated. The weights of the FFN are trained via gradient descent using as cost function the Bellman error stored along batch of trajectories.
neuron i 1, I,.... N, follows the dynamics
N Σ
Ty (1 + = xz + 5t
+ +
Jayny + T,(1)
where ((-) is non-linear transfer function, 5t is the integration time step of the dynamics, and (t(t) are currents injected by the controller in the RNN neurons. The recurrent connections of the RNN are fixed and sampled from normal distribution, Jij ~ NO00 g² /N). We use saturating transfer function (.)) = tanh(-), which leads to chaotic dynamics when the internal recurrent connections are strong enough (25 (see Fig. 2h). Results for RNNs with non-saturating (ReLU) transfer functions
are shown Appendix F
At time t. the controller samples random action a from state-dependen stationary policy x x--(z)) The action an M M-dimensional discrete vector A(z); we will consider below the presence of terminal states æt, where the number of available actions is drastically reduced. Based on the generated action a, the controller injects into the i-th neuron the current
4(1) =
Kuay
where K5 ~ UO,, 1) are positive input weights sampled from uniform distribution and p is parameter that scales the strength the current. Thus, internal actions are expanded via the random matrix K into N N-dimensional currents. This expansion allows to study the harder problem of controlling the RNN's dynamics using actions with reduced dimensionality. ie., M < N, but our framework also works for projections, M N.
The controller follows MOP, that is, it aims occupying action-state path space Here we restrict ourselves to action paths, and show the general action-state framework in Appendix [ We assume that the controller gets an intrinsic reward of In π ((a)z) (x)) for generating action when the network is in state x at time I, being the largest when the generated action has low probability under the current policy æ. The controller does not greedily maximize this immediate intrinsic reward every time step. Instead, the policy is chosen maximize the value function, defined as the expected

---

discounted sum of future intrinsic rewards
V (z) = BrNnip
> (
Br Br
(3)
t=0
with discount factor < < Note that this expression takes the form of a sum of future action entropies, where (())))))) = A())(s) =(a[z) In (a]xx) The expectation is over all paths with initial condition z:(0) generated by sampling actions from the policy π and following the state transitions probability p = p(x((t + ())= a(t)) defined by Eq. I By virtue of maximizing action path entropy, the MOP agent occupies current and future action space as broadly as possible [44] 45). Discounted cumulative future action entropy stands as the only measure of occupancy that adheres to the intuitive notion that the occupancy over an action path is the sum of the occupancies over any of its subpaths, the so-called additive property [44].
(x, (()))())))).)))))........)).)...
The optimal policy x² (144) Appendix C maximizing the value function is
e² Mlrlaaaa)))
x <a(a))) ((x)
(4)
where 7(x) = EleA(s) e E.' pla' (x²))""(x) is the partition function and (*(x) is the optimal value following the optimal policy, defined as
(x) In ((x) = In Σ
(5)
aEA((z
In our specific implementation with deterministic dynamics (Eq. D. the transition probability p(a"|az a) is a delta function, and so the successor state x' is uniquely determined by the current state x and the action a, x' x'(x, a). Our algorithms work well also for RNNs with noisy dynamics (Appendix EE.
We will define different problems by choosing specific state-dependent action sets, so that the available set of actions a depends on E A(z). Specifically, we define terminal states, denoted x!, as absorbing states the network cannot escape from and where doing nothing is the only available action. These terminal states might represent detrimental state regions, e.g., too high neural activity, or other adverse activity patterns resulting in significant external penalties, such as the falling of an agent the floor. With doing nothing being the only action, entering a terminal state is an irreversible process leading to an intrinsic reward of always zero from that point onwards. i.c., In 1 0, as no further action entropy can be generated. Therefore. by definition, V₂ (x*) = for any policy. Terminal states can be considered 'dead' states of the network, and they will be naturally avoided to keep maximizing future action path entropy. In our implementations, non-terminal states share the same action set of M -dimeenioonal binary actions ag E {-1, } Vk = I,..., M.. This does not imply that all non-terminal states are equally desirable; via the computation of the value function, the network naturally exhibits less preference for 'bad' states that increase the likelihood of encountering terminal states in the future. We will show that the structure of terminal regions, along with the network dynamics, leads to complex, rich, variable behaviors without the need to specify an extrinsic reward function. In essence, MOP tells agents what not to do, and thus it does not restrict behavior. In contrast, standard extrinsic reward maximization tells agents what to do, inevitably limiting behavior. Note that maximizing action path entropy entails striking balance between maximizing immediate and future entropy, with behaviors that can become locally very deterministic if this globally opens up larger repertoires of possible action courses, i.e., larger future action entropy.
((a = nothing/z!)
2.2 Value function approximator by minimizing the Bellman error
As our problem involves a high-dimensional continuous state space (N = 100), we use a feed- forward network (FFN) to approximate the optimal value function V (x) in Eq. with V(x, w). The FFN with parameters w consists of one hidden layer of NAid neurons, an input layer with input the activities x of the RNN, and one single output neuron with activity V(z,, w) (Fig. 1).
In order to optimize the parameters of the FFN, we consider Va( w). the expected evolution of the approximated value function satisfying the Bellman consistency equation, defined as
Vg(a,, w) = In Σ oEA(z)
e Ex p(x' (r,o))()
(6)

---

If the approximated value function V(z, w) were equal to the optimal value (*)),, its expected evolution would coincide with the value function itself, ie.. Vis(t) (+) (see Eq. 5 Thus, we optimize the weights of the FFN by minimizing loss function, defined as the summed squared errors between Vi(() and ((, w) over trajectories in each epoch I 1, ,,..., Nop.
Naras
(((w)
1
wi)
N Ntraj
end t=1
where the squared error is accumulated over a batch of Ntraj paths (x(*) (0), (1)... xl) (t).... and the path lifetime ( is the minimum between the time when the network reaches a terminal state and the maximum episode duration T'. Each path is generated by sampling actions from the policy in Eq. 4 with the same initial condition x(0) - x. The policy depends on the specific values of the FFN weights at epoch I. The parameters of the network are updated each epoch I using Adam as optimizer.
Our results have proven to stable also for FFNs only receiving as an input Nap < N activities randomly selected from the N didimensional state, denoted ø. If needed, the input can be extended to include any required extra-information or E can be constrained to specific neurons. For instance, in the context-dependent constraints problem defined in Sec. [3..1] we added extra units the FFN input layer to flag (via a one-hot vector) the context. Including in the readout neurons (I1, T2) improved stability and performance.
2.3 The reward-maximizing network
To provide a comparison for the NeuroMOP network, we introduce the R network, which aims at maximizing the discounted sum of future extrinsic reward. To ensure fair comparison, incorporate the notion of survivability present in MOP by assigning to the R network in state æ and taking action the extrinsic reward
r(z,0) = {1 if x'x # x if a' =
8
where (xx, a) the state evolution of the RNN as defined in Eq. 1 To allow the generation of variable trajectories also by this network, stochasticity in the action selection is implemented by endowing the network with an e-greedy policy defined as
arrgiax V. (x' ()) with probability 1-6
a (-(x))
random
with probability c
where (x( is the expected future cumulative reward when following this policy, which can be written recursively
Ve(x)= = Eaws, [r(x, a) + VV()'))
(10)
In this framework is hyperparameter controlling the amount of (random) action variability generated by the network. To better compare the two networks behaviors, the choice of is such that the average lifetime of the two networks in each problem is comparable. The value function is approximated using a one-hidden layer FFN as Sec. 2.2 Analogously to MOP, in order to train the network we generate a batch of Niraj paths 7 starting with the same initial conditions (0)) and minimize the loss function defined as the summed squared error between the approximated value Va w) and its expected evolution following the Bellman equation Eq. 10
3 Results
3.1 Energy constraint
To test whether NeuroMOP can produce maximal variability under strict constraints, we first study scenario where a terminal state is reached when the overall level of the RNN's activities is high. Specifically, xf are all the states where the energy, defined function of x, exceeds a certain value E(x!) > L (see Appendix B In this way, implement the idea that high activity is detrimental,
5

---

(a)
Free net
(b)
NeuroMOP net early mainingg
(c)
NeuroMOP
(d)
late training
0-
500
1000
600
1000
500
1000
520
1000
0.15
0.15
-5.44
0.15
-0.54
0.15
0.11
0.11
0.11
ML M04
0.11
NLAII
0.11
0.11
0.10
0:10
0.03
0.03
0
1000
<2.27
0.00
1000
-2.27
0.00
<2.27
500
1000
600
1000
500
1000
500
1000
(e)
(8)
(h)
0.3
Free
G 0.
3
600
Oii-en Doi Elsi-L stane space regions
80
60
Figure 2: Energy constraint. (a) In the free network, the RNN shows chaotic activity (top panel) with high energy consumption (bottom) above threshold (dashed line). In the top panel, each line represents the activity of randomly sampled neuron of the RNN. A single trial is shown. (b) Early in training (~ 10 epochs), the NeuroMOP network quickly reaches terminal state by crossing the energy threshold (dashed line. bottom panel). Indeed, the large action entropy throughout the trajectory suggests no state-dependent action entropy adjustment. Inset provides a zoom of the energy close to the boundary. (c) After training, the NeuroMOP network is able to avoid terminal states for the whole duration of the trajectory (T = 1000) by reducing its action entropy whenever closer to the energy threshold. Inset as in (b). (d) The R network employs completely different and risk-averse solutions. (e) Probability density function of the average state occupancy. (f) When far from the energy threshold (E(x) < L), the NeuroMOP network exhibits maximum effective dimensionality (ED, ~ M = 8), but loses one degree of freedom (EDa ~ M = 7) when approaching the threshold corresponding to terminal states (E(z) L, i.e. E(x) [L SL, L], with öL 0.001, arbitrary). The R network only lives far from the threshold injecting mainly inhibitory currents and it exhibits a low effective action dimensionality. (g.h) With training. both networks increase the average standard deviation of the individual trajectories (o), with MOP displaying larger variability (g). Together, they
learn to reach the end of the episode lend T = 1000 (h). Averages over Nav = 10 networks with batches of Neraj = 10 trajectories; errors are standard errors of the mean (SEM).
either because it is costly 48 or because it leads to neural saturation, impeding sensory encoding 40 We consider a free network (the RNN without external current, Eq. [ with I = 0) in chaotic state, where the repetitive saturation of the neurons activities leads to high energy consumption over time (Fig. [b) The NeuroMOP network learns to control the input current in order to generate maximal input entropy while, at the same time, avoiding terminal states and thus surviving for the whole length of the stimulation (Fig. [2p, early training; Fig. 2 late training). Importantly, we observe that the NeuroMOP network changes dynamical regimes depending on how far the energy consumption is to threshold (Fig. 2 bottom panel, inset): when the energy is close to threshold, the action entropy reduces, and therefore the policy becomes more deterministic. In contrast, when the energy is far from threshold, action entropy rises again and the policy increases its stochasticity. Moreover, the policy dimensionality, as measured by the effective dimensionality of the currents (see Appendix A). is lower when close to the threshold compared to further away (Fig. 2D, projecting noise to lower dimensional manifold such that actions (i.e., currents) that would push the network above threshold are suppressed. Overall, these results show that the NeuroMOP network flexibly changes from highly stochastic to more deterministic policy depending on the network state.
Comparing NeuroMOP with the R network, we find that the R network employs different solutions, showcasing a preference for risk-avers solutions (Fig, 21). As a long lifetime is encouraged by the

---

(9)
(b)
(c)
NeurMOP net
D.2
5.54
XLAIXI
x00
0-
0-
-0.2
0.00
1000
-0.2
0.2
(d)
R
(c)
1000
1000
0.2
5.54
NexrMMP
Rc decoy)
Cold
600
800
8 0
8 0-
MLAIO
-0.2
0.00
10-3
10000000000000--1100000000100111 10
0
100
-0.2
0
0.2
opochs
X1
X1
Figure 3: Constrained neural space. (a) Terminal states are defined as the boundaries of a square in the activity space of two randomly selected RNN's neurons (#1, (b-c) As a result, NeuroMOP network confines the activities 21 and 22 within the square boundaries (panel b, magenta traces), while it 'draws' the square by filling its inside (panel c, colored line, representing one trajectory of the two readout neurons). In contrast, in the space of any other pair of neurons Zy) i,j activities spread in space (grey line, representing one trajectory). A zoom-in of the readout space shows that the NeuroMOP network adapts the action entropy based the state proximity to the boundaries (colorbar, c, right panel). (d) The network, following an c greedy policy, fails to avoid the terminal states except for extremely small values of effectively reducing its action stochasticity to zero. Lifetime computed after training the network for 100 epochs. (e) Matching lifetimes for both NeuroMOP and R network with an e-greedy policy with exponential decay (see Appendix B The network learns to satisfy the boundaries constraints after the exponential decay has dropped the randomness of the action selection (e) zero. (f) Same in (c) for the R network. The R network only 'draws' one side of the square with the two readout neurons (æ1, I2), while the other neurons, as well receiving the external currents, are driven towards the saturating states. Averages are over Nav = 10 networks with batches of Neraj = 10 trajectories. Errors are SEM.
extrinsic reward, the network learns steer the RNN's energy very far from the terminal state, so that the spontaneous action fluctuations given by the stochasticity of the random policy would not harm the overall performance by keeping it far from threshold. The policy found by the R network consists in injecting mostly inhibitory currents, driving the RNN towards the point of minimum energy and effectively 'silencing' the RNN.
Although both networks are able to avoid the terminal states, they lead to different behaviors and space occupancy: while the R network tends to suppress the RNN's activity, the NeuroMOP network exploits the overall range of activities permitted by the terminal states (Fig. 2 Again, the NeuroMOP network is able to do by adapting its action entropy terminal state imposes constraint on the network's 's activity. In contrast, the R network operates only far from the threshold and, by showing clear preference for inhibitory actions, exhibits an effective dimensionality significantly lower than the maximum possible one, i.e., FDD M (Fig. 2). As consequence the different action selection strategies, the RNN's neurons within the NeuroMOP network display greater variability than those within the R network, with larger average standard deviation over the trajectories (Fig. g The choice of e, representing the level of stochasticity, of the R network is such that the two networks have comparable lifetimes (Fig. 2 h).
state dependent manner. Proximity the
3.2 Constrained neural space
Terminal states can be arbitrarily imposed on the activities of individual neurons or any subset of them, not only globally as in the previous scenario. Here, test NeuroMOP in new problem
7

---

(a)
(b)
Circle
Plus
Diamond
Mean accuracy
600
8
X
X
0
fond
400
-1
-1
-1
Cirole
Plus
Diamand
Heart
Square
Oval
X1
Heart
Square
Oval
(c)
5.54
600
0.3
MAAp
8 0
X
X 0
Lend
400
5
-1
-1
-1
0
100
100
X1
epochs
epochs
Figure 4: NeuroMOP can constrain a subset of neural activities within different regions of the neural space in a context-dependent manner. The network is informed of the shape it needs to draw via a one-hot vector fed into the value function.
(a) Example of single network drawing C different shapes by confining its readout activities (I1, (22) within the corresponding activity regions (T 5000). Notably, action entropy is both state and context dependent. One trajectory per context is shown. (b) Mean accuracy, measured the mean lifetime in each context th reflects varying shape difficulty, consistent across networks. (c) With training, the NeuroMOP network learns to approach the arbitrary training end of the simulation lend = T = 600 (left panel) and to increase the average standard deviation of the individual trajectories (o) (right panel). Averages over N = 10 networks, with batches of Niraj 10 trajectories. Errors are SEM.
where terminal states are set directly on two randomly selected readout neurons (r1, x2) of the RNN. Specifically, a terminal state encountered any time |x1 > or |222| L.
The structure of the terminal states generates interesting behavior the NeuroMOP network: the network 'draws' square in the (22) activity space by filling the available area while avoiding the square's s boundaries (Fig. (3). In other words, the network occupies all the space allowed by the terminal states after learning. The NeuroMOP network confine the activities of the two readout
neurons even within very small regions of the activity space (Fig. 3p, magenta lines). Notably, as neurons are driven by actions that aim to maximize future cumulative entropy and terminal states here exclusively set on T1 and Ø2, all other neurons (2:, Vi + 1,2) occupy a much larger region of the activity space (grey lines, in their own spaces (y)) In the (11, space, the network reduces its action entropy when in proximity to terminal states, corresponding to the square boundaries (Fig. Bc). Due the activity correlations induced by the shared input current, controlling the readout neurons along the anti-diagonal of the square presents challenges for the NeuroMOP network. In those regions, the NeuroMOP network learns the necessity of highly deterministic action selection to avoid terminal states (Fig. Be. right panel).
While NeuroMOP adapts the stochasticity of its policy to occupy the maximum available space, find that the R network, following e greedy policy, fails to do the same for most values of e (Fig. (d. The R network's 's lifetime is comparable to that of the NeuroMOP network for extremely low values of (notice decreasing scale), for which the consequent randomness of the action selection is effectively zero. To give more flexibility the agent, we allowed the R network first explore phase space by using an epoch-dependent with an exponential decay. Starting from a larger eo at epoch 0, and slowly decreasing it, can match the two lifetimes (Fig. 3D). Despite this, the inherent greediness of the action selection forbids the R network to occupy all the available activity region (Fig. 3d, resulting in a largely repetitive and stereotyped network behavior.
3.2.1 Context-dependent neural space constraints
We next wondered about the versatility of the NeuroMOP network to confine neural activity within even more complex boundaries. We introduce context-dependent neural space constraints (see Appendix B). where in each context the set of terminal states in the readout space of two random
8

---

activities x2) define different shapes, with different sizes, orientations and border complexity. We augment the input layer of the value approximator described in Sec. 2.2 with C 6 additional nodes, where C' represents the number of different shapes where activity has to be confined. Thus, the feedforward value approximator now receives, alongside the network state z, C dimensional one-hot vector indicating the current context.
The same feedforward network, opportunely informed of the context, correctly approximates the value function. Consequently, the NeuroMOP network also adapts its action entropy in a context- dependent manner (Fig. 4.. Notably, while avoiding terminal states, the NeuroMOP network learns to occupy the available state region in each given context. As expected, some terminal states rise greater challenges for the NeuroMOP network to be avoided (Fig. 4p). a feature that is independent on the number of stored contexts C. Overall, the NeuroMOP network successfully avoids terminal states while increasing the variability in the network with learning (Fig. P
4 Discussion
We have introduced NeuroMOP, novel theory that puts forward the idea that neural variability arises from the active generation of future neural activity entropy. We have explored this theory by introducing a mechanism for maximizing and controlling variability the chaotic regime. Contrary to the common idea that excessive variability may impede performance. our model showcases that injecting maximal controlled variability into RNNs actually permits to solve different 'tasks', indirectly defined by the structure of terminal states. By allowing for a diverse array of actions according to the state, this variability enables the network to explore a wider solution space, potentially leading to more effective adaptations 50 51 52).
-ighly-dimensional RNN in
We tested our network in a series of scenarios. First, we introduced an energy threshold on the network activity. Energy constraints may have been likely selected by evolution, as brain activity is costly both during information processing and at rest [53 48. By observing that the network always keeps energy consumption close to the threshold without exceeding it, our results align with the
idea that sustained, controlled energy consumption could actually be beneficial (54). In second series of problems, we showed that the NeuroMOP network can avoid terminal states in the readout space while increasing the variability in the subspaces where no boundaries are set. Therefore, long lifetime is achieved by flexibly switching between deterministic and stochastic dynamical regimes when needed. Additionally, we show that our algorithm capable of solving problems often tackled through extrinsic rewards, such as balancing a cartpole (Appendix G. or scenarios where more deterministic behavioral modes are required, like traversing a narrow corridor in neural space (Appendix H.. In addition, we show that introducing extrinsic rewards in the MOP framework largely reduces the variability of the network behavior (Appendix D.
When comparing MOP with other systems that also generate variability, like a (R) network with epsilon-greete noise, we find that R networks can only avoid terminal states after extensive training and only by quenching the source of randomness. We remark that MOP agents face as well the drawback of stochasticity as their policy follows a Boltzmann distribution (Eq. 4). and thus a non-zero probability is assigned to all actions regardless of the state. Despite that, MOP agents overcome this tendency by adapting their randomness via the computation of the value function, which trades-off immediate with future variability. This is in contrast with R agents, where the stochasticity parameter e is state independent. These results suggest that state adaptation of stochasticity is a relevant property we might expect in intelligent systems.
By keeping the weights of the RNN fixed, we depart from the common practice of training networks. Weights training usually leads to activities exploiting the saturation state of the neurons [35] (3). We conjecture this dynamics to be unrealistic, as biological neurons largely display activities that are well below their maximum values [35] [6 49). Analogously, saturation is undesirable even in artificial neural networks due to the vanishing curvature of the loss landscape. By favoring states and actions with low probability, the NeuroMOP network leads to the more uniform occupation as possible, avoiding saturation and encouraging neurons to stay in a 'healthy' regime, i.e., a regime suitable for computation [57]. As well, we have demonstrated that NeuroMOP can control high-dimensional chaotic RNNs Future research should investigate how MOP-driven input currents affect the RNN's regime. We anticipate that MOP currents will stabilize neural trajectories, consistent with operating at

---

the edge of chaos [58] Characterizing chaoticity as a function of the input properties (e.g. magnitude) 59
a promising direction.
In our system, an external stochastic input current generator is designed to maximize its cumulative entropy impinging onto the neurons of an RNN. The idea of specialized circuits serving the role of stochastic input generator is not novel. In songbirds. for instance, the LMAN brain region, part of the neural circuit controlling songs' production, has been largely postulated fulfill the function of injecting variability into the downstream motor pathways [60] 61] Allegedly, increased variability in the motor neural activity favors behavioral exploration in the songs' production. Remarkably, has also been shown that, during courtship, adult birds significantly reduce their vocal variability compared to their solitary singing 62). This switch of behavior from random to more deterministic modes aligns with our hypothesis of the existence of directed variability in the brain.
Finally, NeuroMOP offers several testable predictions regarding the nature of neural variability we should expect in the brain. Firstly, it predicts that neural variability will persist even after extensive training, which aligns with studies reporting large spiking variability even in well-trained non-human primates (21 [9). Despite this persistence, our model also suggests that neural variability may decrease when terminal states are sufficiently close, as the network is expected to transition into more deterministic mode to avoid those states [30] 63). Finally, our model predicts that reward signaling systems in the brain will also signal intrinsic motivation rewards. This is partially supported by recent studies demonstrating that spontaneous movements elicit dopamine release .. Further, we postulate that the visitation of all activity states may increase flexibility and help generalization. Consistent with that, certain activity states are observed to be replayed in the absence of any stimulation the brain, and several mechanisms in RNNs have been proposed for this phenomenon 644 65). NeuroMOP predicts the deterministic reactivation of activity patterns and memories that are relevant for generating higher future behavioral entropy, but the more stochastic reactivation of less relevant memories.
Limitations In the proposed framework, we choose not to approximate the policy, but instead to rely on an 'oracle' to provide the best action following the derived exact analytical expression (Fig. 1, h box). Our model could be extended to include a neural network to also approximate the policy using actor critic approaches [66]. Therefore, we do not delve in the process of learning the policy itself. Exploring the policy learning process represents a significant direction for future work. Despite relying on the exact policy, the input current selection has still high computational cost. In order to partially mitigate this, we introduced the random matrix K, which transforms low M M-dimensional binary actions into high N didimensionall currents. Via this matrix, we were able to reduce complexity from ~ O (2 to ~ O (2*). thereby significantly speeding up the computation, without compromising the convergence of the algorithm. The extension of NeuroMOP to more realistic spiking and Poisson-like variability is another major possible direction. Finally, another interesting direction that we have not addressed here is how learn the structure of terminal states, and how nearby 'bad' states surrounding terminal states can be learnt and used to speed up learning.
Conclusion Our results demonstrate that maximizing cumulative future action entropy while avoiding terminal states leads to interesting behaviors without the need of defining an extrinsic reward function. Our work shows that NeuroMOP networks can flexibly switch between stochastic and deterministic modes as needed to avoid terminal states. These results contribute to novel theory of neural variability based on future entropy production, reconciling stochastic and deterministic behaviors within a single framework. Our work highlights a significant limitation in classical neuroscience studies, where limited behavioral repertoires are promoted by the task design and experimental trials terminate upon reaching the goal. In ecological settings, in contrast, agents continuously generate interim goals and elicit new behaviors. NeuroMOP offers powerful model of neural activity underlying natural behavior.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Howard Hughes Medical Institute (HHMI, ref 55008742), Ministry of Science and Innovation, State Research Agency, European Union (Project financed by MCIN/AEI/1O, 1303//00 10001 1033/ ERDE, EU) and ICREA Academia to R.M.-B. and AGAUR-FI ajuts from Generalitat de Catalunya/ESFF (2024 FI-B3 00020) to C.M.
References
[1] C. Button, M. MacLeod, R. Sanders, and S. Coleman. Examining movement variability in the basketball free-throw action at different skill levels. Res Q Exerc Sport, 74(3):257-266 2003.
[2] S. Miller. Variability in Basketball Shooting: Practical Implications. Proceedings Archive, 2000.
ISBS Conference
[3] K. Davids, P. Glazier, D. Araújo, and R. Bartlett. Movement systems as dynamical systems: the functional role of variability and its implications for sports medicine. Sports Med, 33(4):245- 260, 2003.
[4] R. Bartlett, Wheat, and M. Robins. Is movement variability important for sports biomecha- nists? Sports Biomech, 6(2):224-243, 2007.
[5] S. Recanatesi, U. Pereira-Obilinovic, M. Murakami, Z. Mainen, and L. Mazzucato. Metastable attractors explain the variable timing of stable behavioral action sequences. Neuron, 110(1):139- 153.e9, 2022.
[6] E. Schulz, R. Bhui, B. C. Love, B. Brier, M. T. Todd, and J. Gershman. Structured, uncertainty- driven exploration in real-world consumer choice. Proceedings of the National Academy of Sciences, 1 116(28):1 -901313908,, 2019.
[7] M. J. Kahana, E. V. Aggarwal, and T. D. Phan. The variability puzzle in human memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 44(12):1857-1883, 2018.
[8] M. M. Churchland, A. Afshar, and K. V. Shenoy. Central Source of Movement Variability. Neuron, 52(6):1085-1000 1096, 2006.
[9] R. van Beers. Motor learning is optimally tuned to the properties of motor noise. Neuron, 63(3):406-417, 2009.
[10] Y. Mandelblat-Cerf, R. Paz, and E. Vaadia. Trial-by-trial variability of single cells in motor cortices is dynamically modified during visuomotor adaption. J. Neurosci., 19043)):5033 2009.
[11] J. Markowitz, W. F. Gillis, M. Jay, Wood, R. W. Harris, R. Cieszkowski, R. Scott, D. Brann, D. Koveal, T. Kula, C. Weinreb, M. A. M. Osman, S. R. Pinto, N. Uchida, S. W. Linderman, B. L. Sabatini, and S. R. Datta. Spontaneous behaviour is structured by reinforcement without explicit reward. Nature, 614(7946):008-117 2023.
[12] K. Doya and T. J. Sejnowski. A Novel Reinforcement Model of Birdsong Vocalization Learning. In Advances in Neural Information Processing Systems, volume 7, 1994.
[13] Cignetti, Schena, and A. Rouard. Effects of fatigue on inter-cycle variability in cross- country skiing. Journal of Biomechanics, 42(10):1452) 45591459, 2009.
[14] Legg, D. B. Pyne, Semple, and N. Ball. Variability of Jump Kinetics Related Training Load in Elite Female Basketball. Sports, 5(4):85, 2017. 2017.
[15] F. Attneave. Multistability in perception. Scientific American, 225(6):62-71, 1971.
[16] R. Moreno-Bote, Rinzel, and Rubin. Noise-induced alternations in an attractor network model of perceptual bistability. Journal of neurophysiology,
2007.

---

[17] J. Drugowitsch, R. Moreno-Bote A.K. Churchland, M.N. Shadlen, and A. Pouget. The cost of accumulating evidence in perceptual decision making. J. Neurosci, 32(11):3112-362 2012.
[18] Cazettes, L. Mazzucato, M. Murakami, Morais, E. Augusto, A. Renart, and Z. Mainen. A reservoir of foraging decision variables in the mouse brain. Nat Neurosci, 26(5):840-849 2023.
[19] D. Campos, F. Bartumeus, V. Méndez, S. Andrade, and X. Espadaler. Variability in individual activity bursts improves ant foraging success. J R Soc Interface, 13(125):2010885)) 2016.
[20] W. R. Softky and C. Koch. The highly irregular firing of cortical cells is inconsistent with temporal integration of random EPSPs. J. Neurosci., 13(1):334-350, 1993.
[21] D. J. Tolhurst, J. A. Movshon, and A. F. Dean. The statistical reliability of signals in single neurons in cat and monkey visual cortex. Vision Res, 23(8):755-785 1983.
[22] A. Arieli, A. Sterkin, A. Grinvald, and A. Aertsen. Dynamics of ongoing activity: explanation of the large variability in evoked cortical responses. Science, 273(5283) 1888))):8888881111 1996.
[23] A. A. Faisal, L. P. J. Selen, and D. M. Wolpert. Noise in the nervous system. Nat Rev Neurosci, 94):29223333, 2008.
[24] R. Moreno-Bote. Poisson-like spiking in circuits with probabilistic synapses. PLoS Compul Biol, 10(7):e1033222, 2014.
[25] H. Sompolinsky, A. Crisanti, and H. J. Sommers. Chaos in Random Neural Networks. Phys. Rev. Lett., (1(::2259 262, 1988.
[26] C. van Vreeswijk and H. Sompolinsky. Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science, (5223):122244 1722-1766,7-- 1996.
[27] A. Renart, R. Moreno-Bote X.-J Wang. and N. Parga. Mean-driven and fluctuation-driven persistent activity in recurrent networks. Neural Comput, 19(1):1-46, 2007.
[28] G. Orbán, P. Berkes, J. Fiser, and M. Lengyel. Neural Variability and Sampling- -Baeed Proba- bilistic Representations in the Visual Cortex. Neuron, 92(2):330-543, 2016.
[29] E. Zohary, M. N. Shadlen, and W. T. Newsome. Correlated neuronal discharge rate and its implications for psychophysical performance. Nature, 370(6485); (10064655:140-1444 140-143, 1994.
[30] M. M. Churchland, B. M. Yu, P. Cunningham, L. P. Sugrue, M. R. Cohen, G. S. Corrado, W. T. Newsome, A. M. Clark, P. Hosseini, B. B. Scott, D. C. Bradley, M. A. Smith, A. Kohn, J. A. Movshon, K. M. Armstrong, T. Moore, S. W. Chang, L. H. Snyder, S. G. Lisberger, J. Priebe, I. M Finn, D. Ferster, S. I. Ryu, G. Santhanam, M. Sahani, and K. V. Shenoy. Stimulus onset quenches neural variability: a widespread cortical phenomenon. Nat Neurosci, 13(3):369-378, 2010.
[31] L. Mazzucato, A. Fontanini, and G. La Camera. Dynamics of multistable states during ongoing and evoked cortical activity. J. Neurosci. 35(21):8214-8231, 2015.
[32] A. Litwin-Kumar and B. Doiron. Slow dynamics and high variability in balanced cortical networks with clustered connections. Nat Neurosci, 15(11):1488-1555 2012.
[33] R. Laje and D. V. Buonomano. Robust timing and motor patterns by taming chaos in recurrent neural networks. Nar Neurosci, 16(7):925-933) 2013.
[34] H. Jaeger and H. Haas. Harnessing nonlinearity: predicting chaotic systems and saving energy Science, 304(5667):78 80, 2004.
communication. communica
in wireless
[35] D. Sussillo and L.. F. Abbott. Generating Coherent Patterns of Activity from Chaotic Neural Networks. Neuron, 63(4):544-555 200.
2009.
[36] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with Stochastic Actor. In Proceedings of the 35th International Conference on Machine Learning, pages 1861-1870, 2018.
12

---

[37] E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning. pages 2681-2691. PMLR. 2019.
[38] A. Renart and C. K. Machens. Variability in neural activity and behavior. Curr Opin Neurobiol, 25:21 1-220, 2014.
[39] S. Haar, O. Donchin, and I. Dinstein. Individual Movement Variability Magnitudes Are Explained by Cortical Neural Variabilityy. J. Neurosci, 37(37):076-9085, 2017.
[40] L. Waschke, N. A. Kloosterman, J. Obleser, and D. D. Garrett. Behavior needs neural variability, Neuron, 109(5):751-766, 2021.
[41] R. Darshan, W. E. Wood, S. Peters, A. Leblois, and D. Hansel. A canonical neural mechanism for behavioral variability. Nat Commun,
2017.
[42] P. Dagenais, S. Hensman, V. Haechler, and M. C. Milinkovitch. Elephants evolved strategies reducing the biomechanical complexity of their trunk. Curr Biol, (1(21):4727-4737.2 2021.
[43] M. T. Kaufman, M. M. Churchland, S. I. Ryu, and K. V. Shenoy. Cortical activity in the null space: permitting preparation without movement. Nat Newrosci, 17(3):400-44 2014.
[44] Ramírez-Ruiz D. Grytskyy, C. Mastrogiuseppe, Y. Habib, and Moreno-Bote. Complex
behavior from intrinsic motivation to occupy future action-state path space. Nat Commun, 15(1):6368, 2024.
[45] R. Moreno-Boo and J. Ramirez-Ruiz. Empowerment, free energy principle and maximum occupancy principle compared. In NewrlPS 2023 workshop: Information-TTeorrtie Principles in Cognitive Systems, 2023.
[46] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity Is All You Need: Learning Skills Without A Reward Function. In International Conference on Learning Representations, 2019.
[47] S. Mohamed and D. Jimenez Rezende. Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 28, 2015.
[48] J. E. Niven and S. B. Laughlin. Energy limitation as a selective pressure on the evolution of sensory systems. Journal of Experimental Biology, 211(11):1922-804, 2008
[49] S. Laughlin. A Simple Coding Procedure Enhances a Neuron's 's Information Capacity. Zeitschrift für Naturforschung C, 36(9-10):9 10):9 10-912, 1981.
[50] H. G. Wu, Y. R. Miyamoto, L. N. G. Castro, B. P. Ölveezky, and M. A. Smith. Temporal structure of motor variability is dynamically regulated and predicts motor learning ability. Nar Neurosci, 12(2):31--311, 2014.
[51] S. Kirkpatrick, C. D. Jr. Gelatt, and M. P. Vecchi. Optimization by Simulated Annealing. Science, 220(4597), 1983.
[52] S. Fusi. Hebbian spike-driven synaptic plasticity for learning patterns of mean firing rates. Biological Cybernetics, 87(5-6), 2002.
[53] D. Attwell and S. B. Laughlin. An energy budget for signaling the grey matter of the brain. / Cereb Blood Flow Metab, 21(10): 2001.
[54] C. Chintaluri and T. P. Vogels. Metabolically regulated spiking could serve neuronal energy homeostasis and protect from reactive oxygen species. Proceedings of the National Academy of Sciences, 120(49):e23065251120 120, 2023.
[55] I. Ohzawa, G. Sclar, and R. D. Freeman. Contrast gain control in the cat's visual system. J Neurophysiol, 54(3):651 1677, 1985.
[56] G. Sclar, Lennie, and D. DePriest. Contrast adaptation in striate cortex macaque. Vision Research, 29(7)7477755,, 1989.
13

---

[57] A. Lazar, G. Pipa, and J. Triesch. SORN: self-organizing recurrent neural network. Frontiers in Computational Neuroscience, 3, 2009.
[58] M. Cencini, F. Cecconi, and A. Vulpiani. Chaos: From Simple Models to Complex Systems.
Series on advances in statistical mechanics. World Scientific, 2010.
[59] S. Takasu and T. Aoyagi. Suppression of chaos in partially driven recurrent neural network. Phys. Rev. Res., 6:013172, 2024.
[60] M. H. Kao, A. Doupe, and M. S. Brainard. Contributions of an avian basal ganglia-fonebrain circuit real-time modulation of song, Nature, 433(7026))638-647 2005.
[61] B. Ólveczky, M. Otchy, J. H. Goldberg, D. Aronov, and M. S. Fee. Changes in the neural control of complex motor sequence during learning. Journal of Neurophysiology. 106(1):386-397, 2011.
[62] C. Woolley and A. J. Doupe. Social Context-Induced Song Variation Affects Female Behavior and Gene Expression. PLoS Biol, 6(3)=622, 2008.
[63] R. Nogueira, S. Lawrie, and R. Moreno- Bote. Neuronal Variability Proxy for Network State. Trends in Neurosciences, (1(4):170-111 2018.
[64] T. Asabuki and T. Fukai. Predictive learning rules generate a cortical-like replay of probabilistic sensory experiences. eLife, (13):RP92712, August 2024.
[65] N. H. Krishna, C. Bredenberg, D. Levenstein, B. A. Richards, and G. Lajoie. Sufficient conditions for offline reactivation in recurrent neural networks. In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023.
[66] V. Konda and J. Tsitsiklis. Actor-Critic Algorithms. Advances in Neural Information Processing Systems, volume 12, 1999.
[67] K. Rajan, L. F. Abbott, and H. Sompolinsky. Stimulus-dependen suppression of chaos in recurrent neural networks. Phys. Rev. E, 82:01 2010.
[68] M. Farrell, S. Recanatesi, G. Lajoie, and E. Shea-Brown. Recurrent neural networks learn robust representations by dynamically balancing compression and expansion. In NewrIPS 2019 workshop: Real Neurons & Hidden Units: Future directions at the intersection of neuroscience and artificial intelligence, 2019.
[69] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. 2016. 10X.5.6006001540.
[70] R. V. Florian. Correct equations for the dynamics of the cart-pole system. In Center for Cognitive and Neural Studies (Coneural), 2005.
14

---

Appendix
A Additional Methods
Effective dimensionality We model an external controller sampling actions living in an M dimensniio space. In the NeuroMOP network, the controller aims at occupying action-state path space, and therefore expect the sampled actions maximally occupy the action space. To quantify the effective occupation of action space, we introduce the effective dimensionality [67 68 of the actions as
Ax) ED. (3)
(11)
where Å (i = 1,.,,..,,NN are the eigenvalues the covariance matrix of the sampled actions. Intuitively, the effective dimensionality quantifies the number of dimensions needed to explain the observed variance in the sampled actions by identifying the dimensions of spread of the signal. the absence of constraints, the NeuroMOP network would maximally occupy space, uniformly in all the directions of the action space, leading to comparable eigenvalues A = I...., M and an effective dimensionality close to the full dimensionality of the action space itself (ED,, = M). The presence of constraints introduce directions actions that will be avoided by optimal networks. Along these constrained directions, the sampled actions variances, hence the corresponding eigenvalues, significantly reduced, resulting in a lower effective dimensionality, i.e. ED. < M. Since actions sampled from state-dependent stationary distribution, the effective action dimensionality may vary according to the state. To investigate that, we quantify the effective dimensionality as a function of different states by restricting the covariance matrix, from which the eigenvalues in Eq. 11 the actions sampled in specific regions the state space.
Average standard deviation We quantify the induced variability in RNN's activities by mea- suring the fluctuations of individual neurons. For this reason, we introduce the average standard deviation as
(a)
1
(T) 1=1
(12)
Ntraj
N
where o! the standard deviation of the activities of neuron i Vi I,..., N along trajectory т.
Parameters of the simulation We simulate RNNs in the chaotic state 25 35 with the parameters
reported in T unless otherwise specified. The parameters defining the algorithm, including the specifics of the feedfoward network and the details of the optimization, reported in Table 2
B Terminal states
Energy constraint
Terminal states reached whenever the RNN's current energy expenditure
exceeds = 0.11 (arbitrary units). Energy is defined as E(x) = , V Σi (xt + 1) the euclidean norm of the activity, translated so that the lowest activity state x; 1 Vi = 1,..., N has zero energy. Thus, terminal states æt are all the states where E(xt) L. Analogous results have been
N
Table 1: Hyperparameters for the RNN.
Parameter
Value
N
100
nonlinearity ((-)
tanh, ReLU
ôt
0.05
T
1.0
g
5.0
Pranh
2.0
PReLU
5.0
15

---

Table 2: Hyperparameters for the algorithm, including the parameters of the FFN.
Parameter
Value
action dimensionality M
8
discount factor Y
0.9
number of hidden layers (FFN)
hidden units per layer
256
imput units Napp
20
FFN nonlinearity x²
ReLU
see Sec. Terminal states
training epochs Nep
see Sec. Terminal states
number of agents Nag
10
trajectories per batch Ntraj optimizer learning rate
10
Adam
0.01
obtained with different definitions of energy function. Neward-maximizing (R) networks follow an --greedy policy with 0.3. Both networks are trained for Nep 60 epochs.
Constrained neural space Terminal states are all states where |<| > L or |22| > L. with (x1, x2) being two arbitrary neurons of the RNN and 0.1. The R networks follow an e e-greedy policy with exponential discount, i.e. at epoch / the probability of sampling a random action decreases as 0.9. The networks are trained for Nep 100 epochs.
6j = Q-1 8, with eD 0.5 and
Context-dependent neural space constraints Taking two length scales L+ = 0.2 and L__ = 0.1, we define terminal states in the different contexts as below. In each epoch, one of the possible contexts is randomly sampled, and kept fixed for all the trajectories in the epoch. More stable yet slower convergence obtained if different contexts are sampled in each trajectory of the batch. The NeuroMOP network is trained for Nep 100 epochs. Terminal states in each context are reached whenever the following conditions are met:
square context: |21| L_
222 L_; (|r1) > L_ and [222| > L-);
plus context: |
L
|22| > Lu
circle context:
o + 2 >
L4;
diamond context:
+
|.2
>
l4;
2
oval context:
m1
+
L-
1;
heart context:
(x7
+ x2
L.
x
x/.3 > 0.
C Optimal policy and value
We show here the derivation of the analytical expression for the optimal policy π (a[z) in the case of an agent following MOP and maximizing the action-state path entropy [44). Then, we use this analytical solution derive the Bellman consistency equation. While in the main on agents maximizing the action space occupancy, here we take the more general formulation considering both the action and state space occupancy maximization. We include the state entropy in our NeuroMOP network using a small noise approximation in Appendix
focused
The MOP agent gets "n In (a(t)l=(t)) p (z(t+ 1) xxt), a(t)), with discount factor 0 and state 2 0 weights, and where r = (z(0), a()), a(1), xt)),
intrinsic reward
path equal
R(T)
< action at)..... denotes a path of states and actions. Although we use two parameters, a and 3, effectively the number of parameter is only one, their ratio, which measures the relative strength of state over action entropy. Note that the action path entropy maximizer agent is recovered by taking 3 and a 1. The objective of the agent is to maximize the value function V (zz), defined as the expected return of the
16

---

intrinsic reward as ((x) = Byor, [R(T) = z]. The Bellman recursive equation enables us to write the value function as the sum of the intrinsic reward the agent receives the state z and the expected discounted sum of future intrinsic rewards, i.e., the value function in the next state, taking the form
V(x)
EY [R(T)[=(0) z]
= 7 In (T" (a(t))=(t)) (z(t 1))x((). [x(0) - = (a(1 1))x((1) 1))x((),
= E
Σ
1=0
T(alr)p(r|(0,0 a)
=(0)=x
ΣΣ x(1z))((1(2 in (= (x)
+ 7 (T (o(1+ 1)]=((0)
- E (nd Σ <(aa)) Σ pla' (1pp(())
b (R(') =x] aH(AA)) + B Σ x ((ax)) a) + *((p))) Σ [p(e(z,a)V(()),
where we recognize the entropy over the action space H(AAr)= space H(S(x,a) = En' p(a' (aaa|ar,0) a) In p (a') (al|x,,)) a).
= (n (()a)) and state
The optimal policy n" (-I-) is the policy maximizing this value function. Therefore, we look for the critical policies (-l-)) under the constraints <--ls) 0 and E. <aaax) Finding the critical
points of the Lagrangian function defined as L = V(r) (2, T(lax) 1) involves solving
OC
(v
Axxx = =0,
(13)
θτ(a r))
where A(a, .r) indicates that the derivatives and the policy are computed in the same state æ. Deriving the value function with respect to (a(x) gives the desired Lagrange multiplier X. By writing V (x) using the Bellman recursive equation we get that
(v)
x(x,2) = (ala)
T(a'lz) (alu((all))) + ('
(a(
->V(2'))
W=T=
-aalnn((ap))) + Σ p(a' (-$3 + (V))2x))
+ Σ n (d'lz) Σ p(x1 (')a((
a ln (a|x)) - > p(xl a) (np(e((z,a)) + Eplella a) + h(14)
where we collected
= h(x) all the terms that are not dependent on a. Introducing the partition function 7(x) = exp (1 (X(x, x) - M(x))
EaEA(e) exp (1 (BH(S)a, a) a (x.))))) as the normalization constant, the critical policy takes the expression
(H((S),,)++ + E Elld'an [oleda,.\Vooodd))
<aaxx) = Z(x)~ exp
(15)
a
17

---

The corresponding critical expected return is obtained by substituting the critical policy in the Bellman recursive equation as
(x)
Σ (a|p))
Σ (a|2)) Σ p(22|2,,)) a -
-aln (100 |
+ + [va(1']
alz((x)= Σ (d|z) (H(()22 +2 Σ plallo, plallo,))V<(())) +
Σ (a)))) go [C(xr)]
Σ ((((SH(() En plz' (x)))
a ha (xx) a In
(16)
aεA((w)
where we see the expectation value over future states simplifies in the last step.
We now prove that this stationary point corresponds maximum of the value function. For this, first note that the value function is continuous and has continuous derivatives with respect to the policy, and therefore the maximum lies either on the boundaries of the policy constraints or it is indeed the critical value. Given a state x, the policy boundaries are the points where an (initially available) action à unavailable, i.c. T(ā return along a boundary leads to the same solution defined in Eq.16) but the unavailable action à does not appear in the sum over all the possible a. As the expected return an increasing function with the elements making up the sum, the critical value V<(x) is greater than the expected return along the policy boundaries. The critical value function is therefore the optimal value function, i.e. [x) = V* (a:), and the critical policy is indeed the optimal policy, i.e., л (a|x) x (a|or) Uniqueness of the critical value comes from concavity of the value function.
0. Thus, computing the critical value of the expected
As discussed in Sec. 2.2 we deal only with approximations of the optimal value function, and consequently our V(z, W) will not exactly satisfy the Bellman consistency equation. We extend the evolution via the Bellman operator defined in Eq. 6 to the case of the action-state occupancy maximization by defining (z, w) as
Vzs w) = a In Zv, (x, w) = Σ exp ( (BH(())))) a) +7 Endele,a (w))
(17)
As the Bellman consistency equation is satisfied by the optimal value (*(x), we take the best representation of the weights w of the value approximator be the one that minimizes the difference between the value Vax, w) and its evolution through the Bellman operator Va( w) by minimizing the loss function defined in Eq. 7
D Approximation of the state entropy term
We introduce here an approximation for the state entropy term in NeuroMOP networks that maximize cumulative future action-state entropy. The state entropy term could bring a fundamental contribution in the overall desired large occupancy of the state space when the magnitude of the action signal is weaker. To quantify the state entropy term, we leverage the non-linear dynamics of the RNN. By taking the small noise limit, we assume that a network maximizing state entropy would exhibit preference for the regions where the non linear dynamics induces larger changes in phase-space volumes, resulting in larger occupation in the state space. We rewrite the RNN's s dynamics Eq. [] in differential form as du' = dx 81V f (x. a)da where f=(x, a) = + Φ(Jr + I(a)) is the dynamics of the RNN for fixed policy i (-1æx) and where we made explicit the dependence from the MOP actions. We quantify the changes in the occupation by looking at the changes in the volume in
18

---

(8)
(e)
0.14
800
2.0.1
0-
400
600
800
pdf
0.0 0.2 0.5
(b)
0.14
0.04
400
12.0.11
0.02
0.06
800
600
pct
0.0 0.2 0.5
(c)
0.14
(g)
0.4
8-08
201
0.011
E 0.2
0.08
pcf
0.0
(d)
(h)
0.14
8-155
X01
80.11
0.08
600
600
0.0
Figure 5: Effect of introducing a 'reward' term approximating the state-entropy. (a) The free network is characterized by high energy consumption and the exploitation of the saturating states, with the RNN's s neurons alternating between activities -1 and (b-d) Same as in (a), but for increasing values of the / E {0.0, 0.8, 1.5]. With increasing f3, there is larger average occupancy of the non- saturating regime of the neurons. Activities and corresponding energies correspond to single trials, while the occupancy is averaged over the trajectories of Nag 10 agents.
(e) Lifetime as function of 33 With reduced action magnitude, for small values of B, small fraction of agents (~ out of the Nav 10) fails to avoid the terminal state. (f) The exploitation of the non-saturating regime leads the NeuroMOP network to increase also (Tax), ), i.e., the average standard deviation of the 'jumps' done by the activities in two consecutive temporal points in the dynamics. (g) Average standard deviation over the trajectories is not affected by 3. (h) The effective dimensionality of the action signal is reduced for 3 1.5 in order to drive the dynamics in the 'rewarded' saturating region. Averages are over Na = 10 networks trained for Nep = 60 epochs, with batches of Neros = 10 trajectories. Standard deviations computed over batches of Ntraj 50 trajectories. Errors are SFM.
the state space
Vol(dx') det(1 + 8tV fz, (2.)
(1+ (I+ Vol(dz),
(18)
where we exploited the fact that for small öt the RNN dynamics behaves as a perturbation of an identity transformation and we can approximate the determinant with the trace. This "extra' term fr(V (I, a)) represents the contribution of the RNN dynamics in increasing the occupation of the state space. We take this term as the approximation of the state entropy H(S) jr, a) and add the intrinsic return the MOP agent receives along trajectory. The value function is then modified as
(x) ET-TTP Σ (az (A.((()) + (TVVS (0)) =
(19)
where introduced the 'temperature" hyperparameters and 3 regulating the amount of action and state entropy. Finding the optimal policy maximizing this value function reduces to solving the same constrained minimization problem defined in Eq. [3] with the value function V, (x) defined here. The resulting Lagrange multiplier follows the same expression as in Eq. [4] where the state entropy term H(S)a, ar a) is substituted by our approximation Tr(Vf=(x, a)). The optimal policy maximizing the value function defined above is
(a/x)
1 ((x)
(FT(V/ p(a p(el
(20)
Effectively, we will set a = 1 and measure the state entropy temperature in units of a.
We test the effect of introducing the state entropy term in the NeuroMOP network while satisfying the energy constraint, as defined in Appendix B We start with an RNN showeasing chaotic dynamics, with large energy consumption, when left free to evolve with no external currents, i.c., with the
19

---

Free
(b)
NeuroMOP
(c)
(d)
1000
0.20
too
3 0.15
0.10
-1
500
1000
500
1000
40
epochs
epochs
0.15
0.15-
5.54
0.11
0.11
E0)
NeurMOO
0.03
<3.94
500 1000
1000
pof
Figure 6: Energy constraint problem with noisy RNNs. (a) The RNN with external control (I= 0) now has an intrinsic source of noise in the dynamics (n = 0.1, parameter scaling the external Gaussian noise term É(!)) (top panel). The energy is also subject small fluctuations due the noisy nature of the activities (bottom). (b) Given the external source of random noise, the NeuroMOP network learns generate long lifetimes by keeping the energy close, but far enough from threshold. (c) The NeuroMOP network learns to avoid the terminal states and generate variability, but in this case the lifetime does not reach (on average) the maximum episode length. (d) On the other side, the average standard deviation of the trajectories increases with learning. (e) Average probability density function of the occupancy in the activity space. Averages over N 10 networks with batches of Ntraj 10 trajectories, errors are SEM.
dynamics defined in Eq. with I (Fig. (4). Without the state-entropy term (B = 0), the choice of the parameter p (p 1.2) rescaling the external current is such that the injected external currents cannot lead to the full occupancy of the state space. Thus, the NeuroMOP network learns to avoid the terminal states but still largely exploits the saturating states of the transfer function (Fig. 6b). With increasing B. the NcuroMOP network gradually expands its occupation in the activity space, favoring those regions leading to larger activity changes (Fig. (o.d). First, we note that we match the average lifetimes for different values of /3 (Fig. 6b) To test the effect of introducing the state entropy term, define for each neuron i the activity 'umps' Ar; the difference in activity between two consecutive time points, i.e., (t, = xt 1) (t) Vi 1,.... N. Therefore, we introduce the standard deviation of these series averaged across neurons, i.e., (aaz). and find that it increases with the contribution of the (approximated) state entropy (Fig. 6D. Conversely, the average standard deviation remains roughly constant (Fig. (g. To drive neurons towards the highly sensitive region of the transfer function, the NeuroMOP network reduces its effective action dimensionality even when far from the threshold, deviating from the maximum available dimensionality when / 1.5 (Fig. 6b).
E RNNs with noisy dynamics
We extend the NeuroMOP network controller of noisy RNN, following the same dynamics as in Eq. I but with an additional Gaussian noise term as
N Σ
2,(1 + = (1((1) + 51
Zi
+ +
Jo + + + (10)++(()(()))))))))
(21)
where E; E / (0, 1) is an i.i.d. normal random variable acting on each neuron i, and n is the noise amplitude. With the introduction noise in the network, we consider an environment whose transition probability is defined over continuous set of possible values. Therefore, sampling from the optimal policy requires performing the integral that appears in the r.h.s of x=" =²(a() z(aee" e Jer p(x (x')),
20

---

net
(b)
NeuroMOP net
(c)
(d)
(e)
0:15
5.54
1000
HLA(A)
x60
xto
ED2
Cou
1.54
1000
1000
1000
FUAt state space regions
opocha
Figure 7: Energy constraint problem with non-saturating transfer function (o(.) = ReLU (-)) (a) The free network (Eq, [ with I =0) is characterized by exploding patterns of the RNN's activity. Consequently, its energy consumption quickly diverges as well (not shown). (b-c) The NeuroMOP network learns to keep the RNN's activity bounded (b) by avoiding a terminal state of high energy consumption (c). Here, we choose arbitrary threshold of L 0.13. (d) Effective action dimensionality when far (E(z) < L) and close (E(x) L) to the terminal state. (e) Average lifetime as a function of the training epochs. Averages over Na 10 networks with batches of Ntraj = 10 trajectories, errors are SEM.
Here, we describe an approximation the integral, suitable for our transition dynamics. For small öt, we can locally linearize the transfer function (--), and replace normal variable with Bernoulli variable with p = 0.5. With this approximation, the integral simplifies to a sum of positive and a negative contribution of the noise over the dynamics, namely
f (t'p(="(z,,))V( (Val//aa.. -V)
(22)
where (x, a = +1 defines the evolution of the dynamics as in Eq. 21 when the external noise takes the values +1 or 1, respectively.
The dynamics in Eq. 21 in the absence of external control (1 0) generate noisy patterns of activation, with neurons, and consequently the energy, randomly fluctuating (Fig. 6a). We test the NeuroMOP network in the same energy constraint problem defined in Appendix B Due the inherent random fluctuations in the activities, long lifetimes are only granted if the RNN is kept sufficiently distant from the terminal state. The NeuroMOP network adopts this strategy, reducing its action entropy even before approaching proximity to the boundary (Fig. (p). Nevertheless, the stochastic nature of the RNN can still lead the network to enter the terminal state after training: while the average lifetime approaches the arbitrary length of the simulation, it always remains below threshold (Fig. (b). The NeuroMOP network learns to increase the average standard deviation of the individual trajectories (Fig. 61I,, and to broadly occupy the activity space (Fig. 6b).
F RNNs with non-saturating transfer function
The ability of the NeuroMOP network to avoid terminal states and generate variability is independent on the choice of the RNN parameters and dynamics. To illustrate this point, we introduce RNNs following the dynamics defined in Eq. [ but employing non-saturating transfer function, specifically (.)) = ReLU(-). In the absence of external control (no external currents,
0), the RNN activity exhibits runaway excitation of its neurons (Fig- [pp), resulting in unbounded levels of energy consumption. We set thus an arbitrary threshold of high energy and test the NeuroMOP network's ability to inject variable currents while avoiding the terminal states xl where E(x!) > L. The network successfully bounds the RNN's activity pattern (Fig. (7) by keeping the energy below threshold for the whole duration of the episode (Fig. 70). Notably, the inherent diverging drive of the network given by the linear transfer function, makes it highly responsive to positive input. To avoid this, the NeuroMOP network reduces action entropy throughout the whole episode, not only in proximity of the energy threshold. The energy constraint imposes in the action space a no-go direction for all states in state space due to the high susceptibility in the RNN. Thus, the effective action dimensionality loses one degree of freedom independently of the distance from the terminal state (Fig. 71). The NeuroMOP network rapidly increase its lifetime through learning (Fig. 7p).
21

---

(a)
(b)
-082
>0.01
(c)
density
toxt
0.82
spochs
Figure 8: Balancing cartpole. (a) Scheme of the cartpole. The controller network has binary actions (forces) {-40, 40) on the cart. (b) Probability density function of the occupation of the cart position and pole angle 0. The MOP network balances the pole while generating variability in its state variables. (c) Lifetime increases with the training of the value function. Averages over Nav = 10 networks, errors are SFM.
Table 3: Hyperparameters for the balancing of the cartpole
Parameter
Value
cart mass M pole mass m
1.0
0.1
pole length gravity acceleration, g
1.0
9.81
discount factor y
0.98
number of hidden layers (FFN)
I
hidden units per layer input units Ninp FFN nonlinearity training epochs Nep number of agents Nag
10
4
ReLU
10
10
trajectories per batch Niraj
20
optimizer
SGD
learning rate !
0.02
G Balancing the Cartpole
We test the ability of MOP to control and generate diverse behavior in a system with physically realistic dynamics. We consider MOP 'network' balancing cartpole 69 70 composed of moving cart with pole attached and free to rotate (Fig. (p). The system has four degrees of freedom: the cart position the pole angle 0 and the corresponding velocities, å and respectively. The value function approximated via a feedforward network receiving as input the four variables (hyperparameters are reported in Table 3). When not in terminal state, the MOP network acts directly on the cart with two possible actions, which are binary forces F E {040, 40), with dynamics
sin f + cos(f) (-F [F +sin
M+M
(23)
cosos
m++M
F + ml62 sin cos
#
(24)
M +
where M is the mass of the cart, m and the mass and the length of the pole and g is the gravity acceleration. Note that no damping is applied to the cartpole, i.c., the system is frictionless. The MOP network enters a terminal state when either the cart position of the pole angle overshoot their threshold, specifically when |z| > 1.8 or ( > 0.62 (radiant units). We threshold the values of the velocities such that they never exceeds the values |.2| 6 and |0| = 3. We find that the MOP network can generate variability the cart position and the pole angle (Fig. gb) while balancing the pole, ensuring enough distance to the terminal states (Fig. ge).
22

---

(8)
5.5
1000
0.1
ted
500
-0.1
0.4 1.5
-0.1 0.1
0.2
15
Figure 9: Two rooms arena connected by a narrow corridor in the neural space (æ1, x2). (a) The NeuroMOP network occupies both rooms of the arena and acts deterministically to cross the corridor (example of one trajectory with T 5000), and (b) learns quickly to avoid the complex terminal states (note low action entropy at the corridor). Averages over Naw 10 networks with batches of Ntraj = 10 trajectories, errors are SEM.
# 0.1
(b)
P 0.5
-33
000
1000
-33
+0 000
1000
600
600
0.3
0.34
-.3
25
50
-0.5
0.8
25
50
Xt
opochs
opochs
Figure 10: Adding extrinsic reward term to MOP changes the behavior in the constrained neural space problem. Terminal states are the states outside the boundaries of square in the neural space (21, 222, centered in zero and with total side length = 0.4. The NeuroMOP network gets extrinsic reward r = +1 whenever in an inner square centered in (011, and with side length 1, = 0.1. We introduce a parameter μ regulating the balance between extrinsic reward and the action-entropy term. (a) Probability density function the occupation (left) and lifetime (right) for μ = 0.1. (b) Same as in (a), but for /a 0.5. Averages over Nav 5 networks with batches of Ntraj 10 trajectories, emoos are SFM.
H Crossing a narrow corridor
MOP agents' ability generate future entropy is guaranteed by their capacity flexibly switch between stochastic and deterministic behavior. We provide an additional example of this flexibility by showing the behavior of the NeuroMOP network in problem where highly deterministic actions (locally) required. Specifically, we consider the same neural space (r1, x2) we introduced in Sec. 3.2 but now terminal states are defined such that the network can only live in a region space defined by two circles (c.g., rooms) connected by narrow available region of the neural space (e.g.. corridor). We observe that the intrinsic motivation for occupancy drives the NeuroMOP network to visit both rooms. To succeed in that, the NeuroMOP network largely reduces its action entropy to cross the narrow corridor, to later increase its stochasticity when the larger rooms (Fig. (h). Importantly, crossing the narrow neural space does not affect the agent's ability to avoid the terminal states (Fig. (g).
I Adding an extrinsic reward
We showed that the NeuroMOP network is able show complex behavior (e.g., crossing a narrow corridor) without the need to specify any reward function. Here we show that NeuroMOP is compatible with the addition of an extrinsic reward function. In this case, the value function modified as
D Σ
V= (z) =F ETTTT
(aH((A)))) + ( a(t)
(25)
23

---

where the hyperparameters a, 3 and regulate the weight on action entropy, state entropy and extrinsic reward maximization, respectively. By solving the same constrained minimization in Eq. 13 we obtain the analogous formulation of the optimal policy maximizing the value function described above as
1 (14( En plel),,)) ((x)
('(or)
(26)
As before, set 0 and = 1 and measure the weight of the extrinsic reward in units of a.
We have already shown that by setting specific terminal states structures we can have MOP agents generating behaviors without the need to specify any reward function. We test now the effect of providing the NeuroMOP network with an additional extrinsic reward. We consider the same problem as in Sec. 3.2 where the NeuroMOP network activity has to be confined in square region of the neural space (I), here centered in zero and of side length = 0.4. Here, we modify the scenario by adding additional state-dependent reward r(x) = 1 that the NeuroMOP network receives every time its activity 2' in a smaller square of the (z1, x2) space, centered (0.1, 0.1) and with side length 4,- 0.1. By modulating the term regulating the importance of the extrinsic reward, observe different behaviors. When μ is small enough (μ = 0.1), the NeuroMOP network shows only slight preference for the rewarded region of the square, maintaining overall uniform occupancy (Fig. TO a). In contrast, as increases (μr 0.5), behavior collapses to the occupation of the extrinsically rewarded square (Fig. T0 b). Finally, as the rewarded region is not in conflict with terminal states, a more conservative behavior also associated with a decreased variability of the lifetime (Fig. (0). Our results highlight significant distinction between MOP and the standard extrinsic reward maximization framework: the introduction of extrinsic rewards is associated to collapse of the repertoire of the observed behavior into simpler behavioral patterns. Certainly, while the agent in both cases acts to maximize what is asked to, the critical question remains what constitutes interesting behavior, as have argued throughout this paper.
Pseudocode
We show here the pseudocode of the algorithm (Pseudocode I). While sampling from the policy, the network receives a teaching signal regarding the presence of terminal states. Note that we enforce that the value function approximation has value zero in all terminal states regardless of their approximated value. Also, to enhance stability, the contribution to the loss function coming from visited terminal states have been weighted by factor order of magnitude larger.
24

---

Data: Random J, w. K, 2o
for i 1 to N_ep do
Initialize L O:
for traj: :1 to N traj do x Io:
for :: 1 to T do
V(a,, w) x. w,
if x > Fth then
L =+ 1. w) - 0)2:
tend + t;
break
end
else
2 Ea (z
V2 w) InZ;
C C+ 0.1 (V(z, w) Vo(2, 2))²
end
a T(1z);
RNN(x,);;
end end
Adam(w, L)
end
Algorithm 1: Pseudo-code for the training of the weights w of the feedforward network.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's contributions and scope are introduced in the Abstract and in the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the corresponding paragraph of the Discussion
section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [Yes]
Justification: Assumptions and proofs, as well as analytical derivations, are shown the Appendix. All used equations numbered and rrfss-referenced between main text and appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yss]
The introduced architecture is described in Fig. Parameters and specifica- tions of the simulations are also reported in the corresponding paragraph of Appendix A Methods to evaluate the networks' performance are also extensively discussed.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
27

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: We include in the supplemental material the codes to generate the data and the corresponding figures.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcceeeooooooop httpa///gu.deee//eeuubbbbiiiii sesiissinnn swiniisssoopiiiyyy Policy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We report in Table 2 in Appendix A all the details regarding the type of optimizer and the hyperparameters of the optimization problem.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report error bars as the standard error of the mean (SEM) across multiple
agents.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Computations run easily on commercial CPU,
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpps//neuurps.... http:////cpppp.eeeiiccceeeeeeeee
Answer: [Yes]
Justification: This a completely theoretical work on neural variability. No external dataset has been used. The implications of this work the same any other theoretical work, e.g. reinforcement learning.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The implications of this work are the same as any other theory.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not pose any risk for misuse of this theory.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No existing assets have been used in this work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
30

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce any assets that could be potentially used outside this work.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not include human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
31

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "NOimxQDLCK", "reviewer": "Reviewer_QFbP", "summary": "This paper proposes a mechanism to induce variability in \"reservoir\" recurrent neural networks without impinging upon task performance, by maximising the cumulative entropy of future states and actions/behaviors. These actions are provided by a controller network to the reservoir as input currents. The authors demonstrate through experiments that the induced variability does not come at the cost of adhering to constraints on energy or specific neuronal activities, or task performance (there is no explicit reward in these tasks for the proposed framework, and so this is measured by the survival time, i.e., timesteps until a terminal state is reached or a constraint is violated). Comparisons with networks without the input current modulation or those with explicit rewards show that the former are unable to properly satisfy task constraints, while the latter find overly conservative or \"risk-averse\" solutions that suppress variability. The demonstrations also show that the proposed framework leads to networks switching between more deterministic modes of computation (lower action entropy) near terminal states/constraints and more stochastic (higher action entropy) ones otherwise. Overall, this paper provides a novel perspective on how neural variability could be maximized while still allowing for accurate performance and avoiding terminal states, especially when an explicit reward function is unavailable or undesirable.", "questions": "See the Weaknesses section.", "limitations": "The authors have adequately discussed limitations related to computational complexity and not learning the policy in the Discussion section.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-motivated, clear, and provides a unique perspective on how there could be controlled variability in a system without negatively affecting task performance. The figures and overall presentation are good and clearly demonstrate the validity of the central claims.\n2. The experimental results include specific controls for the proposed mechanism – the authors show results for networks without any input current modulation and also networks with explicit constraints imposed by a reward function.\n3. In the appendix, the authors show that their central claims are largely valid even when there are other sources of variability such as intrinsic noise in the networks.\n4. While the tasks are simplistic, they are well-designed and quite interpretable, allowing the claims to be validated easily through the visualizations.", "weaknesses": "1. This is a limitation acknowledged by the authors, but it seems like the computational complexity for the framework is quite high, so it is difficult to evaluate how this would scale to more complex tasks or multi-task settings.\n2. To my knowledge, and perhaps I have missed this, but the authors have not provided clear connections to the biological inspirations for the proposed mechanism, such as perhaps neuromodulatory mechanisms, or comments on its biological realism. Could the authors elaborate on this and are there any testable predictions for this model of neural variability?\n3. While the simplicity of the tasks is an asset, it would also be important to see what happens when there is a greater diversity among tasks in a multi-task setting (see Yang et al. [1]), and when performance is not only linked to constraint satisfaction or \"survival\". For example, would this framework impede performance when one of the tasks explicitly requires less variability as is perhaps the case in a memory task (see Yang et al. [1] again for examples)?\n4. This is a minor point, and a suggestion rather than a weakness, but there are some interesting recent works that could be mentioned to strengthen the background:\n    1. Takasu & Aoyagi [2] discussed an input current modulation mechanism and how it affects the Lyapunov exponents of reservoir networks' dynamics – specifically, suppressing chaos and ensuring networks are at the edge of chaos to enable effective information processing (and is thus related to the variability in these networks; also related to [59] from the paper). It would be interesting to briefly compare/contrast the proposed mechanism/goals to that proposed in [2].\n    2. In lines 33-37, the authors discuss works where internal synaptic noise is proposed as a mechanism for neural variability, and mention that some works use this assumption to \"describe variability during spontaneous activity–in the absence of sensory stimuli\". Works such as Asabuki & Fukai [3] and Krishna et al. [4], where such a mechanism is assumed and used to describe properties of spontaneous activity, could be discussed (in addition to [18, 20] from the paper) to provide a better idea of the implications of such mechanisms.\n\n**References:**\n1. Yang et al. “Task representations in neural networks trained to perform many cognitive tasks.” Nature neuroscience vol. 22,2 (2019): 297-306.\n2. Takasu & Aoyagi. “Suppression of chaos in a partially driven recurrent neural network.” Phys. Rev. Research 6, 013172 (2024).\n3. Asabuki & Fukai. “Learning rules for cortical-like spontaneous replay of an internal model.” bioRxiv (2023): 2023-02.\n4. Krishna et al. “Sufficient conditions for offline reactivation in recurrent neural networks.” The Twelfth International Conference on Learning Representations (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textit{Weaknesses: This is […] multi-task settings.}$\n\nWe agree with the reviewer’s comments regarding the high computational cost of our framework, as acknowledged in the manuscript. This complexity arises from a specific choice we are committed to in our current approach, which employs an exact representation of the policy (Eq. 4). However, there are several ways to improve computational complexity, by furtherly reducing the dimension of the already low-dimensional action space ($N_c=8$), or by approximating the policy by a neural network. These limitations are already commented in the discussion section.\n\n$\\textit{To my knowledge, […] neural variability? }$\n\nWe thank the reviewer for the question.\nWe will address this comment on these points in the next revision by adding paragraph P1 you will find in the global responses of the rebuttal. \n\n$\\textit{While the simplicity of the tasks […] for examples)?}$\n\nWe thank the reviewer for the question. We predict that our MOP framework will lead to deterministic behavior when required by any task, as it would be in a working memory (WM) task where a memory item needs to be stored. In such cases, we do not expect that a MOP neural network would generate variability that impairs performance. However, consider a more interesting scenario where many memory items need to be stored in a limited WM system, but it is unclear which one should be stored at any time. In this situation, a system like MOP, which promotes variability on the stored items while keeping functionality of the network, might better suited to solve the task. The limited time does not allow us to explore this, but it represents a relevant direction for future work.\n\nTo complete our response, we have added two additional simulations that partially address the reviewer’s comments. First, we show that in a standard cartpole task (Fig. A of the rebuttal figures) MOP agents are capable of generating variable behavior while controlling non-linear dynamics and confining the corresponding state variables in a desired space region – a reward maximizer agent will lead to just the up-right position without any variability. Second, in a scenario with two arenas connected by a corridor in the neural space (Fig. D of the rebuttal figures), we find that a MOP agent can generate deterministic actions to move neural activity through the narrow corridor. These examples demonstrate once again that MOP networks are capable of deterministic behavior when needed. \nWe will add paragraph P2 of the rebuttal global response to the Discussion section.\n\n\n$\\textit{This is a minor point […] background: Takasu and Aoyagi [2] ...}$\n\nWe thank the reviewer for the suggestion. The proposed paper investigates the chaoticity in RNNs as a function of the input statistics, using a mean field approach to compute the maximum Lyapunov exponent (MLE). The study finds maximum memory capacity for MLE close to zero, indicating that the network operates at the edge of chaos. This aligns with previous literature suggesting that information transmission can be maximized near the chaotic state boundary. Testing the behavior of the MLE in our framework is a promising direction for future work. Our input current statistics do not allow for a closed mean-field form, hence we propose a numerical computation could be performed as in Laje and Buonomano 2013. We expect MOP networks to maintain consistent neural trajectory distances despite external perturbations, hence approximating an MLE close to zero and supporting the reviewer's points. We will add to the Discussion section paragraph P3.\n\n$\\textit{In lines 33-37, …}$\n\nWe thank the reviewer for pointing out relevant literature on the role spontaneous activity and reactivation play in the brain, which will include in the next version of the manuscript.\nThe mentioned papers offer valuable insights into reactivation patterns using RNNs, driven by observations that spontaneous activity in the brain resembles stimulus-evoked activity. Reactivation is thought to play a relevant role in the brain, such as facilitating the stabilization of memory patterns or the generalization of acquired knowledge. \n\nAsabuki et al, utilize reservoir computing and introduce a learning rule based on neuron responses’ prediction, which allow for the formation of an internal model representation of sensory experiences and enabling reactivation without inputs. Relevantly, the learning rule minimizes the Bayesian surprise by reducing the KL divergence between prior and posterior distribution. \n\nKrishna et al. show that continuous-time, noisy RNNs trained to track state variables in neuroscience-related tasks (as spatial navigation), learn to match the desired output while compensating for intrinsic noise. The defined dynamics produces patterns of reactivation even in absence of external input. In particular, it is the very presence of noise that fundamentally shapes the dynamics and leads to the emerging property of the reactivation patterns. \n\nBoth approaches enforce reactivation patterns by indicating the network the ‘favorable’ patterns to be reproduced, either by minimizing the Bayesian surprise or by informing the loss function with a time-dependent target. In our framework, the goal-directed behaviors naturally emerge from the intrinsic motivation of generating action path entropy and the resulting neural patterns are as variable as the terminal states defining the task allow. If we assume that reactivation of patterns is linked with the replay of activity that is relevant for survival, then we may expect MOP networks to favor those states by assigning greater values to those activity states. \n\nWe appreciate the reviewer’s suggestions for future research directions and will add paragraph P4 to the Discussion section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b8TwYpSVEY", "reviewer": "Reviewer_Hyea", "summary": "This paper applies the maximum occupancy principle (MOP) -- previously introduced as a normative theory of behavioural variability -- to recurrent neural networks, thereby proposing MOP as a normative theory of neural variability. The MOP postulates that an agent seeks to maximize future occupancy of its state-action space. A key insight of the previously published MOP paper (ref 42) was that an MOP-following agent naturally learns “good” behaviour by actively avoiding terminal states (e.g. death) as those imply very reduced future state occupancy. This effect is demonstrated again here by showing that MOP-following RNNs can be made to avoid specific activity patterns, but remain maximally variable otherwise. In terms of methods, the main challenge here was to approximate the MOP value function for nonlinear RNNs -- the sole determinant of the optimal policy. The authors do so by training a NN value function approximator on a regression objective derived from the self-consistency (Bellman) equation that the value function must satisfy. The framework is then applied to a few toy setups, including a context-dependent drawing task. Some technical limitations are discussed.", "questions": "Minor typos I picked up:\n- sentence on l.54 (“This theory frames...”) is grammatically weird.\n- l.128: inevitably constraint → constrain\n- l.135: of parameters → with parameters\n- l.178: a terminal states → state", "limitations": "Some technical limitations are discussed but it would be nice to more clearly spell out the conceptual limitations (c.f. above).\n\nEDIT: following discussion with the authors, I am raising my score from 5 to 6.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The idea of applying MOP to RNNs is potentially interesting, as it provides a new normative theory of neural variability that will be interesting to confront to neural data -- this paper provides some technical foundations for investigating this hypothesis further. The paper is technically well executed.", "weaknesses": "There is some, but not an awful lot of, added value relative to ref 42.  The MOP is a creative new framework, but the idea that biological agents learn what _not_ to do _instead of_ learning what to do seems a hard sell. The drawing tasks of Figs 3+4 seem carefully designed to demonstrate that MOP-following networks can achieve some “positive” functionality by exclusion, but I have a hard time imagining how the framework would scale to even simple control tasks like swinging a pendulum up; many such tasks are defined by what the agent must do, and many suboptimal states are actually not at all absorbing / terminal. I think the paper probably ought to discuss these limitations in more depth.\n\nThere is at least one other normative theory of neural variability that ought to be mentioned in the intro more explicitly: sampling-based probabilistic inference, where variability represents uncertainty. (Ref 26 is cited for \"nonlinear network interactions leading to variable activity patterns\" but has nothing to do with networks. Echeveste et al 2020 by the same group might be more appropriate in this context.)\n\nIn summary, I think this idea is potentially interesting -- I view it as a putatively useful theoretical framework for studying how brains learn from bad outcomes (which engage a very different system from the brain's dopaminergic reward system). However, the paper as it currently stands is rather incremental and perhaps not of broad appeal to the NeurIPS community; I would strongly encourage the authors to explore the ramifications of the neural-MOP framework for neuroscience, articulating predictions for neural variability in specific setups where neural data is available for confrontation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textit{There is some, […] discuss these limitations in more depth.}$\n\nWe appreciate this criticism, which allows to discuss more deeply some fundamental features of our MOP network. Let us take the example of the inverted pendulum that the reviewer has brought up. Our intrinsic motivation approach with value-function approximation can generate interesting behavior in this example, including balancing the pole upright. \n\nTo show this, we have substituted the environment in the MOP network, i.e., the RNN, with a cart-pole constituted of a cart moving along a linear track and a vertical pole attached to it (Fig. A of the rebuttal figures). The cart-pole receives five possible actions: an input to the left, to the right, a strong input to the left, a strong input to the right, and no input, all acting directly only on the cart. Terminal states are encountered whenever the angle between the pole and the normal to the cart exceeds a certain threshold ($|θ_{th} |=0.74 rad$) or the cart hits any of the borders of the linear track ($|x_{th} |=2.4 a.u.$). The value function is approximated via a FFN receiving as input the variables of the cartpole (position, linear velocity, angle, angular velocity) and following the training described in the manuscript. As learning progresses, MOP improves its ability to avoid terminal states (Fig. A). Our algorithm can balance the pole, i.e., avoiding terminal states, while at the same time showing variability in the pole angles and carts positions. This example shows that MOP with terminal states can generate quite interesting behavior, without the need of telling the system what exactly to do. A reward maximizer agent will not lead to variable behavior and instead would only show the up-right position. How far one can go with MOP across different dynamical systems and generate interesting variability is a matter of future exploration. \n\nMoreover, we want to remark that we did not intend to say that biological agents learn what not to do instead of learning what to do. As mentioned in lines 127 and 128 of the manuscript, we want to mention the fundamental difference of having extrinsic rewards versus intrinsically motivated agents, with the consequences it has on the generated variability. By introducing extrinsic reward, agents are constrained to choose a single behavior (the optimal one) and follow it, while introducing terminal states we let the agent free to develop all possible behaviors allowed to solve the task.\nWe will add the clarification paragraph P5 of the rebuttal global response to the Discussion section.\n\n$\\textit{There is at least one other normative theory […] }$\n\nWe thank the reviewer for the suggestion, and we will correct the reference in the next version of the manuscript. We will also mention the literature on sampling-based probabilistic inference.\n\n$\\textit{In summary, I think that the idea is potentially interesting […]  }$\n\nWe agree with the reviewer that we are building on a previously published work to extend the idea of the Maximum Occupancy Principle to neural activity. However, we believe that both the technical and conceptual contributions of this paper are significant for the NeurIPS community.\nFirstly, we demonstrate how to control a high-dimensional, chaotic network to perform various tasks using external input currents based on the MOP framework, without the need to train network weights or rely on extrinsic reward signals from the environment. This approach is not limited to simple tasks, as we have shown.\nSecond, we have presented a radically novel theory for neural variability, which is a major theoretical step. There is no precedent in the literature of this new hypothesis and worked out framework. \nIt is possible that these two points were not clearly explained, so we will rewrite old sentences to improve understandability: \n\nWe show that the principle of maximizing occupancy can be extended beyond behavior and may also underpin neural activity. Our work introduces a radically novel theory of neural activity, where neural variability is a cardinal feature rather than a drawback. To illustrate this, we trained MOP agents to control high-dimensional chaotic RNNs in order to perform different tasks defined by the structure of terminal states and the environment, without modifying the internal weights of the network and or providing any extrinsic reward.\n\n$\\textit{I would strongly encourage the authors […] }$\n\nWe thank the reviewer for the encouraging comments.\n\n$\\textit{Questions:\nMinor typos … }$\n\nWe thank the reviewer for picking up these typos, we will correct them all in the manuscript.\n\n$\\textit{Limitations:\nSome technical limitations are discussed but it would be nice to more clearly spell out the conceptual limitations (c.f. above). }$\n\nWe will add the text indicated in the above responses to address this comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NyCR67SbkT", "reviewer": "Reviewer_ivma", "summary": "In natural behaviors, there’s usually variability despite being able to perform tasks with high performance. This paper aims to understand whether it’s possible for neural networks to have high variability while maintaining high task performance and being able to switch to deterministic behavior modes when needed. \nThis paper uses an RNN with fixed weights to model the state of the environment and how it’s affected by the action of the agent. The  agent is modeled by a controller which aims at optimizing the occupancy of the action-state space. This is achieved by having a reward that increases when the agent selects an action that is less likely with the current policy. The optimal value function, and in turn the policy, is approximated by using a single hidden layer feedforward NN.", "questions": "1.\tHow does the structure (the number of hidden layer neurons, depth) of the NN that serves as value function approximator, affect the results?\n2.\tIn Fig 3(c), why is the action entropy low in the lower right and upper left corners but not the other two corners? one would naively expect them to be symmetric.", "limitations": "limitations are covered in the 'Weakness' section.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a reward function that maximizes the action path entropy, and provides interesting examples of how this network performs  in three different example tasks. The writing is clear and easy to follow.", "weaknesses": "1.\tThe tasks are mostly limited to setting several terminating states, so that the MOP network learns to avoid the terminating states while maximizing the entropy. I wonder how general this type of tasks is, or how more interesting RL tasks may or may not be formulated this way. \n2.\tThe motivation of this paper is unclear to me. The authors aim to show that NNs can have high variability with good performance, in order to match natural behaviors, and to propose possible mechanisms for neural variability. However, there is no comparison with experiments to show how well the behavior of the NN matches natural behaviors, or to show how the proposed reward may be superior. The authors also do not explain the generation of neural variability, but instead directly enforce variability in the MOP network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textit{ Weaknesses: 1.\tThe tasks are …}$\n\nWe thank the reviewer for giving us the possibility to clarify the role of terminal states in our algorithm and to highlight the general applicability of our framework to various RL tasks that are not typically formulated in this manner. To illustrate this, we performed an additional simulation where we applied MOP to solve the classical task of balancing the inverted pendulum (Fig. A of the rebuttal figures). We show that the feedforward network quickly learns the approximate the value function and to balance the pole without the need of any extrinsic reward. \n\nFurthermore, our framework is compatible with the incorporation of extrinsic rewards. We have already exploited this property in Appendix D by employing an approximation for the state entropy as an extra reward term (Fig. 5). To further clarify this, we added a simulation where the MOP network gets an extrinsic reward $r = 1$ for occupying the top right region of the square in the $(x_1, x_2)$ space (Fig. B of the rebuttal figures).  The reward function now consists of two terms, an extrinsic reward component and an entropic one.  By adjusting the temperature parameter balancing the two terms, we may observe different behaviors (see left and right panels in Fig. B of the rebuttal figures document), as the network differently balances the internal drive for occupation with the drive to maximize extrinsic reward. \nWe will add these examples in the Appendix to illustrate the generality of our approach. \n\n\n$\\textit{ 2.\tThe motivation … }$\n\nThe reviewer raises a significant point. Our model employs rate-based recurrent neural networks, a feature that makes the direct comparison with neural data unlikely and of difficult interpretation. A significant direction for future work would be extending our framework to spiking neural networks, where the statistics of the spiking could be easily interpreted and confronted with the Fano Factor and the Coefficient of Variation from experimental data. \nNevertheless, with our current setting of rate-based dynamics, we find qualitative similarities by looking at fluctuations, as BOLD signals showcasing large variability at the global scale. Moreover, a state-dependent adaptation of neural variability has been largely reported, with variability (as measured by the Fano Factor) significantly reducing when the stimulus is presented (Churchland et al. 2010). From the perspective of the subjects performing the experiments, switching to a deterministic mode to faithfully encode the sensory stimulus is what guarantees survival. \n\nWe will add these predictions by adding paragraph P1 of the rebuttal global response to the Discussion section.\n\n$\\textit{ Questions: 1.\tHow does the structure …}$\n\nWe thank the reviewer for giving us the possibility to highlight the robustness of the value approximator with respect to changes in the chosen parameters. In particular, our value approximator is a feedforward network of one hidden layer, i.e., we chose the simplest structure allowing to solve nonlinearly separable problems. We found that this simple feedforward network can correctly approximate the value function for a large set of parameters. In terms of number of hidden neurons, we found that this parameter does not strongly affect the outcome of the simulation even when this number was significantly reduced. More relevantly, we proved the stability of the approximator by feeding into the value network only a subset of the neurons from the recurrent neural network. We will clarify the robustness of our results against hyperparameter choices in the subsection regarding the parameters of simulation in Appendix A by adding the following:\n\nWe found our algorithm to be robust across various parameter configurations, including the number of neurons in the hidden and input layers, as well as the total number of layers. An exhaustive characterization of parameter behavior was not conducted, as fine-tuning was not necessary to effectively train the approximator.\n\n\n$\\textit{ 2.\tIn Fig 3(c), …}$\n\nThe reviewer is correct. There is a major observed asymmetry, which could be explained by the fact that all neurons in the network share a common low-dimensional input, i.e. each component of the action signal acts all on the RNN neurons. Therefore, the resulting activities are expected to be highly correlated. We expect these correlations to be attenuated via acting on the read-in matrix K, for instance imposing sparseness on this matrix or increasing the dimensionality of the action signal. This analysis represents a relevant direction for future work which will allow to potentially exclude the presence of other factors (e.g., initialization of the weights in the value function) influencing this asymmetry.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7gkPGcWzE", "reviewer": "Reviewer_TEkX", "summary": "The authors propose a principle for selecting actions to drive recurrent neural network activities which aims at maximizing the variability of the neural activity while avoiding unwanted states. They define unwanted states as states where no action is possible, and use a reinforcement learning framework to select the input and analyze the coupling between action choice and network dynamics. They apply their networks and input selection to a few tasks where maximizing the entropy within some boundaries is defined as success, and show that the network performs those tasks.", "questions": "Finally, the authors mention that MOP might take deterministic actions to increase entropy later. This would be, in my opinion, an important contribution, but I haven't seen this being done explicitly in the current paper. Why not try to have two circles connected by a very thin line, to showcase this?\n\nThere is a edge case that bothers me. If we consider the perfect value function, all action paths within the boundary have a limited entropy, but an action path that leads to leave the boundary has technically infinite entropy (it is never taken, thus its probability would be zero, hence infinite entropy). If this is correct, how does the neural network for the values avoid this problem?", "limitations": "There is no negative societal impact.\nI think the limitation of learning a policy is fair, but there are more important concerns, namely", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The framework is, to the best of my knowledge, original. It is also sound in its analysis, and well explained.", "weaknesses": "I have two concerns, the first one is (for lack of a better word) teleological, the second more practical.\nMy first concern is that the authors use the word performance often and refer to their networks as solving the task. But from what I can read, the \"task\" of filling as much of the space (MOP) as possible was never given to the agent/input controller. Thus, if the task of the network is to maximize its entropy while remaining on a bounded region, and the R network is not taught to maximize entropy, we can hardly argue that the R network failed. I get that the point that MOP can be useful for some problems focusing on exploration, but it is a bit odd to talk about solving tasks and performances when such performance was not told to the agent, but rather an agent was build for that task (or for another one in the case of R). Please correct me if I misunderstood.\nMy second concern is that an agent with a binary value function which chooses random actions (value 1) except if those have a punishment (value 0). Such value might be easy to learn (it only requires learning a boundary, which is smooth in the problems presented), and if the state space of the actions is large, random choices could be very close to MOP (because a random action sequence is likely to go through many states). While this is not necessarily true, it is worth checking, as it would make the whole MOP less impactful.\n\nTechnical remarks:\n- The addition of stochasticity to the R network is a bit tricky, because the MOP agent never had the problem that it might \"accidentally\" jump into the terminal state if it did not want to. Thus, for \"survival\" it might be better to simply stay in some very small region that is as far as possible from the terminal states, because there we find the smallest chance of accidentally going to the wrong region. A better option would be to give R a random action and then ask if it wants to take it. But I suspect that this would give a high variability.\n\nMinor issues about literature and references\n- When the authors mention that usually recurrent neural networks tend to have neurons with saturation, it seems like an unfair comparison. A network trained with a specific task do not have an incentive to maximize the number of states, but this could be added to the loss function (if there is one explicitly) or simply enforced by some intrinsic plasticity rule, as some works in reservoir computing have done (both in Echo State Networks for machine learning and biological models such as SORN). Also, it might simply be the case that for a given task it is better to be close to the saturation.\n- In the discussion the authors rightfully mention the variability found in songbirds. But they do not note that it agrees with the works that they mention in the introduction (ref 29-35) which use variability only during learning, and that they use as a current limitation motivating their work. It would seem natural from a neuroscience point of view that variability is suppressed during courting by some executive brain area, rather than assuming that the area that generates the song naturally knows when to change behaviors. \n- The argument that terminal states are those where the agent has no available actions is a bit limiting. Any task that corresponds to reaching a goal (for example a location) leads by definition to a reduced action space, at least in practice (the agent would not leave that position). A note about this would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textit{Weaknesses: I have […] controller.}$\n \nThe reviewer is correct, the only goal of the MOP agent is to maximize future action-path entropy, Eq. 3. The ‘task’ of filling out space was never given but it emerges naturally as the MOP agent seeks to maximize action-path entropy while avoiding terminal states. Our aim is to demonstrate that MOP agents can generate interesting not-enforced behaviors from a simple objective. We acknowledge that our use of the term 'tasks' may be confusing, and we will clarify it in the next version. \n\n$\\textit{Thus, […] misunderstood.}$\n\nThe reviewer raises a valid point, and we appreciate the opportunity to clarify. The R network indeed does not fail in its task, as it is designed to maximize extrinsic rewards (Eq. 10) and learns that increasing its survival leads to the accumulation of larger cumulative reward. Both MOP and R networks have an intrinsic drive for survival, so it is fair to compare their ability to generate as well emerging behaviors. With equal survival time, MOP agents, driven by an intrinsic occupancy goal, explore states more broadly than the R network. As experimental complexity increases, matching survival times requires to quench the source of variability of R agents. In summary, while R agents achieve their task of maximizing rewards, they lack emerging variability, which is a desired property once performance metrics have been optimized. We will clarify this in the manuscript. \n\n$\\textit{My second […] impactful.}$\n\nSee our comments below.\n\n$\\textit{Technical remarks: The addition of … }$\n\nWe apologize for the misunderstanding. MOP agents act according to policy in Eq. 4, which follows a Boltzmann distribution and assigns a non-zero probability to all actions. Hence, they do face the possibility of accidentally jumping into terminal states. Consequently, the comparison between MOP and R agents is not as unfair as the reviewer implies. We will add paragraph P6 of the global rebuttal to the manuscript to clarify this.\n\n$\\textit{Thus, for survival …}$\n\nAs discussed above, both MOP and R agents can accidentally fall outside the boundary but differ in their risk sensitivity. MOP agents, through action exploration, learn better the value function near boundaries, allowing them to be riskier and increase state space occupation. In contrast, R agents, due to their inherent randomness, play it safer and stay more confined, strongly limiting their learning.\n \nFollowing the reviewer’s suggestion, we created a super-R agent with binary value function, i.e., it takes random actions at any time and it can choose to avoid wrong actions leading to a boundary. Not surprisingly, this agent is riskier and comparable to MOP in terms of average occupancy (Fig. C of the rebuttal) and state occupancy entropy ($H_{MOP}=7.34, H_{super-R}=7.38$, where occupancy probability is approximated with the visitation frequency). We think that this super-R agent does not represent a fair comparison, since it hard-wires the avoidance of terminal states, unlike MOP agents.\n\n$\\textit{Minor issues ... When […] saturation.}$\n\nWe agree that RNNs lack an inherent incentive to maximize the number of states, and hence, exploiting the saturation state could be a beneficial strategy in some tasks. However, this is unlikely to be observed in biological neurons, which benefit from maximum information transmission [47]. The reviewer mentions relevant literature where networks are encouraged to live in a ‘healthy’ regime. While we acknowledge that synaptic plasticity can stabilize the network’s regime, our framework only involves learning the value approximator. Hence, the involved time scales are instantaneous and achieved through active control rather than synaptic change. We comment on this in paragraph P7. \n\n$\\textit{In the discussion […] behaviors.}$\n\nThe reviewer correctly notes that much of the literature, including studies on songbirds, focuses on learning. Nevertheless, flexible switching from stochastic to deterministic behavior is observed also in adult birds, where different levels of variability appear in directed and undirected singing even after learning [57]. While various explanations exist for this phenomenon (e.g., variability suppression by executive areas), this example aligns with our thesis that flexible switching between stochastic and deterministic modes may play a cardinal role in biological networks. We will clarify this by adding paragraph P8.\n\n$\\textit{The argument […] beneficial.}$\n\nWe thank the reviewer for the opportunity to highlight our perspective shift. Our framework does not include the notion of reaching a goal. We hope to address a partial limitation in classical neuroscience tasks, where tasks are designed as to have a fixed point of behavior and episodes terminate upon reaching it. This approach is valuable for specific studies, as characterizing the neural basis of motor reaching, but departs from real-world scenarios where agents’ goals are continuously reallocated. We believe studying this reallocation is crucial and presents future challenges for computational neuroscience. By allowing MOP to dynamically develop its own survival behavior we move towards a more flexible and realistic model. To clarify this, we will add paragraph P9.\n\n$\\textit{Questions: Finally, […] this?}$\n\nWe followed the suggestion and implemented a two circles arena connected by a narrow corridor in the neural space (Fig. D of the figures). We show that the MOP network, starting from the left room of the arena, crosses the narrow corridor by reducing its action entropy, to furtherly increase its action entropy in the right room of the arena. We will add this result to the Appendix of the manuscript. \n\n$\\textit{There is a edge case […] }$\n\nAs commented above, the MOP policy follows a Boltzmann distribution, hence even the perfect value function would not lead to an immediate intrinsic reward of infinity. We will clarify this by adding paragraph P6 to the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their insightful comments and for giving us the opportunity to clarify important aspects of our framework. Based on the feedback received, we would propose to modify the Discussion section incorporating the following additional paragraphs.

$\textbf{Addition P1}$ : Our model offers several testable predictions regarding the nature of neural variability we should expect in the brain. Firstly, it predicts that neural variability will persist even after extensive training, aligning with studies reporting large spiking variability even in well-trained non-human primates [19, 27]. Despite this persistence, our model also suggests that neural variability may decrease when terminal states are sufficiently close, i.e., when the system enters into a deterministic mode (Churchland et al. 2010, Nogueira et al. 2018). Finally, our model predicts that reward signaling systems in the brain will also signal intrinsic motivation rewards. This is supported by recent studies demonstrating that spontaneous movements elicit dopamine release [11]. 

$\textbf{Addition P2}$ : We have shown that our MOP network is able to solve different tasks while generating variability. In Appendix, we test our algorithm in tasks where more deterministic modes are required, as when balancing the cartpole or when crossing a narrow corridor in the neural space. 

$\textbf{Addition P3}$ : We have demonstrated that the MOP network can control high-dimensional RNN activity in the chaotic regime. Future research should investigate how MOP-driven input currents affect the RNN's regime itself. We anticipate MOP currents will stabilize neural trajectories, consistent with operating at the edge of chaos, as indicated by a Maximum Lyapunov Exponent close to zero. Characterizing chaoticity as a function of the input properties (e.g., magnitude) (Takasu & Ayoagi 2024) is a promising direction.

$\textbf{Addition P4}$ : We postulate variability to play a relevant role in neural activity as visitation of all activity states may increase flexibility and help generalization. The phenomenon of underlying occupancy of previous states via the replay of favorable patterns of activity is commonly observed in the brain. Various mechanisms in reservoir computing have been proposed to model the emergence of such reactivation in network activity (Asabuki and Fukai, 2023; Krishna et al., 2024). Within our framework, we anticipate that MOP will facilitate the reactivation of all activity patterns relevant to survival, exhibiting deterministic behavior when required by the task.

$\textbf{Addition P5}$ : In our framework, we challenge the canonical approach of training agents to solve tasks by telling what to do via extrinsic rewards. By introducing terminal states, we provide the agent solely with the information of what not to do, allowing agents to develop the full behavioral repertoire compatible with the terminal states and the structure of the environment. We show that MOP can solve various tasks, including those typically addressed with extrinsic rewards, such as balancing a cartpole (see Appendix).

$\textbf{Addition P6}$ : We emphasize that a MOP policy, by following a Boltzmann distribution (Eq. 4), assigns a non-zero probability to all actions, and therefore agents can inadvertently fall beyond the boundaries that define the terminal states. Therefore, MOP agents face the drawback of stochasticity, but they adapt their randomness via the computation of the value function. This is in contrast with R agents, where the stochasticity parameter $\epsilon$ is state independent. These results suggest that state adaptation of stochasticity is a relevant property we might expect in intelligent systems.

$\textbf{Addition P7}$ $\textit{Line 285}$: “[…] uniform occupation as possible, avoiding saturation and encouraging neurons to live in a ‘healthy’ regime, i.e., a regime suitable for computation (Lazar et al. 2009).”.

$\textbf{Addition P8}$ $\textit{Line 292-293}$: “[…] Remarkably, it has also been shown that, during courtship, adult zebrafinches significantly reduce their vocal variability compared to their solitary singing [55, 57]. This state dependent variability adaptation persists even after learning. This switch of behavior from random to more deterministic modes aligns with our hypothesis of the existence of directed variability in the brain.”.

$\textbf{Addition P9}$ : This work highlights a significant limitation in classical neuroscience studies, where a fixed point of behavior is built into the task and episodes terminate upon reaching the goal. In ecological settings, agents continuously reallocate their goals and generate new behaviors. By allowing MOP to dynamically develop various behaviors, we offer a more flexible and realistic model of natural behavior.

Meta-review: This paper explores a novel approach to thinking about the behavioural variability seen in animals. Specifically, the authors examine the potential for maximizing variability in a neural network while maintaining good performance on a given task. To do this, they train a neural network to maximize future action-state entropy while avoiding punishing states - i.e. they train a controller to inject inputs into an RNN in order to fully explore its activity space. The authors compare this model to a more standard model trained to maximize external rewards. They examine performance on some relatively simple tasks, including an energy constraint task and a drawing task.

The reviewers felt that the paper was well-written and presented some interesting ideas. They raised a variety of concerns, principally related to the simplicity of the tests used, relevance to biology, and what is being contributed beyond previous works. However, in rebuttal, the authors were able to address some of these concerns, and the scores raised, bringing the final average score to 6.25. In the AC's assessment, this paper presents some interesting ideas, and the rebuttal did a good job of addressing some of the most important concerns. Given this, and the final scores, a decision to accept was reached.
Decision: Accept (poster)