Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics
Authors: Reece Keller
Alyn Kirsch
Felix Pei
Xaq Pitkow
Leo Kozachkov
Aran Nayebi
Venue: NeurIPS 2025 poster ()
Keywords: NeuroAI
intrinsic motivation
zebrafish
neural-glial
embodied agents
reinforcement learning
autonomy.
Primary area: neuroscience_and_cognitive_science
TL;DR: A novel intrinsic motivation method based on world-model memory mismatch enables embodied agents to exhibit robust autonomous behaviors that closely match whole-brain neural data from zebrafish.

Abstract: Autonomy is a hallmark of animal intelligence, enabling adaptive and intelligent behavior in complex environments without relying on external reward or task structure. Existing reinforcement learning approaches to exploration in reward-free environments, including a class of methods known as *model-based intrinsic motivation*, exhibit inconsistent exploration patterns and do not converge to an exploratory policy, thus failing to capture robust autonomous behaviors observed in animals. Moreover, systems neuroscience has largely overlooked the neural basis of autonomy, focusing instead on experimental paradigms where animals are motivated by external reward rather than engaging in ethological, naturalistic and task-independent behavior. To bridge these gaps, we introduce a novel model-based intrinsic drive explicitly designed after the principles of autonomous exploration in animals. Our method (3M-Progress) achieves animal-like exploration by tracking divergence between an online world model and a fixed prior learned from an ecological niche. To the best of our knowledge, we introduce the first autonomous embodied agent that predicts brain data entirely from self-supervised optimization of an intrinsic goal—without any behavioral or neural training data—demonstrating that 3M-Progress agents capture the explainable variance in behavioral patterns and whole-brain neural-glial dynamics recorded from autonomously behaving larval zebrafish, thereby providing the first goal-driven, population-level model of neural-glial computation. Our findings establish a computational framework connecting model-based intrinsic motivation to naturalistic behavior, providing a foundation for building artificial agents with animal-like autonomy.

Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics
Reece K Alyn Kirsch Felix Pei
Xaq Pitkow 1.3
Leo Kozachkov*h1 Aran Nayebia.12.1
Neuroscience Institute, Carnegie Mellon University
Robotics Institute, Carnegie Mellon University
3 Machine Learning Department, Carnegie Mellon University
'IBM Thomas J. Watson Research Center, IBM Research
{rdkeller, fcpei, akirscht, xpitkow, anayebi] Gandrew. cmu.edu edu leokoz80gmail. com
Abstract
Autonomy hallmark of animal intelligence, enabling adaptive and intelligent behavior in complex environments without relying on external reward or task structure. Existing reinforcement learning approaches to exploration in reward- free environments, including a class of methods known as model-basee intrinsic motivation, exhibit inconsistent exploration patterns and do not converge an exploratory policy, thus failing to capture robust autonomous behaviors observed animals. Moreover, systems neuroscience has largely overlooked the neural basis of autonomy, focusing instead on experimental paradigms where animals are motivated by external reward rather than engaging in ethological, naturalistic and lask-independent behavior. To bridge these gaps, introduce novel model-based intrinsic drive explicitly designed after the principles of autonomous exploration animals. Our method (3M-Progress) achieves animal-like exploration by tracking divergence between an online world model and a fixed prior learned from an ecological niche. To the best of our knowledge, introduce the first autonomous embodied agent that predicts brain data entirely from self-superviser optimization of an intrinsic goal- withhut any behavioral neural training data demonstrating that 3M-Progress agents capture the explainable variance in behavioral patterns and whole-brain neural-glial dynamics recorded from autonomously behaving larval zebrafish, thereby providing the first goal-driven, population-level model of neural-glial computation. Our findings establish computational framework connecting model-based intrinsic motivation to naturalistic behavior, providing a foundation for building artificial agents with animal-like autonomy.
Introduction
Animals exhibit remarkable autonomy, navigating complex environments through self-directed, inter- nally driven behaviors rather than solely responding external rewards immediate physiological needs. Unlike typical artificial agents designed to optimize explicit, predefined task objectives in well-defined problem settings, animals intrinsically explore and adapt in open-ended, naturalistic environments where goals are neither clear nor stable. This capacity for autonomous behavior allows
"ooresponding author.
These authors jointly supervised this work.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

hoto- fleed
Figure 1: Simulation of the zebrafish agent in physics-based virtual environment. A) The 6-link embodiment geometry [ in a environment with dynamic fluid forces. B) The agent controls the torque exerted by motors at each joint (5 DoF) to swim and navigate its environment. C) A custom cosmetic skin to mimic the appearance of larval zebrafish. D) A virtual environment matching the experimental parameters of the open loop protocol Mu et al. 2 The root joint located at the head is fixed during training.
organisms, including humans, to flexibly engage in abstract thinking, exploratory learning, and innovative problem-solving. Central this autonomy is the ability to generate intrinsic goals, seek novel experiences, and continuously refine internal models of the world that inform future actions. Understanding the interplay between extrinsic and intrinsic motivations remains a fundamental challenge in both systems neuroscience and artificial intelligence, particularly for building robust agents capable lifelong autonomy.
However, existing methods in reinforcement learning for intrinsic motivation using world-models such as curiosity-driven exploration and variants [3- -exhibit inconsistent behavioral patterns due to difficulties distinguishing between controllable and uncontrollable stimuli, coping with non- stationary environments, and avoiding the "noisy TV" problem (7]. For example, agents driven by reinforcing prediction-error alone gets locked into pursuing inherently unpredictable or irrelevant aspects of the environment, hindering their ability to develop robust exploratory behaviors [3] Moreover, existing methods typically fail to produce structured and stable behavioral transitions that characterize genuine autonomous behaviors observed in animals and children []]
At the same time, systems neuroscience has historically overlooked the cellular basis of autonomy, favoring experimental paradigms centered on explicit external rewards motivate task-dependent behaviors rather than free, unconstrained exploration. As result, the cellular mechanisms underlying naturalistic autonomy -artimmlllal those involving whole-brain interactions remain poorly under- stood. Growing experimental, computational, and theoretical evidence suggests that non- nouronall cells, especially astrocytes, play critical role in the generation of intelligent behavior 2 14 Owing to their close association with neurons, distinctive connectivity (with a single astrocyte capable of interfacing with up million nearby synapses) and ability to perform integrative computations using hierarchy of timescales, astrocytes are well-positioned to support adaptive, naturalistic, goal-directed behavior 20).
A Dataset for Animal Autonomy To investigate these questions, we study autonomy in larval zebrafish-a uniquely valuable animal model due to their optical transparency, which affords whole- brain calcium imaging via light-sheet microscopy (21] Given that the cellular basis of autonomous behavior remains largely unknown, whole-brain imaging allows us to search across the entire recorded population of over 250,000 cells (roughly 125K neurons and 125K astrocytes) to measure how our proposed model aligns with brain activity 2 We examine a particular cognitive behavior known as futility-induedd passivity, experimentally induced state of "giviing up" observed across the animal spectrum 2 24). zebrafish, futility-inee induced passivity occurs when their swimming effort in a virtual environment fails to simulate optic flow, key visual feedback signal associated with motor- action under ethological circumstances. After some time in the passive state, zebrafish reattempt to swim-an exploratory behavior assess whether the open-loop dynamics of the environment have changed. This cycle between active and passive states continues over the course of a trial.
Mu et al. 2 offers rigorous experimental evidence suggesting this transition to passivity is driven neuron-glial interactions the lateral medulla oblongata (L-MO) that detect and accumulate sensori- motor feedback error. Noradrenergic (NE) neurons signal swim failures which slowly activate radial
The dataset is open-source and freely available here

---

astrocyte with extending processes to the NE cluster of L-MO. At a critical level of intracellular cal- cium, astrocyte processes downstream of L-MO activate GABAergic neurons that suppress premotor neurons relevant for swimming. With extensive manipulation experiments, Mu et al. [2] demonstrate that autility-inducel passivity is not due to fatigue, struggle, or positional homeostasis. Specifically, when visual feedback from closed-loop trials was replayed during the open-loop condition, zebrafish still transitioned to a passive state. This indicates the behavior is mediated by an internal model that is used to detect active sensorimotor feedback error, providing strong evidence for model-based intrinsic motivation.
Contributions In this work. we study neural-glial computations and their relationship to au- tonomous animal behavior by training an embodied zebrafish agent with model- -ased intrinsic motivation and studying its emergent behavior and internal representations. We introduce novel intrinsic reinforcement learning algorithm termed Model-Meeorr--Msmmane alongside virtual environment that captures the fundamental physies of the zebrafish embodiment. Our approach leverages an internal model that continually compares the agent's online memory formed by its current sensory experience against an ethologically relevant prior memory; intrinsic reward then reinforces transitions between behaviors that maximize the divergence between memories relative to its temporal history. 3M-Progress is uniquely capable of producing stable, ethologically- relevant behavioral transitions among several state-of-thee thettr-rrt exploration algorithms in reinforcement learning.
Progress (3M-Progress)
By training embodied agents with AM-Progress, we successfully replicate both the robust behavioral patterns and whole-brain neural-glial dynamics in autonomously -eeauvvii zebrafish. Capturing nearly all of the variance in neural and astrocytic activity, this marks the first predictive and normative model of neural-glial computation. Our agent was not trained any behavioral neural data, and thus represents the first autonomous embodied agent that predicts brain data completely from optimizing a self-supervised, intrinsic goal. To summarize, our technical contributions are:
3M Progress, a novel intrinsic reward that leverages an ecological dynamics prior to guide exploration in new environments.
Emergence of an autonomous behavior known futility- inducedd passivity in 3M- Progress agents, closely matching larval zebrafish behavior.
Alignment between whole-brain calcium response and 3M-Progress agents, providing the first goal-driven model of neural-glial computation.
A general modeling perspective positioning intrinsic reinforcement learning as a computa- tional framework for understanding autonomy animals.
2
Related Work
Neural-Glial Models Although glial cells especially astrocytes are increasingly recognized as crucial to adaptive brain function [2]
25 26]. computational models of 27 Existing models of neuron-astrocyte dynamics typically fall into two categories: phenomenological models that reproduce specific experimental findings like calcium oscillations or epileptic activity 28 31] and simplified, "bottom-up" mathemat- ical models that explore theoretical principles based on astrocytes' unique morphology and anatomy 35 12 [3.. While important, these models are not directly applicable to our setting because they are (a) not yet directly trained on ethological tasks, embodied, or quantitatively validated against real brain data, and (b) typically focused on a single astrocyte rather than a population of astrocytes. In contrast, we adopt "top-down" approach: we train general purpose recurrent architecture to control an embodied agent to perform ethologically-reees relevvatt behavior and find that this imposes strong constraints on the learned representations, allowing us to identify units whose activity patterns closely match those observed in neurons and astrocytes of larval zebrafish.
9
17
14
19
20
neural-glial interactions remain underdeveloped
purpros
Curiosity-driven Exploration Exploration using self self-supervised world-models has demonstrated promising success several standard reinforcement learning domains, and even more recently in language modeling [66. Methods like learning progress [6] and Random Network Distillation (RND) 4 were primarily evaluated using either handcrafted object-centric or state observations with low- din-dimensionality embodiments, limiting the applicability to pixel-based environments or
3

---

continuous control. The Intrinsic Curiosity Module (ICM) [ uses a pixel-encoder trained with an inverse-dynamies loss to predict features rather than raw states, but was evaluated on discrete pixel-based environments like Doom and Atari. While some recent works, such as LEXA [37 or Plan2Explore 38 extend intrinsic curiosity (specifically, Disagreement (5I) to continuous control from visual inputs, success of the exploration policy is defined relative to downstream task generalization. Since these methods do not independently evaluate the quality of the exploration policy, it remains unclear whether these algorithms are powerful enough to learn complex behaviors. In contrast, we investigate completely open-ended autonomous behavior in reward-free, continuous MDPs with nigh-dimensional observations, turning the focus the ability of the exploration algorithm to develop ethological, interpretable behaviors independent of any downstream task.
Embodied Al in Neuroscience Several works have leveraged embodied AI to bridge computational models with neuroscience, including virtual animal models such as the virtual rodent [39 40, which facilitates grounded studies of motor control by replicating rodent motor behaviors across various tasks using imitation learning; the virtual fruit fly, a biomechanically detailed model matching both the visual system and basic flight capacity used study a diverse range of behaviors driven by imitation learning [3]:43]: ; the Open Worm project [44], a biophysically accurate simulation of the C-elegan nematode, but has yet be combined with deep learning and task-optimization; and Zador et al.'s Embodied Turing Test position paper [45]. which emphasizes developing Al models whose sensorimotor capabilities rival those of their biological counterparts. There are several existing works that apply mask-optimization to control details musculoskcletal models 446) use robots to implement and validate neural circuits in zebrafish (4), and model animal-like social behavior or object perception using digital twins 50 [1].
Our work extends these directions by providing the first predictive and normative computational formalization of neural-glia -giial interactions in embodied agents, thereby validating a circuit model recently proposed by Mu et al. [2]. Most importantly, our model is trained entirely via intrinsically- motivated exploration, unlike previous approaches that constrain behavior and their resulting neural representations by supervised learning [39] To the best of our knowledge, this marks the first completely autonomous, embodied agent model of behavioral and brain data in neuroscience, pointing towards a promising computational framework for understanding naturalistic, task-independent behavior in biological systems using intrinsic reinforcement learning.
3 Methods
Virtual Zebrafish Environment Animals are physically coupled to the environment through their embodiment; this coupling is often referred to as the sensory-motor or perception-action feedback loop. Physical embodiment imposes strong constraints on both the sensory stream from which an agent learns meaningful representations of the world and the actuation system by which the agent manifests behavior. Following this top-down view of biological systems, we construct an embodied agent and custom virtual environment in the MuJoCo physics engine [52] specifically designed after the ethology of the zebrafish (Figure TD. Leveraging the procedurally generated n-link swimmer and built-in inertial fluid model from the Deepmind Control Suite (dm-control) [1, we construct an ethological environment (Figure ( in which the agent can freely behave in the presence of both passive and active fluid currents, similar to the dynamic water environments to which zebrafish are native. To evaluate our agent in the futility-iduced passivity task, we construct a second environment closely matching the open-loop experimental protocol in Mu et al. (2) (Figure ID). In this environment, agents passively experience a high-contrast grating moving away from the egocentric point-of-view, which simulates backward motion. In the closed-loop condition, the agent is head-free and can learn positional- homemllllomssttiiss policy to counteract the perceived backward flow, In the open-loop condition, the agent head-fixed and its swim commands produce no movement. The passive speed of the moving grating, its colors, and sizing relative to the zebrafish body were determined from experimental parameters in Mu al. (2] These environments provide sensory and actuator configurations that closely match the basic structure of free swimming as well as the head- fixed protocol, facilitating a meaningful comparison between artificial and biological agents.
Agent Design The zebrafish agent is equipped with a recurrent sensory-cognitive architecture to support perception and action continuous, high-dimensional environments. Because the au- tonomous zebrafish behavior recorded by Mu et al. [ is primarily driven by visual input, we restrict
4

---

A
vision
((1))
care module
V(hp)
T Torld
1+tl
J
proprio encoder
((())
policy module
world model
NE-MO Neurons
h
Redial astrocyles
intrinsic drive module
GABA Neurons
Sg
Motor Neurons
o x f(,-
echolloicel
unethological
3M: Mode--Merary-Mismatchh
3M- Progrrrsssss
= (1 D ye
memories
0g
Das joy 11 an
Figure 2: Agent architecture and 3M-Progress. A) Egocentric visual input (I+) is encoded via small residual network Φ1. Proprioceptive state observations (Jz) are encoded via a small multi- layer perceptron Φu with shorteut paths both the core and policy module. Sensory features are passed into recurrent LSTM core (4f) and policy (h7) modules that learn state value function and stochastic policy, respectively. The intrinsic drive module consists that parameterizes a forward dynamics model on sensory features observed from environment with dynamics Torld. B) AM-Progress uses two memories created from environments with differing transition dynamics. Divergence between the ethological prior we and the current world-mode war defines 3M, which is then is used as input to leaky integrator è to generate intrinsic reward r].
small multi-layer perceptron
the sensory stream to vision encoder operating images with similar resolution to the visual acuity of zebrafish (53). Sensory features are used by three distinct cognitive networks with modularized objectives. The core and policy modules are implemented as actor-critic architecture using Long Short-term Memory (LSTM) networks (54] followed by feedforward decoders trained end-to-end with Proximal Policy Optimization (PPO) to output torques that are used as the control input to the agent's motors [55] Specific agent implementation details can found in Appendix Although PPO is model-free reinforcement learning algorithm, the intrinsic drive module (IDM) learns world model through online experience that approximates the world state-action transition dynamics in sensory feature-space (as described below). However, this internal model functions only to generate intrinsic reward and is not used for planning or model-based control.
Model-based Intrinsic Reinforcement Learning In classical reinforcement learning, policies are obtained by mapping a desired behavior to maximizing a reward function (56]. This function one component of the Markov Decision Process (MDP) that formally specifies task M as a tuple (S,A, A,T, T, po. r), where S the space of environment states, A is the space of actions, T: S A P(S) is the transition dynamics (where P(S) the set of probability densities S), Po is the distribution over initial states, and r is the reward function. Importantly, when reward defined as part of the task MDP, it is extrinsic and provided from the environment. However, learning complex behaviors using extrinsic reward can fail for many real-world situations where is extremely sparse (winning a long-horizon game like Go) intractable begin with (intellectual pursuits such as knowledge acquisition) without powerful exploration mechanism to guide behavior. Model-based intrinsic motivation is class of such mechanisms that leverage predictive world models to convey exploration-releyy relevvnt information using prediction-erros to the form an intrinsic reward rf. In the absence of any extrinsic reward rf, policy learning with intrinsic motivation is completely self-supervised. In this work, we consider intrinsic motivation driven by forward dynamics world models w. Let 0 parameterize neural network and wg S A P(S). For simplicity, we assume fixed variance Gaussian density We N (ő($:+1) Φ(st). ar He, al). Here, o(-) denotes the
5

---

A
Training slep (1e6)
JM-Progress
y-Progress
Disagreement
10
20
1.1/
80
60
Intan-etina A4 greant Astien Fronsiion Pretke Trasatoo
Achve
09
Zebrafish
Behawforal Aligmment
Model-fren
3M-Progress
Model-boes bodel-besed
y y-Prageess
15
Transitions
10
Homenttale
Mextin
10
20
40
Training Steps (1e6)
Figure 3: Model-behavioral alignment. A) Swim power traces of artificial agents with different intrin- sic drives throughout training. B) Pearson's r correlation between agent swim power (joint torques) and zebrafish swim power (motor neuron activity) for active and passive behavioral transitions. C) (Top) Timecourse of active-passive transitions zebrafish compared against stationary behavior from progress-driven agents for single rollout. (Bottom) Average number of behavioral transitions per rollout across training for different intrinsic drives.
concutenated sensory embeddings from ør (-) and Φ-(+). This class methods is appealing from two complimentary perspectives. On one hand, it is computationally straightforward; learning internal model of an MDP's transition dynamics under the agent's policy- solicy-states states with high prediction error under the internal model indicate regions of the state-action space that are poorly understood or rarely visited. the other hand, it is well motivated by experimental evidence in both neuroscience and psychology: numerous empirical studies suggest humans and animals depend on predictive models the world for decision-making [17 primary substrate for intrinsic motivation; however, we emphasize that a world model is best viewed as only a substrate, as exactly how should be used for intrinsic motivation remains unclear.
natural way to measure novelty of the states visited
Thus, world models are well-positioned
Model-Brain and Inter-Animal Alignment To rigorously evaluate our embodied agents against whole-brain data collected by Mu al. (2), neural-glial recordings from zebrafish were first aligned to periods specifically capturing behavioral state transitions between active swimming and futility- induced passivity. Correspondingly. latent states and predicted neural-glia responses from our agents were aligned to these same experimentally defined epochs using vvent-tiggeeed averaging around transition events. To quantify model--bain alignment, we employed stringent "One-to-One mapping, in which each neural or astrocytic unit from the zebrafish brain was matched directly to the single most correlated artificial unit from the virtual embodied agent. Although this mapping typically too restrictive for capturing inter-animal alignment in heterogeneous neural populations for brain-region- ceiii--rein---scciiti specific (c.g. not whole brain) data in other animals (as shown in prior sensory [62] and cognitive systems [63] where full linear regression is necessary),
found was fully sufficient here due to the exceptional whole-brain reliability of zebrafish neural-glial responses during futility- induced passivity. Indeed, inter-animal alignment computed from pairwise correlations between individual zebrafish was nearly 100%, establishing a robust empirical ceiling for evaluating model performance.
Inter-animal alignment was computed as the correlation of neural-glial response patterns across pairs of zebrafish within aligned epochs, thus formally defining the upper bound of predictivity achievable by any candidate model (the mathematical details of the chosen metric can be found in Appendix F). This approach ensures that successful computational models achieve neural predictivity

---

indistinguishable from biological measurements, within the bounds set by natural biological variability. Together, these procedures allowed us to precisely quantify model-brain alignment; achieving close correspondence between artificial and biological responses under this strict One-to- -oe-t-OOe mapping criterion provides strong evidence that our intrinsic motivation-driven embodied agents effectively capture the detailed neural-glial dynamics underlying autonomous behavioral transitions in zebrafish.
Model-Behaviore Alignment To quantify similarity between biological zebrafish and artificial agents, we use Pearson's 's r as metric between the respective swim power readouts for each system surrounding state transitions. a 10ms window of recorded tail motor nerve signals following Mu et al. 22. We use the same passive and active transitions windows identified by Mu et al. (2 to compute the inter-animal behavioral alignment across 11 subjects by applying the metric to swim power after smoothing and normalization over a 20 second window surrounding the transition time. For the artificial agents, we take the norm of their joint torques swim power and identify behavioral transitions as swim power above a threshold of I Newton. This threshold was determined empirically by recording joint activations during active and passive behaviors in the default swim task in the dm-control suite 6-link swimmer environment []] Model-behavine consistency is then computed by applying the metric between segmented zebrafish data and the agent's 's swim power surrounding these transitions over a 20-step window.
zebrafish, swim power was calculated as the standard deviation over
changes in
4 Animal-like Exploration from First Principles: Lessons from Zebrafish
Autonomy in animals enables intelligent and robust decision-making in complex environments, even in the presence of high-entropy or unfamiliar dynamics such as noisy stimuli or laboratory habitats. Reflecting on characteristics of autonomous exploration in zebrafish, we propose two simple desiderata for intrinsic drives that capture animal-like autonomy:
1. Animals do not perseverate on unpredictable stimuli or pursue stimuli they cannot causally interact with. Mu et al. (2 demonstrate that zebrafish transition to passive behavioral states when motor commands elicit unpredictable sensory-feedhack (unlearnable dynamics) or when sensory-feedback is withheld altogether (uncontrollable dynamics). An intrinsic objective for animal autonomy should avoid unpredictable or uncontrollable stimuli.
2. Animals exhibit consistent decision-making strategies across repeated encounters of the same context. Zebratish exhibit stable behavioral stite-swtching in the open-loop experimental protocol across trials and subjects, ultimately converging on single exploration policy (2]. An intrinsic objective for animal autonomy should converge to a stable behavioral policy.
Intrinsic objectives that rely on prediction-error alone 3 reward stochastic environment dynamics and incentivize learning transitions in which the agent has no causal contro Learning progress 66, [64] and Disagreement (5) overcome this by leveraging temporal dynamics or statistics of world model ensemble, but together with ICM and RND, are formulated as functions of the world model training loss and are thus -ta-sssttttt-llaaraeee
repeated behavioral strategies drive the training loss towards zero, and so any single behavioral strategy is transient since it is not consistently reinforced, Although these properties are suitable for exploration in some robotics domains, particularly when the policy is supplemented with an extrinsic task reward (3 5. they fail to capture the nature of autonomous exploration in animals.
3MMProgress In order to overcome these drawbacks, we introduce Borild-Mmory--isseee Progress (3M-Progeess), novel intrinsic drive that incorporates these simple normative properties of animal autonomy inspired by zebrafish. Curiosity, disagreement, and learning progress couple intrinsic reward to moving world model, yielding non-stationary policy objective. The RL problem becomes a two player minimax game, where the actor seeks states that increase prediction-error and the learner reduces it, leading to reward landscape that flattens and precludes stationary optimal policy Thus, each of the existing curiosity-drive exploration algorithms we consider fails to converge to an exploratory policy. However, learning progress 6 is unique among these algorithms
Although 3 proposed an inverse dynamics feature space to avoid representing uncontrollable stimuli, it's efficacy has not heen demonstrated outside of simple pixel-based environments with discrete actions.
Seee Appendix C.I for mathematical details.
7

---

Mi--hiind Aigennent
Irier Ante--Animell Aiigrment
Model basel-
Model free
Data derived
Bodel-Behawiorr Alignment
0.9-
Modill Brain Aligement
0.0
aM aMM
Prograte
ICM
RHO
Disggraamen
Homootaal
Morraa
Randaa
witsle tradeddoakllee Aug.
PID Contrale
0.0
02
0.4
0.
1
Gamaalin
While
Model-Brain Alignment
Figure 4: Model-brain alignment averaged across active and passive transitions. A) Noise-corrected Pearson's 's r correlation between whole-brain neural and glial units and artificial units from trained agents. B) Model scores behavioral and whole-brain alignment.
in its ability to avoid perseveration unpredictable and uncontrollable stimuli by leveraging the temporal dynamics of prediction-error. This is achieved by maintaining two world models, an online model and long-term memory implemented as exponential weight decay the online model. The slow timescale memory provides a moving baseline that, when compared with the online model, creates an intrinsic reward that flattens as the temporal dynamics of the online model's predictions flatten-in other words, a computationally efficient and biologically plausible implementation of time-derivative.
To combine the adaptive behavior afforded by derivative- operations with an intrinsic goal that admits a stationary solution, we propose decouple the two memories completely. We achieve this by learning the long-term memory Wg in a pretraining environment and deploying it as a fixed prior while the online model wir is learned in new environment with new transition dynamics. The frozen "ethological memory" functions as a static memory primitive: comparing its predictions with the online memory produces residual that encodes match the pretraining environment, which can then be leveraged to partition the state-action space in the new context into regions where memories systematically agree or disagree 6 This reflects the idea that animals develop in an ecological niche with characteristic environment dynamics, distilled via experience
explicit bias toward transition dynamics that
internal world model wig. In new environments with different transition dynamics, such as the experimental protocol in Mu et al. (2), animals can use this prior to seek or avoid regions that match their ecological niche as they learn new world model wa. To capture this, we define the model-memory
memory-mismated mismatch (Ez), exponential filter (é,), and niche-aware aware intrinsic motivation ri as
E DKL (wo (Ø(s++1) ((s)), a.) war (6($441) @(s)), at)l
(1)
& = (1 7) + yEe,
(2)
r E22 - Erl
(3)
where DKL denotes the Kullback-Leibler divergence and is the filter timescale. The non-negativity of this divergence coarsely partitions the state-action space into niche-seeking (model-memory
agreement: é, s 0) and niche-avoidance (model-memory disagreement: e, > 0). We design
the intrinsic reward mechanistically after y-Progress (6) to maintain a moving baseline, but filter
model-memory mismatch miiioyymatmy rather than model parameters.
Due to the moving baseline, the reward does not perseverate on any single partition, either agreement or disagreement. The symmetry enforced by the absolute value encourages periodic exploration between partitions by reinforcing deviations from the moving baseline in either direction. Unlike learning progress, this formulation does not saturate as learning unfolds since 6, is computed using fixed memory. In fact, as the prediction error from War stabilizes with more environment interactions, the signal-to-noise ratio in e, only increases, resulting in more robust behavioral patterns. Note that the absolute value specific choice of an activation function. The shape of this function and the progress horizon 7y determine the relative time spent in each partition. All experiments and
Seee Appendix C.2 for mathematical details.
8

---

baseline algorithms include an action penalty re - 2 to encourage exploration of passive behavior. Figure illustrates a specific example inspired by zebrafish, where wi learned in an ethological environment- the agent can freely behave and experiences passive fluid forces induced by self-motion as swims and and active fluid forces as learns positional homeostasis by resisting an opposing current. The agent is then put into an unethological experimental protocol where it is head-fixed and its swim commands do not clicit sensory feedback. As the agent behaves this environment, it distills a new memory War from experience.
3M-Progress as Normative Model of Neural-Glial Computation Owing to the formulation of 3M-Progress, the state-value function must implement units in the core module that are functionally equivalent to neurons in the Noradernergic cluster of the Medulla Oblongata (NE-MO) and radial astrocytes identified by Mu et al. [2] (2)) 3M-Progress detects sensory-motor motor mismatch using a prior memory an expectation of how action is coupled to sensory-feedben
under ethological environment dynamics (2 This is functionally equivalent to signaling failed swim-attempts by NE-MO neurons in zebrafish. Similarly, the exponential filter is discrete-time leaky integrator on modee--memory-mismal (NE-MO input), which is functionally equivalent to radial astrocytes that accumulate NE-MO signals during failed swim-attempts and decay during passivity (2C).
5 Experimental Results
3M-Progress Agents Replicate Behavioral Patterns Observed in Zebrafish We first as- sessed the ability of intrinsic motivation methods to replicate detailed behavioral patterns Behavioral alignment
observed in biological zebrafish during autonomous exploration. was quantified by comparing locomotor trajectories and state transitions (active to pas- sive and back) between artificial agents and zebrafish across multiple trials and subjects. 3M-Progress discovers ethologically- relevant behavioral state transitions by 10 million environment steps and sooner, whereas other intrinsic drives display tran- sient strategies and stabilize on complete passivity or activity over training (Figure 3A). Agents trained with 3M- PM-Prrgeess exhibited the highest model-behavior alignment, successfully capturing the dy- namics of state transitions recorded in the biological data (Figure (B.. Additionally, agents trained with traditional intrinsic motivation methods such as ICM, RND, Disagreement, and y-progress showed significantly lower alignment and failed to capture the characteristic stable cycling between active and passive states (Figure 3-C)).
Agent
octiv
pazzhe
Caldurn sesporoes (3RFF)
CHECENE
0.00-
Model responte
PC2
Cal
0po-
nz0-
Time since paseiit (s)
paceidy
Caldurr (211))
atllw
Model resconss
-2.00
PP2
-3.04
-Gil
Time since passMty (6)
Time since passkity
Figure 5: Latent dynamics of 3M-Progress agent's inter- nal activations compared with normalized whole-brain neural-glial response in zebrafish. A) Principal com- ponents during passive and active transitions in the agent. B) Normalized average whole-brain neural-glial response during passive and active transitions in a ze- brafish subject.
3M-Progress Agents Saturate Explain- able Variance of Whole-Brain Neural- Glial Dynamies To evaluate how closely intrinsic motivation-driven agents matched zebrafish neural-glial dynamics, we com- pared model predictions to whole-brain cal- cium imaging data recorded during behavioral transitions of futility-induced passivity. Neural- alignment was quantified using a One-to-One mapping, where each recorded biological neuron and astrocyte was matched to the single most correlated artificial unit from the model. Strikingly, 3MM-rogress: agents captured nearly all of the explainable variance in neural and astrocytic activity, markedly outperforming existing intrinsic motivation algorithms, as well as baseline controls (Figure 4A). Model-free intrinsic drive controls include energy cost (homeostatic), entropy bonus (MaxEnt), and randomly intialized agent. Data-derived controls include a Gaussian Process fit to neural-glial responses, proportional canattionnn-nntgaaa--e--ee integral-derivative (PID) control, average neural-glial population response,

---

and white-noise More details on these controls can be found in Appendix [ In fact, the 3M- EM-Progress agent the only model that is almost completely aligned with both the behavioral alignment and neural-glial alignment, highlighting that accurate behavioral modeling can tightly constrain detailed neural dynamics (Figure 43). Taken together, 3M- Progress agents pass the NeuroAl Turing Test on this dataset, a criterion emphasizing models that match both behavior and internal function [65]
Latent Dynamies of 3M-Progress Agents Reflect Underlying Neural-Glial Computations Given that 3M-Progress best matched both behavioral and neural-glial alignment among all candidate models, we then characterized the agent's internal dynamics by performing Principal Component Analysis (PCA) on its in silico neural-ght population during behavioral state transitions. PCA revealed that the dominant latent dimensions of the agent's core module closely mirrors whole-brain neural-glial dynamics measured from biological zebrafish, whereby glial responses accumulate evidence of motor futility via noradrenergic signaling to drive behavioral suppression, and neural responses reflect transient activation patterns associated with detecting mismatches between expected and actual sensory outcomes during unsuccessful swim attempts. This analysis demonstrates that the 3M-Progres, mechanism for intrinsic motivation not only generates realistic behavior, but also robustly captures fundamental internal neural-glial computations underlying autonomous exploration and behavioral state transitions (Figure 5).
Discussion
Our work seeks to identify and computationalize intrinsic goals that enable autonomous, task- independent behavior in animals. Leveraging unique whole-brain dataset recorded in larval zebrafish during an autonomous behavior known as futility-induced passivity, we identify two simple principles of intrinsic goals for autonomous agents: avoid perseveration on stimuli that are uncontrollable and unpredictable, and converge on a robust decision Decision-making strategy. We introduced 3M- 3M-Progress, a novel intrinsic drive that operationalizes these principles by continually tracking divergence between an online world model and an ethologically relevant prior. Learned from experience in an ecological niche that captures the basic physics of a naturalistic zebrafish habitat, this prior guides exploration in new environments by partitioning the behavioral space into niche-seeking and avoiddig modes.
Unlike prior intrinsic motivation methods that suffer from behavioral inconsistency and non- stationarity, 3M-Progress reliably generated stable cycling behaviors closely matching those observed in biological zebrafish. Moreover, 3M-Progress agents were uniquely successful in capturing whole- brain neural-glial -giial activity among all candidate models. We showed that artificial agent's ' internal latent dynamics mirror neural-glin -lial computation, wherein astrocytic responses accumulate evidence of motor futility through noradrenergic signaling to trigger behavioral suppression, while neural populations transiently encode mismatches between expected and observed sensory outcomes. To the best of our knowledge, our work marks the first goal-driven model of neural-glia computation, as well as the first completely self-supervised embodied agent that predicts behavioral and brain data.
Our findings suggest two complementary evolutionary constraints for developing robust, animal- autonomous agents: (1) maintaining an intrinsic drive informed by memory, and (2) continually monitoring divergence between this memory and new sensory experiences. Functionally, tracking this mismatch within reinforeement learning frameworks allows artificial agents identify ineffective strategies, update internal models adaptively, and discover new behaviors. Such mechanisms offer significant potential for enhancing the autonomy of artificial systems, especially in open-ended environments lacking clear external rewards or goals.
Limitations and Future Work Our current analyses primarily focused on behavioral transitions within constrained virtual environments, limiting ecological realism. Future work could extend both the artificial environments and biological experiments to richer ecological settings and more complex behavioral repertoires. providing stronger and more diverse constraints on computational models of autonomy. The biomechanical realism of the body could also be expanded to include muscles and motor circuits, providing realistic constraints on the low-level controller. Additionally, while our model captures essential neuron-glial interactions at a population level, it abstracts away detailed biochemical signaling mechanisms and anatomy of astrocytes and neurons. Incorporating more biologically detailed models of these processes could aid in providing predictions about these mechanisms at a finer scale. Finally, 3M-Progress can be generalized in a variety of ways that extend beyond -induey passivity (see Appendix C.3
we leave this to future work.
10

---

Acknowledgements
We thank Misha Ahrens, Chris Doyle, and Yu Mu for helpful discussions, as well as the anonymous reviewers for their helpful feedback on the initial manuscript draft. A.N. was supported in part by a grant from the Burroughs Wellcome Fund. X.P. and R.K. were supported in part by the National Science Foundation and DoD OUSD (R & E) under Cooperative Agreement PHY-2229929 (The NSF Al Institute for Artificial and Natural Intelligence. ARNI) and the Simons Collaboration in Ecological Neuroscience
References
[1] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprini N.01:0800.000000 2018.
[2] Yu Mu, Davis V Bennett, Mikail Rubinov, Sujatha Narayan, Chao-Tsung Yang, Masashi Tanimoto, Brett D Mensh, Loren L Looger, and Misha B Ahrens. Glia accumulate evidence that actions are futile and suppress unsuccessful behavior. Cell, 178(1):27-43, 2019.
[3] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self- supervsee prediction. In International conference on machine learning, pages 2778- 2787. PMLR, 2017.
[4] Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:18100 /2894, 2018
[5] Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagree- ment. In International conference machine learning, pages 5062-5071, PMLR, 2019.
[6] Kuno Kim, Megumi Sano, Julian De Freitas, Nick Haber, and Daniel Yamins. Active world model learning with progress curiosity. In International conference on machine learning, pages 5306-5315. PMLR, 2020.
[7] Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE transactions on autonomous mental development, 2(3):230-247, 2010.
[8] Nick Haber, Damian Mrowca, Stephanie Wang, Li Fei-Fei, and Daniel Yamins. Learning to play with antrinsiaally-motivatt self-aware agents. Advances in neural information processing systems, 31, 2018.
[9] Jun Nagai, Xinzhu Yu. Thomas Papouin, Eunji Cheong, Marc R Freeman, Kelly R Monk, Michael H Hastings, Philip G Haydon, David Rowitch, Shai Shaham, et al. Behaviorally consequential astrocytic regulation of neural circuits. Neuron, 109(4):576-596, 2021.
[10] Laurie M Robin, José F Oliveira da Cruz, Valentin C Langlais, Mario Ferann-Fermandez, Mathilde Metna-Laurent, Arnau Busquets-Garcia. Luigi Bellocchio, Edgar Soria-Gomez, Thomas Papouin, Marjorie Varilh, et al. Astroglial ebl receptors determine synaptic d-serine availability to enable recognition memory. Neuron, 98(5):935-94) 2018.
[11] Michael R Williamson, Wookbong Kwon, Junsung Woo, Yeunjung Ko, Ehson Maleki, Kwanha Yu, Sanjana Murali, Debosmita Sardar, and Benjamin Deneen. Learning-associated associated astrocyte ensembles regulate memory recall. Nature, pages 1-9, 2024.
[12] Leo Kozachkov, Ksenia V Kastanenka, and Dmitry Krotov. Building transformers from neurons and astrocytes. Proceedings of the National Academy of Sciences, 120(34)::2211150122, 2023.
[13] Leo Kozachkov, Jean-Jacques Slotine, and Dmitry Krotov. Neuron-astrocytte associative memory. Proceedings of the National Academy of Sciences, 122(21))e2417788122,, 2025. doi: 10.10733pnas.24177888 URL htt.:htpss///e pnas wttps///////o///....ooo/ 1073/ 1073/pass 2417788122
[14) Ciaran Murphy-Royal. ShiNung Ching, and Thomas Papouin. A conceptual framework for astrocyte function. Nature Neuroscience, pages 2023.

---

[15] Zhiguo Ma, Tobias Stork, Dwight E Bergles, and Mare R Freeman. Neuromodulators signal through astrocytes alter neural circuit activity and behaviour, Nature, 539(7629):428 2016.
[16] Jillian Stobart, Kim David Ferrari, Matthew JP Barrett, Chaim Glück, Michael J Stobart, Mare Zuend, and Bruno Weber. Cortical circuit activity evokes rapid astrocyte calcium signals on a similar timescale to neurons. Neuron, 98 2018.
[17] Adi Doron, Alon Rubin, Aviya Nechama Novick, Tirzah Kreisel, Yaniv Ziv, and Inbal Goshen. Hippocampal astrocytes encode reward location. Nature, 609(7928):72--778 2022.
Netai Benaim, Tom Carmi, Ron Refacli,
(18) Roberta De Ceglia, Ada Ledonne, David Gregory Litvin, Barbara Lykke Lind, Giovanni Carriero, Emanuele Claudio Latagliata, Erika Bindocci, Maria Amalia Di Castro, laroslav Savtchouk, Ilaria Vitali. et al. Specialized astrocytes mediate glutamatergic gliotransmission in 2023.
the cns. Nature, 622((981)):200
[19] Kyungehul Noh, Woo-Hyun Cho, Byung Hun Lee, Dong Wook Kim, Yoo Sung Kim, Keebum Park, Minkyu Hwang, Ellane Barcelon, Yoon Kyung Cho, Justin Lee, et al. Cortical astrocytes modulate dominance behavior in male mice by regulating synaptic excitatory and inhibitory 2023.
balance. Nature Neuroscience,
[20] Peter Rupprecht, Sian N Duss, Denise Becker, Christopher M Lewis, Johannes Bohacek, and Fritjof Helmchen. Centripetal integration of past events in hippocampal astrocytes regulated by locus coeruleus. Nature Neuroscience, pages 1-13, 2024.
[21] Misha B Ahrens, Michael B Orger, Drew N Robson, Jennifer M Li, and Philipp J Keller. Whole-brain functional imaging at cellular resolution using light-sheet microscopy. Nature methods, (15):413-4200 2013.
[22] Joseph V Brady, Robert W Porter, Donald G Conrad, and John W Mason. Avoidance behavior and the development of duodenal ulcers. Journal of the Experimental Analysis of Behavior, (1):69, 1958.
[23] Robert M Camp and John D Johnson. Repeated stressor exposure enhances contextual fear memory in beta-adrenergic receptor-dependent process and increases impulsivity in non-beta receptor-dependent fashion. Physiology & behavior, 150:64-68, 2015.
Morales-Delgado, [24] Natalija Popović, Nicanor Morales Delgado, Ernesto De la Cruz-Sánchez, and Miroljub Popović. Rats conserve passive avoidance retention level throughout the light phase of diurnal cycle. Physiology & Behavior, 268: 14234, 2023.
[25] Cagla Eroglu. Astrocytes, hidden puppet masters of the brain. Science, 388(6748):705- 706. 2025. doi: 10. 1111///ceenee.axx110000 URL htlphtttss//mme http:://wwwww..nncee/eeeeeoe science org/doi/abs//1 1126/science adx7102
[26] Katheryn B. Lefton, Yifan Wu, Yanchao Dai, Takao Okuda, Yufen Zhang, Allen Yen, Gareth M. Rurak, Sarah Walsh, Rachel Manno, Bat-Erdene Myagmar, Joseph D. Dougherty. Vijay K. Samineni, Paul C. Simpson, and Thomas Papouin. Norepinephrine signals through astrocytes to modulate synapses. Science, 388(678))7776 783, 2025. doi: 10.1 wtt.:/iiince.adgaa...... URL httpss:/www. wwwp:w/wcenncc.... http://wwww.....o./////////////e
wwtp///iieeenceee adg544
[27] Ksenia Kastanenka, Rubén Moreno-Bott Maurizio De Pittà, Gertrudis Perea, Abel Eraso- Pichot, Roser Masgrau, Kira E Poskanzer, and Elena Galea. A roadmap integrate astrocytes into systems neuroscience. Glia, 68(1):5-26, 2020.
[28] Leo Kozachkov and Konstantinos P Michmizos. The causal role of astrocytes in slow-wave rhythmogenesis: A computational modelling study. arXiv preprint arXiv: 2017.
[29] Maurizio De Pittà, Mati Goldberg, Vladislav Volman, Hugues Berry, and Eshel Ben-Jacob. Glutamate regulation of calcium and ip3 oscillating and pulsating dynamics in astrocytes.
Journal of biological physics, 3544):383-411 2009.
12

---

[30] Mati Goldberg, Maurizio De Pittà, Vladislav Volman, Hugues Berry, and Eshel Ben-Jacob. Non- linear gap junctions enable long-distance propagation of pulsating calcium waves in astrocyte networks. PLoS computational biology, (080:e900009; 2010.
[31] Vladislav Volman, Maxim Bazhenov, and Terrence J Sejnowski. Computational models of neuron-astrocyte interaction epilepsy. Frontiers in computational neuroxcience, 6:58, 2012.
[32] Leo Kozachkov and Konstantinos P Michmizos. Sequence learning in associative neuronal- astrocytic networks. In Brain Informatics: 13th International Conference, BI 2020, Padua, Italy, September 19, 2020, Proceedings 13, pages 349-360. Springer, 2020.
[33] Vladimir Ivanov and Konstantinos Michmizos. Increasing liquid state machine performance with edge-of-chaos dynamics organized by astrocyte-mi modulated plasticity. Advances in neural information processing systems, 34:25703-25711 2021.
astrocyte-modulated
[34] Maurizio De Pittà and Nicolas Brunel. Multiple forms of working memory emerge from synapse-astroeye interactions in neuron- -gia network model. Proceedings of the National Academy of Sciences, 1 119(4)):c2209912)) 2201912119, 2022.
[35] Lulu Gong, Fabio Pasqualetti, Thomas Papouin, and ShiNung Ching. Astrocytes as a mechanism for meta- plasttiiy and sereettuall-gguidee network function. arXiv preprint arXiv:2311.03508 2023.
[36] Haoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun, Hua Wu, and Haifeng Wang. Curiosity- driven reinforcement learning from human feedback. arXiv preprint arXiv:2501.1146: 2025.
[37] Russell Mendonca, Oleh Rybkin, Kostas Danilidis, Danijar Hafner, and Deepak Pathak. Dis- covering and achieving goals via world models. Advances in Neural Information Processing Systems, 34:24379-2433
2021.
[38] Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pages 8583-8592. PMLR, 2020.
[39] Josh Merel*, Diego Aldarondo Jesse Marshall Yuval Tassa, Greg Wayne, and Bence Ölveczky. Deep neuroethology of a virtual rodent. International Conference on Learning Representations, 2020.
[40] Diego Aldarondo, Josh Merel, Jesse D Marshall, Leonard Hasenclever, Ugne Klibaite, Amanda Gellis, Yuval Tassa, Greg Wayne, Matthew Botvinick, and Bence P Ölveczky. A virtual rodent predicts the structure of neural activity across behaviors. Nature, pages 1-3, 2024.
[41] Roman Vaxenburg, Igor Siwanowicz, Josh Merel, Alice A Robie, Carmen Morrow, Guido Novati, Zinovia Stefanidi, Gert-Jan Both, Gwyneth M Card, Michael B Reiser, et al. Whole- body simulation of realistic fruit fly locomotion with deep reinforcement learning. bioRxiv, pages 2024 03, 2024.
[42] Victor Lobato-Rios, Shravan Tata Ramalingasetty, Pembe Gizem Özdil, Jonathan Arreguit, Auke Jan ljspeert, and Pavan Ramdya. Neuromechlly, a neuromechanical model of adult drosophila melanogaster. Nature Methods, 19(5):620-6)) 0277772666666.- 2022.
[43] Sibo Wang-Chen, Victor Alfred Stimpfling, Thomas Ka Chung Lam. Pembe Gizem Özdil, Louise Genoud, Femke Hurtak, and Pavan Ramdya. Neuromechfly v2: simulating embodied sensorimotor control in adult drosophila. Nature Methods, 21(12))2353-23) 2024.
[44] Gopal P Sarma, Chee Wai Lee, Tom Portegys, Vahid Ghayoomie, Travis Jacobs, Bradly Alicea, Matteo Cantarelli, Michael Currie, Richard C Gerkin, Shane Gingell, al. Openworm: overview and recent advances in integrative biological simulation of caenorhabditis elegans. Philosophical Transactions of the Royal Society B, (735)))))
2018
[45] Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, et al. Catalyzing next-generation artificial intelligence through neuroai. Nature communications, 14 (1):1597, 2023.
13

---

[46] Adriana Perez Rotondo, Alessandro Marin Vargas, Michael Dimitriou, and Alexander Mathis. Modeling sensorimotor processing with physies-informed neural networks. bioRxiv, pages 2024-09, 2024.
[47] Olivier Codol, Jonathan A Michaels, Mehrdad Kashefi, J Andrew Pruszynski, and Paul L Gribble. Motornet, a python toolbox for controlling differentiable biomechanical effectors with artificial neural networks. Elife, 12:RP88591, 2024.
[48] Alessandro Marin Vargas, Axel Bisi, Alberto S Chiappa, Chris Versteeg, Lee E Miller, and Alexander Mathis. Task-driven neural network models predict neural dynamics of propriocep- tion. Cell, 187(7):1745-17)) 1761, 2024.
[49] Xiangxiao Liu, Matthew D Loring, Luca Zunino, Kaitlyn E Fouke, François A Longchamp, Alexandre Bernardino, Auke Ijspeert, and Eva A Naumann. Artificial embodied circuits uncover neural architectures of vertebrate visuomotor behaviors. Science Robotics, 10(107): eadv4408, 2025.
[50] Justin N Wood, Lalit Pandey, and Samantha MW Wood. Digital twin studies for reverse engineering the origins of visual intelligence. Annual Review of Vision Science, 10(1))144-117) 2024.
[51] Joshua D McGraw, Donsuk Lee, and Justin N Wood. Parallel development of social behavior in biological and artificial fish. Nature Communications, 15(1):10613, 2024.
[52] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RS/ international conference on intelligent robots and systems, pages 5026-500 5003-003..110 IEEE, 2012.
[53] Marion Haug. Oliver Biehlmaier, Kaspar Mueller, and Stephan CF Neuhauss. Visual acuity in larval zebrafish: behavior and histology. Frontiers in zoology, 7:1-7, 2010.
[54] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9()): 1735-1780, 1997.
[55] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[56] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
[57] Kevin Kermani Nejad, Paul Anastasiades, Loreen Hertäg, and Rui Ponte Costa. Self-supervised predictive learning accounts for cortical layer-specificity. Nature Communications, 16(1):6178, 2025.
[58] Nathaniel D Daw, Samuel Gershman, Ben Seymour, Peter Dayan, and Raymond Dolan. Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6): 2204-1215. 2011.
[59] Samuel J Gershman. The successor representation: its computational logic and neural substrates. Journal of Neuroscience, 38(33):7193-700, 2018.
[60] Andreja Bubic, Yves Von Cramon, and Ricarda Schubotz. Prediction, cognition and the brain. Frontiers in human neuroscience, 4: 1094, 2010.
[61] Lee DetWit, Bart Machilsen, and Tom Putzeys. Predictive coding and the neural response to predictable stimuli. Journal of Neuroscience, 30026))8702-8703, 2010.
[62] Aran Nayebi Nathan CL Kong*, Chengxu Zhuang, Justin Gardner, Anthony Norcia, and Daniel LK Yamins. Mouse visual cortex limited resource system that self-learns an aaaoogicaall-ggenert representation. PLOS Computational Biology, 19, 2023.
[63] Aran Nayebi, Alexander Attinger, Malcolm Campbell, Kiah Hardcastle, Isabel Low, Caitlin Mallory, Gabriel Mel, Ben Sorscher, Alex Williams, Surya Ganguli, Lisa M Giocomo, and Daniel LK Yamins. Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. Advances in Neural Information Processing Systems, 34, 2021.
14

---

[64] Joshua Achiam and Shankar Sastry. Surprise-base based intrinsic motivation for deep reinforcement learning. arXiv preprint arXiv:1703.01732, 2017.
[65] Jenelle Feather*, Meenakshi Khosla*, N Murty", and Aran Nayebi*, Brain-model evaluations need the neuroai turing test. arXiv preprini arXiv:2502. 16238, 2025.
[66] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_ mim_ontrol: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[67] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02444 024438080828,, 2015.
[68] Diederik Kingma. Adam: method for stochastic optimization. arXiv preprint 1412v91412.6988) 2014.
[69] Aran Nayebi, Rishi Rajalingham, Mehrdad Jazayeri, and Guangyu Robert Yang. Neural foundations mental simulation: Future prediction of latent representations on dynamic scenes. Advances in Neural Information Processing Systems, (6:70548-70566 2023.
[70] Alex H Williams, Erin Kunz, Simon Kornblith, and Scott Linderman. Generalized shape metrics on neural representations. Advances in Neural Information Processing Systems, 34:4738-4750, 2021.
15

---

Appendix
A
Model-Brain Alignment per Transition and per Module
Model Medal-Brain Bou.BBaiin Aadil-BrainAggnnnnee Aignmen
Mooos Briin Aignment
CrCare Dockin
Dosil
Mooss Briin Algnnninnnnnnnen Algnment
Modia wwool-Briiin. Afgmment
Pragees
#rtee Aggs
PIO Contatir
Whar coter
Progma
Figure 6: Model-Brain alignment for active and passive transitions and per module, excluding per
module read-in and readout layers where applicable. A) Alignment for Active Transitions. B)
Alignment per Agent Module for Active Transitions. C) Alignment for Passive Transitions. D)
Alignment per Agent Network Module for Passive Transitions.
B Latent Dynamies of Baseline Agents
Model Model response a...)
100
5
10
10
5
-30
5
10
100
3
10
PC
P-Prregwos
ICM
RND
Disagreement
Homeostaro
Raedoom
100
Time since passivity (s)
Figure 7: Latent dynamics of each agent-based control model. Dashed line indicates time since passivity. Coloring of PCs neural-glial cell-types (Fig. of the 3M-Progrest agent PCs.
Fig. 5) was chosen according to cell-types
16

---

C Formal Intuitions
C.1 Curiosity-driven Exploration is Not Enough
Consider an MDP with discount E (0, 1), occupancy (s a) (1-7) Eizo ? (1=($)=s,,
and transition kernel T(s' s, s.a). Let predictive world model with parameters @ be trained online to minimize
[ 0) = (ss,a,,')dd T ((((s, a, s';
and suppose the policy receives intrinsic reward determined by this predictor: r'(s, a, 8';0) = g( (f(s, a, f)) where 2 0 and g R20 - R2o is monotone increasing with g(0) = 0. The policy
updated to maximize
(mT 0) ((,,,,")~dt (s,,
Under either alternating or simultaneous updates that decrease L in and increase J in æ, the process cannot converge to nontrivial, uniquely defined exploratory policy. The only stationary outcome a degenerate collapse in which is constant on the visited support: otherwise the policy-moded coupling remains non-stationary and induces drift. That is, any stationary point (Λ* (*) satisfies one of the following cases:
1. Reward collapse. If 0* minimizes L(T*,0) on the support of dz. and the model class is realizable on that support, then ((=:0") O almost surely, hence r'(:;0") = 0 almost surely and J 8*) = 0 for all x. The objective is flat and does not select a unique exploratory policy.
2. No stationary policy. If we freeze the model at 0*, then r'; is fixed reward. If residual error remains, then f(:;0)) (hence r") is not almost surely constant, so there exists a measurable set U on which the intrinsic advantage Art (s, > O for some actions. By the policy-gradient identity.
VJ(T", (") = (1-7) Ed, (AT (s, a) V lo s)].
action with Aq- a) 0 and decrease it on other actions to preserve Σ. T" (a s) == 1. This choice makes the inner product (Ag. (s, V log x- 8)) strictly positive on U, hence the expectation above is positive and the policy gradient is nonzero. Therefore π is not a local maximizer of the stationary objective. Any such occupancy shift then triggers predictor updates that reduce pint on U, moving the high-reward region and preventing a nontrivial fixed point.
For some state U, suppose increase x (a 8) slightly
In both cases, these intrinsic signals do not converge a stable, uniquely-deiee defined exploratory policy. Either training drives the intrinsic signal to a constant (degenerate) value on the visited distribution, or residual heterogeneity in the reward keeps creating ascent directions that are then neutralized by predictor updates, preventing stabilization.
C.2 3M-Progress Partitions the Behavioral Space
Consider two reward-free MDPs, M1 = (S, A, Po) and M2 = (S,A, Tz, Po), that differ only in their transition densities. Further, suppose there exists a U S x A such that Ti(- s, a) (2( a) almost everywhere in s' for all (s, a) E U. Let s, and g(- s,a)) be world models trained on data from Ti and Tz, respectively, with pointwise consistency:
DEL (T s, a.a))pp( s. a)) & 0, DKL (Ta(- s, a)))l s,)) & 0.
Assume for each (s, a), both kernels share the same support with no vanishing probabilities (i.e. Be > 0 s.t. T2(s' 8, a) 2 whenever (s' s, > 0). Then it follows that for all (s, a) E U,
DKL (p(- 18. a))a)ll( s,)) 2 0,
and that for all (8 E Ue
DKL (pl- a))))l(( s,))) 3. DKL (Tx (Ti (T2) > 0.
17

---

C.3 Beyond Futility-indueed Passivity
Although we demonstrate 3M-Progress specific autonomous behavior known Autility-inducel passivity, our algorithm applics to any exploration behavior in which a dynamics niche is reasonably specified. In experiments, futility-induced passivity arises completely from the choice of the pretraining environment and the online learning environment; we choose these environments such that the transition dynamics between environments agree when the agent is passive, thus defining the ecological niche that guides exploration (see Appendix C22 simply put, we choose environments such that there exists a subset U in which the transition dynamics locally agree). In general, the pretraining environment should encode a meaningful dynamics prior the agent can use for continual learning with environments whose physics vary systematically from the pretraining environment.
For example, suppose we pretrain an agent and world model on foraging task (c.g. the virtual rodent environment in (66]). In a new environment that includes the opportunity for foraging and any of its constituent locomotor primitives (running, turning. jumping, etc.), the basic Distribution-matchinn objective of 3M- EM-Progress rewards the agent for trajectories whose dynamics are predictable under the pretrained world model (namely, foraging and constituent motor primitives):
r| f(é ca): E DKL (Wwo); é (1 7) + 7/t
where is learned online and 0 is pretrained. Niche-aware exploration is primarily mediated by the activation function f. When f is monotonic non-decreasing, such as rectified linear unit, 3M-Progress is niche-seeking with the niche defined by the dynamics prior wg. Conversely, if f is monotonic non-increasing, 3M-Progrest is niche-avoiding and explores dynamics outside the prior. Non- monooonee functions allow some amount of symmetry between niche-seeking and niche- avoidance depending on the specific function shape.
The utility of this approach can also be appreciated when the pretraining stage involves a large diversity of dynamics, either from multiple environments and tasks or single environment with multiple tasks. A world model with sufficient computational capacity that captures these diverse dynamics can be flexibly used in new environments to motivate exploration in variety of ways. In the simplest case, for example, suppose we pretrain an ensemble of dynamics models {we, J on
N separate environments and tasks. Maintaining independent temporal filters éf for each prior, deterministic intrinsic motivation can be defined as rf m f(? - , Alternatively, one can imagine various sampling schemes over the ensemble in order to drive specific exploration styles, such as e-greedy or max-entropy. This extends EM-Progress niche-aware exploration over multiple niches, thereby allowing the flexibility of multiple modes of behavior in single objective function. Each addition of dynamics prior further partitions the behavioral space in the online environment, and allows the dynamics characteristic of each pretraining environment to be composed to form an exploration landscape of attractors (niche-seeking. or repellors (niche-avoidance) for online learning.
D Description of Control Models
D.1 Model-based Controls
The Intrinsic Curiosity Module (ICM) defines intrinsic reward as the Shannon surprise of the forward model, r := log (Ø) Φe, az). I denotes an augmented inverse feature space that is learned on-top of sensory embeddings from & ICM trains an additional embedding layer of using an inverse dynamics model parameterized by 0, by optimizing an MSE loss L(Ør) Ema [0); of. 1 (()))ajj)) -a
Random Network Distillation (RND) defines a fixed random nonlinear projection of sen- sory features g(Ø) and trains predictor network 9 using an MSE loss rf := L(ORND) E., (lg( - ((RND:; dx)] With the distillation objective as the intrinsic reward, RND does not reinforce behaviors by scoring their predictability by a forward dynamics model as in ICM; instead, the random memory provides a simple exploration bonus for visiting novel states under the policy distribution.
18

---

Disagreement
S learn an ensemble of world models {we, 1 and defines intrinsic reward as rf = Var ((re) : [M]) for N randomly initialized world models. Ensemble variances scores the stochasticity of the environment and reinforces state-action pairs for which models disagree.
y-Progress 7 leverages the temporal history of Shannon surprise to define intrinsic reward using prediction gain, rf == log -enem, where &new parameterized a world model after learning on new
woald
transitions withheld from lagging model Ortd-
D.2 Model-Free Controls
One straightforward way to achieve a passive behavioral transition is to simply add an action-cost that outweighs any other positive reward signal. In the presence of fixed or nonexistent extrinsic reward signal which the agent has no control over (such as in the open-loop protocol), an action-cost encourages the agent to become passive, corresponding constraint or homeostatic regulation energy. We implement this cost as the magnitude of the force exerted by the agent's motors, c(a)) A \||a, I12- In all our experiments, we set X 1.
Homeostatic Agent
metabolic
Maximum Entropy Agent Maximum entropy RL. a general exploration strategy that provides a bonus reward proportional to the entropy of the current policy. That is, r = XH (T st)]. In all our experiments, we set
Random Agent To test a random baseline model for embodied control, we use a randomly initialized agent with the same model architecture (described in section Figure A and in PPO implementation details below).
Whole-brain Average We the average recorded neural-glial activity during specified be- havioral transition in one larval zebrafish to predict whole-brain neural-glial activity recorded from another larval zebrafish undergoing the same transition. The alignment under the metric described in section 3 is computed using the average response of each cell-type (neural and glial) from the source animal to predict the corresponding cell-type in the target animal. We two subjects and report the total alignment as the sample-weighh weighted average between scores from both cource-target pairs.
Maussian-Process We fit separate Gaussian Process (GP) to each cell-type (neural and glial) for each subject, using radial basis function (RBF) kernel and centering the prior mean at the average whole-brain response. Alignment under the metric described in section i computed between the whole-brain data from the target cell-type from an individual subject and its corresponding GP as the source. We use two subjects and report the total alignment the saiple-weighted average between scores from both source-target pairs.
White-noise At each timestep t, the model's predicted next state is give by Te+1 2, + Te- where ÿ) ~ (00 1) is white-noise. Because the metric that saturates inter-animal alignment is barela, the mean and variance of this random walk arbitrary.
PID Controller To implement circuit-based word model proposed by Mu al. for zebrafish brain's 's transition from active to passive states, employ threshold- based "bbng-bang" controller that switches the fish's 's swim power P(t) between an active waveform (base()) and complete cessation (P(t) 0) once cumulative "futility" signal exceeds fixed GABAergic threshold. This controller can be interpreted as the high-gain limit of a saturated PID controller.
Perceived hydrostatic drift is modeled as a constant Ud- The motor plant converts swim power into counter-drift locomotor velocity with gain gMs.
vs (1) = Va 9MS P(I).
Visual mismatch the product of forward stimulus velocity and ongoing motor drive but is rectified to ignore overshoot,
(t) if (t) 0,
e(t)
0,
otherwise.
19

---

A leaky integrator with time constant accumulates the mismatch (i.e., the futility):
(()) -xF((t) + kre(t),
where AF = T/TF and hp is a gain. second leaky integrator converts sustained futility into an inhibitory drive,
G(I) - -)oG() ko max(0, F()) 0f);
Powe (t),
G(t) - AG.
P())
0,
G(t) > 0g.
To mimic experimental perturbations we JMS 0 between on and LoL off, effectively clamping optic-flow feedback.
E Implementation Details
All code can be found in htttp:://githubb http:///wo....ooooooooaaoa/aaaaa
Proximal Policy Optimization (PPO) In all our experiments, we train our agents with PPO using
clipped surrogate objective [55] with CLIP 0.2 and normalized advantage function computed using Generalized Advantage Estimation (GAE) [67 with X X = 0.95 The policy network is an MLP with two hidden layers [128, 64] optimized by Adam (8] with learning rate a 0.0003 and gradients computed over 5 epochs of 1000-step trajectories with a 250-step batch size vectorized across 64 environments. This MI.P parameterizes the policy as 5-dimensional diagonal Gaussian distribution which is sampled to produce actions during training. For evaluation (the experiments in section 5), the policy is deterministic by taking actions as the mean of the distribution. Actions take the form of continuous real-valued torques on [-1,1]. The intrinsic rewards are normalized by dividing by a running estimate of the standard deviation of the sum discounted rewards with discount factor 7 0.99, which then supervises the value network MLP with two hidden layers [128, 64] to estimate the expected discounted return using this same discount factor. Both policy and value networks compute on hidden states from separate LSTM blocks (Figure (AA) using shared embedding from the sensory feature extractors. Image embeddings are obtained from 64x64 pixel observations passed through three-layer ResNet resulting in outputs with spatial resolution of 16x16 (closely matching the visual acuity of larval zebrafish at the final layer). Image embeddings were flattened and concatenated with proprioceptive features including joint positions and rotational velocities and their embeddings from a two-layer MLP with hidden size [64, 64).
Intrinsic Drive Module (IDM) The IDM architecture details vary depending which intrinsic drive implements, and is described on a case-by-case basis in the sections below. Here, we describe the commonalities between intrinsic drives, which include the optimizer, forward dynamics loss, general forward model architecture, and memory buffer parameters. Each forward dynamics model across intrinsic drives is implemented as a two-layer MLP with hidden sizes [512, 512], trained to predict the true observation one time-step into the future from the current observation using an MSE loss (where inference is done in feature-space), and optimized using Adam (68] with learning rate = 0.001. The IDM maintains a memory buffer of the last 100 observation embeddings from each environment in the vector and trains it's constituent networks on this buffer every 20 steps. The IDM is trained for 1e steps before the intrinsic rewards are observed by the agent.
3.M-Progress The ethological memory created (with the default configuration outlined above) by training a swimmer agent on a simple navigation task in head-free version of the experimental protocol outlined in section 3 This environment chosen to provide the forward model with similar visual input space as the head-fixed version while maintaining the ethology of unconstrained swimming which the agent experiences naturalistic fluid forces and positional displacement in response to swim commands. The task is implemented as shaped reward proportional to the distance of the agent from a target location that in front of the agent, such that the swim-to-target behavior results in stabilizing the constant backwards flow of the high-contrast grating. The episode is long
20

---

enough that optimizing this reward allows the agent to become passive was the target is reached. Together, our setup allows the agent to experience attto-cctin--tate triplets (current state, current action, and resulting state) associated with active and passive behaviors. This provides a close correspondence between sensory-motor coupling in our virtual ethological environment and the closed-loop experimental condition in Mu et al. (2). where larval zebrafish swim against the passive backwards flow of high-contrast gratings motivated by positional homeostasis. Although our agent is motivated to swim by different signal than its biological twin (i.e., moving towards target location rather than a homeostatic drive that resists displacement from an opposing current), the design of the virtual environment renders the sensory-mote stream experienced by the internal world model equivalent between scenarios, since in both cases optic flow is counteracted by forward swim motion. Because this sensory motor stream and it's resulting world-model alone define the ethological memory, the discrepancy in the signal that drove behavior has no bearing on training the new policy and value network in the open-loop environment (Figures D..B).
In the open-loop environment, the agent randomly initializes a new world-model memory that is trained online with the default configuration outlined in the IDM section. The ethological world-model memory is loaded from earliest checkpoint where the agent achieved optimal swim-to-target behavior is computed the MSE between the
and it's weights are frozen. The medddll-mmoy-mmmmmmae predictions from each memory, filtered by an exponential moving average with timescale y = 0.99, and the difference filtered and unfiltered predictions are passed through L1 activation.
For RND both target and predictor networks are im- plemented using the default configuration in the IDM section. The predictor network is trained to predict random feature projections from the target as described in section 3
Random Network Distillation (RND) 4
Intrinsic Curiosity Module (ICM) (3] In addition to a forward model implemented using the default configuration in the IDM section, the ICM implements an inverse dynamics model as an MLP with identical architecture and optimization routine to train one-layer MLP on top of sensory features. The forward and inverse networks are cotrained by minimizing a joint objective 3LF + (1 LI, where Le and L, are the forward and inverse MSE loss functions, respectively. In our experiments, we set 3 = 0.2.
Disagreement We N = randomly initialized independent forward models using the default configuration described in the IDM section. Disagreement is computed as the mean variance across feature dimensions.
Using a randomly initialized forward model implemented using the default con- figuration in the IDM section, the trailing memory is created by copying these initial weights and updating them using an exponential moving average with timescale 7. In all our experiments, we use y 0.99.
y-Progress 6
F Inter-Subject Noise Correction Derivation
Herein we describe how the metric M should correct for noise if there is trial-to-trial variability. This is unified and adapted from Nayebi* et al. [62], Nayebi et al. 63 69.. If you prefer to skip the derivation, for common choices of metric M, such as Pearson correlation, RSA, and especially any metric that satisfies transitive closure (70), one will need to correct by the square root of the product of the mapping consistency and internal consistency of the units, in order properly approximate the true value of M in the limit of infinite trials.
To make this correction explicit, suppose we have neural responses from two animals (or subjects) and B. Let uf be the vector of true responses (either at a given time bin or averaged across a set of time bins) of animal p E AA {A, B.... receive noisy observations of ?,, so let 2 j. be the jth set of n trials of t/. Finally, let M (z; y), be the predictions of a mapping M (e.g.. PLS, or any type of regression) when trained on input to match is the prediction of mapping
stimulus set i [train, test]. Of course, we only
output and tested on stimulus set i. For example, M (traini :B ( test M on the test set stimuli trained to match the true neural responses of animal B given, as input, the
true neural responses of animal A on the train set stimuli. Similarly, M (sf (Sf. if sferain)) test prediction of mapping M on the test set stimuli trained to match the trial-average of noisy sample 1
is the
21

---

on the train set stimuli of animal B given, as input, the trial-average of noisy sample I on the train set stimuli of animal A. Then we have that:
Minu = (Corr (M (ferain: Merain test test))
predictivity
Corí( M (5 trainn? I,,rrain test
B
B test
Mest
(orr B tettain A (s 2,train rest
x Corl
& i 1letst
test
mapping consistency
internal consistency
where the average (.) is taken over bootstrapped split-half trials and train-test splits, and Corr(::-) denotes the Pearson correlation of the two quantities. Corr(::) denotes the Spearman- Brown cor- rected value of the original quantity (since it is computed split-halves of the trials, unlike the numerator, which is evaluated on the full trial set). The analogous correction for RSA holds, where the RDM/RSM of the responses is instead used for s, and M the identity mapping, M (z; test Frest-
When constructing Mea for model-brain mappings (rather than brain-brain mappings), we just replace A with the model responses, which deterministic.
The above correction in (4) fully implemented in the brainmodel brainmodel_utils _ttiss pack- age (https://github.. com/ www.negentssseaeee wtwpoagenttt--bbbbrraadoooooeeoe function. This function can be imported as follows:
specifically in the
get_linregrees_consiiee from brainmodel_umd utils. metrics... consistency import get_linregress coricesssscsssseeeey
The r rr_x________ ____n_sbbbbb___________b________ sb value returned by this function corresponds to the ratio in (4). Refer to the README and the function doestring for usage details across range of linearly regressed and non-regressed (e.g. RSA) metrics.
F11 Single Subject Pair
Suppose we have neural responses from two animals (or subjects) A and B. Let tā be the vector of true responses (either at a given time bin or averaged across set of time bins) of animal p E == {A, B.... on stimulus set E (train, test). Of course, we only receive noisy observations of t,, so let S be the jth set of n trials of !.. Finally, let
v); be the predictions of mapping M (e.g., PLS) when trained on input to match output and tested on stimulus set i. For example, M (1 LB ( test is the prediction of mapping M on the test set stimuli trained match the true neural responses of animal B given, as input, the true neural responses of animal A on the train set stimuli. Similarly, M ( trained to match the trial-average noisy sample on the train set stimuli of animal B given, as input, the trial-average of noisy sample on the train set stimuli of animal A.
is the prediction of mapping M on the test set stimuli B s,trrin) test
With these definitions in hand, the inter-animal mapping consistency from animal A to animal B corresponds to the following "true" quantity to be estimated by Men in the limit of infinite trials:
M := Corr (M (timin (rain))
test (tesst)
where Corr(:..) the Pearson correlation across stimulus set. In what follows, we will argue that Eq (5) can be approximated with the following ratio of measurable quantities, where we split in half and average the noisy trial observations, indexed by and by 2:
Mine == Corr (M (tiaini train) test beeet
Corr [M (sf,train
B (.train) test
(6)
Mest
Corr (M (M(S (sf,traim:
i st.train) test
.M
(sA (5 2raiin 2.train
Corr
(sf, s R.est)
test
In words, the inter-animal consistency (i.e., the quantity the left side of Eq (6)) corresponds to the predictivity of the mapping on the test set stimuli from animal A to animal on two different (averaged) halves of noisy trials (i.c., the numerator the right side of Eq (6). corrected by the
22

---

square root of the mapping reliability animal A's responses to the test set stimuli on two different halves of noisy trials multiplied by the internal consistency of animal B.
We justify the approximation in Eq ( by gradually replacing the true quantities (t) by their measur-
able estimates (8), starting from the original quantity in Eq (5). First, make the approximation that:
Corr (M (téain: (train) se,sss) Corr (M (ttraini tB (train)
btest) x Corr (frest 8 sg,tes)
(7)
by the transitivity of very positive correlations. Namely, in scenarios where correlations are very close to I, form transitivity holds, meaning if variable A is highly correlated with variable B, and variable B with variable C, then variable A is also highly correlated with variable C. This is desired situation, as low or negative correlations indicate neurons that are not self-consistent. Moreover, calculating certain metrics in these cases result in undefined values due to operations like taking the square root a negative number. Assuming high correlations is reasonable, especially when the number of stimuli is large. Next, by transitivity and normality assumptions in the structure of the noisy estimates and since the number trials (n) between the two sets is the same, we have that:
Corr (si,test)
S 2test)) Corr (5i, &test) x Corr (t St
Corr (te Stett))
(8)
In words, Eq (8) states that the correlation between the average of two sets of noisy observations trials each is approximately the square of the correlation between the true value and average of one set of noisy trials. Therefore, combining Eq 7 and Eq (8) it follows that:
Corr (M (terain (trai) 2,test
Corr (M (t terain) test (test)
Corr (11,test 2,test
( S
From the right side of Eq (9), we can see that we have removed t but we still need to remove the M (t t *train) test term, this term still contains unmeasurable (i.e., true) quantities. We apply the same two steps, described above, by analogy, though these approximations may not always be true (they are, however, true for Gaussian noise):
Corr [M ( sf.raii)) test $2,test)
~ Corr (s M (traini ttrain test
x Corr (M (ttrain: (rain)) test M (s),traini sI.train) test
Corr (M (sf s (...aiin) rest M (s (s S test
Corr M (sf.train 8 s,,rainn) test M (Iéainn ((rain) test
which taken together implics the following:
Corr (M (sf, $1 traiinn) test
B 22,test
Corr (M (1 1
B test $2,test)
Corr (M
(I.trainni train
A Sf
.M S 2,train
2,train
test
(10)
Eq 9 and Eq 10 together imply the final estimated quantity given in Eq (6)
F.2
Multiple Subjeet Pairs
For multiple animals, we consider the average of the true quantity for each target in B in Eq across source animals the ordered pair (A, B) of animals and B:
Mine (Corr (M (t 1 test (BBa))
AEA.,A,))
Corr M (s/, (s
B sttrain) test 22,test
Meu =
V /oor (M s4 ( $ s,.rrain)
M P.traini 2.train test/
x Corr
(s (
sB
2,tst
test
AEAIAA.)EAAA
23

---

We also bootstrap across trials, and have multiple train/test splits, in which case the average on the right hand side of the equation includes averages across these well.
Note that each neuron in our analysis will have this single average value associated with when target animal (B), averaged over source animal//subammpled source neurons, bootstrapped trials, and train/test splits. This yields a vector of these average values, which we can take median and standard error of the mean (s.e.m.) over, as we do with standard explained variance metrics.
F.3 RSA
We can extend the above derivations to other commonly used metrics for comparing representations that involve correlation. Since RSA(x, y) Corr(RDM(z)) RDM(y)) then the corresponding quantity in Eq 66 analogously (by transitivity of maximally positive correlations) becomes:
Mase (RSA (M (tirain "crain) test ((ex)) AEA:(A,))AAA
RSA M (s),traini
S (.train)) test $2 test
B
Mesa
VRSA (M(sf.taai)
(sf s Irraii))
M
8 2,train 22,train
x
RSA
B
teste 2e
test
AEA:AAB)AAXA
(11)
Note that in this case, each animal (rather than neuron) in our analysis will have this single average value associated with it when it was target animal (B) (since RSA is computed over images and neurons). where the average over source animals/subsampled source neurons, bootstrapped trials, and train/test splits. This yields vector of these average values, which we can take median and s.e.m. over, across animals E A.
For RSA, we can use the identity mapping (since RSA is computed over neurons as well, the number of neurons between source and target animal be different compare them with the identity mapping). As parameters are not fit, we can choose train = test, so that Eq becomes:
RSA (5)
Mine (RSA (1A,19))AAAAABBEAAA Mest
RSA (sf, sf) x RSA (f)
AEA:AAB)AAA
(12)
F.4 Pooled Source Animal
Often times, may not have enough neurons per animal ensure that the estimated inter-animal consistency in our data closely matches the "true" inter-animal consistency. In order to address this issue, holdout animal time and compare it to the ppeuuo-population aggregated across units from the remaining animals, as opposed to computing the consistencies a pairwise fashion. Thus, is still the target heldout animal in the pairwise case, but the average over A is over sole "pooled" source animal constructed from the pseudo-population of the remaining animals.
Pooling data across subjects to create larger pseudopopulations is common practice [? and helps researchers better isolate core representational principles that are conserved across individuals when data collection modalities limit the number of collected neurons per session.
F.5 Spearman-Brow Correction
The Spearman- Berarman-BBowwn correction can be applied each the terms in the denominator individually, as they are each correlations of observations from half the trials of the same underlying process to itself (unlike the numerator). Namely,
Corr (X, Y)
Corr (X,Y)= = + Corr (X, Y)
24

---

Analogously, since RSA(X, Y)= = RDM(()) then we define
RSA () = (=)() RDM())
2RSA Y)
+ RSA (X,Y)
25

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We confirm that the claims made in the abstract and introduction do not exaggerate or fabricate the scope of our contributions and results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
26

---

Justification:
We confirm that we include a paragraph on limitations in the conclusions
section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not provide any theoretical results in our paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We fully detail the architectures, environment, and algorithms used, as well as the data used for model-brain alignment comparisons.
Guidelines:
27

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will provide full access to our code, as well as links to access the larval zebrafish data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.ccppulic//guus http://////iiiee///deuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
28

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide a complete characterization of all the methods in our paper, with more details in the supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report error bars on all our plots when applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We include details on the computational resources we used.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////aurrssssccc/ppbll/beia
Answer:
[Yes]
Justification:
We strickly adhere the NeurlPS code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper addressed fundamental scientific questions, and is not directly aimed towards societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
30

---

Justification:
Our paper does not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We make sure to properly credit everybody.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification:
We do not provide any new assests.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
31

---

Answer:
[NA]
Justification:
We do not conduct any crowdsoureing experiments.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not use participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
We do not use LLMs for the mentioned purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy (https://neurips: http:///s..ccoooeeee/ne///////// for what should or should not be described.
32


Reviews:
[{"review_id": "WenPLV0T2Y", "reviewer": "Reviewer_wCww", "summary": "The paper proposes 3M-Progress, a model-based intrinsic motivation reward that allows a virual zebrafish to exhibit autonomous behaviors matching behavioral and neural-glial dynamics recorded in autonomously behaving larval zebrafish. A 6-link virual zebrafish was embedded in customized fluid-dynamics MuJoCo environment was used in the experiments. Agents trained with 3M-Progress are compared to baselines implementing other intrinsic reward signals: ICM, RND, Disagreement $\\gamma$-progress as well as model-free entropy-based exploration term and random policy. The results showed 3M-Progress was significantly better at matching behavioral and neural-glial dynamics from real zebrafish recordings in both active and passive transitions.", "questions": "Is there a reason $\\theta_{old}$ is only trained in a separate phase from $\\theta_{new}$? What would the effect of using moving average be here?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- This work is a timely cross-disciplinary contribution, as it is one of the first work to bring goal-driven embodied agent settings to model neuron–astrocyte interaction that is quantitatively validated against whole-brain recordings, which significantly improve ethological validity over prior work.\n- 3M is a clever and novel formulation of intrinsic drive in animals, as the memory-based divergence elegantly avoids the “white-noise” issues of prediction-error signals and yields stable, repeatable behavioral cycles. \n- The empirical evaluation is very solid with multi-level metrics, and also the results are statistical significant with non-trivial gains over other methods. \n\nWeaknesses:\n- The current environments are constrained to only allow passive transitions and fluid movement to simulate active transition observations. This limits the ecological realism, but I do understand the effort of creating such environment and agent is non-trivial.", "was_revised": true, "final_justification": "The author has addressed my concerns and answered my question. I maintain my initial assessment that it's a good paper with insights that would be beneficial to share with the community.", "rebuttal": "### Overview\n\nWe thank the reviewer for their positive position on our paper. Below, we clarify and questions and concerns presented in the review. \n\n\n### Response to Specific Concerns\n\nWe agree that the ecological realism of the evaluation environment can be improved (though we note that this differs from the pretraining environment). The evaluation environment was chosen to match the parameters of zebrafish experiments conducted in Mu et al. (2019), including the head-fixed constraint on movement as well as the striped grating constraint on visual input. This allows us to make meaningful behavioral and neural comparisons between agents trained in simulation with zebrafish acting in vivo. For the pretraining environment, however, agents we're trained in a naturalistic environment with ecological visual inputs (mossy/grassy floors) and fluid forces. \n\nFuture work will include extending the 3M-Progress algorithm to other behaviors, both in zebrafish and in other animals. The core bottleneck to this procedure, however, is the existence of open source neural and behavioral data for autonomous behavior in animals in order to validate a model of animal autonomy. To the best of our knowledge, the dataset from Mu et al. (2019) is the first of its kind.\n\n### Response to Questions\n\n**Q1: Is there a reason $\\theta_{old}$ is only trained in a separate phase from $\\theta_{new}$?**\n\nYes, the pretraining stage to obtain $\\theta_{old}$ is the crux of how our 3M-Progress differs from existing intrinsic motivation algorithms. Traditional algorithms (like learning progress) learn both $\\theta_{old}$ and $\\theta_{new}$ online in the evaluation environment, and use the difference between the old and new models as a learning signal. In contrast, we propose that $\\theta_{old}$ be trained in an environment with entirely different transition dynamics than the evaluation environment. Motivated by the idea that animals are endowed with strong priors on how environment dynamics unfold in ethological circumstances, $\\theta_{old}$ provides a fixed prior on the transition dynamics the agent expects to encounter. This is achieved by training in a separate phase characterized by a separate environment. In the evaluation environment, $\\theta_{old}$ is frozen and compared with the online model $\\theta_{new}$, and this residual forms the learning signal. This signal tells the agent how different the dynamics in the new environment are from the old environment, and it can be used to motivate the agent to seek out behaviors that align the dynamics between the two. \n\n**Q2: What would the effect of using moving average be here?**\n\nBoth 3M-Progress and learning progress employ exponential moving averages, but on different variables. Learning progress obtains $\\theta_{old}$ by an exponential filter over the current world model parameters  $\\theta_{new}$. The effect of this is a signal that loosely corresponds to the rate of change of their prediction-error and avoids perseveration on dynamics with constant prediction error (e.g., white noise). On the other hand, we fix $\\theta_{old}$ while $\\theta_{new}$ is learned (as described above), and apply the exponential moving average to $D_{KL}(\\omega_{\\theta_{old}}\\mid\\mid\\omega_{\\theta_{new}})$. This results in a signal that tracks how predictions compare to the predictions from a pretrained model and avoids perseveration on dynamics with constant KL divergence (e.g., white noise)."}, {"review_id": "u1tK8Z4v2Y", "reviewer": "Reviewer_Ypm3", "summary": "This paper introduces a novel intrinsic reward based on modeling the divergence between an ethological prior and a current world model. This intrinsic reward model is applied to studying larval zebrafish, specifically their different behavioral patterns and their whole-brain neural-glial dynamics. Using this model, the paper demonstrates far greater alignment with zebrafish behavioral data compared to other intrinsic reward models, as well as outperforming them in predicting neural imaging data.", "questions": "- What kinds of behavior does this reward elicit in other environments of the DeepMind Control Suite?\n- What is the behavior of the zebrafish agent in moments of high/low reward?\n- Are there differing computational constraints for the different intrinsic reward models? If so, what implications does that have for the biological plausibility of the model?", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Overall, this paper does an excellent job in clearly establishing the value of the 3M-Progress method, implementing a suite of baselines for each experiment and reporting error bars showing a clear improvement in comparison to those baselines. There is little to fault with regards to the quality, significance, and originality of this paper.\n\nWeaknesses:\n- The placement of baselines in L167-182 is confusing, as I initially had the impression that these were a component of the actual model being introduced. This content could all be moved to the supplementary material.\n- More time could be spent elaborating on the 3M-Progress reward model, specifically regarding the intuition behind the scenarios in which the reward might be especially high or low.\n- A couple of minor typos: “sizing relaitve” in L134, “ecnoded” in the Fig. 2 caption.", "was_revised": true, "final_justification": "The authors have clearly answered my questions in their rebuttal, and agreed to reorganize the paper in a way that addresses my concerns about clarity.", "rebuttal": "### Overview\n\nWe thank the reviewer for their time and their positive position on our paper. We address the concerns and questions presented in the review below. \n\n### Response to Specific Concerns\n\n1. Organization: We agree that the organization of the baselines in L167-182 was confusing and not clearly explicated. We have moved this content to the supplementary material, where we explain these algorithms in more detail. \n2. Clarity: We agree that adding more intuition on the 3M-Progress reward would be helpful to the reader. Following the initial motivation and explanation in the **3M-Progress** section, we have added a section describing how the reward function changes as the agent pursues different behaviors. \n3. Typos: We thank the reviewer for their careful attention and have fixed the typos. \n\n### Response to Questions \n\n**Q1: What kinds of behavior does this reward elicit in other environments of the DeepMind Control Suite?**\n\nBecause 3M-Progress is designed for continual exploration in environments with dynamics that structurally differ from the pretraining environment, the standard DeepMind Control Suite is not a suitable benchmark since it was designed for policies to learn an environment-specific control task, where training and testing are always in-distribution. However, it is not uncommon to introduce domain shifts or task variation if policy generalization is the central question (e.g., the DMControl Generalization Benchmark), and we agree that this would be an interesting line of future work.  \n\nThe focus of our paper, however, is to show that existing intrinsic motivation algorithms do not generalize to autonomous animal behavior, and we propose 3M-Progress as a working solution. Rather than testing 3M-Progress on a suite of RL benchmarks, we are testing existing algorithms on an animal autonomy benchmark. \n\nAlthough 3M-Progress can be benchmarked in standard RL domains, it is beyond the scope of the paper since we are not introducing 3M-Progress to compete with existing algorithms in standard RL domains. To reiterate, we are flipping this view by testing curiosity-driven algorithms in real-life animal environments for which we have behavioral and neural data to validate against. \n\nIn order to concisely illustrate how 3M-Progress can be used in other contexts, we devote some time in the methods section to explain a generalization of the algorithm to new behaviors and other environments. Specifically, we show how to generalize 3M-Progress from a single pretraining environment to $N$ environments, across a variety of sampling procedures to choose behaviors in the evaluation environment. \n\n\n**Q2: What is the behavior of the zebrafish agent in moments of high/low reward?**\n\nIn moments of high reward, the agent is pursuing a behavior that is aligned with its prior memory. In moments of low reward, the agent is pursuing the same behavior over an extended amount of time. This is best illustrated by the rightmost panel of Figure 2B. The agent first computes $\\epsilon_t$, the KL divergence between its current predictive model and its pretrained one. This divergence will be high when the agent chooses actions that produce transition dynamics that differ from those of the pretraining environment---we call this memory disagreement. Initially, the moving average baseline is zero, and so sustaining a behavior with high KL divergence will be rewarded. Eventually, the moving baseline will catch up according to the timescale determined by the progress horizon, $\\gamma$. As it does so, the KL divergence for this behavior decays. The agent is thus incentivized to pursue a behavior with low KL divergence in order to deflect away from the current baseline and recieve a positive reward. This corresponds to pursuing transition dynamics that are as close as possible to that of the pretraining environment---we call this memory agreement. Applying the absolute value as an activation function creates a cyclic transition behavior. \n\nWe completely agree that more time should be spent explaining these properties in detail, and thank the reviewer for pointing this out. As mentioned before, we have added an additional section for this in the paper. This includes the description of 3M-Progress mentioned in response to Q1.\n\n**Q3: Are there differing computational constraints for the different intrinsic reward models? If so, what implications does that have for the biological plausibility of the model?**\n\n3M-Progress imposes an additional computational constraint by requiring a pretrained world model memory. Whereas the other intrinsic reward models learn everything online, 3M-Progress uses a two stage approach. We argue this feature makes our method more biologically plausible than a completely online method since animals come equipped with inductive biases when they solve new decision-making tasks or encounter new environments, rather than starting from a *tabula rasa* network. These biases are determined both by genomic information as well learned from online experience as the animal develops locomotor maturity. \n\nLearning progress assumes an agent implements long-term memory by decaying a world model learned from online experience, and computes intrinsic reward by comparing predictions between the long-term memory and the current model to track temporal history. This is biologically plausible in the sense that animals are free to create online memories that decay in time.\n\nThe Intrinsic Curiosity Module (ICM) computes intrinsic reward as the prediction error of a world model learned in a feature space optimized for inverse dynamics prediction. This design choice is biologically plausible, as there is evidence of inverse dynamics features in animal neural data [1]. \n\nRandom network distillation trains a neural network to predict random features, and this prediction-error is used as the intrinsic reward. This is biologically plausible, since animals may encode sensory input with random projections [2]. \n\nDisagreement maintains an ensemble of $N$ world models, and the intrinsic reward is computed as the variance in their predictions. This is not biologically plausible, since it is intractable for animals to create and learn $N$ memories simultaneously for a single task (e.g., $N>3$ is already intractable for RL systems [3, 4]). \n\n[1] Aldarondo, Diego, et al. \"A virtual rodent predicts the structure of neural activity across behaviours.\" Nature (2024)\n\n[2] Ganguli, Surya, and Haim Sompolinsky. \"Compressed sensing, sparsity, and dimensionality in neuronal information processing and data analysis.\" Annual review of neuroscience (2012)\n\n[3] Kim, Kuno, et al. \"Active world model learning with progress curiosity.\" International conference on machine learning. PMLR, 2020.\n\n[4] Doyle, Chris, et al. \"Developmental curiosity and social interaction in virtual agents.\" arXiv preprint (2023)."}, {"review_id": "UtpdYxS0yd", "reviewer": "Reviewer_fjGy", "summary": "The paper describes a model of learning, behavior, and neural activity in zebra fish; the fish simulation is swimming head-fixed in a simulated dm-lab environment. Although this is an ambitious project requiring a large amount of engineering work at multiple levels, the paper focuses in particular on the learning algorithm which enables natural bi-modal swimming behavior, and brain-like neural activity. The learning model requires training a vision encoder, a proprioception encoder, a policy model, a world model (used to define an intrinsic reward), and a value model (used to predict the intrinsic reward for PPO). The core of the analysis looks into the alternative definition of the world model and intrinsic reward to enable a realistic behaviour. The 3M progress is described as a working solution. The resulting behaviour nicely oscillates between “passive” and “active” behaviour. The artificial neural network activity is then compared with the recorded neural and glial cell activity, and the activity correlation is also highest with this algorithm.", "questions": "Questions are part of my main feedback above; the main ones are:\n\n- I wonder if a simpler model would also achieve that (like oscillating between maximizing action and fatigue, without modeling curiosity or proprioception)\n\n- How is the model-brain alignment correlation metric defined mathematically?", "limitations": "I see no potential negative impact on society.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The project is extremely ambitious and requires a lot of engineering work. While the project is impressive and achieves multiple conceptual milestones on a high level, I believe this paper will have a greater impact if it were written better. For instance, a clearer description of the contributions and findings is needed, and some of the mathematical notation is also unclear. I believe the technical work is deep, and this paper could become a very good paper with a substantial re-writing of the contributions and the technical description.\n\nOne high-level concern: In hindsight, the paper falls partially into some of the difficult existential problems of large-scale simulation work in computational neuroscience. While the algorithm is sound and the simulation is impressive, I am doubtful that the 3M-progress will remain as “the good” model of head-fixed zebra fish. In the end, the behaviour appears to be a bi-modal oscillatory behaviour between active tail and not moving. Although M3-Progress seems to be a working solution, I wonder if a simpler model would also achieve that (like oscillating between maximizing action and fatigue, without modeling curiosity or proprioception), and I also wonder if unintended details of M3-P are responsible for the results. For instance, could the oscillating behaviour just be an effect of the lag between the new and old parameters?\n\nA possible simple way to address my high-level concern is to make a clear list of technical contributions and provide more details to make them individually usable or reproducible. I believe that great individual technical contributions are shadowed by the grand claims, which are less solid. I am thinking of technical contributions that have a higher chance to be useful to other researchers, like the embodied zebra fish dm-lab simulation (if this is a contribution of this work), the novelties of the intrinsic RL algorithm, or for setting a benchmark in a clear quantitative metric (e.g., correlation of neural activity from the Zebra fish dataset).\n\nSuggestions to improve the writing, which is currently sub-optimal. Addressing some of them is very likely to increase my grade.\n\nA) Clarifying the contributions: make clearer what the list of technical contributions is in the intro.\n\nB) Unclear technical description of the model:\n\nB1) When reading paragraph 139 about the PPO algorithm, more basic things like the action space were not well defined\n\nB2) The notation Phi is not clear; it is used with small variations in Figure 2a, l. 166, 169, 174 are sometimes Phi(S_t, sometimes Phi^I, sometimes Phi_t) but it never defined rigorously. I feel that it broadly means “some network model” which is not a good notation. There should be clear notation to dissociate the vision encoder, proprioception encoder, encoder used in other algorithms, etc…\n\nB3) Paragraph line 167: It is not clear if this section is a summary of related work, if this is a contribution from the paper. I then understood that these are the baseline models, but it is quite obscure when reading the paper linearly in logical order.\n\nB4) The notations: Var( {omega} ) is not very clear. omega is a Gaussian density, that’s okay, but what is the Var( {omega} )\n\nB5) The whole paragraph between line 210 and 211 is quite vague. It is written as if that were proof. But I believe it’s also part of the definition of the 3M progress algorithm? What is L_theta_old after in line 210.\n\nB6) Definition of intrinsic reward has no equation number. Why not simply define r = abs(epsilon_hat - epsilon)? The usage of a general f is not helping, but is adding to the general confusion.\n\nC) More details for the quantitative results, and More info is needed to reproduce the results or compare results on the same dataset.\n\nC1) It would be useful to define mathematically the way the Pearson correlation is computed. Is this trial averaged in some way? Is that the definition of correlation, smoothing time in some way? Currently, all the “model-brain alignment” metrics are very obscure because high single-trial correlation over time for two units would be very strange because the behaviour itself is mostly different in each trial. So each unit is most likely not correlated to itself in different behavior histories. Ideally, one would need more than one metric, see for instance [3]. I also find it unsatisfactory that the chosen metric is saturated for most models (see Supplementary Figure 6 and 7)\n\nC2) Also, a mathematical definition of the “one-to-one” alignment would be needed, as it is needed to compute the correlation metrics and reproduce the quantitative results. Are you using some optimal bijective mapping between two networks (as in [3])? Is it first assumed that the number of neurons in the model and the brain are identical via random sampling before applying the optimal assignment? These kind of choices are changing a lot the quantitative results, so they should be explained clearly.\n\nD) I wish the results would be analyzed more deeply. The paper tends to claim grand achievements (“the model passes the neuro ai turing test”). I would not recommend that, but why not? Yet, it would be at least necessary to give all quantitative and technical details to reproduce and justify this wording. And scratch below the surface to illustrate the stated results. Here are some suggestions: D1) Grand claims like “the model passes the neuro ai turing test”, need to be clearly linked to a set of behaviour and neural activity measurements which are clearly defined and detailed.\n\nD2) I wish an ablation study had been run on the M3-Progress to better understand what element of the model is responsible for the quantitative reports.\n\nE) Missing reference to previous work: In addition to the fly model from Janelia, there have been other attempts to model the organism and its environment before the “embodied neuro ai” terminology, like the “swimming neuron” [1] or the open worm project [2]. Reference [3] also modeled neural activity and behavior at large in mice.\n\n[1] https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010899\n[2] https://openworm.org/\n[3] https://proceedings.neurips.cc/paper_files/paper/2023/hash/ec702dd6e83b2113a43614685a7e2ac6-Abstract-Conference.html", "was_revised": true, "final_justification": "The authors have written in their rebuttal that they will describe and highlight their technical contributions better and clarify the model description when it was mathematically ambiguous.", "rebuttal": "### Overview\n\nWe thank the reviewer for their detailed, thorough, and constructive feedback. We found the advice and suggestions of the review very helpful for improving the quality and clarity of the paper. We have implemented all of the reviewer's suggestions. Below, we outline the specific changes we made per suggestion, and answer the reviewer's questions. \n\n### Response to Specific Concerns\n\n**A.** We agree that the technical contributions of the paper were overshadowed in the introduction of the paper. We have rewritten the introduction to be specific, less grandiose, and include a list of technical contributions:\n\n1. 3M-Progress, a novel intrinsic motivation algorithm that tracks divergence between an online world model and frozen dynamics prior. \n2. 3M-Progress agents reproduce whole-brain neural-glial calcium dynamics and behavior, providing the *first* goal-driven model of neural-glial computation.\n3. A virtual environment that closely replicates zebrafish experiments in silico, serving as a platform for further investigating whole-brain neural-glial mechanisms.\n\n**B**. We agree that there were several areas in the technical description of the model that were lacking. We addressed each of the reviewer's concerns as follows. \n\n**B1-B2**. We have added a description of the action space to paragraph 139, as well as clear definition of $\\phi$ to avoid confusion. Specifically, we altered Figure 2A to denote two separate feature encoders, $\\phi_I$ and $\\phi_J$, denoting visual and proprioceptive encoders, respectively. Then, we specified that when $\\phi$ is used without a subscript, it refers to the concatenated features from each encoder, $\\phi_I$ and $\\phi_J$. Finally, we make clear that these encoders form the basic architecture of the agent and remain the same for all the intrinsic reward algorithms. In addition, we point the reader to specific details of the encoding models in Appendix D. \n\n**B3.** We agree that the placement of the description of baseline algorithms was not properly demarcated and interrupted the logical flow of the section. To fix this, we moved the description of the baselines to Appendix C.1, where we are able to provide additional detail on their respective algorithms and the intuition behind their intrinsic rewards. As a result, the **Methods** cleanly flows directly from the mathematical description of Model-Based Intrinsic Motivation to the description of 3M-Progress.  \n\n**B4.** We agree that the notation for Disagreement, $Var(\\\\{\\omega_{\\theta_i}: i\\in [N] \\\\})$ was unclear because each $\\omega_{\\theta_i}$ is a Gaussian density; we are actually taking the variance across the sample set $\\\\{x_i \\sim \\omega_{\\theta_i} : i \\in [N]\\\\}$. Since our implementation uses the mean of each Gaussian as a point estimate of $\\omega$, we now write $Var(\\\\{\\mu_{\\theta_j}: j \\in [N]\\\\})$.\n\n**B5.** We agree that the paragraph at 210 was vague, and unclear whether it was part of the 3M-Progress definition. The intention of this paragraph was to provide a formal justification for how choosing the pretraining and evaluation environment determine which behavioral partitions the agent discovers. However, it wasn't well connected to the existing text. We have removed this paragraph from the main description of 3M-Progress. Instead, we have rewritten it as a formal proof and clearly motivate it in another section where we explain the intuitions behind 3M-Progress reward dynamics (we describe this in more detail in our response to D2). \n\n**B6.** We have added equations numbers to the intrinsic reward, and reformatted the description so that each element (KL Divergence, exponential filter, and intrinsic reward definition) appears together at once. We agree that using the extra symbol $f$ was confusing. This choice was originally made to emphasize that different activation functions can be used with 3M-Progress to elicit different behaviors. However, this could be more clearly motivated in the text since we did not provide a description of how such activation functions could be used. To fix this, we have changed both Figure 2A and the description of 3M-Progress to use only the absolute value function to avoid confusion. In addition, we added a clear description of how each component (including the activation function) affects the learned behavior---we explain this change in more detail in our response to D2. \n\n**C.** We agree with the reviewer that the definition of the metric used for model-brain and inter-animal alignment was not defined in the paper. We added a complete description to Appendix E.\n\n**C1.** The Pearson correlation is not computed from single trial data, but from trial averaged responses between subjects within the same behavioral block as identified and labeled by Mu et al. 2019. So, high correlation over time for two units is common in this case, since the units taken between subjects are undergoing the same behavior. We added a full derivation of the one-to-one correlation metric to Appendix E---more on this in C2. \n\nRegarding the saturation of the chosen metric, we explain why this occurs and provide alternative evaluation criteria in our \"Response to concern #4\" within our rebuttal for Reviewer M8K4. To avoid repetition, we kindly refer the reviewer there for complete details. \n\n**C2.** The mathematical definition of the one-to-one alignment has been added to Appendix E and referenced in the relevant sections. The mapping is not bijective, as it does not assume an equal number of neurons between the model and the brain. Rather, the alignment maps each neuron in the animal brain to any unit in the model, and this mapping is restricted to any individual model unit in the 100th percentile of correlation. That is, unlike typical model-brain alignment metrics that rely on linear regression between a set of model units in order to predict a single neuron, our metric uses a single unit. \n\n\n**D.** We agree with the reviewer that some of the achievements, especially the NeuroAI Turing Test, could be better defined in the text. In addition to adding a precise definition (Appendix F) and explaining how our model meets that definition in this context, we have also rewritten the paper with less grandiose language overall in order to make the contributions and results more clear. \n\n**D1.** We have added the definition of the NeuroAI Turing Test to the results section. Passing this test means that according to a chosen metric and a given task, the behavioral and neural alignment between the model and animal is close to, or the same as, the behavioral and neural alignment between two individual animals. \n\n**D2.** We agree that the elements of 3M-Progress could be better explained to understand how each component contributes to the model-brain and model-behavior alignment. However, one can view some of the baseline algorithms as ablations themselves. For example, $\\gamma$-Progress can be viewed as 3M-Progress without the pretrained memory and symmetric activation function. It's failure to capture the behavioral switching indicates that learning both memories online is not sufficient. ICM can be viewed as 3M-progress without the second memory by using predictions from one world model alone. \n\nWe agree that a 3M-Progress specific ablation would still be helpful. We have conducted additional experiments to illustrate how each component contributes to matching the behavior, and thus to explaining the neural-glial data. For example, raising the progress horizon, $\\gamma$, leads to longer, sustained activity in each behavioral transition. Changing the activation function to something asymmetric, like ReLU, leads to one-way transitions towards model-memory-agreement. We add these ablations and discuss their consequences in the Appendix. \n\n\n**E.** We have added the recommended references to the related work section. \n\n### Response to Questions\n\n**Q1: I wonder if a simpler model would also achieve that (like oscillating between maximizing action and fatigue, without modeling curiosity or proprioception)?**\n\nOne seemingly plausible possible explanation for the behavior is that the zebrafish cycles between the positional homeostatic drive to move forward until it becomes fatigued due to an action cost. However, Mu et al. 2019 explicitly ruled out this hypothesis with extensive experiments, and found that the futility-induced passivity behavior was not driven by fatigue nor positional homeostasis. Rather, both the behavioral and neural evidence support an error-tracking mechanism implemented by neural-glial circuits. \n\nThis provides the basis for curiosity-driven intrinsic motivation, since error-tracking in a novel environment implicitly assumes the presence of a predictive model. Given this basis, our paper focuses on how to use a predictive model for intrinsic motivation, and show that existing curiosity-driven approaches from reinforcement learning do not capture this specific case of autonomous animal behavior. \n\nFurther, model-free methods were consistently lower in whole-brain alignment than model-based methods, supporting our hypothesis based on the data in Mu et al. 2019 that this behavior is driven by world models. In particular, a PID controller (Appendix C), designed explicitly to integrate feedback error in a model-free way and switch between two control states (active and passive), is among the *worst* performing models on whole-brain data (Figure 3B). \n\nThus, although a simpler model could achieve the behavior, it would be inconsistent with the whole-brain data and neural-glial circuit proposed in Mu et al. 2019. \n\n**Q2: How is the model-brain alignment correlation metric defined mathematically?**\n\nWe adopt the model-brain alignment metric defined in [1], with details defined in page 18, \"Noise-corrected neural predictivity\". \n\n[1] Nayebi, Aran, et al. \"Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation.\" PLOS Computational Biology (2023)"}, {"review_id": "snDpfw867q", "reviewer": "Reviewer_M8K4", "summary": "In this work, the Authors introduce 3M-Progress, a novel framework for intrinsic motivation, and compare the predictions of this framework to previously published behavior and neural activity of larval zebrafish in the futility-induced passivity task. They show that the proposed model, compared to several existing baselines, best matches the animals' behavior in this task and reproduces the animals' neural activity marginally better than the baselines. The proposed model is mapped on the interaction between neurons and astrocytes in the brain.", "questions": "- What other biologically plausible models could explain the observed behavior?\n\n- What would be the steps to independently distinguish the models based on their neural activity?\n\n- What additional effect of the observed behavior or what additional behaviors would the model predict?\n\n- Perhaps I've overlooked this in the text, but it would be great to see / highlight the proposed circuit based on the results of this paper and to test / discuss its presence in the brain.", "limitations": "- Limited evaluation: a single model organism, a single behavioral observation, and a single biologically plausible model.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The research here pursues an interesting and relevant topic of how intrinsic motivation affects decision making. Although the concept of intrinsic motivation has been used in machine learning for quite some time now, it couldn't be readily used to model biological motivations, despite being loosely inspired by them.\n\n- In this work, the Authors compare their model to neuroscience data and consider a number of baseline models, which is, to the day, the best way to propose and validate neuroscience-related hypotheses. They show that the proposed model is the best among the baselines to match the animals' behavior in the task and is in the same ballpark with the other considered models in predicting the animals' neural activity.\n\n- The text is well-structured and, for the most part, written clearly, allowing the readers and the reviewers to know exactly what was done in this project and thus allowing to come up with a well-grounded valuation of this work.\n\nWeaknesses:\n\n- The language of the introduction and, overall, non-descriptive parts of the paper appears inflated, overstating the scope of the provided results. The paper correctly states that existing models in the field *\"fail to produce robust autonomous behaviors observed in animals\"*, *\"reproduce specific experimental findings\"*, or offer *\"simplified, bottom-up mathematical models\"*; that current models are limited to *\"constrained environments with either handcrafted object-centric inputs, or externally-defined rewards\"*, or *\"non-embodied settings\"*. The current work, in contrast, is claimed to *\"bridge this gap\"* by *\"leveraging biologically-realistic, open-ended scenarios rather than simplified tasks\"* to yield *\"a naturalistic theory of neural-glial computation, offering insights into evolutionary pressures underlying the detailed heterogeneity observed at level of individual neural and glial cells.\"* Which seems inconsistent because i) the model is compared to a single experimental observation in a single model organism and ii) the models used in RL/robotics are quite sophisticated. This is fine, however, not to mislead the reader who will learn from this text, the scope of the claims need to be adjusted.\n\n- While this work compares the animals' behavior to multiple baselines, which is the proper thing to do, these baselines, as the Authors point out, are not necessarily biologically plausible. As for the specific model proposed here, that matches the behavioral observations in zebrafish larvae, the design choices in the model are done specifically to account for particular biological observations and thus may not generalize to broader scenarios, which would be a typical test generally applied to ned models. Specifically, the distributional matching, enforced by the KL divergence in Equation on Line 210 directly addresses the problem of \"Noisy TVs\", the exploration hijacked by the white noise, and the constraint in Equation on Line 214 with the symmetric function f described in Line 218 directly enforces the periodicity of behavioral phases observed in larval zebrafish and shown in Figure 3A. Thus, while it makes the proposed model principled and valid by design, it's unclear if (1) the model will generalize to new observations and (2) whether the observed effect won't be explained by other models.\n\n- Checking the models against the neural activity is, again, a highly appropriate step, however, the proposed analysis may not be sufficient to decide between the models based on the similarity to the observed neural activity. Specifically, all the model-based frameworks in Panel A exhibit high correspondence to the neural activity recorded in larvae. Yet, they lead to different behaviors. Would the latent dynamics of the other models look similar in Figure 5? Does that mean that the bulk of the neural activity is then irrelevant for this behavior? Should we then rely on this similarity at all?\n\nOverall, it's an interesting and promising work, however, I believe that (1) the strength of the claims need to be adjusted to reflect the scope of the work and (2) then the work's merit needs to be discussed among the reviewers with regard to NeurIPS's expectations in light of the work's results revisited during step (1).", "was_revised": true, "final_justification": "Post-rebuttal: some (not all) of the concerns were addressed; raising the score accordingly\n\n_______________________________________________________\n\nPost-rebuttal: second iteration: the remaining concerns that I had in the original review were addressed; raising the score.", "rebuttal": "### Overall Response\nWe thank the reviewer for their detailed, thorough, and constructive feedback. We found the advice and suggestions of the review very helpful for improving the quality and clarity of the paper, and we have implemented all the reviewers suggestions. Below, we provide responses to the reviewer's concerns and specific questions. \n\n### Response to Specific Concerns\n\nWe agree that some of the language of the introduction was grandiose and nonspecific, and should be rewritten to reflect the specific contributions of the paper. \n\n### Response to concern #1\nRegarding the comparison of our model to a single experimental observation in a single model organism, we agree that this is a limiting factor in demonstrating the generality of our method. However, to the best of our knowledge, the dataset provided in Mu et al. 2019 is the only example of whole-brain recording during an autonomous cognitive animal behavior. So, it is not feasible to test this method in other behaviors in other organisms since the validation data does not yet exist. \n\nWe would also like to point out that the baseline algorithms are state-of-the-art methods in RL/robotics used for model-based exploration. The goal of our paper is to use this unique dataset to demonstrate that these existing approaches to autonomous exploration in reinforcement learning, especially curiosity-driven methods, may not capture the nature of autonomous behavior real animals. The zebrafish dataset is one example of this gap, and can already serve to motivate new algorithms to understand animal autonomy.\n\nIn the future, we are excited for new experiments in other organisms to be released so we can continue to develop and validate our model on a wide variety of data. Until then, we are currently working on demonstrating how 3M-Progress can be used as an exploration mechanism in general reinforcement learning problems. However, to emphasize the generality of 3M-Progress, we have added a section in the main paper describing it's implementation and intuition for other embodiments and new environments.\n\n### Response to concern #2\n\nWe agree with the reviewer's observation that the design choices for 3M-Progress were specifically inspired by observations in the zebrafish. However, the 3M-Progress algorithm itself is a general intrinsic motivation strategy that can be used for other embodiments and tasks. The focus of our paper, however, is to show that existing intrinsic motivation algorithms do not generalize to autonomous animal behavior, and we propose 3M-Progress as a working solution. Rather than testing 3M-Progress on a suite of RL benchmarks, we are testing existing algorithms on an animal autonomy benchmark. \n\nAlthough 3M-Progress can be benchmarked in standard RL domains, it is beyond the scope of the paper since we are not introducing 3M-Progress to compete with existing algorithms in standard RL domains. To reiterate, we are flipping this view by testing curiosity-driven algorithms in real-life neuroscience tasks for which we have behavioral and neural data to validate against. \n\nFinally, we would like to remark that the purpose of our work is not to propose 3M-Progress a final or complete model of autonomous exploration, but rather to provide a platform for future work of reverse-engineer autonomous behavior in animals, since our agent can be deployed in new settings than the ones it was trained in.\n\n### Response to concern #3\nRegarding whether the observed effect won't be explained by other models, we first point out that none of the considered algorithms, both model-free and model-based intrinsic motivation, captured the autonomous behavior. Here, biological plausibility just refers to whether the algorithm could be feasibly implemented by an animal brain. For example, Disagreement assumes that an agent maintains $N$ world model simultaneously learned online, which is generally intractable for biological agents. \n\nHowever, the remaining algorithms (learning progress, ICM, max-entropy, PID control) are biologically plausible in principle and yet they do not capture the behavior. Further, we could design a simple algorithm that does capture the behavior, such as balancing an action cost with a homeostatic drive, which seems biologically plausible and avoids curiosity or introducing world models. The PID controller (Figure 4A) is one such example, designed to simply accumulate feedback error until hitting a threshold, where it switches between one of two possible states. However, Mu et al. 2019 explicitly ruled out such hypotheses with extensive experiments, and found that the futility-induced passivity behavior was not driven by fatigue nor positional homeostasis. Rather, both the behavioral and neural evidence support an error-tracking mechanism implemented by neural-glial circuits. \n\nThis provides the basis for curiosity-driven intrinsic motivation, since error-tracking in a novel environment implicitly assumes the presence of a predictive model. Given this basis, our paper focuses on how a predictive model should be used for intrinsic motivation, and show that classic curiosity-driven approaches from reinforcement learning do not capture this specific case of autonomous animal behavior. \n\n### Response to concern #4\nRegarding the saturation of the chosen metric, this can be attributed to our choice of evaluation criteria, which we explain below. Although the zebrafish behavior itself involves multiple transitions that occur in sequence over a single trial, we instead chose to compare behavior at the level of individual transitions for convenience. \n\nFirst, we point out that no intrinsic reward algorithms exhibited stable behavioral transitions within an episode over the course of training except for 3M-Progress (Figure 2A). For example, where 3M-Progress agents exhibit a passive-to-active transition, no such transition occurs for other agents in the same episode. This is because the baseline agents algorithms are unstable as they exhibit spurious or transient behaviors. Thus, in order to facilitate comparisons between individual transitions in 3M-Progress agents and baseline algorithms, we took a generous approach by allowing the missing transition in baseline algorithms to come from any episode from any model checkpoint. This means that despite the baseline agents not meeting the full behavioral deserderata of stable transitions within a *single* episode, the individual transitions are on equal footing with 3M-Progress agents, yielding a relatively high model-brain alignment (albeit, still lower than 3M-Progress). \n\nOne way to address this is to require that both active and passive transitions must come from the same episode for all baseline algorithms, as was done for 3M-Progress and as the animal exhibits. Since other algorithms fail to produce both transitions in a single episode, we simply choose the behavior following the identified transition to represent the missing transition. For example, RND often exhibits a valid passive transition early in the episode, and remains passive for the remainder of time. For the active transition, we take any window of time following the passive transition to represent the active transition. Using this criteria, RND, next highest performant model overall after 3M-Progress, falls from a model-brain alignment of $\\approx 0.93$ to $\\approx 0.51$ on active transitions. This evaluation criteria reflects the fact that the zebrafish behavior is characterized by sequences of transitions within a trial. \n\nWe can further distinguish between models on the basis of their latent dynamics (Figure 5). While we did not originally include the latent dynamics of the other intrinsic reward algorithms, we have added this to Appendix F. Despite the model-brain alignment for ICM, RND, $\\gamma$-Progress, and Disagreement being fairly high (but still lower than 3M-Progress), their latent dynamics fail to capture the neural-glial circuit observed in Mu et al. 2019. \n\n### Response to Questions\n\n**Q1: What other biologically plausible models could explain the observed behavior?**\n\nWe answer this question above in the \"Response to Concern #3\" section. \n\n**Q2: What would be the steps to independently distinguish the models based on their neural activity?**\n\nWe answer this question above in the \"Response to Concern #4\" section. \n\n**Q3: What additional effect of the observed behavior or what additional behaviors would the model predict?**\n\nGiven any dynamics prior that encodes intrinsic preferences, 3M-Progress agents will seek out behaviors in new environments relative to their alignment with their prior. The specific behavior they pursue primarily depends on the pretraining environment(s) from which the prior memory is obtained.\n\n**Q4: Perhaps I've overlooked this in the text, but it would be great to see / highlight the proposed circuit based on the results of this paper and to test / discuss its presence in the brain.**\n\nWe agree that the proposed circuit is core component of our paper, and we thank the reviewer for the opportunity to clarify this for the reader. 3M-Progress is a direct absorption of the model-based error-tracking and alignment---each of it's components reflect a computational motif of the neural-glial circuit proposed in Mu et al. (2019). We explain these details in the last paragraph of the \"3M-Progress\" section within the Methods portion, as well as in the \"Latent Dynamics of 3M-Progress Agents Reflect Underlying Neural-Glial Computations\" section within the Results portion. However, in order for the proposed circuit to be clearly outlined in the paper, we have added a section in the Methods portion that summarizes these details to explain how 3M-Progress provides a normative circuit model."}]

Author rebuttal:

Meta-review: This paper introduces a novel intrinsic reward mechanism based on modeling the divergence between an ethological prior and a current world model. The proposed approach is applied to the study of larval zebrafish, specifically examining their diverse behavioral patterns and whole-brain neural–glial dynamics. The research addresses an interesting and relevant question concerning how intrinsic motivation influences decision-making. Most of the reviewers’ comments have been adequately addressed by the authors, making this paper suitable for publication.
Decision: Accept (poster)