Title: Intrinsic Goals for Autonomous Agents: Model-Based Exploration in Virtual Zebrafish Predicts Ethological Behavior and Whole-Brain Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: NeuroAI

intrinsic motivation

zebrafish

neural-glial

embodied agents

reinforcement learning

autonomy.
Abstract: Autonomy is a hallmark of animal intelligence, enabling adaptive and intelligent behavior in complex environments without relying on external reward or task structure. Existing reinforcement learning approaches to exploration in reward-free environments, including a class of methods known as *model-based intrinsic motivation*, exhibit inconsistent exploration patterns and do not converge to an exploratory policy, thus failing to capture robust autonomous behaviors observed in animals. Moreover, systems neuroscience has largely overlooked the neural basis of autonomy, focusing instead on experimental paradigms where animals are motivated by external reward rather than engaging in ethological, naturalistic and task-independent behavior. To bridge these gaps, we introduce a novel model-based intrinsic drive explicitly designed after the principles of autonomous exploration in animals. Our method (3M-Progress) achieves animal-like exploration by tracking divergence between an online world model and a fixed prior learned from an ecological niche. To the best of our knowledge, we introduce the first autonomous embodied agent that predicts brain data entirely from self-supervised optimization of an intrinsic goal—without any behavioral or neural training data—demonstrating that 3M-Progress agents capture the explainable variance in behavioral patterns and whole-brain neural-glial dynamics recorded from autonomously behaving larval zebrafish, thereby providing the first goal-driven, population-level model of neural-glial computation. Our findings establish a computational framework connecting model-based intrinsic motivation to naturalistic behavior, providing a foundation for building artificial agents with animal-like autonomy.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the Authors introduce 3M-Progress, a novel framework for intrinsic motivation, and compare the predictions of this framework to previously published behavior and neural activity of larval zebrafish in the futility-induced passivity task. They show that the proposed model, compared to several existing baselines, best matches the animals' behavior in this task and reproduces the animals' neural activity marginally better than the baselines. The proposed model is mapped on the interaction between neurons and astrocytes in the brain.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The research here pursues an interesting and relevant topic of how intrinsic motivation affects decision making. Although the concept of intrinsic motivation has been used in machine learning for quite some time now, it couldn't be readily used to model biological motivations, despite being loosely inspired by them.\n\n- In this work, the Authors compare their model to neuroscience data and consider a number of baseline models, which is, to the day, the best way to propose and validate neuroscience-related hypotheses. They show that the proposed model is the best among the baselines to match the animals' behavior in the task and is in the same ballpark with the other considered models in predicting the animals' neural activity.\n\n- The text is well-structured and, for the most part, written clearly, allowing the readers and the reviewers to know exactly what was done in this project and thus allowing to come up with a well-grounded valuation of this work.\n\nWeaknesses:\n\n- The language of the introduction and, overall, non-descriptive parts of the paper appears inflated, overstating the scope of the provided results. The paper correctly states that existing models in the field *\"fail to produce robust autonomous behaviors observed in animals\"*, *\"reproduce specific experimental findings\"*, or offer *\"simplified, bottom-up mathematical models\"*; that current models are limited to *\"constrained environments with either handcrafted object-centric inputs, or externally-defined rewards\"*, or *\"non-embodied settings\"*. The current work, in contrast, is claimed to *\"bridge this gap\"* by *\"leveraging biologically-realistic, open-ended scenarios rather than simplified tasks\"* to yield *\"a naturalistic theory of neural-glial computation, offering insights into evolutionary pressures underlying the detailed heterogeneity observed at level of individual neural and glial cells.\"* Which seems inconsistent because i) the model is compared to a single experimental observation in a single model organism and ii) the models used in RL/robotics are quite sophisticated. This is fine, however, not to mislead the reader who will learn from this text, the scope of the claims need to be adjusted.\n\n- While this work compares the animals' behavior to multiple baselines, which is the proper thing to do, these baselines, as the Authors point out, are not necessarily biologically plausible. As for the specific model proposed here, that matches the behavioral observations in zebrafish larvae, the design choices in the model are done specifically to account for particular biological observations and thus may not generalize to broader scenarios, which would be a typical test generally applied to ned models. Specifically, the distributional matching, enforced by the KL divergence in Equation on Line 210 directly addresses the problem of \"Noisy TVs\", the exploration hijacked by the white noise, and the constraint in Equation on Line 214 with the symmetric function f described in Line 218 directly enforces the periodicity of behavioral phases observed in larval zebrafish and shown in Figure 3A. Thus, while it makes the proposed model principled and valid by design, it's unclear if (1) the model will generalize to new observations and (2) whether the observed effect won't be explained by other models.\n\n- Checking the models against the neural activity is, again, a highly appropriate step, however, the proposed analysis may not be sufficient to decide between the models based on the similarity to the observed neural activity. Specifically, all the model-based frameworks in Panel A exhibit high correspondence to the neural activity recorded in larvae. Yet, they lead to different behaviors. Would the latent dynamics of the other models look similar in Figure 5? Does that mean that the bulk of the neural activity is then irrelevant for this behavior? Should we then rely on this similarity at all?\n\nOverall, it's an interesting and promising work, however, I believe that (1) the strength of the claims need to be adjusted to reflect the scope of the work and (2) then the work's merit needs to be discussed among the reviewers with regard to NeurIPS's expectations in light of the work's results revisited during step (1).\n\nQUESTIONS:\n- What other biologically plausible models could explain the observed behavior?\n\n- What would be the steps to independently distinguish the models based on their neural activity?\n\n- What additional effect of the observed behavior or what additional behaviors would the model predict?\n\n- Perhaps I've overlooked this in the text, but it would be great to see / highlight the proposed circuit based on the results of this paper and to test / discuss its presence in the brain.\n\nLIMITATIONS:\n- Limited evaluation: a single model organism, a single behavioral observation, and a single biologically plausible model.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a model of learning, behavior, and neural activity in zebra fish; the fish simulation is swimming head-fixed in a simulated dm-lab environment. Although this is an ambitious project requiring a large amount of engineering work at multiple levels, the paper focuses in particular on the learning algorithm which enables natural bi-modal swimming behavior, and brain-like neural activity. The learning model requires training a vision encoder, a proprioception encoder, a policy model, a world model (used to define an intrinsic reward), and a value model (used to predict the intrinsic reward for PPO). The core of the analysis looks into the alternative definition of the world model and intrinsic reward to enable a realistic behaviour. The 3M progress is described as a working solution. The resulting behaviour nicely oscillates between “passive” and “active” behaviour. The artificial neural network activity is then compared with the recorded neural and glial cell activity, and the activity correlation is also highest with this algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nThe project is extremely ambitious and requires a lot of engineering work. While the project is impressive and achieves multiple conceptual milestones on a high level, I believe this paper will have a greater impact if it were written better. For instance, a clearer description of the contributions and findings is needed, and some of the mathematical notation is also unclear. I believe the technical work is deep, and this paper could become a very good paper with a substantial re-writing of the contributions and the technical description.\n\nOne high-level concern: In hindsight, the paper falls partially into some of the difficult existential problems of large-scale simulation work in computational neuroscience. While the algorithm is sound and the simulation is impressive, I am doubtful that the 3M-progress will remain as “the good” model of head-fixed zebra fish. In the end, the behaviour appears to be a bi-modal oscillatory behaviour between active tail and not moving. Although M3-Progress seems to be a working solution, I wonder if a simpler model would also achieve that (like oscillating between maximizing action and fatigue, without modeling curiosity or proprioception), and I also wonder if unintended details of M3-P are responsible for the results. For instance, could the oscillating behaviour just be an effect of the lag between the new and old parameters?\n\nA possible simple way to address my high-level concern is to make a clear list of technical contributions and provide more details to make them individually usable or reproducible. I believe that great individual technical contributions are shadowed by the grand claims, which are less solid. I am thinking of technical contributions that have a higher chance to be useful to other researchers, like the embodied zebra fish dm-lab simulation (if this is a contribution of this work), the novelties of the intrinsic RL algorithm, or for setting a benchmark in a clear quantitative metric (e.g., correlation of neural activity from the Zebra fish dataset).\n\nSuggestions to improve the writing, which is currently sub-optimal. Addressing some of them is very likely to increase my grade.\n\nA) Clarifying the contributions: make clearer what the list of technical contributions is in the intro.\n\nB) Unclear technical description of the model:\n\nB1) When reading paragraph 139 about the PPO algorithm, more basic things like the action space were not well defined\n\nB2) The notation Phi is not clear; it is used with small variations in Figure 2a, l. 166, 169, 174 are sometimes Phi(S_t, sometimes Phi^I, sometimes Phi_t) but it never defined rigorously. I feel that it broadly means “some network model” which is not a good notation. There should be clear notation to dissociate the vision encoder, proprioception encoder, encoder used in other algorithms, etc…\n\nB3) Paragraph line 167: It is not clear if this section is a summary of related work, if this is a contribution from the paper. I then understood that these are the baseline models, but it is quite obscure when reading the paper linearly in logical order.\n\nB4) The notations: Var( {omega} ) is not very clear. omega is a Gaussian density, that’s okay, but what is the Var( {omega} )\n\nB5) The whole paragraph between line 210 and 211 is quite vague. It is written as if that were proof. But I believe it’s also part of the definition of the 3M progress algorithm? What is L_theta_old after in line 210.\n\nB6) Definition of intrinsic reward has no equation number. Why not simply define r = abs(epsilon_hat - epsilon)? The usage of a general f is not helping, but is adding to the general confusion.\n\nC) More details for the quantitative results, and More info is needed to reproduce the results or compare results on the same dataset.\n\nC1) It would be useful to define mathematically the way the Pearson correlation is computed. Is this trial averaged in some way? Is that the definition of correlation, smoothing time in some way? Currently, all the “model-brain alignment” metrics are very obscure because high single-trial correlation over time for two units would be very strange because the behaviour itself is mostly different in each trial. So each unit is most likely not correlated to itself in different behavior histories. Ideally, one would need more than one metric, see for instance [3]. I also find it unsatisfactory that the chosen metric is saturated for most models (see Supplementary Figure 6 and 7)\n\nC2) Also, a mathematical definition of the “one-to-one” alignment would be needed, as it is needed to compute the correlation metrics and reproduce the quantitative results. Are you using some optimal bijective mapping between two networks (as in [3])? Is it first assumed that the number of neurons in the model and the brain are identical via random sampling before applying the optimal assignment? These kind of choices are changing a lot the quantitative results, so they should be explained clearly.\n\nD) I wish the results would be analyzed more deeply. The paper tends to claim grand achievements (“the model passes the neuro ai turing test”). I would not recommend that, but why not? Yet, it would be at least necessary to give all quantitative and technical details to reproduce and justify this wording. And scratch below the surface to illustrate the stated results. Here are some suggestions: D1) Grand claims like “the model passes the neuro ai turing test”, need to be clearly linked to a set of behaviour and neural activity measurements which are clearly defined and detailed.\n\nD2) I wish an ablation study had been run on the M3-Progress to better understand what element of the model is responsible for the quantitative reports.\n\nE) Missing reference to previous work: In addition to the fly model from Janelia, there have been other attempts to model the organism and its environment before the “embodied neuro ai” terminology, like the “swimming neuron” [1] or the open worm project [2]. Reference [3] also modeled neural activity and behavior at large in mice.\n\n[1] https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1010899\n[2] https://openworm.org/\n[3] https://proceedings.neurips.cc/paper_files/paper/2023/hash/ec702dd6e83b2113a43614685a7e2ac6-Abstract-Conference.html\n\nQUESTIONS:\nQuestions are part of my main feedback above; the main ones are:\n\n- I wonder if a simpler model would also achieve that (like oscillating between maximizing action and fatigue, without modeling curiosity or proprioception)\n\n- How is the model-brain alignment correlation metric defined mathematically?\n\nLIMITATIONS:\nI see no potential negative impact on society.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-24"}

comments: SUMMARY:
This paper introduces a novel intrinsic reward based on modeling the divergence between an ethological prior and a current world model. This intrinsic reward model is applied to studying larval zebrafish, specifically their different behavioral patterns and their whole-brain neural-glial dynamics. Using this model, the paper demonstrates far greater alignment with zebrafish behavioral data compared to other intrinsic reward models, as well as outperforming them in predicting neural imaging data.

STRENGTHS AND WEAKNESSES:
Strengths:
- Overall, this paper does an excellent job in clearly establishing the value of the 3M-Progress method, implementing a suite of baselines for each experiment and reporting error bars showing a clear improvement in comparison to those baselines. There is little to fault with regards to the quality, significance, and originality of this paper.

Weaknesses:
- The placement of baselines in L167-182 is confusing, as I initially had the impression that these were a component of the actual model being introduced. This content could all be moved to the supplementary material.
- More time could be spent elaborating on the 3M-Progress reward model, specifically regarding the intuition behind the scenarios in which the reward might be especially high or low.
- A couple of minor typos: “sizing relaitve” in L134, “ecnoded” in the Fig. 2 caption.

QUESTIONS:
- What kinds of behavior does this reward elicit in other environments of the DeepMind Control Suite?
- What is the behavior of the zebrafish agent in moments of high/low reward?
- Are there differing computational constraints for the different intrinsic reward models? If so, what implications does that have for the biological plausibility of the model?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes 3M-Progress, a model-based intrinsic motivation reward that allows a virual zebrafish to exhibit autonomous behaviors matching behavioral and neural-glial dynamics recorded in autonomously behaving larval zebrafish. A 6-link virual zebrafish was embedded in customized fluid-dynamics MuJoCo environment was used in the experiments. Agents trained with 3M-Progress are compared to baselines implementing other intrinsic reward signals: ICM, RND, Disagreement $\\gamma$-progress as well as model-free entropy-based exploration term and random policy. The results showed 3M-Progress was significantly better at matching behavioral and neural-glial dynamics from real zebrafish recordings in both active and passive transitions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This work is a timely cross-disciplinary contribution, as it is one of the first work to bring goal-driven embodied agent settings to model neuron–astrocyte interaction that is quantitatively validated against whole-brain recordings, which significantly improve ethological validity over prior work.\n- 3M is a clever and novel formulation of intrinsic drive in animals, as the memory-based divergence elegantly avoids the “white-noise” issues of prediction-error signals and yields stable, repeatable behavioral cycles. \n- The empirical evaluation is very solid with multi-level metrics, and also the results are statistical significant with non-trivial gains over other methods. \n\nWeaknesses:\n- The current environments are constrained to only allow passive transitions and fluid movement to simulate active transition observations. This limits the ecological realism, but I do understand the effort of creating such environment and agent is non-trivial.\n\nQUESTIONS:\nIs there a reason $\\theta_{old}$ is only trained in a separate phase from $\\theta_{new}$? What would the effect of using moving average be here?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}