Of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
Authors: Shuo Han
German Espinosa
Junda Huang
Daniel Dombeck
Malcolm A. MacIver
Bradly C. Stadie
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement learning
Biological agents
Comparative study
Variance-penalized temporal difference learning
Experience replay
Behavioral alignment
Primary area: reinforcement_learning->everything_else
TL;DR: We identify key differences between mice and RL agents in predator-avoidance tasks and develop new methods to make artificial agents behave more cautiously like their biological counterparts.

Abstract: Recent advances in reinforcement learning (RL) have demonstrated impressive capabilities in complex decision-making tasks. This progress raises a natural question: how do these artificial systems compare to biological agents, which have been shaped by millions of years of evolution? To help answer this question, we undertake a comparative study of biological mice and RL agents in a predator-avoidance maze environment. Through this analysis, we identify a striking disparity: RL agents consistently demonstrate a lack of self-preservation instinct, readily risking ``death'' for marginal efficiency gains. These risk-taking strategies are in contrast to biological agents, which exhibit sophisticated risk-assessment and avoidance behaviors. Towards bridging this gap between the biological and artificial, we propose two novel mechanisms that encourage more naturalistic risk-avoidance behaviors in RL agents. Our approach leads to the emergence of naturalistic behaviors, including strategic environment assessment, cautious path planning, and predator avoidance patterns that closely mirror those observed in biological systems.

Of Mice and Machines:
A Comparison of Learning Between Real World Mice and RL Agents
Shuo Han German Espinosa? 2
Junda Huang
Daniel A. Dombeck 3 Malcolm A. MacIver? Bradly C. Stadie
Abstract
decision-making agents, they arose primarily from compu- tational principles' rather than biological experimentation, Although subsequent research has attempted to draw paral- Icls between RL algorithms and biological learning mech- anisms (Pozzi et al., 2018; Neftci & Averbeck, 2019; Tan et al., 2023), suggesting the biological plausibility of meth- ods like Q-learning, fundamental question remains: To what extent do RL agents truly capture the decision-making characteristics of biological organisms?
Recent advances in reinforcement learning (RL.) have demonstrated impressive capabilities in com- plex meeiiion-mmkkig tasks. This progress raises natural question: how do these artificial systems compare to biological agents, which have been shaped by millions of years of evolution? To help answer this question, we undertake a compara- tive study of biological mice and RL agents in a predator- avvodance maze environment. Through this analysis, we identify a striking disparity: RL agents consistently demonstrate lack of self- seff-preservation instinct, readily risking "death" for marginal efficiency gains. These risk-taking strategies are in contrast to biological agents, which exhibit sophisticated risk-assessment and avoidance behaviors. Towards bridging this gap between the biological and artificial, we propose novel mechanisms that encourage more nat- uralistic risk-avoidance behaviors in RL agents. Our approach leads to the emergence of natural- istic behaviors, including strategic environment assessment, cautious path planning, and preda- tor avoidance patterns that closely mirror those observed in biological systems.
To investigate this question, we designed a controlled exper- imental environment where mice navigate a complex space while avoiding a robotic predator-like autonomous agent (hereafter referred to as "predator" for brevity). A simu- lation environment replicates this with the prey controlled by an RL algorithm. The predator is reactive agent in both the experiment and simulation. Through careful behav- ioral analysis, offline RL modeling (Levine et al., 2020) of mouse behavior, and Exploratory Data Analysis over visita- tion density graphs, we identified several striking disparities between biological and artificial agents. Most notably, RL agents demonstrate remarkable lack of self-prrservation instinct, often choosing marginally more efficient paths that bring them dangerously close to the predator. In contrast, biological agents exhibit sophisticated risk-assessment be- haviors, with mice spending a significant amount of time gathering environmental information and evaluating preda- tor positions before movement.
1. Introduction
Inspired by these observations, we developed two novel mechanisms to bridge the behavioral gap. First, we intro- duce modified replay buffer mechanism that amplifies and frequently resamples near-death experiences during train- ing, mimicking post-traumatic stress responses observed in biological systems. This enhancement enables agents to develop more appropriate risk-avoidance behaviors. Second, propose a fundamental modification to the TD-learning framework incorporating action uncertainty through Q-value variance estimation. These additions create naturally risk- averse agents that better reflect the cautious decision- Docision-making
Mathematical foundations for reinforcement learning (RL) emerged through several landmark contributions: Samuel's early work on learning systems, Bellman's 's dynamic pro- gramming principles, and later Sutton's temporal differ- ence learning (Samuel, 1959; Bellman, 1966; Sutton, 1988). While these seminal advances have enabled powerful
'Department of Statistics and Data Science, Northwestern Uni- versity, Evanston, USA 'Department of Mechanical Engineering, Northwestern University, Evanston, USA. 'Department of Neurol- ogy, Northwestem University, Evanston, USA. Correspondence to: Shuo Han <TTansiHHane @ TTTansioHanu.nnottvweee uun.....estem.......
'hhile Sutton et al. (1998) has some biological inspiration, there is clear distinction to be drawn between biologically in- spired and derived from measured biological data. We investigate the latter viewpoint in this volume.
Proceedings of the nnnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
patterns observed in biological systems. Empirically, these mechanisms enhance the behavioral similarity between arti- ficial and biological agents, increasing the visitation pattern overlap with mice from 20.9% to 86.1%. The enhanced agents exhibit conservative movement patterns with approx- imately 45% less distance traveled during initial environ- ment entry, closely matching the cautious behavior of real mice.
Stat
Soar
Figure I. Real Mouse Setting
Given the fundamental behavioral differences we observed between RL and biological agents, we were left wonder- ing whether more advanced AI systems might naturally exhibit more biologically-aligned behavior. To investigate this, we introduced a third agent derived from a large lan- guage model (II.M)a a system trained on vast amounts of human-written text and capable of sophisticated world modeling. (Devlin, 2018; 8; Brown et al., 2020; Hurst et al., 2024) Surprisingly, when controlling a simulated version of the mouse, the LLM exhibited risk-taking characteristics much more aligned with traditional RL agents than biologi- cal ones. This finding suggests that biological alignment in decision-making behaviors may be a broader challenge in AI systems, independent of their training approach or world knowledge.
Figure 2. RL Setting: Mouse's view (left). predator's view (right).
facilitating real-time adjustments for the robot's trajectory through combination of A" pathfinding and PID control. The robot initiated each episode from starting position outside the mouse's field of view and actively navigated the arena by algorithm 1. The experimental arena featured both rewards and threats: water rewards were available at specifie locations (Figure 1, doors), while mice had to simultancously avoid being caught by the pursuing robot. The mice were rater-restricted, which motivated them to explore the arena despite the predator's presence. The ex- periment involved eight lab mice (4 male and 4 female), each performing multiple 30-minute sessions daily. This data collection procedure follows previously established methodologies (T.i et al., 2024).
Our key contributions can be stated as follows:
We perform systematic analysis of behavioral dif- ferences between biological and artificial agents in high-risk seuudo-rredtoorprey environment.
We derive novel mechanisms for incorporating biologi- cally inspired risk assessment and experience process- ing into RL objectives.
Results show that incorporating these mechanisms lead to trajectories that are closer to biological behavior. This is true even when compared to other surprise- avoiding objectives from the literature such as SMiRL. (Berseth et al., 2019)
Algorithm 1 Autonomous robot predator behavior
1: while experiment is running do Find spawn cell
3: Move robot to S
4: while episode is running do
2. Experiment Setup
5: if mouse is visible then
This study assessed strategic evasion behaviors in mice through task approximating predator-prey interactions
6: Move robot to last seen mouse cell
7: else if mouse is not visible then
within hexagonal arena, namely "cellworld" (Lai et al., 2024). The physical setup included custom robotic predator-like threat that pursued the mice, simulating preda- tory threat through aversive air blasts, while the mice at- tempted to reach designated reward area without being "captured"" (within 27.5 em or 0.1 units of the robot). The arena was mapped with hexagonal grid of 331 magnetic cells, each 11 cm apart, where obstacles were strategically placed to encourage adaptive evasion wrratgies(MMugn & MacIver, 2020).
Find cells not visible to robot Randomly select a non-visible cell N Move robot to N 11: end if
10:
12: end while
13: end while
For direct comparison with the reinforcement learning (RL.) model, we created a simulated environment, "cellworld gymnasium", building upon the Gymnasium API (Towers et al., 2024) and Pygame, which closely mirrors the physical experiment setup. This simulated environment replicates the hexagonal grid layout of the real arena and incorporates
To record precise positioning and behavior, a high-speed tracking system monitored both the robot and mice at 90 Hz,
2

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
the same reward structure: the virtual "mouse" receives reward of +1 upon reaching the goal without capture and penalty of if is "puffed" by the simulated predator-like robotic threat. 2 To further ensure fidelity, both the mouse and the predator in the RI. environment move at speeds proportionally scaled to the average speeds observed in the physical trials. Additionally, the RL agent has partial observations, detecting the predator's location only when the predator is within its field of view. All distances within the arena are normalized such that unit corresponds to the diameter of the hexagonal arena.
lis, 1999) algorithm that simultaneously optimizes a policy and value function by maximizing trade-off between expected reward and entropy (a: entropy coefficient), pro- moting exploration (Haarnoja et al., 2018):
(TT (((aa)> [r(s, a) - a log क (a|s)]
3.1.2. MODEL--BASED METHODS
Model-based methods learn a world model P(s''ss, a) to predict environment dynamics (Ha & Schmidhuber, 2018; Okada & Taniguchi, 2022). TDMPC-2 combines tempo- ral difference (TD) learning with model predictive control (MPC) (Morari & Lee, 1999; Nagabandi et al., 2018), using the world model's ss trajectories to jointly optimize policies and value functions (Hansen et al., 2022: 2023). The al- gorithm operates by first using an encoder 2 = h(s) to transform states into latent representations, then employing latent dynamics 2' = d(z, a) to predict the next latent state. It includes a reward predictor r- R(z, a) for estimating immediate rewards and a value predictor q" = Q(=, a) for predicting future rewards in the latent space. A policy prior à = p(2) proposes actions to maximize the predicted value. This hybrid approach improves both sample efficiency and policy performance.
3. Reinforcement Learning (RL)
Reinforcement Learning (RL) provides a robust framework for sequential decision-making (Sutton al., 1998), where an agent learns to maximize cumulative rewards through interactions with an environment. Formally, the RL. problem is defined as a Markov Decision Process (MDP), character- 7). Here, states S represents all possible states, actions A denotes available actions, tran- sition dynamics Pss'|ss, a) defines the transition probability from state s' given action a, r(s, a) specifies the reward for taking action in state 8, and discount factor y E 1) balances future versus immediate rewards.
ized by the tuple (S, P,,,7)
TDMPC-2 optimizes two objectives. The model objective C()) minimizes the discrepancy between predictions and actual values:
The objective of RL is to find an optimal policy x" (als) that maps states to action distributions, maximizing the expected cumulative discounted reward:
[(0) ((
Σ x (11 sg((ss))() t=0
J(x) = E,
Σ (T(
-C((r)))) + CE((i u))
3.1. Online RL: Learning through Interaction
where B denotes the replay buffer and CE is the cross- entropy loss. The policy objective (p()) maximizes returns while maintaining exploration:
Online RL involves the agent iteratively interacting with the environment to gather data and refine its policy. This paradigm further divided into:
H
3. MODEL.FRREE METHODS
(0) = (s.a))a B
Σ
X ((Q(ea)(a)))BBB(((()))) (H(((11=)))
These methods optimize the policy directly without con- structing an explicit model of the environment's dynamics. Notable examples include:
For action selection, TDMPC2 uses MPPI (Williams et al., 2017) to optimize over a horizon H:
nº, o" = arg max EN( H Q(#+H, (a.0)
Deep Q-Network (DQN): It utilizes a neural network to approximate the valion-valle function Q(s,, a). The update rule is based on the Bellman equation (Hester et al., 2018):
H-1
h R(2 an)
Q(s, + r(s,a) + max
max Q(s',a'). a'
4. Comparison Between Mouse Behavior and RL Agent Behavior
Soft Actor-Criti (SAC): An actor-critic (Konda & Tsitsik-
The equal magnitude (+1/-1) for rewards and penalties was chosen to reflect that both water reward and air blasts are relatively mild stimuli.
By directly comparing mouse behavior with RL agents in the "cellworld" setting, we aim to characterize key behavioral
3

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
differences in their threat evasion strategies. Specifically, analyze trajectory patterns quantify wall-following tendencies, examine state density distributions under- stand waiting behaviors near obstacles and starting points and compare action sequences before and after predator detection. Through these analyses, we seek to uncover the fundamental differences between mice and RL agents in risk management during predator-prey interactions.
goal-oriented paths with minimal environmental interaction, with 0% of trajectories hugging walls. The state-visitation density of RL agents is heavily concentrated along these direct routes, showing little of the preemptive caution or environmental awareness displayed by the mice.
We begin by training TDMPC-2 the "cellworld gymna- sium" environment and comparing the behaviors generated to those of the mice. On superficial level, the performances of the mice and TDMPC-2 are similar, with the mice achiev- ing an 86% success rate* across environments and the RL algorithms achieving about 80%. However, when we ex- amine the exact nature of these successes, we see there is dramatic divergence between the two agents.
Figure 5. An example of distinct behaviors: Mouse (left) exhibits hesitation (dense segments near entrance), while RL agent (right) takes direct path. Additional examples are the Appendix.
Vell holwwnn 2004
This behavioral disparity is particularly evident in situations where the predator not visible. As illustrated in Figure 5, mice exhibit distinct waiting behavior, shown by dense trajectory segments near the entrance (left), while the RL agent takes direct path through the arena (right). This waiting behavior is further confirmed by the density plots in Figure 4. RL and mice agents start from the same position, but only the mice show high-density concentrations near the entrance, obstacles, and walls. While mice tend to wait and observe in such scenarios, RL agents maintain their direct approach, suggesting a fundamental difference in risk assessment and strategic planning between biological and artificial agents.
Figure Trajectory plots: RL (left) Mouse (right). Blue indi- cates wall- wall-following trajectories (thigmotaxis), while red indicates non-wall-following trajectories.
We also find that on average RI. agents explore only 21.3% of the available space. This is in contrast mice, which visit 77% of the entire The visitation pattern overlap between RL agents and mice only 20.9%.
0.0
Dielence Change
Dielsnos Crange
-02
0.4
TDMPC-2
-04
IDMPC-2
Mouse
Mouse
Density
Sleps Before Sewing
Alter Firel Seving Predistor
0.6
YPosioo
YPostion
0.5
T0MPC-2
Mouse
0.5
0.4
Step
Slep
Figure 4. Density plot of RL (left) Mouse (right)
Step
0.3
Step
Step 3
Step
As shown in Figure 3, mice trajectories exhibit distinct patterns concentrated along walls (thigmotaxis, commonly observed in many animals and corresponding to measures of anxiety (Mugan & MacIver. 2020) and obstacles, with approximately 70% trajectories following wall-following paths. In contrast, RL agents' trajectories show direct,
(TDMPC-2)
(Mouse)
Figure 6. Top row: Mean changes in goal-agent distance before and after predator detection (left, right), with shaded areas showing standard deviation. Bottom row: spatial distribution of first three steps for each agent, with black stars showing mean positions. (1 unit the diameter of the hexagonal arena)
PProportion of runs in which the agent reached the goal without heing captured.
Figure highlights key behavioral differences between units (10% the diameter the hexagonal arena) of wall.
DDfined as trajectories where over 70% states are within

---

of Mice and Machines; A Comparison of Learning Between Real World Mice and RL Agents
TDMPC-2 and mice. In reduction of distance to goal (top row), TDMPC-2 shows excessive movement both before (-0.2 to -0.4) and after (-0.2 to -0.35) predator detection, while mice maintain conservative patterns (-0.05 to -0.15), reflecting natural cautiousness. Spatially (bottom row), mice trajectories exhibit structured, radial exploration patterns, whereas TDMPC-2's movements appear scattered and dis- organized.
Threat Experience Rebalancing: Inspired by how rodents show heightened sensitivity dangerous situations, investigate whether higher proportion of negative experi- ences is necessary for effective learning. We employ vari- ous percentage settings to ensure that each training batch contains a specified proportion of negative transitions dur- ing sampling. Our experiments (Figure 7) shows that 50% negative experiences in the training batch yields optimal performance. This finding provides valuable insights into the importance of negative experiences during early learning phases, similar to how young animals develop risk assess- ment behaviors.
Another key difference lies in learning efficiency after neg- ative encounters. RL agents typically require multiple air blasts before adapting their behavior to avoid the preda- tor, while mice exhibit immediate and persistent avoidance after a single encounter. Of course, this disparity is natu- ral-mice have millions of years of evolutionary risk avoid- ance encoded in their genetics, while neural networks begin learning from scratch. However, this gap highlights im- portant challenge in artificial intelligence: how to develop RL systems that can more readily incorporate risk-avoidance behaviors without requiring extensive negative experiences.
Threat Experience Amplification: The rewards of nega- tive experiences are amplified by a factor of 200 during sampling', reflecting the heightened emotional intensity of trauma in biological systems, which leads to stronger behav- ioral changes.
5. Changing RL Behaviors to Match Biology
Densty
5.1. Trapre-Inspired Safety Memory in RL Agents
Can we teach RL agents to value their own 'lives' more and have a greater fear of the predator?
TOMPC-2 Ampificttion
Ampilication
A common approach is Prioritized Experience Replay (PER) (Schaul et al., 2015). Our experiments demonstrate that PER provides some benefits, but does not fully address the challenges in our scenario. While PER accelerates learning, it has minimal impact on the agents' behavioral policies. This limitation motivated the development of novel extensions.
Figure 8. Standard TDMPC-2 with limited exploration patterns; Right: With Threat Experience Amplification factor 200, showing increased path diversity and enhanced wall-following behavior.
Our computational experimental results demonstrate that the TISB induces more naturalistic behaviors in RL agents. In our long-term experiments, TISB was implemented primar- ily through the Threat Experience Amplification mechanism. Figure 8 illustrates broader exploration and wall-following patterns similar to those observed in mice. However, the agents still lack the characteristic waiting behaviors seen in mice, as evidenced by the significantly lower density the entrance compared to mice. This observation motivates our next approach.
We introduce the Trauma-Inspiree Safety Buffer (TISB), which modifies standard experience replay to mimic how biological systems learn from trauma (Al Abed et al., 2020). In nature, rodents overweight negative experiences during risk assessment (LeDoux, 2013). Our approach implements this through two mechanisms:
100
5.2. Learning to Wait Through Variance-Penalized Temporal Difference (TD) Learning
Episode Reward
Our analysis of mouse behavior reveals that pausing before potentially surprising situations is key survival strategy. Mice tend to wait and gather information when they cannot see predators or when entering new areas. This observa- tion suggests that effective survival behaviors might emerge
-50
-100
Seacies Hanary Ao Solecten Micnen 20%
-150
1000
2000
3000
4000
5000
'We conducted experiments with different amplification factors: smaller values showed limited behavioral changes, while factors above 200 slowed down training without providing substantial benefits.
Training Steps
Figure 7. Experiments over 5000 steps show that 50% negative experiences yicld optimal performance (5 random sceds).
5

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
from an agent's desire encode uncertainty in decision- making circuits (Rushworth & Behrens, 2008) and minimize surprise in its environment. One established framework for achieving such behavior is Surprise Minimization in Rein- forcement Learning (SMiRL) (Berseth et al., 2019), which explicitly encourages agents to seek predictable states.
our partially observable environment, where states naturally fall into distinct risk categories (e.g., safe dangerous), this uniform penalization becomes problematic. By treating all transitions equally, SMiRL may inadvertently discourage beneficial state transitions, such as moving from danger- ous to safe states, and interfere with TISB's path-finding capabilities.
To embed SMiRL in TDMPC-2, we assign the intrinsic reward
To address these issues, we propose a more selective ex- tension approach based on our observation that Q-value variance effectively signals states requiring caution. Our em- pirical analysis shows that Q-value variance across actions spikes sharply when the predator appears or approaches, nat- urally indicating states where cautious behavior is crucial. This insight leads us to develop a variance- penaaceeee TD target that selectively adjusts penalties based on the agent's uncertainty in different states. We formalize this approach as:
(5)) - log s<t)),
which penalizes unlikely states under the agent's s learned history. Estimating the full conditional density p(st s<t) is prohibitively costly and numerically brittle in our high- dimensional, rapidly changing environment, so we approx- imate the negative fog-likelihood with the mean-squared prediction error in the model's latent space. Under the com- mon fixedvvariace Gaussian assumption, the MSE differs from the negative log-likelihood by only an additive con- stant (Girin et al., 2019), preserving the intuition that large prediction errors signal rare or "surprising" states. Because the latent representation learned by TDMPC-2 is especially sensitive to abrupt events-sude such as a predator suddenly entering view-the MSE provides a responsive and compu- tationally lightweight proxy for surprise, yielding a stable and efficient realization of the SMiRL objective.
TD barget rx (1 a a
Here, is a coefficient that controls the strength of the vari- ance penalty. During training. we compute this variance by uniformly sampling grid of actions from the action space, calculating Q-values for each action, and then computing the variance of these Q-values. This process results in a penalty term that increases in states where different actions lead to highly variable outcomes.
20
Our method diverges from SMiRL in two critical ways: in- stead of modeling state transition probabilities, we directly measure uncertainty through Q-value variance across ac- tions, aligning penalties with the agent's decision-makinn makiin confidence. Rather than statically penalizing all state transi- tions via rewards, we dynamically adjust penalties within the TD target computation, enabling risk-averse behavior to emerge contextually, particularly in high-uneertaing: states such as predator proximity.
TDMPC-2
RMIRL without Tiss
SMIRE with TISB
Figure 9. Density plots: TDMPC-2 (left), SMiRL without TISB (middle), and SMiRL with TISB (right). SMiRL may negate TISB's effect, indicated by only minor changes in the main density regions.
Unfortunately, incorporating SMiRL into TDMPC-2 did not have the desired effect of making RL agents more conser- vative and cautious. As shown in Figure 9, while SMiRL shows some increased density in the central area compared to the baseline TDMPC-2, suggesting slightly more diverse path choices, the core trajectory pattern remains largely un- changed, as evidenced by the similar high-density regions (bright pink areas) across all conditions. Most importantly, SMiRL failed to demonstrate the crucial waiting behavior at the entrance, which was our primary goal. Furthermore, when combined with TISB, SMiRL appears to counteract TISB's find-finiing benefits that were previously observed in TDMPC-2, resulting in only minor changes to the density distribution compared to the more substantial strategy shifts we were aiming for.
In summary, both approaches can be viewed as implement- ing different forms of uncertainty minimization:
SMiRL: max E
Σ
rt (rr- a1 log s<(i)
Our approach: 17
max E
Σ
Σ
rl (re as
Consequently, our approach encourages the agent to avoid states with high decision uncertainty, effectively balancing exploration and risk aversion. Intuitively, it incentivizes be- haviors such as cautious planning and avoiding unnecessary risks.
We attribute these limitations to SMiRL's fundamental mech- anism of penalizing surprise across all state transitions. In
6

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
0.0
0.8
Original TDMPC2
0,0
VP-TDMPC2 (Ours)
Mouse Data
-0.1
Datance Change
Dietance Change
Distancc from Start
0.8
-02
-0.3
0.4
+0.3
-04
0.2
Original TDMPC2
Original TDMPC2
-0.5
(Ours)
VP-TDMPC2 (Ours)
Mouse Data
Mouse Data
0.0
2 Sleps After First Seeing Predator
0
2
5
6
Sleps Before First Seeing Predator
Step Number (Inilial Steps)
Step
Step 2
0.7
Step 2
Step
Step
0.8
0.7
0.5
Positon
0.5
0.5
0.4
0.3
0.3
0.6
Position (Original TDMPC2)
Position (VP-TDMPC2 (Ours))
(Mouse Dala)
Figure 10. Top row: Mean changes in goal-agent distance hefore and after predator detection (left, middle), and initial movement distances (right). Shaded areas show standard deviation. Bottom row: Spatial distribution of first three steps for each agent, with black stars indicating mean positions. The black stars show that mouse data and VP-TDMPC2 maintain more consistent, cautious initial positions, while baseline TDMPC2's s mean positions spread out more rapidly, indicating less conservative carly behavior. (1 unit = the diameter the hexagonal arena)
Figure 11 compares the three agents under investigation. Numerical calculations of the visitation pattern overlap show that the TDMPC-2 agent has 20.9% overlap with mice, while VP-TDMPC-2 achieves 86.1% This metric reflects similarity in exploratory coverage (whether both agents have been to the same states), rather than a one-to-one match in how frequently or what manner they visit those states.
in real mice. For closer examination of the differences between mice behavior and RL, we analyzed the initial part of the trajectory from the start gate.
Our analysis (Figure 10) reveals behavioral parallels with mice exploration patterns. Before encountering the preda- tor, VP- VP-TDMPC-2 demonstrates natural hesitation patterns, with distance changes closely matching mice behavior (- 0.1 to -0.2), while the baseline TDMPC-2 shows excessive movement (-0.2 to -0.5). The spatial distribution plots (bot- tom row) further illustrate that our method captures key characteristics of mice behavior: initial hesitation near en- try points some thigmotactic movement along boundaries, and structured radial exploration patterns.
T
VP-TD-MPC2(Dur)
To further quantify these behaviors, we conduct additional analyses focusing two metries: waiting behavior' and episode length. Across 1,000 trajectories, standard TD- MPC showed 0% incidence of waiting, while real mice
Figure 11. Final states visitation pattern comparison of original TDMPC-2 (left), to our VP-IDMPC2 (middle: trained with TISB and Pariance-Penalized TD learning) and Mouse (right).
'Trajectory comparisons (Figure 15) in the appendix provide additional evidence of behavioral similarity between VP- TDMPC-2 and real mice.
Notably, VP-TDMPC-2 exhibits geometric patterns and bal- anced coverage resembling mice's systematic exploration, though it lacks the extreme thigmotaxis (wall-following with minimal distance throughout the entire trial) observed
DDefined movement with distance changes under 0.1 units within the first six steps.

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
exhibited about 27.7% and our VP-TDMPC-2 achieved about 32.4%.
tant questions about the extent and nature of their internal representations. Does the behavior of an L.MM resemble that of mice, or is it closer to that of an RL agent? Such an experiment may provide insights into the world model of LLMs and their biological alignment. In this study, we use ChatGPT-4 to perform the experiments.
Episode length also differed substantially (Figure 13 in Ap- pendix). Standard RL agents completed the task in an av- erage of 7.09 steps, while VP-TDMPC-2 averaged 13.89 steps, closely aligning with the 14.04 14.04-step average observed in mice. Longer episodes may reflect more cautious or deliberate planning under risk.
Parallel to the RL setting, the LLM agent receives state information in two forms: visual representation of the cur- rent environment and a textual description of the task. The LLM agent operates under the same partial observability constraint, where the predator's 's location is only revealed when it falls within the agent's 's field of view. Our LLM agent makes decisions at each timestep, identical to TDMPC2, en- suring a fair comparison. To maintain experimental integrity and assess the LLM's inherent capabilities, the prompts are designed to be neutral without any suggestive cues about optimal strategies.
Together, these quantitative findings suggest that our pro- posed mechanisms enhance the agent's ability to model biologically plausible risk-sensitive behaviors.
5.3. Challenges in Closing the Mice and Machines Gap
As shown in Figure 11, the final converged policies still exhibit qualitative differences. Mice display stronger wall- following behavior, whereas our improved RL agents tend to make greater use of the central regions.
Our hypothesis that this behavioral difference arises from the structure of the RL state space. After thoroughly explor- ing the outer wall during training, the agent no longer needs to maintain constant wall contact for safety. Upon reaching the top area of the arena, it can confidently conclude that the task is solved. Additionally, the original TDMPC2 agent relied almost entirely on the central regions. Although our improved RL agent has learned exhibit some cautious, mouse-like behavior, it has also learned that the central area offers more efficient paths to the goal when safety is ensured. In contrast, mice continue to follow the wall regardless of task mastery, likely due to rodents' innate behavioral priors (Champagne et al., 2010).
Figure 12. LLM agent trajectories (left) and scatter plots (right). Experiments were run with fewer trials due to API cost constraints, but this was sufficient to illustrate the observed pattern. Interestingly, like RL agents, the LLM agent often exhibits reckless behavior (Figure 12). At the beginning of each trial, the agent enters the environment without apparent strategic considerations, despite expressing intentions such as: "Mov- ing cautiously towards the goal while avoiding obstacles and staying alert to the unseen predator's movements."
We observe that performance can be sensitive to the penalty- weight hyperparameter: setting it higher may encour- age overly cautious policies, whereas lower settings can sometimes produce risk-prone behavior. Moreover, our predator-pree prey environment is deliberately highly stochas- tic-even with fixed seeds-to better mimic real predator randomness, which can occasionally lead to less stable train- ing compared to standard RL benchmarks. These limitations highlight the broader challenge of aligning artificial behav- iors with natural ones, both in terms of behavioral fidelity and parameter robustness.
When encountering the predator, the agent does take eva- sive actions, with internal reasoning resembling: "Moving horizontally to avoid obstacles and maintain distance from the predator while progressing toward the goal." " However, the observed behavior significantly differs from that of mice. While the L.M agent demonstrates reactive adjustments to visible threats, its overall approach lacks the nuanced, adaptive strategies often exhibited by biological agents like mice. We believe this suggests the internal world model of LLMs might lack biological alignment along certain axes, particularly regarding anxiety and fear.
6. What about LLM agents?
Given the remarkable capabilities of transformer models in sequence modeling and reasoning (Brown et al., 2020), we are particularly interested in examining how large language model (LLM) agent behaves in controlled envi- ronment. Recent work has shown that LLMs like GPT-4 can exhibit behavior consistent with human-like theory of mind in controlled tasks (Strachan et al., 2024), raising impor-
7. Related Work
Reinforcement learning has emerged as a fundamental framework for understanding decision-making processes in neuroscience and bio-behavioral research (Botvinick et al., 2020; Subramanian et al., 2022; Fan et al., 2023). Re- searchers (Schultz et al., 1997; Niv, 2009) discovered that
8

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
dopamine neuron activity patterns in the midbrain closely mirror the reward prediction error signals central tempo- ral difference learning. Neuroimaging studies (Lockwood & Klein-Flüüge 2021) have revealed neural correlates of Q-values in the ventral striatum and orbitofrontal cortex, establishing these regions as key components in value-based decision-making.
the context of decisions where the outcome is irreversible (succumbing to predator, for example), standard RL sys- tems lack an explicit notion of mortality (Ororbia & Friston, 2024). Without permanent failure states, RL agents tend to over-explore and underweight rare but fatal risks, prior- itizing immediate goals over the survival-oriented caution observed in biological threat responses. Irreversibility in outcomes has motivated using imagination to inform policy learning in planning algorithms (Racanière et al., 2017), and is a possible basis for the evolution of imagination and planning in the brain (MacIver & Finlay, 2022).
The discovery of mirror neurons, which exhibit similar acti- vation patterns during both action execution and observation (Buccino et al., 2004), has provided biological founda- tion for imitation learning, influencing both robotic systems (Zahra et al., 2022) and third-person imitation learning al- gorithms in RL (Stadie et al., 2017). Building on biological inspiration, researchers have also integrated curiosity-driven learning mechanisms (Stadie et al., 2015; Oudeyer & Smith, 2016) into RL algorithms to enhance exploration and leamn ing efficiency. Recent advances in distributional reinforce- ment learning suggest that the brain encodes not merely the mean expected reward, but rather the complete distribution of possible rewards (Muller et al., 2024).
To bring RL agents closer to mice, we introduce two mecha- nistic innovations: the Trauma-Inspired Safety Buffer and variance- penaaneeee TD learning. The TISB's 's dual mecha- nisms provide key insights: Threat Experience Rebalanc- ing reveals the optimal proportion of negative experiences during early learning, while Threat Experience Amplifica- tion enhances wall-following behaviors similar to mice's defensive strategies. Meanwhile, the variance penalty objec- tive induces information gathering behaviors reminiscent of biological risk assessment. These mechanisms not only make agent behaviors more mice-like but also advance the interpretability of RL systems by grounding their decision- making processes in biologically plausible behaviors.
information-gatheringg
Comparative studies have increasingly focused on aligning biological and artificial behavior. For instance, Vaxenburg et al. (2024) and Singh et al. (2023) analyze navigation tasks across species and models. Our use of model- RL draws on findings by Daw et al. (2011), who demonstrated that animals employ internal models in decision-making with striatal signals resembling TD prediction errors. Fur- thermore, Mattar& & Daw (2018) showed that prioritized memory access can explain biological planning. which di- rectly motivates our TISB framework. In parallel, Blanchard et al. (2011) identified neural mechanisms for risk assess- ment in rodents, aligning with our variance- penalized ob- jective. Together, these studies suggest that while standard RL falls short of replicating mouse-like behavior, biologi- cally grounded modifications offer a promising path toward closing this gap.
We do admit that our work indeed focuses on a spe- cific predator-prey domain, and the proposed mechanisms were intentionally designed within this context. However, predator-pre prey dynamies represent fundamental survival in- teractions observed across both natural and artificial sys- tems, as highlighted by Marrow et al. (1996) and Tsutsui et al. (2024). These studies emphasize such scenarios as key paradigms for investigating adaptive behavior. Importantly, we view this work as an exploratory step toward aligning artificial agent behavior with biologically inspired patterns, rather than as a definitive benchmark for performance or generality.
variance-penalized
Several promising directions emerge from our find- ings. While our methods improved wall-following behav- ior, certain patterns-iike extreme wall-kissing trajecto- ries-remain challenging to replicate. The dynamic predator presence complicated our attempts with intrinsic rewards, suggesting the need for novel architectures to capture these biological strategies. An observed "baiting" behavior is of special
8. Discussion
In this paper, we present a detailed parallel comparison be- tween machines and mice in an environment characterized by risks, goals, and obstacles. The parallel comparison al- lows us to explore behavioral differences in depth: mice show distinct path-planning strategies, often sacrificing path efficiency for safety, while RL agents gradually learn to follow distance-optimal routes. Most importantly, we ob- serve waiting and peeking behaviors in mice at entrances, behind obstacles, and other locations where the predator is not visible, while regularly trained RL agents never exhibit such behaviors.
mice intentionally attract predator attention before retreating (Lai et al., 2024). While TDMPC- 2 incorporates planning horizons, replicating such deceptive strategies requires deeper integration of opponent model- ing, pointing to opportunities to develop better prediction mechanisms.
These behavioral discrepancies may arise from fundamen- tal architectural gap. Unlike biological agents that evolve in
9

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
Acknowledgments
research. Behavioural brain research, 214(2)333--32,, 2010.
We thank the anonymous reviewers for their helpful feed- back in clarifying our conclusions about generalization, shaping our motivation, guiding our approach quantifying risk awareness, and improving the related work section.
Daw, N. D., Gershman, J. Seymour, B., Dayan, P., and Dolan, R. J. Model-based influences on humans' choices and striatal prediction errors. Neuron, 69(6):1204-122 2011.
Impact Statement
Devlin, J. Bert: Pre-training of deep bidirectional trans-
This paper presents work whose goal is to advance the field of Machine Learning. While there are many potential so- cietal consequences of our work, we want highlight the following: All experiments involving eight laboratory mice (4 male/4 female) employed 30-minute behavioral sessions designed to minimize stress, strictly following NIH humane treatment standards. Experimental procedures were in ac- cordance with NIH guidelines and approved by the corre- sponding institutions' Animal Care and Use Committee.
formers for language understanding. arXiv preprint arXiv: DXXiv:811000805, 2018
Fan, C., Yao, L.,, Zhang, J., Zhen, Z., and Wu, X. Advanced reinforcement learning and its connections with brain neuroscience. Research, 6:0064, 2023.
Girin, I.., Roche, F., Hueber, T., and Leglaive, Notes on the use of variational autoencoders for speech and audio spectrogram modeling. In DAFx 2019- 22nd International Conference on Digital Audio Effects, pp. 1-8, 2019.
References
Ha, D. and Schmidhuber, World models. arXiv preprint arXiv: /8K3: 1803. /0122, 2018
Al Abed, A. S., Ducourneau, E.-G., Bouarab, C., Scllami, A., Marighetto, A., and Desmedt, A. Preventing and treating ptsd-like memory by trauma contextualization. Nature Communications, 11(1):4220, 2020.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Bellman, R. Dynamic programming. science, 153(3731): 34-37, 1966.
Berseth, G., Geng, D., Devin, C., Finn, C., Jayaraman, D., and Levine, S. Smirl: Surprise minimizing rl in dynamic environments. arXiv preprint arXiv:1912.05510, 2019.
Hansen, N., Wang, X., and Su, H. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203.04955. 2022.
Blanchard. D. C.. Griebel, G., Pobbe, R., and Blanchard, J. Risk assessment evolved threat detection and analysis process. Neuroscience & Biobehavioral Reviews, 35(4):991-995 2011.
Hansen, N., Su, H., and Wang. X. Td-mpe2: Scalable, ro- bust world models for continuous control. arXiv preprint 1rXi::2310.66822 2023.
Hershey, R. and Olsen, P. A. Approximating the kull- back leibler divergence between gaussian mixture mod- els. In 2007 IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSPPPP volume 4, pp. IV-317. IEEE, 2007.
Botvinick, M., Wang, X., Dabney, W., Miller, K. J., and Kurth-Nelson, Z. Deep reinforcement learning and its neuroscientific implications. Neuron, 100(4))603-616 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, Piot, B., Horgan, D.,, Quan, J., Sendonaris, A.. Osband, I.. et al. Deep q-learning from demonstrations. In Pro- ceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Buccino, G., Vogt, S., Ritzl, A., Fink, G. R., Zilles, K., Fre- und, H.-J., and Rizzolatti, G. Neural circuits underlying imitation learning of hand actions: an event-related fmri study. Neuron, 42(2):323-333 2004.
Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Champagne, D. L., Hoefnagels, C. C., De Kloet, R. E., and Richardson, M. K. Translating rodent behavioral repertoire to zebrafish (danio rerio): relevance for stress
Konda, V. and Tsitsiklis, J. Actor-critic algorithms. Ad- vances in neural information processing systems, 12, 1999.
10

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
Kostrikov, I., Nair, A., and Levine, S. Offine reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169 2021.
prefrontal cortex. Nature Neuroscience, 27(3):403-408, 2024.
Nagabandi, A., Kahn, G., Fearing, R. S., and Levine, S. Neural network dynamics for model-based deep rein- forcement learning with model-free fine-tuning. In 2018 IEEE international conference on robotics and automa- tion (ICRA), pp. 7559-7566. IEEE, 2018.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33: 1179-1191, 2020.
Lai, A. T., Espinosa, G., Wink, G. E., Angeloni, C. F., Dombeck, D. A., and MacIver, M. A. robot-rodent interaction arena with adjustable spatial complexity for ethologically relevant behavioral studies. Cell reports, 43 (2), 2024.
Neftci, and Averbeck, B. B. Reinforcement learn- ing in artificial and biological systems. Nature Machine Intelligence, 1(3):133-143, 2019.
Niv, Y. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139- 2009.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (000),0000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Okada, M. and Taniguchi, T. Dreamingv2: Reinforcement learning with discrete world models without reconstruc- tion. 2022 IEEE/RSI International Conference on In- telligent Robots and Systems (IROS), pp. 985-991. IEEE, 2022.
LeDoux, J. E. Emotion circuits in the brain. Fear and anxiety, pp. 259-288, 2013.
Ororbia, A. and Friston, K. Mortal computation: A foundation for biomimetic intelligence. arXiv preprint arXiv:2311.09580 2024.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives open problems. arXiv preprint ar arXiv:2005.0164: 2020.
Oudeyer, P.-Y. and Smith, L. B. How evolution may work through curiosity-driven developmental process. Topics Cognitive Science, 8(2):492 502, 2016.
Lockwood, P. L. and Klein-Flügge, M. C. Computational modelling of social cognition and behaviour reinforce- ment learning primer. Social Cognitive and Affective Neuroscience. 16(8):761-77, 2021.
Pozzi, I., Bohté, S., and Roelfsema, P. A biologically plau- sible learning rule for deep learning in the brain. arXiv preprint 1rXiv::8// 1.01768, 2018.
MacIver, M. A. and Finlay, B. L. The neuroecology of the water-to-land transition and the evolution of the vertebrate brain. Philosophical Transactions of the Royal Society B, 377(1844)20200523, 2022.
Racanière, S., Weber, T., Reichert, D., Buesing, L., Guez, A., Jimenez Rezende, D., Puigdomènech Badia, A., Vinyals, agents
Marrow, P., Dieckmann. U., and Law, R. Evolutionary dynamics of predator-prey systems: an ecological per- spective. Journal of mathematical biology, 34:556-578, 1996.
O.,, Heess, N., Li, Y., et al. Imagination-augman for deep reinforcement learning. Advances neural information processing systems, 30, 2017.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines:: Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1-8, 2021. URL http: html.
Mattar, M. G. and Daw, N. D. Prioritized memory access explains planning and hippocampal replay. Nature neuro- science,
2018.
//jmlr org. wttpg//apexs/////////2
Morari, M. and Lee, J. H. Model predictive control: past, present and future. Computers & chemical engineering, 23(4-5):667-682. 1999.
Resulaj, A., Kiani, R., Wolpert, D. M., and Shadlen, M. N. Changes of mind in decision-making, Nature, 461(7261): 263-266, 2009.
Mugan, U. and MacIver, M. A. Spatial planning with long visual range benefits escape from visual predators in com- plex naturalistic environments. Nature communications, 11(1):3057, 2020.
Rushworth, M. F. and Behrens, T. E. Choice, uncertainty and value in prefrontal and cingulate cortex. Nature neuroscience, 11(4):389-397, 2008
Muller, T. H., Butler, J. L., Veselic, S., Miranda, B., Wallis, D., Dayan, P., Behrens, T. E., Kurth-Nelson, Z., and Kennerley, S. W. Distributional reinforcement learning in
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM Journal of Research and Development, 3(3):211-22)) 1959.

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Priori- tized experience replay. arXiv preprint 1rX 2015.
Williams, G., Aldrich, A., and Theodorou, E. A. Model predictive path integral control: From theory parallel computation. Journal of Guidance, Control, and Dynam- ics, 40(2):344-357, 2017.
Schultz, W., Dayan, and Montague, P. R. neural substrate of prediction and reward. Science, 275(5306): 1593-1599, 1997.
Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. Combo: Conservative offline model-based policy optimization. Advances in neural information 2021.
Singh, H., van Breugel, F., Rao, P., and Brunton, B. W. Emergent behaviour and neural dynamics in ar- tificial agents tracking odour plumes. Nature machine intelligence, 5(1):58-70, 2023.
processing systems,
Zahra, O., Tolu, S., Zhou, P., Duan, A., and Navarro- Alarcon, D. A bio-inspired mechanism for learning robot motion from mirrored human demonstrations. Frontiers in neurorobotics, 16:826410, 2022.
Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing ex- ploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.008/ 2015.
Stadie, C., Abbeel, P., and Sutskever, Third-person im- itation learning, arXiv preprint a 770.. 703.01703, 2017.
Strachan, J. W., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., et al. Testing theory of mind in large language models and humans. Nature Human Behaviour, 8(7): 1285-1295, 2024.
Subramanian, A., Chitlangia, S., and Baths, V. Reinforce- ment learning and its connections with neuroscience and psychology. Neural Networks, 14:2271-287 2022.
Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3:9 3:9-44, 1988.
Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Tan, X., Han, L., Gong, H., and Wu, Q. Biologically inspired complete coverage path planning algorithm based on q- learning. Sensors, 231)))4647 2023.
Torabi. F., Warnell, G., and Stone, P. Behavioral cloning from observation. arXiv preprint 1005009805.01954, 2018.
Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 2024.
Tsutsui, K., Tanaka, R., Takeda, K., and Fujii, K. Collabo- rative hunting in artificial agents with deep reinforcement learning. Elife, 13:e85694, 2024.
Vaxenburg, R., Siwanowicz, I., Merel, J., Robie, A. A., Morrow, C., Novati, G., Stefanidi, Z., Both, G.-J., Card, G. M., Reiser, M. B., et al. Whole-body simulation of realistic fruit fly locomotion with deep reinforcement learning. bioRxiv, pp. 2024-22 03, 2024.
12

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
A. Environment Details
In our environment setting, we employ both discrete and continuous action spaces, tailored for position control. The continuous action space is utilized for algorithms such as SAC and TDMPC-2, while the discrete action space is designed for algorithms like DQN or other discrete action-based methods.
In the continuous action space, we have three-dimensional action space. The first two dimensions represent the target coordinates (x, y) within the arena, indicating the position the agent should move to. The third dimension serves as waiting or movement indicator: if this value is larger than 0.5, the agent should not move. The agent learns the value of these actions through interaction with the environment.
In the discrete action space, the environment modeled using hexagonal grid, where the arena a large regular hexagon composed of smaller hexagonal cells. Each cell assigned a unique identifier, effectively discretizing the action space. In this setting, each step corresponds to 0.25 seconds in the real-world mouse experiment.
The state space is represented as ten-dimensional vector, which includes the following information:
The simulated mouse's own position, including its x and y coordinates and the direction it is facing.
The simulated robot's 's position, but only when the robot is visible to the mouse.
The distance to the goal.
A binary indicator of whether the mouse has been "puffed" by the robot.
The reward structure is sparse. All distances within the arena are normalized such that unit corresponds to the diameter of the arena. This normalization ensures consistency and scalability across different experimental setups.
Upon each environment reset, the predator initialized at random location outside the prey's field of view. This initialization remains stochastic even when fixed random seed is used, introducing variability in the predator's starting state across episodes. After initialization, the predator then begins hunting the prey following the procedure outlined in Algorithm 1.
B. Experimental Details
All experiments were conducted with a maximum episode length of 300 steps and a total of 100,000 training steps. We use a timestep of 0.25s. This aligns with findings showing that rodents make approximately 2-5 decisions per second during navigation (Resulaj et al., 2009). Through environment setup, we determined that the 0.25s provides an optimal balance between biology realism and learning performance.
B.1. Model-Free Methods
Experiments on SAC and DQN were implemented based on stable-baseline3 (Raffin et al., 2021). For bascline comparisons, DQN was implemented with epsilon- epsilon-greedy exploration strategy, using learning rate of 1e-4 and batch size of 256. SAC was initialized with 1,000 random seed steps for exploration, trained with a learning rate of 3e-4 and batch size of 256. Both algorithms employed two-layer networks with 256 units per layer and parameters updated every step.
B.2. Main Model-Based Method
For our main methods, TDMPC-2 and VP-TDMPC-2, we implemented both experiments based on the original code (Hansen et al., 2023). Both methods utilized 1,000 random seed steps for initial exploration and were trained with learning rate of le-4 and batch size of 512. After training, we collected expert trajectories from TDMPC-2.
For VP-TDMPC-2, we modified the buffer and TD-target the TDMPC-2 agents their original code. To calculate Q-variance, we first sampled 400 actions uniformly from the action space. We used an ensemble of 5 different Q-networks, each predicting Q-values for these actions. The variance was then computed across all Q-values produced by these networks.
We used the horizon length of 3 is in our experiments. Our task mirrors real mouse behavior: excluding waiting periods, mice need 2-3 seconds to reach the goal. Given our simulation timestep (0.25 seconds per action), successful task completion in RL requires 8-12 steps. The gamma factor is 0.995.
The penalty coefficients in the range of 0.1-0.2 achieved the optimal balance between task completion and cautious behavior. For comparison with mice, we trained two separate agents with coefficients of 0.1 and 0.2, respectively, and collected expert
13

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
data from both. To prevent training instability and variance explosion, we implemented variance value clipping at 1,000 during VP-TDMPC-2's TD target updating.
As we have stated in the limitation section, while VP-TDMPC-2 demonstrated improved behavioral alignment with biological agents, we note that its training stability was limited. The model exhibited high variance, and successful cautious behavior only emerged under specific hyperparameter settings. Although our findings are representative, we emphasize that VP-TDMPC-2 remains a proof-of-concepl method rather than a robust, general general-purpose purpose algorithm.
B.3. Why not SAC and DQN
We experimented with DQN and SAC alongside TDMPC2 but focused on TDMPC2 because: (i) Model-based approaches together with TD learning better align with biological seeision-making (Daw et al., 2011), 1), (ii) DQN and SAC showed less stable convergence and inconsistent trajectories, and TDMPC2 produced more stable behaviors.
Notably, with sufficient training, all mentioned RL methods ultimately exhibited similar "reckless" behaviors (lacking waiting periods and showing minimal exploration), confirming that our observations about behavioral gaps are not specific to one algorithm.
B.4. How we evaluate our agent
For the density plots presented in all sections, select the pest-performing model based on validation metrics to ensure that the comparison highlights the real exploration capabilities of each method. If we averaged multiple runs, one with density mainly at the top and another at the bottom, the result could wrongly suggest strong exploration, even if both actually explored poorly.
To ensure fair comparison, both the baseline and improved models were trained following identical procedures, and the gest-performing checkpoint (as measured by validation performance) was selected for each. The collected datasets are of similar scale to highlight the main patterns.
C. Episode Length Distribution Plot
3900
e 10.4
p
1200
0020
5000
2000
Froquency
Frequency
200
0000
1600
1000
400
500
30
30
40
60
10
20
30
40
60
0
10
20
33
40
60
TD-PPG Eploode Length
Mouse Episode Length
VP-TD-MPC Episoce Longth
Figure 1.3. Comparison of episode length distributions across models and mice (filtered for episodes with length between 5 and 50 to exclude trivial failures and outlier trajectories unlikely to reflect meaningful exploratory behavior). TD-MPC2: Mean = 7.09, SD = 1.77, Range [5, 46]; Mouse: Mean 14.04, SD 10.40, Range = [5, 50]; VP-TD- MPC2: Mean 14.25, SD 6.24, Range = [6, 50]; VP-TD-MPC2 reproduces mouse-like exploratory durations more faithfully than standard TD-MPC2, with longer mean episode length and greater behavioral variability.
14

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
D. More Example trajectories
Figure 14. More mouse trajectories, demonstrating consistent waiting behavior at the entrance and strong thigmotactic patterns follo-following behavior). We added slight noise to highlight the waiting path.
Figure 15. More VP-TDM trajectories, exhibiting similar waiting patterns the starting point and emerging thigmotactic tendencies. closely resembling mice behavior. We added slight noise to highlight the waiting path.
15

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
Figure 16. More TDMPC-2 trajectories, showing direct, goal-oriented paths with predator avoidance occurring only in close proximity the goal, lacking the strategic caution observed in mice. We added slight noise to highlight the waiting path, otherwise it would be visually obscured due to overlapping trajectory lines.
E. Fitting Offline RL On Mouse Data and Motivation for KL Divergence
To further validate the behavioral differences between mice and RL agents, we employ offline reinforcement learning to directly learn from mouse trajectory data. By training offline RL algorithms on mouse behavioral datasets, we can create "mouse-like" policies that mimic biological decision-making patterns.
Comparing these learned policies with standard online RL agents provides additional evidence for the fundamental differences in risk assessment and exploration strategies between biological and artificial agents.
Offline RI. aims to learn optimal policies solely from pre-collected datasets without further environment interactions. This approach is essential in scenarios where live data acquisition is prohibitively expensive or risky. A primary challenge in offline RL is addressing the distributional shift between the data distribution daaa (s, a) and the policy- induue distribution dz (8, a).
E.1. Behavior Cloning (BC)
Behavior Cloning (Torabi et al., 2018) 11 is a supervised learning approach that learns a policy ra]s)) directly from the dataset by mimicking the actions taken by the behavior policy. The objective is to minimize the discrepancy between the learned policy and the behavior policy:
min (s,,)))dan (a, n (s))),
where is a loss function measuring the difference between the actions.
Limitations of BC: While BC can effectively mimic certain aspects of mouse behavior, it struggles with capturing underlying decision-makinn strategies, especially in states that underrepresented or absent the dataset. This limitation results in poor generalization and inability to solve the task.
16

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
E.2. Offiine RL Methods
To overcome the shortcomings of BC, more sophisticated offline RL methods modify standard RL objectives to account for the distributional shift:
1. Conservative Q-Learning (CQL): CQL penalizes overestimation of Q-values for unseen actions (Kumar et al., 2020), ensuring the learned policy remains close to the data distribution. The objective is:
min max ar (EEordan anat(a))) ( a)] E(e,a)<dam (((s,a)]] + LRellman (Q) R(H),
where y is an auxiliary action-distribution and R(4)) its regularizer.
2. Conservative offiin model-based policy optimization (COMBO): COMBO (Yu et al., 2021) extends CQL by incorporating a learned model of the environment. It utilizes model-based planning in conjunction with conservative value estimation to improve policy learning:
min a (Egndarntals) [Q(s, a)] - ((s.a) (Q(s + Lellman (Q),
where d represents the state distribution induced by the learned model.
3. Implicit Q-Learning (IQL): IQL (Kostrikov et al., 2021) focuses on recovering optimal actions implicitly by decoupling the value function from policy improvement. uses expectile regression learn the value function:
(vb = (ed 115 (Qo(s, Vo(s))],
(0 (aa,,,))))) [(00(s,0) - -
where L3 is the asymmetric squared loss with expectile T.
Empirical Evaluation
In our predator-prey task, we evaluate the performance of different offline reinforcement learning methods in reproducing mouse behaviors. While both CQL and COMBO successfully reproduce the basic mouse trajectories, they fail to capture the nuanced waiting behavior observed in real mice. This limitation appears stem from their overemphasis on reward optimization, which leads to overly aggressive strategies that deviate from the natural behavioral patterns.
In contrast, IQL demonstrates superior performance by accurately capturing both the waiting actions and the precise trajectory patterns exhibited by mice. This success highlights IQL's enhanced adaptability to complex behavioral dynamics and its ability to balance between reward optimization and behavioral fidelity.
Despite these promising results, the limited availability of mouse trajectory data in our dataset constrains our ability to learn a fully accurate mouse policy. This data scarcity represents fundamental challenge that must be addressed in future work. We anticipate that developing more sophisticated behavior-cloning techniques will be essential for faithfully reproducing the subtle decision-making patterns observed in real mice.
To provide quantitative assessment of the behavioral differences, we measure the discrepancy between the online RL policy and the mouse-derived policy using Kullback- (KL) divergence (Hershey & Olsen, 2007). For each state in a representative subset of the state space, the KL divergence between the two policies is defined as:
TIQ T (a]s)
K (-I=) (-1ss)))) Σ miQ (a]s) log
TR. (a
17

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
where TIQL (a|s) denotes the probability of selecting action a given under the IQL policy (representing mouse behavior), and (als) represents the corresponding probability under the online RL policy. By averaging the KI. divergence across all sampled states, we obtain overall measure of policy divergence.
32
0.8
0.8
24
RL U.AAcionny
2.0
IOL ALLAdoo
1.8
1.2
0.8
0.4
09.0
0.0
0..0
0.0
0.2
0.4
0.8
0.2
0.4
0.8
0.8
RL Action
[QL Action
Probabiiity Density
Probability Density
3.0
2.5
20
0.5
1.0
1.0
0.0
0.4
Acson
0.8
0.0
0.4
Action
0.g
0A Action
0A
0.5
0.2
Action
0.6
0.2
RL Polay Distribution (3D)
IGL Poticy Distrbution (3D)
KL Divergenoe:
Figure KL Divergence between online RL IQL policies (IQL representing mouse behavior)
Our quantitative analysis reveals a substantial divergence between the online RL policy and the mouse-derived policy, with KL divergence value of 6.4386 (Figure 17). This significant divergence underscores the considerable gap between current RL approaches and natural mouse hehavior, highlighting the complexity of accurately modeling biological decision-making processes. These findings emphasize the need for more sophisticated approaches that can better capture the nuanced behavioral strategies employed by mice predator-prey scenarios.
18

---

of Mice and Machines: A Comparison of Learning Between Real World Mice and RL Agents
F. ChatGPT-40 Experiment
Table LLM Agent Prompt
You are controlling a prey agent in a predator -prey environment.
Your goal is reach the green goal point (1.0, 0.5) while avoiding the predator and obstacles.
Environment Details:
You (red dot) must reach the goal (green dot) while avoiding the predator dot)
Black areas are obstacles walls that you cannot pass through The predator (blue dot) is constantly moving and trying to catch you, and there is
larger blue circle indicating the puffed area around the predator
If you can 't see the predator, it means it's hidden behind obstacles
The environment has a grid to help you locate positions (x and y coordinates from to 1)
Each move must have an L2 norm less than 0.2
Your response must be a JSON object with exactly this format:
"move": [
("x": <float>, "y": <float>)
1.
"thoughts"; "<single line explaining your strategy >"
Rules for moves:
1. Provide exactly I move
2. Each move should be a small step
(L2 norm of distance between your next position and current position 0.2)
Example response:
"move": [
("x": 0.20, "y": 0.45]
1. "thoughts"; "Moving toward the goal while avoiding obstacles and keeping distance from potential predator locations."
19


Reviews:
[{"review_id": "NnP0pH0ujT", "reviewer": "Reviewer_GbX7", "summary": "The authors devise several RL agents for a navigation task that involves reaching a goal while avoiding a predator. The task is closely modeled after an actual biological experiment with real mice. Based on the observation of systematic behavioral differences between biological and artificial agents in this predator-prey environment, they suggest two additions to a hybrid RL algorithm, which combines TD learning and MPC involving a deep encoder-decoder model: 1) a replay buffer mechanism that resamples episodes with close encounters with the predator during training more often and 2) a penalty term in the TD error that penalizes reward with the variance of the Q-values across actions in that state. Empirical results show that these two modifications result in an increased overlap of cell visitations between the simulated agent and mice.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough feedback and would like to address several key points:  \n\n**Q: Why PTSB and VP-TDMPC2**\n\nA:  Our PTSB models rodents' responses to adverse stimuli. Even with mild aversive stimuli (air puffs), mice show pronounced fear responses after few exposures, exhibiting PTSD-like memory amplification. LeDoux (2000) demonstrates how rodents overweight negative experiences during risk assessment, which we implement by selectively amplifying memories of predator encounters.\n\nOur Variance-Penalized approach stems from observing mice frequently waiting at the starting position before making decisions - behavior suggesting internal risk evaluation. Rushworth & Behrens (2008) found mammalian brains specifically encode uncertainty in decision-making circuits, with orbitofrontal cortex activation correlating with outcome uncertainty. Behaviorally, animals consistently avoid high-variance options even with identical expected values. We evaluated multiple uncertainty measures before selecting variance as most biologically plausible and computationally effective.\n\n**REF**\n\n- LeDoux, Joseph E. \"Emotion circuits in the brain.\" Fear and anxiety (2013): 259-288.\n- Rushworth, Matthew FS, and Timothy EJ Behrens. \"Choice, uncertainty and value in prefrontal and cingulate cortex.\" Nature neuroscience 11.4 (2008): 389-397.\n\n**Q: Predator Encounter Density and Behavioral Implications**\n\nA: Predator encounters primarily occur in central maze regions, as our density plots show. This distribution explains why our mechanisms produce mouse-like behaviors: when combined with PTSB (amplifying learning from central encounters) and variance-penalized training, agents develop:\n\n-  increased waiting at the starting position to assess risk\n-  stronger preference for wall-following paths that avoid these high-risk central areas\n\nWe will add density plots to illustrate this relationship between encounter locations and resulting behavioral adaptations.\n\n\n\n**Q: Memory Amplification Implementation and PTSB Generalizability**\n\nA: In standard replay buffers, negative experiences typically constitute only about 10% of collected data. Our approach ensures higher representation (50%) in each training batch. This ratio showed optimal training efficiency in our experiments and the principle generalizes to other algorithms like DQN and SAC, though optimal ratios may vary.\n\n\n**Q: Environmental Design Choices and Partial Observability Effects**\n\nA: Regarding predator visibility and observation design: Mice have approximately 270° natural vision, and our experiments documented frequent head-turning behavior that gives them effective near-360° awareness. Our implementation matches this natural capability rather than imposing artificial constraints.\n\nWe tested various observation spaces, including more limited observations (only prey/predator positions when visible), field-of-view restrictions, and observations without goal information. Importantly, all variations produced similar behavioral patterns—RL agents consistently took direct paths without waiting or wall-following behaviors. This consistency suggests the behavioral differences aren't primarily attributable to partial observability.\n\nOur current observation space specifically parallels actual mouse experimental conditions while maintaining computational tractability. For the robot, the same algorithm was used in both physical trials and simulation, enabling direct comparison between RL agents and mice under identical conditions.\n\n**Q: Wall-Following Behavior Limitations**\n\nA: We acknowledge certain limitations. Extreme wall-following remains challenging even when giving the agent extra rewards for being near the wall, particularly with predator presence. However, our approach achieves significant improvements in risk-aware behavior compared to baselines.\n\n**Q: Impact of Varying Rewards and Risk Quantification**\n\nA: The same question is asked by reviewer yzj9, and we have provided detailed answers addressing both reward variation effects and our risk quantification methodology in that response.\n\n**Q: DQN Classification and PTSB Terminology**\n\nA: We agree that RL terminology can be confusing. DQN is off-policy (learns from experiences generated by different policies) but is considered online in standard RL taxonomy because it continuously collects new data through environment interaction. We agree with the reviewer regarding the terminology confusion and will revise accordingly in the final version.\n\n**Q: ICML relevance**\n\nA: ICML welcomes interdisciplinary research. Biological systems have developed sophisticated risk-assessment mechanisms that could inform machine learning approaches in high-stakes scenarios. This research direction aligns with ICML's mission to advance AI capabilities through cross-disciplinary insights.\n\nIn the end, we again appreciate the reviewer's insights and look forward to further discussions!", "claims_and_evidence": "The claims are that \n1) survival performance,\n2) state visitation distributions,\n3) wall following behavior,\n4) initial waiting\nare closer to mice's behavior in the proposed RL model. 1-2 are evaluated quantitatively, and 3-4 are evaluated qualitatively. However, while 1 does not seem to differ (a statistical test may help here), the used measure for 2 shows an improvement, but the typical wall-following trajectories of mice are still not reproduced.", "methods_and_evaluation_criteria": "The algorithmic choices, such as the replay buffer and altered cost function are ad hoc, and at ICML it would be great to see a more in-depth motivation and analysis of these choices.\n\nIt would be very helpful to find more principled and quantitative measures of similarity between trajectories and between strategies of behavior in the predator-prey environment to better quantify and understand the similarities and differences in behavior between mice and models.", "theoretical_claims": "There are no theoretical claims involving theorems.", "experimental_designs_or_analyses": "The specific behavior of the RL agent contrasts with the behavior of the mice. However, many choices in setting up the RL agent have not been further discussed or investigated. First, the reward for encountering the predator is fixed. Second, the agent does not explore the. maze but encouraging exploration e.g. by adjusting \\beta in the third equation on page 3. Third, the environment is clearly partially observable, which is not explicitly modeled. Fourth, the predator is always detected, no matter in what direction it is relative to the agent, including being in its back, which does not correspond to the mice’s field of view. Fifth, the conclusions are drawn based on predator behavior determined by the parameters of algorithm 1, and more aggressive attack behavior may suffice to obtain more \n\n“Experiments (Figure 7) indicate that 50% (ranging from 0% to 100% in increments of 25%) is the optimal percentage for this buffer.” How generalizable is this result, or how much does this pertain to the specifics of the current setup?\n\nWhile the authors report that “TDMPC-2 agent has a 20.9% overlap with mice, while VP-TDMPC-2 achieves 86.1%”, looking at figure 11 seems to suggest very different behavior and still limited wall-following behavior.", "relation_to_broader_scientific_literature": "There is a wealth of literature on modeling the behavior of animals, particularly mice with RL. The literature on replay buffers since DYNA is extensive, similarly the literature on implementing forms of risk aversion in RL. \n\nI would be very careful with naming an algorithm as the present one, “Post Traumatic Stress Buffer (PTSB),” just out of respect for the medical profession and the patients suffering from such conditions. Maybe there is a way to reformulate this.", "essential_references_not_discussed": "Sorry, not sure what I would pick out here as \"essential\".", "other_strengths_and_weaknesses": "This is an interesting and important direction for neuroscience and ethology, but I think that the paper in its current form is not at the core of the interests of the ICML community but would elicit more excitement at a conference such as From Animals to Animats.", "other_comments_or_suggestions": "Characterizing DQN as online RL is a bit difficult, given that it uses the replay buffer, which makes it more of an offline method.\n\nThe color mapping is wrong in the caption of figure 3.", "questions_for_authors": "How was “Memory Amplification” implemented?\n\nWhere do the encounters with the predator happen? A density plot of the encounters with the predator would help understand the changes in agent behavior given the modified algorithms. \n\nWould different rewards in the problem's setup, especially for being caught by the predator, change behavior? \n\nAre the differences in behavior mostly attributable to not using a formulation of partial observability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqXWEfKZ9Z", "reviewer": "Reviewer_bEMR", "summary": "This paper investigates the behavioral differences between biological mice and reinforcement learning (RL) agents in a predator-avoidance maze environment. The authors find that RL agents lack preservation instincts, often taking risky, efficiency-driven paths without assessing potential threats, in contrast to biological mice. To address this discrepancy, the paper introduces two novel mechanisms designed to encourage biologically inspired risk-avoidance behaviors in RL agents: (i) Post Traumatic Stress Buffer (PTSB), which mimics biological trauma responses, and (ii) Variance-Penalized Temporal Difference (TD) Learning, which integrates action uncertainty by penalizing high Q-value variance and promotes more risk-averse decision-making. Experimental results demonstrate the effectiveness of both proposed mechanisms. Additionally, the authors introduce ChatGPT-4 to explore the behavior of large language model (LLM) agents in a controlled environment. However, experiments reveal that LLM-based agents exhibit risk-taking tendencies similar to those of RL models, suggesting that achieving biological alignment remains a broader challenge in AI.\n\n## update after rebuttal\nI keep my initial rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful feedback. Regarding the two main concerns raised:\n\n**Q: On generalization across environments: The experiments are conducted in Cellworld Gymnasium, raising questions about whether the proposed mechanisms would generalize to other, more diverse environments or tasks. Could the authors provide more analysis or change the environment to demonstrate the generalization of the mechanisms?**\n\nA: We acknowledge that generalization is an important consideration. Our research primarily aims to understand fundamental behavioral differences between biological and artificial agents in predator-avoidance scenarios, rather than demonstrating broad environmental generalization. We focused on a controlled environment that precisely mirrors our biological experiments, allowing for direct behavioral comparisons.\n\nWhile broader generalization testing is valuable for future work, our current focused comparison provides crucial insights into the computational principles underlying biological risk assessment. These principles (memory amplification of negative experiences and uncertainty aversion) are likely to transfer to other risk-assessment scenarios, though the specific behavioral manifestations may vary with environment. Future work will examine how these mechanisms generalize across different environmental structures and predator behaviors.\n\nWe truly appreciate your recognition of our systematic quantitative evaluation approach and the biological motivation behind our proposed mechanisms. Our future work will indeed explore applications in more complex environments, particularly in AI safety domains and autonomous navigation systems where risk assessment is critical. \n\n\n**Q: Regarding the ChatGPT-4o analysis: This paper utilizes ChatGPT-4o to conduct the experiments. However, the analysis appears limited, partly due to the reduced number of trial runs, which may lead to unreliable conclusions.**\n\nA: While we conducted approximately 100 trajectories (limited by computational constraints of running GPT-4o), our experimental design ensures comprehensive coverage through systematic variation of predator spawn locations across all relevant scenarios. The remarkable consistency of ChatGPT-4o's behavior across these diverse configurations provides strong evidence for our conclusions. Our analysis reveals that regardless of predator positioning, ChatGPT-4o consistently adopts strategies similar to baseline TDMPC2, reinforcing our findings about the fundamental behavioral gap between artificial and biological agents. We will include additional visualizations showing predator spawn distributions to further support this conclusion.\n\n\n\nWe look forward to further discussions with you!", "claims_and_evidence": "Good.", "methods_and_evaluation_criteria": "Good.", "theoretical_claims": "None.", "experimental_designs_or_analyses": "Good.", "relation_to_broader_scientific_literature": "None.", "essential_references_not_discussed": "none.", "other_strengths_and_weaknesses": "Strengths: \n\n1. The paper presents a systematic and quantitative analysis of the behaviors of mice compared to reinforcement learning (RL) agents in a predator-prey maze, revealing significant differences in risk assessment and exploratory strategies.\n\n2. The writing of this paper is good. The proposed PTSB and Variance-Penalized TD learning designs are well-motivated. \n\n3. The proposed novel methods for altering RL behaviors to align with biological processes are effective and have potential application value in life science research and other related fields.\n\nWeaknesses：\n\n1. The experiments are conducted in Cellworld Gymnasium, raising questions about whether the proposed mechanisms would generalize to other, more diverse environments or tasks. Could the authors provide more analysis or change the environment to demonstrate the generalization of the mechanisms?\n\n2. This paper utilizes ChatGPT-4 to conduct the experiments. However, the analysis appears limited, partly due to the reduced number of trial runs, which may lead to unreliable conclusions.", "other_comments_or_suggestions": "No.", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NIV6kgbFQ9", "reviewer": "Reviewer_TtQ7", "summary": "This paper compares the learned behaviors of biological mice and simulated RL agents in a predator-avoidance maze environment. The authors highlight the disparities in behavior and propose two mechanisms to bridge the gap between a TD-MPC2-trained simulated agent and biological mice. Finally, they compare the performance of an LLM-agent and show its similarity to an RL agent.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's thorough and valuable feedback. Below we address each point systematically:\n\n**Q: Regarding algorithm selection and comparison**\n\nA: We experimented with DQN and SAC alongside TD-MPC2 but focused on TD-MPC2 because: (1) Model-based approaches together with TD learning better align with biological decision-making (Daw et al., 2011), (2) DQN and SAC showed less stable convergence and inconsistent trajectories, and TD-MPC2 produced more stable behaviors. \n\nNotably, with sufficient training, all RL methods ultimately exhibited similar \"reckless\" behaviors (lacking waiting periods and showing minimal exploration), confirming that our observations about behavioral gaps are not specific to one algorithm. \n\n\n**Q: Concerning Prioritized Experience Replay (PER)**\n\nA: While our approach shares conceptual similarities with PER, our experiments showed standard prioritized replay failed to change the RL behavior. The key distinction is our specific amplification of negative experiences to model trauma-like responses, simulating high-risk encounters in a biologically-inspired manner, rather than prioritizing based solely on TD error magnitude.\n\n\n**Q: As for risk and uncertainty**\n\n\nA: We observe that while initial training phases do show high variance across the environment (reflecting exploration uncertainty), this pattern quickly evolves to specifically highlight areas with risk (predator encounters). After sufficient exploration, Q-variance remains low near walls and starting positions but stays consistently high in predator-dense regions, confirming our mechanism captures genuine risk rather than mere exploration uncertainty. \n\n**Q: On Task Specification, Evolutionary Pressure, and Implementation Details**\n\nA:  We agree that evolutionary pressure contributes to these behavioral differences. Our reward structure (-1/+1) mirrors our Cell Reports study paradigm (not cited due to double-blinding), where mice received mild water rewards and air puffs of similar intensity. Increasing penalty values failed to encourage cautious behavior like waiting behavior at the entrance. We chose episodic RL and sparse rewards specifically to maintain consistency with our mouse experiments.\n\nThe 0.995 discount factor was selected to encourage longer-term planning that better matches mouse behavior, though it lacks direct biological explanation. Importantly, behavioral differences persisted across all tested discount factors (0.9-0.995).\n\nWe used H=3 after experimenting with longer horizons (H=6,10), which degraded performance due to compounding prediction errors. When H is too big, the prey can even ignore the predator. \n\nNo baiting behavior emerged even with extended training (up to 10M steps).\n\nThe above shows fundamental risk assessment differences rather than training duration issues.\n\n**Clarification On LLM**\n\nA: Our LLM agent makes decisions at each timestep, identical to TD-MPC2, ensuring a fair comparison despite the computational cost.\n\n\n**We've improved the related work section**\n\nA:  Recent studies by Vaxenburg et al. (2024) and Singh et al. (2023) demonstrate the value of comparing biological and artificial behaviors in navigation tasks. Our choice of model-based RL draws from Daw et al. (2011), showing animals maintain internal models for decision-making, with striatal prediction errors paralleling TD learning. Regarding whether standard RL suffices for mouse-like behaviors, Mattar & Daw (2018) showed that prioritized memory access explains biological planning processes, inspiring our PTSB. Blanchard et al. (2011) documented neural risk assessment mechanisms in rodents aligning with our variance-penalized approach. These studies collectively suggest that while standard RL alone cannot reproduce mouse behaviors, our biologically-inspired modifications bridge this gap.\n\n\n**Regarding weaknesses**:\nOur proposed mechanisms are not ad hoc, they are directly motivated by observed mouse behavior, as detailed in our response to Reviewer GbX7.\n\n\n**REF**\n\n- Vaxenburg, R., Siwanowicz, I., Merel, J., Robie, A. A., Morrow, C., Novati, G., ... & Turaga, S. C. (2024). Whole-body simulation of realistic fruit fly locomotion with deep reinforcement learning. bioRxiv, 2024-03.\n- Singh, S. H., van Breugel, F., Rao, R. P., & Brunton, B. W. (2023). Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nature machine intelligence, 5(1), 58-70.\n- Daw, N.D., et al. \"Model-based influences on humans' choices and striatal prediction errors.\" Neuron 69.6 (2011): 1204-1215.\n- Mattar, M.G., and Daw, N.D. \"Prioritized memory access explains planning and hippocampal replay.\" Nature Neuroscience 21.11 (2018): 1609-1617.\n- Blanchard, D.C., et al. \"Risk assessment as an evolved threat detection and analysis process.\" Neuroscience & Biobehavioral Reviews 35.4 (2011): 991-998.\n\nIn the end, we thank the reviewer again for insightful comments! We look forward to further discussions with you.", "claims_and_evidence": "I'm mostly fine with the claims, but they are generalized across RL agents when, in fact, they may be an artifact of specific task specifications in the simulation and the choice of algorithms (discussed in the upcoming sections). That said, since the authors focus solely on comparing behaviors and do not claim biological plausibility, I suggest only minor rewrites.", "methods_and_evaluation_criteria": "- In Section 3, the authors describe Deep Q Network (DQN) and Soft Actor-Critic (SAC), along with other behavior cloning and offline RL methods in Appendix C. However, all the plots in the main paper only use TD-MPC2. Why is this the case? If the results from the other methods are not used, perhaps the unnecessary text could be removed?\n\n- The ideas of Post Traumatic Stress Buffer (PTSB) and Selective Memory Sampling are reminiscent of Prioritized Experience Replay (Schaul et al., 2015). Did the authors consider this approach? Would it be accurate to say that the proposed approach is one instantiation of prioritized experience replay?\n\n- TD-MPC2 uses Model-Predictive Path Integral (MPPI) to sample actions. In simpler terms, it generates multiple rollouts of planning horizon $H$ and picks the action sequence that yields the highest predicted return, typically using a weighted averaging scheme based on the rollouts' performance. The default implementation of TD-MPC2 only considers a horizon length of 3 or 5, which is very small. What is the choice of of planning horizon $H$ in the experiments? This could potentially impact your results. \n\n- The proposed Variance-Penalized Temporal Difference (TD) Learning aims to reduce risky behaviors by avoiding states with high uncertainty. However, it is unclear whether risk and uncertainty are inherently the same. For example, early in learning, Q-values may appear uncertain due to limited exploration, but this does not necessarily imply that those states are inherently risky.\n\n**References**\n- Schaul, T., Quan, J., Antonoglou, I., & Silver, D. (2015). Prioritized experience replay. arXiv preprint arXiv:1511.05952.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "**Task Specification**\n\nI have some concerns about how the task is specified in simulation. I appreciate the effort the authors put into aligning the simulation with the real-world task. However, the disparity in qualitative behaviors may stem from the use of an episodic RL formulation with discounting and sparse rewards. \n\nAn RL agent taking the shortest path to the goal is likely an artifact of choosing discount factor $\\gamma < 1$ with sparse rewards, as this setup inherently incentivizes shortest-path solutions. While I do not know the biological mechanisms underlying mouse intelligence, it seems highly unlikely that they align with an episodic RL framework with discounting, as used in these experiments. \n\nTo understand the impact of task specification in goal-reaching tasks, Vasan et al. (2024) is a useful reference. Their work demonstrates that three seemingly equivalent task specifications in RL can lead to different final performance outcomes, even when using the same learning algorithm.\n\n**Comparisons with Other RL Methods **\n- TD-MPC2 is a model-based RL algorithm. I am curious whether model-free algorithms such as PPO or SAC achieve similar final performance to TD-MPC2.  \n\n**LLM Agent Description**\n- The action space of the LLM agent is unclear. Does it select an action at every timestep, or does it generate an entire trajectory at once? For a fair comparison with TD-MPC2, the LLM should be evaluated in a setting where it selects an action at each timestep.\n\n\n**References**\n- Vasan, G., Wang, Y., Shahriar, F., Bergstra, J., Jagersand, M., & Mahmood, A. R. (2024). Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learning. Reinforcement Learning Journal, 4, 1841–1854.", "relation_to_broader_scientific_literature": "This aligns with a broader body of work, including Vaxenburg et al. (2024) and Singh et al. (2023), which compare biological intelligence and RL agents, conducting behavioral analyses to assess biological plausibility and differences\n\n**References**\n- Vaxenburg, R., Siwanowicz, I., Merel, J., Robie, A. A., Morrow, C., Novati, G., ... & Turaga, S. C. (2024). Whole-body simulation of realistic fruit fly locomotion with deep reinforcement learning. bioRxiv, 2024-03.\n- Singh, S. H., van Breugel, F., Rao, R. P., & Brunton, B. W. (2023). Emergent behaviour and neural dynamics in artificial agents tracking odour plumes. Nature machine intelligence, 5(1), 58-70.", "essential_references_not_discussed": "The related work section primarily motivates RL as a useful framework for understanding biological intelligence. However, it does not discuss prior works similar to theirs or explain why RL alone is sufficient to generate the behaviors observed in mice.\n\nAdditionally, the section appears somewhat hastily written. For instance, a paragraph begins with the vague statement, \"Fundamental challenges persist\" (line 438). While the related work lists studies that explore the intersection of RL and neuroscience, it does not clearly connect them to the present study. For example, a more relevant discussion could address why model-based RL is specifically chosen to model rat behavior.\n\nPlease read the papers I mentioned in this review and consider citing them if relevant.", "other_strengths_and_weaknesses": "**Strengths**  \n- The paper is well-written and enjoyable to read.  \n- The plots and illustrations are aesthetically pleasing and effectively highlight the behaviors the authors aim to showcase.  \n- The analysis is timely, given the increasing role of AI in society. Comparisons with biological intelligence are valuable and much appreciated.  \n\n**Weaknesses**  \n- The authors do not provide strong explanations or hypotheses for why RL agents behave differently from mice.  \n- The proposed mechanisms for promoting risk aversion feel ad hoc. I encourage the authors to consider additional factors, such as the choice of algorithm and task specification.", "other_comments_or_suggestions": "- I'm open to reconsidering my score if 1) the authors update the related work section to better position their study and share it during the rebuttal, and 2) address my concerns regarding the methods and evaluation criteria.\n\n\n**Typos, etc**\n- The citation in the very first line is incorrect. It is Sutton & Barto (2018). Not just Sutton (2018). In APA: \"Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press\"", "questions_for_authors": "- What is the planning horizon \\( H \\) used in your TD-MPC2 experiments?  \n- What is the chosen discount factor \\( \\gamma \\)?  \n- Evolution optimizes for reproductive success, which inherently emphasizes self-preservation behaviors. In contrast, an RL agent lacks such pressure and is only optimized for success and failure. Do you think this could partly explain the differences in behavior? If so, could this be encoded through rewards?\n- Figure 11: The authors state that the proposed “VP-TD-MPC2 + PTSB” achieves an 86.1% visitation overlap with a mouse. However, the trajectories still appear visually different. Do you have any hypotheses on why this gap remains?\n- Would a \"baiting\" behavior emerge with longer training in the simulation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mjonwO0STZ", "reviewer": "Reviewer_YZj9", "summary": "This paper conducted and analyzed an experiment of \"predator and prey\" by mice and a robotic agent to learn the behavior patterns of biological self-preservation instinct, and designed and evaluated the simulation of reinforcement learning agents and the same \"predator\" robotic agent. The paper implemented negative memory amplification and variance-penalized temporal difference learning to introduce the risk-avoidance to reinforcement learning and decision making. The simulation results showed the two mechanisms made the reinforcement learning agents share more similar visitation pattern with the mice such as radial exploration and wall-following.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough and valuable feedback! \n\nWe would like to address several key points:  \n\n**Q: Motivation for Risk-Awareness in RL**\n\nA:  Our paper aims not to improve RL performance but to understand differences between biological and artificial learning. This basic research is crucial for advancing both fields. Modern RL has diverged significantly from biological principles, and understanding these differences has scientific value regardless of performance metrics. Many AI breakthroughs (CNNs from visual cortex, attention mechanisms from human vision, and Hierarchical RL from human planning) originated from biological insights. We aim to understand how and why biological risk assessment strategies differ from current RL approaches, which serves important scientific goals beyond conventional performance evaluation.\n\n\n**Q: How to Quantify Risk Awareness?**\n\nA: We have provided multiple objective indicators in the paper, including waiting behavior (mice pausing to assess risks), diverse path selection and wall-following behavior (shown in visitation density comparisons), and more conservative movement patterns (changes in average distance before/after predator detection).\nTo further quantify these behaviors, we conducted additional quantitative analyses: \n-  Waiting behavior: Defined as movement with distance change <0.1 units within the first six steps, our analysis of 1,000 trajectories shows standard TD-MPC exhibits 0% waiting behavior, while mice demonstrate 27.7% and our improved method shows 32.4%. \n-  Episode length: Standard RL agents complete tasks with an average length of 7.09 steps, while our improved method shows 13.89 steps and the mouse shows 14.04 steps. \n\nThese quantitative measures demonstrate that our proposed mechanisms enable RL agents to exhibit risk perception abilities more similar to mice. We will include these results in the future.\n\n\n\n**Q : Could you clarify the meanings of 'receives +1 reward at the goal' and 'puffed by the predator,' and would increasing the penalty (beyond ±1) make RL agents more cautious compared to mice?**\n\nA: In our setup, the hexagonal environment has a starting point at one vertex and a goal point on another. The agent (prey) receives a reward of +1 only upon reaching the goal point, which mirrors the real mouse experiment where mice receive water as a reward at the goal. The agent receives a penalty of -1 when \"puffed\" by the predator, which occurs when their distance is below 0.1 units (27.5cm in real environment). This corresponds to the air puff stimulus used in the physical mouse experiments. \n\nThe initial equal magnitude (+1/-1) for rewards and penalties was chosen to reflect that both water reward and air puff are relatively mild stimuli. We did experiment with larger penalty values (up to 200x). However, even with increased penalties, the RL agents still exhibited fundamental differences from mice in that they would not display waiting behavior at the start point. This behavioral difference highlights the need for additional mechanisms beyond simple reward scaling.\n\n\n**Q: How to formulate the Q-value variance?**\n\nA: We would like to highlight that this is explicitly detailed in our paper. The Q-value variance is empirically computed by uniformly sampling actions from the action space and calculating the variance of their Q-values (as stated on the right side of lines 279-284).\n\n**Q: Figure 6 top row and Figure 12 top row, what is the meaning of \"distance changes\" and the x-axis? What is the reason for the change to be linear?**\n\nA: The plots show how goal-agent distance changes before and after predator detection. (This information is in the caption, maybe too small now.) The linear trend emerges naturally because we are considering an average of 1000 episodes. This big number average makes mice’s and also RL’s trend to be linear.\n\n**Q: Why Discuss LLMs?** \n\nA: We included LLMs to compare how different agents behave in similar environments, tying back to the theme of \"machines and mice.\" The results suggest risk-awareness is not just an RL issue but a broader challenge in AI cognition. The LLM experiments are supplementary, not a core focus.\n\n**Q: Some other clarification**\n\nA: The survival rate is the same as the success rate.\n\nIn the end, we greatly appreciate the reviewer's insights and look forward to further discussions!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "1. Line 110 - 113, what is the meaning of \"receives a reward of +1 upon reaching the goal\", moving to the next cells safely? What is the meaning of \"puffed by the simulated predator-like robotic threat\"? It is fair to set 1 to both reward and penalty in the simulation to explain the difference between mice and RL agents, but I am wondering if increasing the penalty would make RL agents more cautious? \n\n2. Line 172 - 179, what is the definition of success rate? Is it the same metric as \"survival rate\" first appeared in 2. Experiment Setup? What is the motivation of making RL agents move more like mice if the success rate is almost the same while the RL agents even traveled less distance?\n\n3. Line 321 - 329, how to formulate the Q-value variance? Is it empirically computed or parametrically formulated?\n\n4. Figure 6 top row and Figure 12 top row, what is the meaning of \"distance changes\" and the x-axis? What is the reason for the change to be linear?\n\n5.Line 603, what is the evaluation criteria or quantification of \"risk-awareness\" for RL agents? It is ambiguous to claim that RL agents behavior should resemble mice's. Besides, how does risk-awareness further benefit the RL agents if the survival rate is not the primary focus?", "theoretical_claims": "Not applied", "experimental_designs_or_analyses": "Please refer to \"Method and Evaluation Criteria\" section", "relation_to_broader_scientific_literature": "This paper proposed two mechanisms in the field of reinforcement learning to introduce risk-awareness instinct to the model. The first one is based on Post Traumatic Stress Buffer (PTSB) (Al Abed et al., 2020) and the second one is extension to Surprise Minimization in Reinforcement Learning\n(SMiRL) (Berseth et al., 2019).\n\nIn 6. What about LLM agents, the paper discussed the performance of large language model under the same simulation setup. I am not convinced by the necessity of discussion about LLM which seems irrelevant to the previous parts about reinforcement learning.", "essential_references_not_discussed": "None to my knowledge", "other_strengths_and_weaknesses": "This paper provided an insightful work including both biological experiments and first-hand data for \"predator-prey\" to illustrate the risk-awareness and self-preservation, and provided feasible modifications to the previous method to make reinforcement learning model to capture similar patterns of mice's.\n\nI have a few major concerns. The first one is about the motivation for including the risk-awareness in RL models, especially since the success rates between the mice and the simulation are similar (and the setup of the RL model in this paper is arguably designed to be \"reckless\", since the bonus and penalty are of the same scale). Second, \"risk-awareness\" is ambiguous and subjective, and it lacks quantification. The authors also need to provide more evidence to claim the benefits of incorporating it into the RL models.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper compares mice and reinforcement learning (RL) agents on specific behavioral measures in a navigation task where subjects must reach a goal while avoiding predators. The authors have done an excellent job modeling the task to closely match the one used in mouse experiments. Reviewers raised concerns regarding the generality of claims about biological and artificial agents, and suggested more systematic comparisons with a wider range of model-based and model-free RL agents. While this paper falls somewhat outside typical ICML submissions, I believe it represents an important research direction in aligning artificial and biological agents. Beyond baseline RL models, the authors also replicate the experiments using ChatGPT-4 as an example of a large language model (LLM). Although the paper and the LLM experiment could be more comprehensive in their comparisons and could better explain the motivations behind their methodological choices, I still consider this an interesting contribution to ICML. Therefore, I recommend weak acceptance, with the expectation that the authors will address the reviewers' comments (specifically from reviewers GbX7 and TtQ7) in the revised manuscript.
Decision: Accept (poster)