Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments
Authors: Riley Simmons-Edler
Ryan Paul Badman
Felix Baastad Berg
Raymond Chua
John Vastola
Joshua Lunger
William Qian
Kanaka Rajan
Venue: NeurIPS 2025 poster ()
Keywords: Deep reinforcement learning
reinforcement learning
foraging
exploration
RNN analysis
open-ended learning
navigation
decision making
behavior analysis
cognitive science
neural dynamics
representations spatial memory
recurrent neural dynamics
joint neural-behavioral modeling
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose methods for studying behavior and neural dynamics in deep RL agents based on neuroscience methods and demonstrate them in a naturalistic simulated foraging environment

Abstract: Understanding the behavior of deep reinforcement learning (DRL) agents—particularly as task and agent sophistication increase—requires more than simple comparison of reward curves, yet standard methods for behavioral analysis remain underdeveloped in DRL.
We apply tools from neuroscience and ethology to study DRL agents in a novel, complex, partially observable environment, ForageWorld, designed to capture key aspects of real-world animal foraging—including sparse, depleting resource patches, predator threats, and spatially extended arenas.
We use this environment as a platform for applying joint behavioral and neural analysis to agents, revealing detailed, quantitatively grounded insights into agent strategies, memory, and planning.
Contrary to common assumptions, we find that model-free RNN-based DRL agents can exhibit structured, planning-like behavior purely through emergent dynamics—without requiring explicit memory modules or world models.
Our results show that studying DRL agents like animals—analyzing them with neuroethology-inspired tools that reveal structure in both behavior and neural dynamics—uncovers rich structure in their learning dynamics that would otherwise remain invisible.
We distill these tools into a general analysis framework linking core behavioral and representational features to diagnostic methods, which can be reused for a wide range of tasks and agents.
As agents grow more complex and autonomous, bridging neuroscience, cognitive science, and AI will be essential—not just for understanding their behavior, but for ensuring safe alignment and maximizing desirable behaviors that are hard to measure via reward.
We show how this can be done by drawing on lessons from how biological intelligence is studied.

Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments
Riley Simmons-Edleri
Ryan P. Badman 122+
Felix Baastad Berg'
Raymond Chua
John J. Vastola 1,2
Joshua Lunger
William Qian
Kanaka Rajan 1,2
Department of Neurobiology, Harvard Medical School
2 Kempner Institute for the Study of Natural and Artificial Intelligence, Harvard University DDeparteent of Mathematics, NTNU School of Computer Science, McGill University & Mila
Department of Computer Science, University of Toronto
Biophysics Graduate Program, Harvard University kanaka_rajjan hwrvard.
(riley_simmons simmonsmmeeiieeee edler,
Abstract
Understanding the behavior of deep reinforcement learning (DRL) agents particularly as task and agent sophistication increase reeeirree more than simple comparison of reward curves, yet standard methods for behavioral analysis remain underdeveloped DRL. We apply tools from neuroscience and ethology to study DRL agents in novel, complex, partially observable environment, ForageWorld, designed to capture key aspects of real-wor world animal boaging-including sparse, depleting resource patches, predator threats, and spatially extended arenas. We use this environment as platform for applying joint behavioral and neural analysis to agents, revealing detailed, quantitatively grounded insights into agent strategies, memory, and planning. Contrary to common assumptions, we find that model- free RNN-based DRL agents can exhibit structured, planning- like behavior purely through emergent dynamics-witte
requiring explicit memory modules or world models. Our results show that studying DRL agents like mimals--nallyznng them with neuroethology-inspree cramics-uncovers rich structure in their learning dynamics that would otherwise remain invisible. We distill these tools into general analysis framework linking core behavioral and representational features diagnostic methods, which can be reused for a wide range of tasks and agents. As agents grow more complex and autonomous, bridging neuroscience, cognitive science, and AI will be essen- tial-not not just for understanding their behavior, but for ensuring safe alignment and maximizing desirable behaviors that hard to measure via reward. We show how this can be done by drawing on lessons from how biological intelligence is studied.
tools that reveal structure in both behavior and neural
"Equal contributions Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
As reinforcement learning (RL) researchers pursue more complex, naturalistic, and open-ended tasks [1, there a growing need for tools to understand both the detailed behaviors and the neural dynamics of deep RL agents. This challenge mirrors long-standing problems in neuroscience, where researchers have developed extensive methods for joint behavioral and neural analysis in animals to understand biological intelligence. While machine learning (ML) has adopted neural representation analysis tools from neuroscience, mature behavioral analysis methods from neuroscience, cognitive science, and ethology 22 3]
remain underused in DRL. Open-ended RL tasks typically involve partial observability, memory. planning, and spatial reasoning -ye we still lack a systematic understanding of the strategies agents to solve them 9 Further, despite thousands of benchmarks from
video games 10] to 2D/3D gridworlds [1] 12 there are no standard methods for analyzing or comparing agent behavior across tasks. RI evaluation still focuses primarily on reward curves and aggregate performance, which, while useful, offer limited insight into how agents solve structured tasks. The lack of behavioral analysis is a key bottleneck-lmmitne insight into why agents succeed or fail. Without understanding behavior, it's difficult to diagnose algorithmic performance. Applied RL also stands to benefit from removing this bottleneck: more complex tasks require both better agent planning and deeper diagnostics to verify expected behavior [13
Notably, this is not just a machine learning problem interprettnenggggg behavior in partially observable, reward-driven environments is also a core challenge in neuroscience and cognitive science. Inspired by how animals are studied in neuroscience and ethology, we propose a behavior-first approach to analyzing DRL. agents. This approach enables richer insight into both learned behavior and internal representations, drawing on joint behavioral-neural analysis tools that have proven essential for understanding biological intelligence. We ground this proposal in Forage World, a novel task suite in which agents forage and survive in procedurally generated, large-scale, partially observable environments with temporally extended dynamics. These pressures reflect the ecological tradeoff's animals encounter during natural foraging [6 17). and align with calls to push foraging tasks in neuroAI beyond the traditional bandit paradigms [88 23). Forage World builds on Craftax [12), combining complex naturalistic environment features with efficient training. We extend the GPU- accelerated Craftax with features such as patchy resources and structured threats to better capture key elements of animal foraging. These additions place strong demands on memory, spatial reasoning, and long-term planning, presenting a non-trivial challenge for modern DRL agents. In addition, foraging serves as a useful metaphor for core robotics tasks such as navigation, cleaning, and search and rescue 224] and underlies many popular RL benchmarks, including Minecraft and NetHack (25).
To support behavior-first evaluation in DRL, we introduce an analysis framework (Table 1 that maps key behavioral and neural targets to diagnostic methods. This framework is showcased by the results in this work and is reusable across neuroAl, interpretability, and broader RL applications.
Our key contributions are:
(1) We develop Forage World, a naturalistic RL task suite with ecological structure -uch depleting resources, predators, partial observability, and open-ended survival goals-to to study planning and memory in model-free agents.
(2) We adapt tools from neuroscience and ethology ethology- including path analysis, generalized linear models (GLMs), and recurrent state decoding- for joint behavioral and neural analysis, enabling biologically grounded interpretability of deep RL agents.
(3) Using these tools, we show that model-free RNN agents exhibit structured, planning-like behavior through emergent dynamics, despite lacking explicit world models.
(4) We release reproducible pipeline and open-source codebase to support further research on internal representations and behavior.
We demonstrate these contributions ForageWorld, analyzing trained agents across five motif classes: exploration, decision-making learning dynamics, internal representations, and generalization.
The remainder of the paper reviews related work (Section 2 introduces task and analyses (Section 3 and presents results (Section 4p.
2

---

Table 1: Understanding DRL agent behavior in open-ended tasks requires context- sensitive analysis frameworks, much like those used in animal neuroscience. This table summarizes our reusable toolkit for probing memory, planning, exploration, and internal representations in DRL agents.
Analysis Target
Key Question
Methods Used
Goal Inference
What goals is the agent pursuing, and how are they shaped by experi- ence?
Decision GLMs, in-context learning study, behavior phase segmentation
Memory Span
How far back in time can the agent remember?
RNN state decoding, memory module ablations
Planning Horizon
How far ahead does the agent plan?
Future decoding, auxiliary loss analysis for predictive objectives
Spatial Structure
How does the agent move through the environment?
Position occupancy entropy, tracking revisitation, path analysis
Network Capacity
Is the model too large or too small for the task difficulty?
Network size/pruning sweeps, compar- ing performance curves
Representations
What information is encoded in the
Decoding, GLMs, encoding profiles of task variables
agent's 's internal states?
2 Background: Open-Ended RL as a Bridge to NeuroAI
2.1 Open-Ended RL and the Need for Behavioral Analysis
Generalization in RL refers the ability transfer knowledge to new environments or tasks, and is critical for both practical applications and theoretical understanding in AI [26]. However, many RL algorithms struggle generalize beyond their training tasks (27 28 This bottleneck has renewed
interest in open-ended RL tasks that require rich exploration, subgoal discovery, and long-horizon
reasoning [1] Open-ente endede RL tasks typically (i) scale with the learner, requiring skill composition in large state spaces, and (ii) incorporate naturalistic structure to better reflect real-world environments Many recent benchmarks embody this open-ended perspetiiee-emphasiiiige compositional skills, dynamic goals, procedural generation, and expansive state spaces [ [3 30-39)
29
2.2 Joint Behavioral and Neural Analyses for Artificial Agents
To date, open-ended RL benchmarks have not drawn sustained interest from neuroscience or cognitive science, leading to lack of deeper analysis frameworks in this research area. One reason that few open-ended tasks reflect how animals explore, plan, and decide [3] 40). Thus, there is need for more naturalistic benchmarks-ant for analysis pipelines that study agents with the same rigor applied to animal behavior and neural computation. Recent neuroscience and cognitive science work has primarily applied brain- irsiininspired tools to ML systems for comparing internal representations [0). This lack of behavioral emphasis mirrors limitations in how internal dynamics are typically studied-ofie in isolation from behavior, or without the tools needed to reveal structure. To date, most of this work has focused on representational similarity, not on joint analysis of behavior and underlying neural computation. A few recent exceptions- mostly outside RL--have begun to explore interpretability with greater focus behavior
[47 51 52 13 53). Researchers have also made
calls for more interpretability work in the RL domain as well (64
Despite this challenge, key lesson from neuroscience is that neural activity alone is often insufficient to explain computation 3 57 60] For example, recent findings show that common similarity metrics can yield inconsistent conclusions [48] 61). Several recent works argue that neuroAI should
prioritize aligning behavior and internal dynamics naturalistic settings, rather than relying on coarse similarity metrics in toy tasks [62 60). We thus adopt behavior-firs firsst approach to studying DRL agents in the naturalistic, open-ended task Forld characterizing their strategies and internal representations using neuroscience-inspres tools as is done in biological systems.
3 Methods
This section summarizes the environment, model architectures, training procedures, and analy- sis framework we used in this work. Full code for ForageWord and our analysis is available at attps://et //gi thub. in the appendix.
wowettar-ooaigggtaaaaaa tettttreefffree tree//ragii..gg with additional details
3

---

A
B
Partially observable window the agent sees (actual task screenshot)
96 x 96 Grid-based Foraging Arena
Patch
Water
Obstacles
Obstacla
Agent with heading arrows
Predator
#+
Food
Predator
Agent
C
7uhh1
2uth PuRi Eal
Diak
Path 1-3 Fat
Est
DAN
Full Path
Patch
Eac
Drink
Arena 80
80
60
Co
Arena Y
Arena
8-
20
80 80
20
Arena
Arena
Arena X
Figure 1: Model-free agents exhibit structured exploration and revisitation behavior in an open- ended, partially observable environment. (A) 9x grid view (left, local observation window) shows the agent's local observation, positioned within the full 96x96 environment (right). Cows (food) diffuse from spawn points and deplete when consumed; lakes (drink) are fixed and unlimited; predators intermittently pursue agents in view. (B) Agent observations include the visual window, inventory, and internal states (c.g., health, hunger, fatigue). This agent is sleeping -typiclly in corners to reduce predator contact. (C) Trajectories from single episode: three early sequential exploratory paths (left), initial revisitation (middle), and full path with revisited patches in orange (right). The paths depicted belong to the same episode, and are directly consecutive from Path 1 through 4 in the behavioral time series logs for the episode. Path I starts at t=0 of the episode.
3.1 World: Naturalistic Foraging with Internal States
We build on the Craftax benchmark [12), modifying it to support more naturalistic behavior under partial observability. Forage World introduces ecological constraints and uncertainty better reflect real-world cognitive demands and enable deeper analysis. In particular, the need to localize food and avoid predators increases the memory demands relative the original Craftax task. Cows have an arena-wide count limit, spawn at fixed locations, and diffuse over time-Icading to temporary, localized food depletion and requiring the agent to leave and revisit patches periodically (Figure
Drinking from procedurally placed water sources maintains hydration. Predators intermittently appear near food patches and pursue agents when within line of sight (Figure Ip. Agents manage hunger, thirst, fatigue, and health. Fatigue recovers only through sleep, starvation and extreme thirst reduce health, and death occurs if the health variable hits zero. Sleep can only be performed when energy is below 50% of maximum. It immobilizes the agent while gradually restoring energy and, applicable, health (Figure T). Arenas per-episode novelty. Unlike Craftax, our reward function is focused on agent survival: to receive reward, the agent must maintain its physiological resources (health, food, drink, energy) above half' their maximum values. This objective correlates strongly with survival time and encourages the agent maintain a buffer in resource levels- promoting overconsumption or maxing all variables. Specifically, the reward function is:
procedurally generated with randomized layouts, ensuring
R(s:) 0.1 x (1 + sign(health; 5) + sign(food, 5)
sign(drink, - 5) + sign(energy, 5))
As defined in Equation 1 the agent's reward is a function of physiological variables: it receives increasing positive reward for each key survival resource maintained above threshold-health, food, drink, and energy.

---

For analysis, we log neural states, actions, and environment variables for each timestep (Table 2). This setup also supports held-out evaluation, enabling generalization assessment. Logged variables span both behavioral and neural axes from our analysis framework (Table 1
3.2 RL Architectures and Model Variants
Our primary architecture was PPO with a GRU-based recurrent core (PPO-RNN), detailed in subsec- tion A.6 In some experiments, we added an auxiliary objective to encourage interpretable spatial representations. In these experiments, the model also outputs its predicted position (æt, 3/2) relative to the first-timestep origin (path integration), via a small fully connected head that shares the RNN hidden state h, with the policy and value heads. We train this auxiliary objective jointly with the PPO loss, using an L. regression loss:
Laux El Poll
(2)
This term is weighted by factor Waxx during optimization. This objective encourages RNN states to encode location, enabling spatial decoding (c.g.. allocentric structure, planning horizon) and downstream analysis. Examining internal dynamics allows link behavior to emergent methods in animal navigation studies.
To introduce biologically-insiiee sparsity, we prune the weights of the RNN core using JaxPruner during training [63]. We targeted 90% sparsity across model layers, approximating biological brain sparsity, which ranges from 70% to 94% across species [54]65] We used magnitude-based pruning. Pruned agents performed comparably to unpruned ones and, in some cases, showed improved decoding of spatial variables from hidden states (Figure 16) consistet with findings that sparse connectivity improves modularity and interpretability [66
3.3 Decoding Allocentric Position and Planning Horizons
To evaluate memory and planning, we test whether hidden states encode past and future spatial positions. If location is internally represented, it may enable revisitation and planning. We first record hidden states h4 and ground-truth positions (x+; yee) throughout each episode. Then, at each timestep I, we train decoder to predict allocentric displacement Y++ (Ar, Ay) between the current hidden state he and the agent's s position At steps in the past or future, using only ht E 52 This method tests whether the RNN encodes allocentric spatial information various planning and memory horizons- analooous to representations in animal navigation systems. Additionally, we train separate decoders for allocentric (relative to origin) and egocentric (relative to body orientation) coordinates to assess which coordinate frame the agent uses internally. We use ridge regression as the decoding model to preserve interpretabilityy Decoders were trained on the first 75% of each episode and evaluated on the final 25% to assess generalization. Accuracy is reported as RMSE, benchmarked against average displacement-basel baseline. All behavioral analyses were conducted in held-out test arenas, with weights frozen throughout, to ensure that hehavior reflects generalization rather than ongoing learning. See subsscctin C.2 and subsection C.3 for full training details.
4 Results
4.1
Structured Exploration and Revisitation in Novel Environments
We begin by examining how agents explore novel environments and transition revisitation strategies. Foraging has long served as a rich behavioral paradigm for studying goal-directed exploration in animal neuroscience [69] 77]. In rodents, foraging in novel, partially observable environments unfolds in distinct behavioral phases: mice initially engage in broad exploration, followed by direct and efficient revisitation of previously discovered locations once the environment mapped [71].
Using trajectory visualizations, behavioral segmentation, and spatial entropy metrics, we find that induces structured, phase-like foraging behaviors in trained DRL agents-analogee to those observed in biological systems. Standard performance metrics confirm that agents are competent at the task, showing long survival times (directly proportional to return) after training (Figure 2}. However, our behavioral analysis reveals that DRL agents equipped only with i.es- without explicit world models -eoolls-develop in-context adaptive learning and exploration strategies that evolve across episodes.
Strikingly, after initially mapping novel arena, agents shift into revisitation behaviors that reflect structured, multi-objective decision-making- maiing-mirroring phase transitions in animal foraging (71) and information seeking (7277). Their early exploration trajectories qualitatively resemble known
5

---

patterns of insect search behavior (Figure 8). Agents generated outwardly spiraling, azimuthally rotating loops from their starting position that gradually expanded to cover most of the arena (Figure 1). Even during revisitation phases, exploration covered the majority of the arena by the end of each episode (Figure 6). These extended trajectories also supported predator avoidance, enabling agents to discover shortcut paths between safe regions and resource patches (Figure 7). Similar patterns have been observed in ants and bees during nest displacement or early flights from the hive (Figure 8).
misisted-moss successful agents had
4.2 Mujti-Objectivve Foraging and Strategic Patch Use
Having characterized agents' exploration and revisitation behaviors, we next examine how they select among remembered locations. After initial exploration, agents displayed clear shift to innitaatioo-rtturnnngg to remembered patches via direct routes. Similar to rodents navigating partially observable mazes [7]], our agents exhibit rapid transition from broad exploration to targeted revisitation (Figure D.. This transition mimics phase-structured foraging seen in rodents, while exploration resembles insect search behavior Figuuee 8). Agents also periodically return to their starting position posenann-p a reorientation strategy in the absence of an explicit spatial map.
Memory and Network Capacity are Necessary for ForageWorld
Pruning Does not Impact Performance
Layer-Size-512 (haselnn)) Jayen-Size-1 Layer-Stze-AA
Prune 90% (baseline)
No-Frune Prune-80re Prune 90%
Laver Stze
15000
15300
Relur
Retun
Frunc
10000
10000
5000
5000
Iterations
PPO Iterations
10k
20k
30k
40k
10k
20k
30k
40k
Path Integration Loss Improves Performance In Large Arenas
Directional Perception Improves Learning Speed
Map-Ste Sia---i--66 (baueiin))
Map-Size de No- Ruthint Map Sizu-4e 40 Map-Size-24
2x11 Perouption Window fbaosfind Directional Parception
Map-Size-4BB 40t-----tttttttt
Map 24------oo Puthied
15000
40000
Return
Recum
Prune Step
30000
10000
20000
5000
10000
PPO Iterations
PPO lterations
10k 20k
30k
40k
10k 20k
30k
40k
Figure 2: Performance metrics and model ablations motivate deeper behavioral-neural analysis. (Top left) World requires memory and substantial network capacity to learn. Replacing the 512-unit RNN with feedforward network significantly impairs performance, does downsizing to a 64-unit network (550k parameters). A midsize 128-unit network also underperforms when pruned, suggesting that high capacity is needed to support sparsity. (Top right) Pruning does not degrade training performance but improves the spatial interpretability of internal representations (see Figure 16). (Bottom left) Removing the auxiliary path integration loss reduces performance in large arenas-but but not small ones. Thus, the ability to predict current self- pos-ipoition appears critical for larger- scale navigation (see Figure for representational effects). (Bottom right) Limiting perception to forward-facing field of view improves early pearhing-perhaps due to reduced input complexity-bi final performance conceals behavioral differences (see Figure 15). (All plots) Curves show the time-weighted EMA mean across 5 random seeds; shaded regions show one standard deviation.
Agents are not given an explicit list of patch locations. Instead, revisitation behavior emerges from recurrent state dynamics encoded during exploration and is reconstructed for analysis from logged trajectories. This result shows that model-free RNN agents can develop memory nemory-guided, goal- directed behavior such as selective revisitation triistttton-wwihhuutt explicit world models or symbolic memory structures. This memory dependence is further supported by substantial performance gap between PPO agents with and without recurrence (Figure 2 Figure 1T Figure 10). Like animals that balance intake against physiological constraints (e.g., stomach volume, satiation), our agents are rewarded for maintaining low hunger, thirst, and fatigue- rather than for rapidly depleting resource patches. Under our baseline task parameters, expert agents ate an average rate of 0.011 (approximately once

---

every 95 steps), drank at 0,034, and slept at 0.262. This emergent spacing enabled us to test whether agents made memory-guided, multi-objective decisions when selecting which patches revisit. We analyzed patch revisitation choices and found that agents integrated spatial and task-relevant features multi-ohjective manner (Figure 3 Figure 7).
Standardized GLM coefficient value
Factors that influence patch revisitation choice
0.4
***
I
****
0.0
**
**
*
-0.4
-0.6
EatRate
DrinkRate
PredRate
Recency
Dwelltime CowCount Uncertainty
Variable Category for Chosen Patch Relative Non-chosen
Figure 3: Memory-guided, multi-objective revisitation strategies emerge in model-free agents without world models. Generalized linear model (GLM) coefficients for patch history variables predicting an agent's choice revisit one patch over others. Choices are defined 50 timesteps before each patch eat event. From left to right: agents prefer patches with fewer prior eat actions (EatRate); show no preference for water proximity (DrinkRate); avoid patches with more predator encounters (PredRate); prefer patches visited more recently (Recency); ; show mild preference for longer dwell time (Dwelltime); prefer patches with more observed cows (CowCount); and prefer patches with higher prior position prediction error (Uncertainty). Significance levels: < 0.05, < 0.01, < 0.001. Figure 13 has model output and VIF analysis. Error bars are 95% confidence intervals (CT).
4.3 Emergence of Behavioral Competencies Over Training
To examine how agent behavior evolves during training, analyzed performance metrics and behavioral patterns across episodes. We find that ForageWorld supports staged learning: agents shift from undirected exploration structured, goal-aligned stratgies-ttansiee in standard reward curves. These patterns suggest temporally organized skill acquisition. To test this, we tracked how different behaviors emerged and stabilized over the course of training. We used a panel of behavioral metrics to characterize the timeline of skill acquisition and identify which behaviors drove early versus late performance gains (Figure
only partially visible
Figure 9).
Early in training, PPO-RNN agents exhibited a "fishing" strategy-remannn remaining stationary near the origin for extended periods. After approximately 20,000 training iterations, agents underwent abrupt shift in behavior across multiple metrics. This transition included longer-distance travel during both early exploration and later revisitation, more strategic trade-offs between water and food collection, a sharp increase in tool-making frequency, and the emergence of gradual gains in predator defense. This shift marks the emergence of higher-level strategies in the training process.
To evaluate the contribution specific components of the environment and model, we compared PPO-RNN baseline to several variants with targeted objectives or capacities removed. One key ablation altered the agent's field of view (FOV) to include only the forward grid rows, rather than the default Craftax FOV of 9x11 rectangle centered on the agent. This change increased the need for information-seele seeking behavior and more closely resembled animal vision, which is predominantly front-facing. We found that front-FOV agents explored farther earlier in training and often engaged in more localized, dense search patterns (Figure 14 Figure 13). We also trained a different DRL objective, off- off-policy RL architecture called parallelized Q-network (PQN) (75] (Appendix E). PQN-LSTM models performed comparably to the PPO-GRU baseline similar learning histories and pathing (Figure 31 Figure 30). but with the noticeable behavioral differences of much lower rates predator killing (L.e. reducing predator HP to 0) (Figure 3Tp. Thus, PPO-GRU seems converge predator fighting strategy, while PON-LSTM converges to predator evasion.
29) and with overall
7

---

Spatial Unc. / Distance
Distance Traveled
Spatial Entropy
Angular Variance
Earty Episade
Early Episode
Late Episade
2.5
Late Episode
Mean Value
0.20
30
0.15
20-
1.0
0.3
0.10
0.05
10
0.5
0.2
0.00
25
75
100
25
0.1
Pred. Enc. Rate
Pred. Rate
Tool Make Rate
Food Drink Score
0.5
2.00
1.75
0.4
3.50
Mean Value
0.3
3.25
1.00
0.2
10-
0.75
0.50
0.25
Food Score Drink Score
0.00
25 50
75
100
125
25
50
75
100
125
25
75
100
125
50
100
125
Ep. Number (1000x)
Ep. Nurmber (1000x)
Ep. Number (1000x)
Ep. Number (1000x)
Figure 4: Distinct behavioral competencies emerge over training. with early gains in exploration followed by refinement of survival strategies. Training dynamics show staged acquisition of task- aligned skills. Early learning emphasizes spatial exploration and arena coverage, while later training refines predator response, tool use, and pathing patterns. Metrics include: spatial uncertainty (normalized by distance from origin); distance from origin (early late, meaning before after the first 1,500 timesteps in an episode); state occupancy entropy of agent position; angular orientation variance across 250 timestep intervals; predator field-of-view exposure; tool-making rate (1 one tool crafted, 2 = both); and food/water satiation levels. Error bars are 95% CI. These plots can be compared to the generally linear performance gains seen in survival times across training Figure
4.4 Recurrent State Representations Support Memory and Planning
We next ask whether agents encode task-relevant structure in their internal representations. Using linear decoders trained on recurrent states, we find that agents represent allocentric position across time-revealing implicit memory and planning capacity not apparent from performance metrics alone. Thus, our framework can (1) decode the memory and planning timescales represented agent RNN states, and (2) identify the internal encoding structure that supports foraging behavior.
growing body work in RL has examined whether model-free RNN agents can perform implicit planning without the explicit world models typically used for plamning tasks (76178). This question also arises in neuroscience, where insects like ants and bees show planning-lik like behavior despite lacking mammalian hippocampal-entonn agents revisit patches based on multiple factors such as proximity, predator history, and position prediction error (Figure 3) Figure 7). An agent's position relative to the origin also be decoded from its current RNN state up to 50-100 timesteps into the past and future- supporting its capacity for long-horizon memory and planning (Figure 5p. Furthermore, approximately 100/512 neurons were found to be position-sensitivy at the single neuron level using generalized linear model (GLM) analysis. The neural activity response position change (GLM coefficient magnitude) increased with radial distance from the agent's 's origin as well, suggesting a distance accumulation circuit that modulates rrvgintreviiitation behavior (Appendix Dj.
map systems (79-81]. Our analyses show that DRL
Decoding accuracy improved over the course training, consistent with the agent acquiring more stable and structured internal representations. In contrast, the agent's position relative itself (egocentric orientation) could not be reliably decoded. This preference for origin-relative encoding may help explain why agents frequently revisit the arena origin during later phases (Figure I).

---

A
B
Example arena for decoding
Dacodar Performance (RMSE)
Decoding Past/ Future (Ax, 4y) Relative to Self
dn,a
50
origin (0,0)
MosYa
agent
distance from
origin, steps into the future
40
1 arcna
30
Averaae displacementt
dn.e= distance from stops
20-
10-
into the future
-10"
-10"
-10" 0
10
10'
10²
C
D
Time Step of Position Decoded (At)
Decoder P Perfornacc (RMSE)
Decoding Past/ / Future (Ax, 4y) Relative to Origin
Decoder Performance (RMSE)
Decoding Past/ Future (Ax, Ay) Across Training
50
.0
I arena
From epech 10k
40
anenas
40
epoch
arcnas
From opach 90%
30
Average displaemmen
30
Average dislecc
20
20-
10-
10-
0
-10²
-10' -10 0
10
10'
10²
-10"
-10
-10
10
10'
10
Time Step of Position Decoded (At)
Time Step of Position Decoded (At)
Figure 5: RNN states encode allocentric position and temporal structure, revealing emergent memory and planning capacity in model-free agents. (A) A single decoding model was trained per agent test whether spatial information was encoded in allocentric (relative to origin) or egocentric (relative to agent) coordinates. (B) Late-training decoding performance for egocentric distance remained at chance level across models. (C) Allocentric position could be decoded above chance up to approximately 50-100 timesteps into the past and future depending on the run. Training arena count per decoder was varied show that models did not rely on arena-specific cues. (D) Allocentric decoding improved over training. Error bars are larger in (D) because decoding was limited to timesteps 1000-600 1000-000 per arena, due to shorter survival early-training (10k epoch) agents. All plots use average displacement per timestep as a chance baseline. Error bars reflect 95% CI.
4.5 Generalization and Modularity in Recurrent Dynamies
We next examine whether the internal representations learned by DRL agents generalize across environments and exhibit modular substructure. These properties are important not only for perfor- mance, but also for iiterretabillty-reeellll
which units encode what, and under what conditions. Decoding performance was moderately higher in pruned (sparse) networks than in fully connected ones (Figure T6), confirming more efficient information encoding sparse architectures ctaftectures-connisseeettt with recent findings [66-68]. Removing the auxiliary objective for position prediction eliminated above-chance decoding performance (Figure 17). suggesting that the associated task performance loss (Figure stems from the model's inability to encode position information using the PPO objective alone. These results suggest that the auxiliary objective improved both agent performance and the interpretability of its internal representations- dual benefit revealed only through neural analysis. Past and future position encoding relied on overlapping but functionally modular subpopulations of RNN units, which diverged more clearly longer time horizons (Figure 18). Notably, a single decoding model generalizes across apisodes--predicee recorded in many different arenas for the same agent. This cross-episode generalization likely reflects the consistent spatial structure of the environment: although arena configurations vary, the 96x grid layout and global orientation are preserved, providing a stable allocentric reference frame.
past and future position from RNN states
In summary, model-free RNN agents trained in open-ended tasks can develop structured internal representations that support memory, planning, and spatial generalization inttiaiiationn-wittout relying on explicit world models. ForageWord induces naturalistic behavioral and neural motifs in DRL agents, and our analysis framework enables these motifs to be detected and interpreted using behavior-fim first tools.

---

5 Discussion
We provide two main findings: (1) ForageWorld induces behavior-neure motifs relevant to planning and memory, and (2) our toolkit reveals these motifs in ways that simple performance metrics cannot.
We also help challenge the common view in neuroscience that sophisticated planning and memory require explicit world models, mammalian-like brains, or symbolic memory [80] [8]] topic that is of growing interest in computer science well 83 [ Despite being trained with model-free objectives, RNN-based agents in our study exhibit revisitation, uncertainty
exploration, and allocentric encoding -behavoosss typically associated with model- agents. While recent work suggests that temporal credit assignment and environment interaction can give rise to planning- like behavior in RNNs [7), our study provides the first systematic evidence that such behavior can emerge in complex, naturalistic environments without world models. Thus, internal interpretability is shaped not just by training outcomes, but by design choices that constrain promote structure.
We also find that architectural choices such as recurrence, pruning, and auxiliary losses significantly influence both performance and interpretability. Pruning improved decoding accuracy (Figure 16) while leaving performance and behavior substantively unchanged (Figure 2); the auxiliary position prediction objective enhanced predator-related behavior and spatial encoding structure Figuuee 4 Figure I1 Figure 17) while also improving performance (Figure 2). This win-win fits with observa- tions from neuroscience biological agents do not simply maximize reward, but also engage in latent path learning and other subgoals [82]. Thus, there are benefits for RL researchers to embracing this complexity of learning objectives and bridging the gaps between artificial and natural intelligence.
These findings also speak to neuroscience research on small nervous systems. Insects, for example, navigate using structured internal cues 80, 85 The fact that similar behaviors emerge in DRL agents with only a few hundred recurrent units suggests that such agents may offer useful models for evaluating the cognitive capacities of small nervous systems. Taken together with behavioral consistency across arenas, the ability single decoding model to generalize across episodes likely reflects stable, compass-like encoding of space mircorini strategies used by insects and other animals to maintain orientation using global cues such as magnetic fields or polarized light [86-88]. This encoding strategy may explain how DRL agents reliably plan navigation across environments with differing local layouts.
More broadly, we dispute the idea that DRL models are too complex for neuroscience-style analysis. The brain is least as complex- and likely more so than current DRL agents (45). If neuroscience tools cannot explain DRL agents with full behavioral and neural access, their utility for understanding the brain limited. Testing these tools in complex simulation environments also addresses practical challenges in naturalistic neuroscience, including low statistical power, high data costs, and limited experimental control [89]. By generating high-volume, ecologically grounded datasets with joint ceural-behavin logging, we provide a platform for evaluating brain analysis techniques. Our diagnostic framework TTblle I) offers a guide for interpreting agent behavior and internal dynamics across diverse RL tasks. We hope this framework serves both as a roadmap for future neuroAl work and as a foundation for more interpretable RL evaluation.
Limitations & Broader Impacts The 96x96 grid size limit in Craftax prevents us from evaluating how our results generalize to even larger arenas. A second limitation is that we could not log the precise locations of all trees, rocks, and lakes, which limited our ability to study landmark-based navigation in detail. A third limitation is that incorporating a wider range of RL architectures with full behavioral neurall logging is currently non-trivial in this environment, which why we focused on PPO- and PQN-based based models with shared components. We believe this work is likely to have a positive societal impact by fostering closer collaboration between neuroscientists, cognitive scientists, ethologists, and RL researchers in the analysis of deep RL agents advancing all fields through shared insight into the structure and function of learned behavior.
10

---

Acknowledgments
We thank Albert Lee and members of the Rajan Lab for the helpful comments and feedback during the course of the project. We also thank the research assistants Yang Wu and Yunxin Wu for their help during revisions. R.S., R.P.B. and K.R. were collectively funded by the NIH (RFIDA056400 UOINSL36507), James S. McDonnell Foundation (220020466), Simons Foundation (Pilot Extension- 00003332-02, McKnight Endowment Fund, CIFAR Azrieli Global Scholar Program, NSF (2046583), Harvard Medical School Neurobiology Lefler Small Grant Award, Harvard Medical School Dean's Innovation Award, and Harvard Medical School Leonard and Isabelle Goldenson Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence Harvard University, F.B.B. received support from Fulbright Program, Jansons Legat, and the Norwegian State Educational Loan Fund.
References
[1] OEL Team, Adam Stooke, Anuj Mahajan, Catarina Barros, Charlie Deck, Jakob Bauer, Jakub Sygnowski, Maja Trebacz, Max Jaderberg. Michael Mathieu, et al. Open-ended learning leads to generally capable agents. arXiv preprint arXiv:2 12808, 2021.
[2] John W Krakauer, Asif A Ghazanfar, Alex Gomez-Marin Malcolm A MacIver, and David Poeppel. Neuroscience needs behavior: correcting a reductionist bias. Neuron, 93(3):480-490, 2017.
[3] Yael Niv. The primacy of behavioral research for understanding the brain. Behavioral Neuroscience, 135(5):601, 2021.
[4] Thomas Wolbers and Jan M Wiener. Challenges for identifying the neural mechanisms that support spatial navigation: the impact of spatial scale. Frontiers in human neuroscience, 8: 571, 2014.
[5] Russell Epstein, Eva Zita Patai, Joshua B Julian, and Hugo Spiers. The cognitive map in humans: spatial navigation and beyond. Nature neuroscience, 20(11):1504-1513 2017.
[6] Arne D Ekstrom, Derek Huffman, and Michael Starrett. Interacting networks of brain regions underlie human spatial navigation: review and novel synthesis of the literature. Journal of neurophysiology, 118(6))3328-35 3344, 2017.
[7] Kai Zhu and Tao Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674-691, 2021.
[8] Eloy Parra-Barrero, Sandhiya Vijayabaskaran, Eddie Seabrook, Laurenz Wiskott, and Sen Cheng. map of spatial navigation for neuroscience. Neuroscience & Biobehavioral Reviews, 152:105200, 2023.
[9] Tamir Eliav, Shir R. Maimon, Ayelet Sarel, Shaked Palgi, Liora Las, and Nachum Ulanovsky. Fragmented replay of very large environments in the hippocampus of bats. Cell, 2025. doi: 10. (0166).ce1.2222..5.022
[10] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-538 2015.
[11] Maxime Sievaller-Boisvert Bolun Dai, Mark Towers, Rodrigo Perez-Vicents Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & Miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. Ad- vances in Neural Information Processing Systems, 36:73383-73394, 2023.
[12] Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning. 2024.
/2w.0/ 16801
11

---

[13] Stephen Chung, Ivan Anokhin, and David Krueger. Thinker: Learning to plan and act. Advances in Neural Information Processing Systems, 11:22896-229333 2023.
[14] Christopher R Fetsch. The importance of task design and behavioral control for understanding the neural basis of cognitive functions. Current opinion in neurobioloyy. 37:16- 22, 2016.
[15] Zipeng Fu, Ashish Kumar, Jitendra Malik, and Deepak Pathak. Minimizing energy con- sumption leads the emergence of gaits in legged robots. arXiv preprint arXiv:2111.01674, 2021.
[16] Nils Kolling, Marco Wittmann, and Matthew FS Rushworth. Multiple neural mechanisms of decision making and their competition under changing risk pressure. Newron, 2014.
[17] Alexander Lai, German Espinosa, Gabrielle E Wink, Christopher F Angeloni, Daniel A Dombeck, and Malcolm A Maclver. A robot-rodent interaction arena with adjustable spatial complexity for ethologically relevant behavioral studies. Cell reports, 43(2), 2024.
[18] Sam Halt-MeMaster and Fabrice Luyckx. Revisiting foraging approaches in neuroscience. Cognitive, Affective, & Behavioral Neuroscience, 19(2):225 230, 2019.
[19] Juliano Morimoto. Foraging decisions as multi-armed bandit problems: Applying reinforce- ment learning algorithms to foraging data. Journal of theoretical biology, 467:48-555 56, 2019.
[20] Charlotte N Boccara, Michele Nardin, Federico Stella, Joseph O' d'eeil,, and Jozsef Csicsvari. The entorhinal cognitive map is attracted to goals. Science, 963(6434):14433144 2019.
[21] Ryan Paul Badman, Thomas Trenholm Hills, and Rei Akaishi. Multiscale computation and dynamic attention in biological and artificial intelligence. Brain sciences, 10(6):396, 2020.
[22] Wan-Chen Jiang, Shengjin Xu, and Joshua T Dudman. Hippocampal representations of foraging trajectories depend upon spatial context. Nature neuroscience, 25(12):1693-1705. 2022.
[23] Laura L Grima, Hannah Haberkern, Rishika Mohanta, Mai M Morimoto, Adithya E Ra- jagopalan, and Emma V Scholey. Foraging as an ethological framework for neuroscience. Trends Neurosciences. 2025.
[24] Alan T. Winfield. Towards an Engineering Science of Robot Foraging, pages 185-192. Springer Berlin Heidelberg, Berlin, Heidelberg, 2009, ISBN 977-3-642--00444-3 doi: 10. 1007/ 16. URL /attps::/doi..rgg// 1007
[25] Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing Systems, 15:18333-1836, 2022.
[26] Richard S Sutton, Andrew G Barto, et al. Reinforcement learning. Journal of Cognitive Neuroscience, 11(1): 1999.
[27] Charles Packer, Katelyn Gao, Jemej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 2018.
[28] Karl Cobbe, Oleg Klimov, Chris Hesse, Tachoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In International conference on machine learning, pages 1282-12 1289. PMLR, 2019.
[29] Minqi Jiang. Tim Rocktäschel, and Edward Grefenstette. General intelligence requires rethink- ing exploration. Royal Society Open Science, 10(6):230539, 2023.
[30] Hua Wei, Jingxiao Chen, Xiyang Ji, Hongyang Qin, Minwen Deng, Siqin Li, Liang Wang, Weinan Zhang, Yong Yu, Liu Linc, et al. Honor of Kings arena: an environment for general- ization in competitive reinforcement learning. Advances Neural Information Processing Systems, 35:11881-1 11882 2022.
12

---

[31] Emma Stensby Norstein, Kai Olav Ellefsen, and Kynre Glette. Open-ender search for environ- ments and adapted agents using map-elites. International Conference on the Applications of Evolutionary Computation (Purt of EvoStar), pages 651-666 Springer, 2022.
[32] Shaofei Cai, Zihao Wang. Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi- task control through goal-aware representation learning and adaptive horizon prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13734-13744 2023.
[33] Mikayel Samvelyan, Akbir Khan, Michael Dennis, Minqi Jiang, Jack Parker-Holder Jakob Foerster, Roberta Raileanu, and Tim Rocktäschel. Maestro: Open- environment design for multi-agent reinforcement learning, arXiv preprint arXiv:2303.033: 2023.
[34] Chi Zhang, Penglin Cai, Yuhui Fu, Haoqi Yuan, and Zongqing Lu. Creative agents: Empower- ing agents with imagination for creative tasks. arXiv preprint arXiv:2312.025519 2023.
[35] Tristan Tomilin, Meng Fang. Yudi Zhang, and Mykola Pechenizkiy. COOM: A game bench- mark for continual reinforcement learning. Advances in Neural Information Processing Systems, 36:67794-6788 2023.
[36] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Skill reinforcement learning and planning for open-wor world long-horizon tasks. arXiv preprint arXiv:2303. 16563, 2023.
[37] Karolis Jucys, George Adamopoulos, Mehrab Hamidi, Stephanie Milani, Mohammad Reza Samsami, Artem Zholus, Sonia Joseph, Blake Richards, Irina Rish, and Özgür Şimşek. Interpretability in action: Exploratory analysis of VPT, Minecraft agent, 2024. URL
https://arxiv. org/ waarg/bs//2/ee/ abs/2407 12161
[38] Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Artem Agarkov, Viacheslav Sinii, and Sergey Kolesnikov. Xland-minigrid Scalable meta-reinforcement learning environments in jax. Advances in Neural Information Processing Systems, 97:43809-43835, 2024.
[39] Guy Davidson, Graham Todd, Julian Togelius, Todd M Gureckis, and Brenden Lake. Goals as neward-produeee programs. Nature Machine Intelligence, 7(2):205-22) 2200, 2025.
[40] Toby Wise, Kara Emery, and Angela Radulescu. Naturalistic reinforcement learning. Trends in Cognitive Sciences, 28(2):144-155 158. 2024.
[41] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj. Rishi Rajalingham, Elias B Issa, Kohitij Kar. Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[42] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22(11::1611-7700 2019.
[43] Martin Schrimpf, Jonas Kubilius, Michael Lee, N Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413-42) 2020.
[44] Soma Nonaka, Kei Majima, Shuntaro C Aoki, and Yukiyasu Kamitani. Brain hierarchy score: Which deep neural networks are hierarchically brain-like? IScience, 24(9), 2021.
[45] Andrew Saxe, Stephanie Nelli, and Christopher Summerfield. If deep learning is the answer, what is the question? Nature Reviews Nearoscience, 22(1):55-67, 2021.
[46] Mitchell Ostrow, Adam Eisen, Leo Kozachkov, and Ila Fiete. Beyond geometry: Comparing the temporal structure of computation in neural eircuits with dynamical similarity analysis. Advances in Neural Information Processing Systems, 36:33824-33888 2023.
13

---

[47] Ida Momennejad, Hosein Hasanbeig, Felipe Vieira Frujeri, Hiteshi Sharma, Nebojsa Jojic, Hamid Palangi, Robert Ness, and Jonathan Larson. Evaluating cognitive maps and planning in large language models with CogEval. Advances in Neural Information Processing Systems, 36:69736-6975,, 2023.
[48] Ansh Soni, Sudhanshu Srivastava, Konrad Kording, and Meenakshi Khosla. Conclusions about neural network to brain alignment are profoundly impacted by the similarity measure. bioRxiv, pages 2024-08, 2024.
[49] Han Wang, Erfan Miahi, Martha White, Marlos C Machado, Zaheer Abbas, Raksha Ku- maraswamy, Vincent Liu, and Adam White. Investigating the properties neural network representations in reinforcement learning, Artificial Intelligence, 330: 130:104100, 2024.
[50] Wilka Carvalho, Momchil Tomov, William de Cothi, Caswell Barry, and Samuel J Gershman. Predictive representations: Building blocks of intelligence. Neural Computation, 36(11): 2225-2298, 2024.
[51] Demet Yesiltepe, P Fernández Velasco, Antoine Coutrot, A Ozbil Torun, Jan M Wiener, Christoph Holscher, Michael Hornberger, Conroy Dalton, and Hugo J Spiers. Entropy and a sub-group of geometric measures of paths predict the navigability of an environment. Cognition, 236:105443, 2023.
[52] Jaedong Hwang, Zhang- Wei Hong, Eric Chen, Akhilan Boopathy. Pulkit Agrawal, and Ila Fiete. Grid cell-inspired fragmentation and recall for efficient map building. arXiv preprint arXiv:2307.0523 37.09999..7...... 2023.
[53] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embed- dings for joint behavioural and neural analysis. Nature, 6 617(7960):360-366. 2023.
[54] Drew H Abney, Caitlin M Fausey, Catalina Guarez-Rivera and Catherine S Tamis-LeMonne LeMonda. Advancing temporal science of behavior. Trends in Cognitive Sciences, 2025.
[55] Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, and Swarat Chaud- huri. Programmatically interpretable reinforcement learning. In International conference machine learning, pages 5045-5054. PMLR, 2018
[56] Zelei Cheng, Jiahao Yu, and Xinyu Xing A survey on explainable deep reinforcement learning. arXiv preprint arXiv:2502.06869, 2025.
[57] Jerry Hirsch. Nothing in neurobiology makes sense-cxcept in the light of behaviour. Contem- porary Psychology, 1986.
[58] Eric Jonas and Konrad Paul Kording. Could a neuroscientist understand microprocessor? PLoS computational biology, 13(1)ee0052668 2017.
[59] Robert Geirhos, Roland S Zimmermann, Blair Bilodeau, Wieland Brendel, and Been Kim. Don't trust your eyes: on the (un) reliability of feature visualizations. arXiv preprins arXiv:2306.0477 2023.
[60] Drew Linsley, Pinyuan Feng, and Thomas Serre. Better artificial intelligence does not mean better models of biology, 2025. https:t:://aaxi org/ htsp//bb/25004 16940
[61] Tianyu Cui, Yogesh Kumar, Pekka Marttinen, and Samuel Kaski. Deconfounded representation similarity for comparison of neural networks. Advances in Neural Information Processing Systems, 35:19138-19151, 2022.
[62] Rosa Cao and Daniel Yamins. Explanatory models in neuroscience, part 2: Functional intelligibility and the contravariance principle. Cognitive Systems Research, 85:101200 2024.
[63] Joo Hyung Lee, Wonpyo Park, Nicole Elyse Mitchell, Jonathan Pilault, Johan Samir Obando Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, et al. Jaxpruner: concise library for sparsity research. In Conference on Parsimony and Learning, pages 515-528. PMLR, 2024.
14

---

[64] Anna D Broido and Aaron Clauset. Scale-free networks are rare. Nature communications, 10 (1):1017, 2019.
[65] Xin-Jie Zhang, Jack Murdoch Moore, Gang Yan, and Xiang Li. Universal structural patterns sparse recurrent neural networks. Communications Physics, 6(1): 2023.
[66] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations (ICLR), 2019.
[67] Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32, pages 14014-14024, 2019.
[68] Neehal Tumma, Mathias Lechner, Noel Loo, Ramin Hasani, and Daniela Rus. Leveraging low-rank and sparse recurrent connectivity for robust closed-loop control. arXiv preprint arXiv:2310.339/5, 2023.
[69] Benjamin Y Hayden. Economic choice: the foraging perspective. Current Opinion in Behavioral Sciences, 24:1-6, 2018.
[70] Thomas T Hills. Animal foraging and the evolution of goal-directed cognition. Cognitive science, 30(1):3-41 2006.
[71] Matthew Rosenberg, Tony Zhang, Pietro Perona, and Markus Meister. Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration. Elife, 10:e66175, 2021.
[72] P-Y Oudeyer, Jacqueline Gottlieb, and Manuel Lopes. Intrinsic motivation, curiosity, and learning: Theory and applications in educational technologies. Progress in brain research, 229:257-284, 2016.
[73] Jennifer Bussell, Ryan P Badman, Christian D Márton, Ethan S Bromberg-Mart LF Abbott, Kanaka Rajan, and Richard Axel. Representations of information value in mouse orbitofrontal cortex during information seeking. bioRxiv, pages 2023-10, 2023.
[74] Ily Monosov. Curiosity: primate neural circuits for novelty and information seeking. Nature Reviews Neuroscience, 25(3):195-208, 2024.
[75] Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. arXiv preprint arXiv:2407.0483 2024.
[76] Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can strong baseline for many POMDPs. arXiv preprint 1rXiv:2110.05033: 2021.
[77] Conor F Hayes, Roxana Rădulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfar- lane, Mathieu Reymond, Timothy Verstraeten, Luisa M Zintgraf, Richard Dazeley, Fredrik Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022.
[78] Stephen Chung. Scott Nickum, and David Krueger. Predicting future actions of reinforcement learning agents. arXiv preprint arXiv:2410.22453 2024.
[79] Rüdiger Wehner and Mandyam V Srinivasan. Searching behaviour of desert ants, genus cataglyphis (formicidae, hymenoptera). Journal of comparative physiology, 142:315-338, 1981.
[80] Lars Chittka. The mind of a bee. Princeton University Press, 2022.
[81] Marilia Freire, Antonio Bollig, and Markus Knaden. Absence of visual cues motivates desert ants to build their own landmarks. Current Biology, 33(13)::802-22805 2023.
[82] Leo Clement, Sebastian Schwarz, and Antoine Wystrach. Latent learning without map-like representation of space in navigating ants. bioRxiv, pages 2024-08, 2024.
15

---

[83] Thomas Bush, Stephen Chung, Usman Anwar, Adrià Gariiga-Aonno, and David Krueger. Interpreting emergent planning in model-free reinforcement learning. arXiv preprint arXiv:250401877, 2025.
[84] Antoine Dedieu, Joseph Ortiz, Xinghua Lou, Carter Wendelken, Wolfgang Lehrach, J Swaroop Guntupalli, Miguel Lazaro-Gredille and Kevin Patrick Murphy. Improving transformer world models for data-efficient RI.. arXiv preprint arXiv:2502.00 0591, 2025.
[85] Sercan Sayin, Einat Couzin-Fuchs, Inga Petelski, Yannick Günzel, Mohammad Salahshour, Chi-Yu Lee, Jacob M. Graving, Liang Li, Oliver Deussen, Gregory A. Sword, and Iain D. Couzin. The behavioral mechanisms governing collective motion in swarming locusts. Science, 337(6737):995-1000, 2025. doi: 10. a222
[86] Rüdiger Wehner. Polarized-light navigation by insects. Scientific American, 235(():106-115, 1976.
[87] Roswitha Wiltschko. Magnetic orientation in animals, volume 33. Springer Science & Business Media, 2012.
[88] Stanley Heinze. Polarized- -ightt processing in insect brains: recent insights from the desert locust, the monarch butterfly, the cricket, and the fruit fly. Polarized light and polarization vision in animal sciences, pages 61-111, 2014.
[89] Samuel A Nastase, Ariel Goldstein, and Uri Hasson. Keep it real: rethinking the primacy of experimental control in cognitive neuroscience. Neurolmage, 222:1 222:117254, 2020.
[90] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1xXiv:7777003444 2017.
[91] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv: 1412.3555, 2014.
[92] Joel Z Leibo, Edward Hughes, Mare Lanctot, and Thore Graepel. Autocurrieula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research. arXiv preprint arXiv:1903.00 000042 2019.
[93] Minqi Jiang, Michael Dennis, Edward Grefenstette, and Tim Rocktäschel. minimax: Efficient baselines for autocurrieula in jax. arXiv preprint arXiv:2311.122 127.1171,, 2023.
[94] Yanyuan Qiao, Wenqi Lyu, Hui Wang, Zixu Wang. Zerui Li, Yuan Zhang, Mingkui Tan, and Qi Wu. Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source LLMs. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 6710-67 1710-6717. IEEE, 2025.
[95] Joshua A Cullen, Caroline L Poli, Robert J Fletcher Jr, and Denis Valle. Identifying latent behavioural states in animal movement with m4, a nonparametric bayesian method. Methods in Ecology and Evolution, 13(2):432-44) 0444332-444444 2022.
[96] Torsten Hothorn, Kurt Hornik, and Achim Zeileis. ctree: Conditional inference trees. The comprehensive R archive network, 8:1-34, 2015.
[97] Juliet Osborne, Alan Smith, Suzanne J Clark, Don Reynolds, Mandy Barron, Ka S Lim, and Andy M Reynolds. The ontogeny of bumblebee flight trajectories: from naïve explorers to experienced foragers. Plos one, 8(1 6(11))e78681, 2013.
[98] Edoardo Balzani, William Broderick, Guillaume Viejo, and Alex Williams. NEural MOdelS, statistical modeling framework for neuroscience. https///giiabbboom/faiioiittttee 2025.
[99] Charles Vorhees and Michael Williams. Assessing spatial learning and memory in rodents. ILAR journal, 55(2): 2014.
16

---

[100] Edvard I Moser, Yasser Roudi, Menno P Witter, Clifford Kentros, Tobias Bonhoeffer, and May-Britt Moser. Grid cells and cortical representation. Nature Reviews Neuroscience, 15(7): 466 481, 2014.
[101] Timothy EJ Behrens, Timothy H Muller, James CR Whittington, Shirley Mark. Alon B Baram, Kimberly L Stachenfeld, and Zeb Kurth-Nelson. What a cognitive map? organizing knowledge for flexible behavior. Neuron, 100(2)4400500) 2018.
[102] Peter M Todd and Thomas T Hills. Foraging in mind. Current Directions in Psychological Science, 29(3):309-315, 2020.
[103] Luca Ambrogioni and H Freyja Ólafsdóttir, Rethinking the hippocampal cognitive map as meta-learning computational module. Trends in Cognitive Sciences, 27($):702-712, 2023.
[104] Payam Piray and Nathaniel Douglass Daw. Reconeiling flexibility and efficiency: Medial entorhinal cortex represents compositional cognitive map. bioRxiv, pages 2024-05. 05, 2024.
[105] Jozsef Csicsvari, Joseph O' O'Neill, Kevin Allen, and Timothy Senior. Place-selective firing contributes to the reverse-order reactivation cal pyramidal cells during sharp waves in open-field exploration. European Journal of Neuroscience, 26(3):704-716, 2007.
[106] H Freyja Ólafsdóttir, Daniel Bush, and Caswell Burry. The role hippocampal replay in memory and planning. Current Biology, 28( 11))3333-,50 2018.
[107] Eric Denovellis, Anna K Gillespie, Michael E Coulter, Marielena Sosa, Jason E Chung, Uri T Eden, and Loren M Frank. Hippocampal replay of experience at real-world speeds. Elife, 10:e64505, 2021.
[108] Payam Piray and Nathaniel Daw. Linear reinforcement learning in planning, grid fields, and cognitive control. Nature communications, 12(1):4942, 2021.
[109] Kiah Hardcastle, Surya Ganguli, and Lisa M Giocomo. Environmental boundaries as an error correction mechanism for grid cells. Neuron, 86(3):827-839, 2015.
[110] Kiah Hardcastle, Niru Maheswaranathan, Surya Ganguli, and Lisa M Giocomo. A multiplexed, heterogeneous, and adaptive code for navigation in medial entorhinal cortex. Neuron, 94(2): 375-387, 2017.
[111] Marielena Sosa, Mark H Plitt, and Lisa M Giocomo. Hippocampal sequences span experience relative rewards. bioRxiv, 2023.
[112] James CR Whittington, Joseph Warren, and Timothy EJ Behrens. Relating transformers mod- els and neural representations of the hippocampal formation. arXiv preprint arXiv:2112.040355 2021.
[113] Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexander Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705): 429 433, 2018.
17

---

7 Appendix
This Appendix provides additional details on the ForageWorld environment and agent training setup, supplementing the descriptions subsection 3.1] and subseection 3.2 of the main text. includes key environment parameters, procedural generation constraints, agent sensory imput, and logging infrastructure used for downstream analysis. We also provide supplementary analyses and figures that expand the main results, including trajectory examples, revisitation models, training dynamics, ablations, and neural decoding structure.
A Forage World: Task and Training Details
We designed ForageWord World to balance between task complexity, transparency to analysis, training efficiency (using JAX GPU acceleration), biological plausibility navigational planning computa- tions, and connection to joint RL and recurrent memory architectures that are of particular interest to neuroscientists. More complex deep RL tasks exist, but many such tasks (for example, Minecraft) are much more difficult to study and in particular more difficult to instrument for behavior analysis. In addition, the increased computational cost of experiments in such environments is prohibitive to some statistical analysis, and many computational neuroscientists in particular do not have access to compute resources sufficient to run experiments such complex environments. Relative to common neuroscience tasks used to study foraging, e.g. n-armed bandit tasks which lack spatial or temporal components, ForageWorld pushes the complexity boundary on the neuro-side of neuroAl while still remaining tractable for neuroscience tools and neuroscientists to study. Further, ForageWorld as exists is not trivial to learn- our agents require billions of timesteps of experience to train to competency the task, and Craftax, the benchmark which build upon to develop ForageWorld has not been solved by any agent or architecture to date.
In the following section, we expand here the ForageWorld environment and agent training setup described in isubsection 3. and (subsection 3.2 of the main text. This includes key environment parameters, procedural generation constraints, agent sensory input, and logging infrastructure used for downstream analysis.
A.1 Procedural Arena Generation
Each arena is procedurally generated at the start of episode using fixed-size 96x96 grid. Spawn points for cows (food) and lakes (drink) are initialized in random patches using Perlin noise, providing episodic variability alongside naturalistic structure. Obstacles are added to break line-of-sight and create navigational constraints. The agent begins at the center of the arena.
A.2 Episode Structure and Termination Conditions
Each episode runs for maximum of 100,000 timesteps or ends earlier if the agent's health drops to zero- is the typical outcome, even for highly competent agents. Agents carn reward by maintaining internal homeostasis baallani hunger. thirst, and fatigue-and and avoiding predator encounters. The reward function is defined subsecttionn... 3.I
A.3 Sensory Input and State Representation
At each timestep, the agent receives 9x 11 grid-centered egocentric view of the environment (see Figure T). along with an inventory vector containing current health, satiation levels (food and water), fatigue, and any collected items. This full observation is encoded by feedforward neural net layer before being passed to the recurrent network.
A.4 Logging Infrastructure
To support joint behavioral and neural analysis, log wide range of signals each timestep, including the full recurrent hidden state, internal reward components, movement trajectories, environ- mental state, and agent decisions. The complete logging schema shown in Taall These logs enable replay, ablation, and representational analysis across multiple analytic pipelines.
18

---

A.5
Training Hyperparameters
Agents were trained using Proximal Policy Optimization (PPO) [90] with Generalized Advantage Estimation (GAE). Each agent trained for billion timesteps, resulting in a variable number of episodes-ttpicalll in the hundreds of wrinsannddddeeddinng Episodes were capped at 100,000 timesteps, though this cap was rarely reached in practice. The PPO rollout horizon was 64, with a minibatch size of 8192 and learning rate of 0.00025. We used the Adam optimizer with default momentum parameters. Additional hyperparameters are provided in Table 3 Policy and value losses were weighted using standard PPO coefficients (see Table 3). For stability, gradient norms were clipped to 1.0.
competence and learning speed.
Each training run used single GPU (NVIDIA A100 or H100) and typically converged within 24-48 hours, depending on configuration. While memory usage varied by configuration, most experiments were feasible on GPUs with at least 24 GB of memory. Logging occurred every 2048 PPO iterations (roughly every 134 million timesteps), and the perforerfonnng checkpoints were selected based on held-out arena performance.
Together, these settings define procedurally rich, ecologically grounded RL training setup that supports the emergence of complex behaviors and structured internal representations, The design balances task realism, computational feasibility, and interpretability the behavioral and neural analyses presented in this work.
A.6 Model Architecture Details for PPO-RNN
Our primary architecture combined Proximal Policy Optimization (PPO) [0] with a recurrent neural network, using Generalized Advantage Estimation (GAE) for stability. PPO optimized a clipped
surrogate objective that balanced advantage maximization with policy stability:
Lelip (0) E.
[ min (re(0) A., clip(r.(0), 1- c, 1 +) A.)]
(3)
where r0)
To (ax (ax) is the probability ratio between current and previous policies, and Å, is Ne.. [+e]+e]
the GAE-computed advantage. The full PPO loss also includes value and entropy terms, following standard practice.
To handle partial observability in the task, the agent uses a gated recurrent unit (GRU) 91 to build memory over time. At each timestep I, the GRU took the current observation or and previous hidden state h producing an updated hidden state ht Both the policy Taa I oss)) and the value estimate V(oct) are computed from h, via separate output heads. This architecture is shared between the actor and critic networks. We used a single-layer GRU with 512 hidden units. GRUs offer a gating mechanism that supports long-range memory retention while being more computationally efficient than LSTMs. This advantage makes them especially well suited for tasks involving memory and interpretability. We followed the implementation provided in the Craftax benchmark [12], which demonstrated that memory-equipped PPO agents perform substantially better on partially observed, long-horizon tasks. In total, our standard configuration contains approximately 6.5 million trainable parameters.
A.7 Future directions in Forage World
The Craftax platform also provides an opportunity for future work yet explored here: incorporating curriculum learning into Forage World and related environments [92 93 The staged emergence
of behavioral competencies we observed (e.g., in Figure 4) suggests that ForageWorld naturally supports curriculum-basee training regimes and Craftax is compatible with JAX curriculum learning
packages. We hope future work will explore curriculum learning [92 [3 Future studies could leverage this structure- and the motif- based analysis framework to probe how learning history shapes internal representations, and how staged training affects generalization, revisitation strategies, and neural dynamics. Along this direction, it would also be interesting to test LLM models' foraging behavior, as navigation is an underexplored topic LLM literature generally (94 (though note that transformers and LLMs would be harder to analyze than the simpler RNNs chosen here for their brain-like recurrent activity). However, Craftax is currently not coded for language- labels of state variables so additional modification would have to be done to support LLM tests.
19

---

Lastly, note that Craftax and ForageWorle do not support multi- environments yet, but this would be an interesting modification explore.
B Interpreting Agent Behavior: Supplementary Figures and Analysis
This section provides supplementary behavioral analyses and supporting visualizations. It expands on the main text results in /ectionn 4] including example trajectories, phase structure segmentation, and multi-objective revisitation strategies. We also present ablations targeting memory, auxiliary objectives, and predator combat support interpretability and robustness claims.
B.1 Structured Exploration and Revisitation Trajectories
We present representative trajectories from trained agents that illustrate the structured exploration and patch revisitation behaviors described in section 4 Figure 6Jshows trajectories from three episodes, with early exploratory loops followed by increasingly direct revisitation as the env environment becomes familiar. These examples demonstrate that agents undergo a consistent shift from broad exploration to memory-guided foraging, even in novel arenas.
B.2 Behavioral Phase Segmentation via Unsupervised Clustering
To understand how agents modulate their movement patterns across time and task demands, we applied unsupervised clustering to the agent's 's locomotion features. We used the bayesmove package identify three latent movement states based on turning angle and step size over 7-timestep moving window [95]. We next used conditional inference trees to identify which task variables best predicted transitions between these states [96].
As shown in Figure 7 the identified states correspond to interpretable movement modes: short-range (state 3), mid-range (state 1), and long-range (state 2) navigation. These were differentially associated with predator presence, eating behavior, and positional uncertainty. In particular, state frequently coincided with predator events, suggesting an evasive or scanning behavior, while state 3 tended to occur near food and was enriched for eat actions. This analysis highlights the agent's 's ability to adapt its movement patterns in response internal and external signals, without requiring explicit mode-switching logic.
B.3 Comparison Real Insect Search Patterns
To contextualize these emergent strategies. we compared agent trajectories to documented search behaviors in ants and bees. As shown in (Figure 8) both species exhibit progressively expanding search loops when navigating novel or ambiguous environments. These loops are typically centered around known location, such as a hive or nest. These patterns strongly resemble the outward- outward-spiraling. azimuthally rotating loops exhibited by our agents during early exploration in novel Forage World arenas. Although the sensory and neural mechanisms differ across biological and artificial systems, the structural similarity in their trajectories suggests that simple control policies-when shaped by partial observability, uncertainty, and survival constraints-may converge similar search dynamics.
B.4 GLM Outputs and Multicollinearity Checks
To support our analysis of patch revisitation strategies, we fitted generalized linear models (GLMs) to evaluate the influence of task variables agent decision-making. Each GLM was trained to classify whether a patch was chosen at a revisitation point, using historical variables-suce as recent reward rates, predator presence, and spatial uncertainty- input features. Each model was trained on data from multiple agents and episodes, with agent identity included as a fixed effect to account for inter-agent variability.
Figure 13 shows the full GLM outputs underlying Figure 3 as well as variance inflation factor (VIF) FiielyynnFFFuree
scores used to assess multicollineearty among the regressors. As expected, uncertainty, predator encounter rate, and recency of visitation emerged as significant predictors, while variables such as dwell time and drink rate contributed less. VIF analysis confirmed that none of the regressors exhibited problematic multicollineearty. While revisitation behavior captures spatial strategy, we also tracked how decision confidence evolved over training.
20

---

A
Pash PAP
PHD DMS
RRP
Pooch
BM
80
60
Arena Y
50
Arena Y
60
Arena Y
40
8-
T
81
Arena
40
Arena X
40
2b
20
Arena X
80
20
100
B
Path2
Dmk
100
Path
100
Denk
8
56
Arena Y
52
Arena Y
60
Arena
40
40
8-
8-
40
100
20
40
50
80
103
Arena X
Arena X
Arena
C
Path
Part 2 Puti Eat onk
Path 1-
Poth
Est
Drin
FullPsh
Patsh
Edl
Drink
50
Arena Y
8
Arena
Arena Y
60
R-
40
100
26
80
100
40
100
Arana X
Arena X
Arena X
Figure 6: Agents show stereotyped early exploration and sustained revisitation behavior novel environments. Representative trajectories from three episodes. Each row (A-C) illustrates three sequential stages from a single episode: early exploratory loops (left), a path sampled during the transition to revisitation (middle), and the full episode trajectory with revisited patch regions highlighted in orange (right). These examples demonstrate consistent exploratory patterns-sude as outwardly expanding loops-and show that exploration persists during revisitation, often aiding predator avoidance. This figure complements the summary view in Figure
B,5 Training Dynamics and Entropy Evolution
To better understand how behavior evolves over training, we tracked the average policy entropy. Figure 9 shows how decision confidence increases over time, reflected in the entropy dynamics that indicate increasing policy stability and confidence selecting goal-directed actions.
These dynamics complement the behavioral and representational changes reported in (section 4 further reinforcing the conclusion that high-level behavioral structure and planning capacity emerge progressively during training.
B.6 Memory and Predator-Related Ablations
To ensure our results were not driven by incidental dynamics or shallow heuristies, we conducted ablation experiments targeting memory mechanisms and predator response strategics. Figure 10
21

---

A
State 1
State 2
State 3
num, pwtpppasssssaaaaaea
food num gam__paallvee_eeeee nearby
rum, passives enarbey food
uncertainty
uncertainty energy drink
energy
uncertainty
drink
energy
enemy__ressee
health
enemy_ppresenl health
health
eeemy_present
0.05 0.15 Importance
Importance
Importance
B
C
100
100
State 1
State 2
State 3
Eat
Soes Predator
80
Arena Y
60
Arena Y
40
40
81
20
60 Arena X
40
80 100
20
40
60
80
100
Arena X
Figure 7: Agents exhibit distinct movement patterns aligned with task variables such as food, fatigue, and predator presence. (A) Variable importance from conditional inference trees (96 predicting transitions between three movement states identified via unsupervised clustering [95]: short-, mid- and long-range locomotion. Key predictors include hunger level (food), positional uncertainty (uncertainly), predator presence (enemy_pree perreesy_,rer,....ee and nearby cow density fnam_passine prassieesneerrby.. (B) representative episode segment colored by inferred movement state. (C) The same trajectory overlaid with predator encounters and eat events. State 1 (mid-range) frequently coincides with predator presence, while state (short-range) clusters near food and eat actions. These results suggest that DRL agents flexibly shift between movement regimes in response to internal drives and environmental witthxt-without requiring weight updates as these results were explored on test arenas with fixed weights.
shows a representative trajectory from a PPO agent without recurrence (i.e., no memory). These agents became trapped in inefficient local loops and had markedly reduced survival times, highlighting the critical role of memory in supporting long- horizon foraging.
In second ablation, ablated the auxiliary position-prediicion objective (path integration). As shown in Figure 11 this had little effect on observed movement patterns. However, significantly degraded decoding performance and eliminated structured spatial representations (see Figure 17 in the neural analysis section).
I Lastly. we tested predator-bline variant in which agents could not damage predators. As shown Figure 4 agents exhibited similar movement strategies and achieved comparable survival times, indicating that predator evasion-an not combat-is the primary learned strategy.
22

---

A
10
-15
B
Key
A
Indiridual flight colony (sadar fixes joined black line).
Field boundaries
Hedgerows area
Patches flowering plants providing nector and pollen
Tall maize crop could cause some obstruction of radar signals
Buikings
Colory location
Radar location
meteorological stations
Location no light trp loogweed plants
B
D
TN
Figure 8: Agent exploration trajectories qualitatively resemble real-world search patterns in ants and bees. (A) Adapted from [79] ant's search trajectory following nest displacement, showing outwardly expanding loops over time. (B) Adapted from [97] bee trajectories during their first, second, and third departures from the
In both cases, the animals exhibit looping paths that rotate azimuthally and expand in spatial scale-closely matching the trajectories observed in DRL agents during early foraging. These parallels suggest that similar search strategies may emerge across biological and artificial systems when operating under uncertainty and partial observability.
C Interpreting Agent Representations: Supplementary Analyses
This section presents supplementary analyses and figures that support the neural decoding results in Isection 4, We examine how pruning and auxiliary objectives influence spatial encoding, and provide additional insight into the structure and modularity of the learned internal representations.
23

---

Entropy Ave.
Log P Ave.
4-
Mean Value
-20
Laprana
3-
-1.
-10
-05
0.0
25 50
75
100
125
0
25
50
75 100 125
Ep. Number (1000x)
Ep. Number (1000x)
Figure 9; Entropy dynamics reveal transition from undirected exploration confident, goal- directed behavior. Evolution of average entropy over training for PPO-RNN agents. As training progresses, entropy declines and the log probability of the selected actions increases, reflecting increased confidence and policy refinement. Entropy slowly increases later in training as the agent discovers ways to maximize both task performance and policy entropy, reflective of the loss function.
Path
Path 2
Path
Eat
Drink
100
80
Arena Y
60
40
20
20
60 Arena X
40
80
100
Figure 10: Agents without recurrent memory exhibit poor spatial coverage and reduced survival. Trajectory from a PPO agent without recurrent architecture. Memoryless agents often became trapped in local loops and survived significantly fewer timesteps than agents with recurrence.
24

---

Spatial Unc. Distance
Distance Traveled
Spatial Entropy
Angular Variance
0.35
0.7
Eary Epicode
2.5
Early in Episodl
0.30
40
0.6
Episade
2.0
Mean Value
0.20
1.5
0:1
20-
1.0
0.5
0.00
0.1
25
125
100
Pred. Enc. Rote
Pred. Kill Rate
Tool Make Rate
Food & Drink Score
2.00
175
20-
Mean Value
0.3
15-
1.00
0.75
0.2
0.00
Drink Scora
25
50
75
300
125
25
50
75
100
125
25
50
75
100
125
100 Ep. Number (1000x)
25
50
75
125
Ep. Number (1000x)
Ep. Number (1000x)
Ep. Number (1000x)
Figure 11: Disabling the auxiliary position-predictto objective limits the predator killing abilities of agents. Training dynamics and movement patterns for PPO-RNN agents without the auxiliary path integration objective. The absence of structured spatial uncertainty plots (top left) due the removal of the predicted position output normally produced by the auxiliary loss, since predicted current position is not outputted by the network in this experiment. The primary difference in learning history dynamics relative to the baseline model is that without the position-prediction objective, the emergence predator killing abilities of agents is worse. This ability requires agents crafting tool, with sword being the highest damage tool, and reducing the predator health to O. Predator killing is difficult because the predators can also kill the agent.
C.1 Impact of Auxiliary Objectives on Decoding
Figure 17 shows that removing the auxiliary path integration objective eliminated above-chance decoding of allocentric position from the RNN state. This indicates that the auxiliary loss was not merely regularizer, but essential for inducing spatially interpretable internal representations.
25

---

Spatial Unc. Distance
Distance Traveled
Spatial Entropy
Angular Variance
Earry Episode
Sarly Eplsode
Late in Episode
2.5
.ate Epicode
0.25
2.0
Mean Value
0.26
30-
1.5
0.15
1.0
0.5
0.0
50
75
100
125
25
30
100
125
25
50
75
1003
1225
25
30
100
125
Pred. Enc. Rate
Pred. Kill Rate
Tool Make Rate
Food & Drink Score
20-
1.5
Mcan Value
1.0
0.2
m
10-
0.1
Fond Scate
0.0
50
101
125
50 15
JU0
12b
100
125
Ep. Number (1000x)
Ep. Number (1000x)
Ep. Number (1000x)
Number (1000x)
Figure 12: Predator evasion dominates combat in learned foraging behavior PPO-RNN agents trained without the ability to damage predators exhibited similar movement patterns and survival outcomes. This indicates that effective predator evasion-rathes than combat-is the primary strategy developed by the agent.
A
B
Comeralized Lineer Hode). Regression Results
Dep. veriable:
Patchterasis
10. obsarvations:
7575
Nodel:
Rasidill::
feature VIF
Dinomial
DE Medel
0
EatRato 1.464861
Nethodt
JRLS
Likethhood:
Se15
1
DrinkRate 1.472354
Tume:
Data::
Ved, May 2015
18:26:37
Deviance:
Peanson did2:
9. 8-85-
20019.
2
PredRate 1.164214
No. Zua Linns
Paedio Soaal (c5):
0.1281
Recency 1.031999
Covartance Typec
nascobutt
4
Dwelltime 1.462161
5
CowCount 1.259648
Intercape
005.5
0.055
0.985
0.182
6
Uncertainty 1.013469
(T
0- 0-8512
0.070
9.728
0.966
-9.057
0.180
Statistic 8.8857728296366347
1 [1
0.0125
0.8758
0.074
9.050
1.013
0.138
0.050
0.305
-0.105
0.233
6.079
0.190
p-value 1.9464488312233996---8
((
9.965
LM Statistic 4396. 4396.655986648892
Drinsfate
0.0971
0.065
LM-Test p-value 0.0
Dradilate
08.0928
8.027
-3.434
6.881
8.146
-0.848
F-Statistic 144.936613222563
Doalitinng
Serwery
-0 148R
06.0635
8 876
8 838
-28 A11
3.151
0.091
08:990
0.111
-0. 167
-8.885
F-Test p-value 8.8
CouCount
0.5386
8.032
0.080
0.468
0.591
3.761
0.886
0.045
Figure 13: GLM results confirm that patch revisitation decisions are shaped by uncertainty, re- ward history, and predator presence. (A) Full GLM output underlying Figure 3] obtained using statsmodels. formula api in Python. The model classified patches chosen (1) not (0) based on average historical variables, using agent II) as a fixed effect. A single model was fit jointly across five PPO-RNN agents, spanning 7978 revisitation decisions. (B) Variance inflation factor (VIF)
scores show no excessive multicollinearity among predictors (all VIF < 10).
26

---

Table Variables logged at each timestep to support downstream behavioral and neural analyses. These include internal state variables, agent decisions, environmental context, predictive outputs, and metadata, and are used to compute behavioral metrics and decoding analyses reported the paper.
Variable
Description
Action
Action selected by the agent
Health
Current health of the agent
Food
Current food level of the agent
Drink
Current water level of the agent
Energy
Current energy level of the agent Did the episode end?
Done
Is Sleeping
Is the agent currently sleeping? Whether the agent is in resting state (disabled in all experiments presented)
Is Resting
Player Position
Agent's ' absolute X/Y position relative to the top-left corner of the 96x96 arena (origin at 0,0)
Recover
The current state of the timer for the agent to recover a point of health The current state of the timer for the agent to lose point of food The current state of the timer for the agent to lose point of drink The current state of the timer for the agent lose point of energy The light level variable, which modifies predator spawn rates (higher when lower, i.e. at night)
Hunger
Thirst
Fatigue
Light Level
Distance Melee
LI distance the nearest melee predator there a melee predator currently on screen? LI distance to the nearest cow/food animal Is there currently screen?
Melee on Screen
Distance to Passive
Passive Screen
Distance to Ranged
L1 distance to the nearest ranged predator there a ranged predator currently on screen?
Ranged on Screen
Num Melee Nearby
How many melee predators are currently on screen?
Num Passives Nearby
How many
currently screen?
Num Ranged Nearby
How many ranged predators are currently on screen?
Delta X & Y (Relative Position)
Agent's position relative to its starting location (center arena)
Predicted Delta X Y Num Monsters Killed
Agent's 's internal prediction of its position relative to its starting location How many predators has the agent defeated this episode so far? Whether the agent has crafted sword to improve combat effectiveness this episode
Has Sword
Has Pick
Whether the agent has crafted a pickaxe to remove stone obstacle tiles
Held Iron
Number of iron resources collected (used to craft improved weapons)
Value
The agent's 's value function output
Entropy
The entropy of the agent's policy for the current timestep
Log Probability
Log probability of the selected action under the policy at the current timestep
Episode ID
Random ID for the current episode
27

---

Table 3: Model and environment hyperparameters used for the baseline configuration.
Parameter
Default Value
How Chosen (if applicable)
Model Parameters
Learning Rate
0.0002
Craftax default
0.99
ĀGAE
0.8
Same as Craftax default
PPO clipping
0.2
Same Craftax default
Wy
0.5
Same as Craftax default
Westroyy
0.01
Same as Craftax default
Activation function
tanh
Same Craftax default
Number of neurons per layer
512
Same Craftax default
Steps per PPO iteration
64
Same as Craftax default
Number parallel environments
1024
Same as Craftax default
Number of epochs per PPO itera- tion
4
Same Craftax default
Number of minibatches per epoch
8
Same as Craftax default Chosen to ensure convergence or trend visibility within 24-48 hour wall-clock training window
Total training timesteps
3,000,000,000
Waux
0.025
Parameter sweep of (0.01, 0.1, 0.025, 1.0), balancing performance and position encoding quality Only JaxPruner type that did not severely degrade performance in testing
Pruning type
Magnitude
Prune Step
20,000
Mid-training: after baseline agent begins performing reasonably, but before convergence
PQN-specific Parameters
Start
1.0
PQN default
Finish
0.05
Parameter sweep of {0.005, 0.01, 0.05, 0.1)
Decay
1.0
Parameter sweep {0.1, 1.0, 2.0)
Total training timesteps
12,000,000,000
Chosen to allow PQN time to ex- plore+converge
Total decay timesteps
6,000,000,000
Parameter sweep of (1B, 6B, 12B)
Num environment steps per update
128
PQN default
À
0.5
PQN default
Environment Parameters
Predators
True
Max Cows
108
Selected from sweep (48, 72, 108) to balance patch-leaving and revisitation in competent agents Disabled actions not relevant to ForageWorld (subset of original Craftax action space)
Use full action space
False
Map Size
96
Chosen from among (24, 48, 96) to maximize the navigation and mem- ory challenge for the agent
Directional Vision
False
28

---

A
Path
Path Path Eat
Drink
100
Path 1-3
Path 4
Eat
Drink
8-
8
Arena Y
80
Arena Y
40
40
8-
R-
Arena X
Arena X
B
100
Path
Path Path 3 Eat
Drink
8
80
Arena Y
60
Arena Y
80
40
8-
8-
20
Arena X
Arena X
C
100
Path
Path
100
Path 1-3
Path Eat Drink
8-
80
Arena Y
g0
g-
Arena
40
8-
8-
40
60
80
100
100
Arena X
Arena X
Figure 14: Front-facing agents exhibit denser local exploration in early training. Each row shows a representative carly-exlloration trajectory for agent with front-facing field of view (FOV), as opposed to the 360 egocentric view used in the bascline configuration. Front-facing agents tended to perform intermittent localized and dense exploratory behavior during the early phases of exploration.
29

---

Spatial Unc. Distance
Distance Traveled
Spatial Entropy
Angular Variance
0.40
50
0.7
Early in Episode Episode
Early Epísode
Late in Episode
0.6
0.30
Mean Value
0.25
30-
1.5
0.15
1.0
0.10
10
0.2
0.00
0.1
Pred. Enc. Rate
Pred. Kill Rate
Tool Make Rate
Food & Drink Score
2.00
Food Score
Drink Score
0.4
20-
Mean Value
0.3
1.25
15
My
1.00
0.1
0.00
25
50
75
300
125
25
50
75
1000
125
0
25
50
100
325
25
75
300
125
Ep. Number (1000x)
Ep. Number (1000x)
Ep. Number (1000x)
Number (1000x)
Front-facing FOV agents exhibit earlier onset of long-range exploration. Variant of Figure for PPO-RNN agents with a front-facing field of view. These agents showed a faster shift toward long-range exploration compared to baseline agents. This modification may serve as a useful manipulation future studies of sensory constraints.
Figure 15:
Decoding Past / Future (Ax, Ay) Relative to Origin
Decoder Performance (RMSE)
50
Sparse network
40
Dense network
Average displacement
30
20
10
-102
-10'
- -10
10
10
10²
Time Step of Position Decoded (At)
Figure 16: Pruned networks yield better allocentric position decoding than fully connected networks. Comparison of allocentric position decoding accuracy (past/future) between sparse (90%) and unpruned PPO-RNN agents. Sparse models show higher accuracy, potentially due to more compact and modular encoding task-relevant variables. Error bars show 95% confidence intervals.
30

---

Decoding Past / Future (Ax, Ay) Relative to Origin
Decoder Performance (RMSE)
50
Sparse network
40
Dense network
Not path integrating
30
Average displacement
20
10
0
-022
-10
-100
0
10 0
10
102 2
Time Step of Position Decoded (At)
Figure 17: The auxiliary position-prediction objective is necessary for spatially interpretable representations. Without auxiliary path integration loss, allocentric position decoding from the agent's 's RNN state drops to chance levels eelii--liminaiing the structured spatial encoding observed in the baseline model. Bars show decoding accuracy for past and future positions; error bars reflect 95% confidence intervals.
31

---

C.2 Coefficient Structure and Functional Modularity
To better understand how position information is distributed across neurons, we analyzed the regres- sion coefficients learned by decoders trained predict past and future positions. Figuur 18 visualizes these ridge regression weights, aligned by neuron ID, for both sparse and dense PPO-RNN agents. In both architectures, overlapping neuron populations contributed past and future decoding, but this overlap diminished at longer temporal offsets. Sparse networks exhibited clearer separation between past- and future-coding units, as well as higher overall sparsity in their weight maps-factors factors that may underlie their superior decoding performance.
A
Coefficient Profiles for Spurse Netweek
B
Coefflcient
Sparse
Put àom30 Tuaas An-00
Rödge cost waight
300
Aa-d
-200
-250
100
200
300
400
500
100
200
300
400
500
Neuroe ID
Nearon
C
D
Cocfficient Profiks for Dense Network
Cocfficient Proílles for Dense Network
400
7all Am-30
Rilge coef weight
Ridge weiggt
200
-000
300
100 200 300
400
500
Neuron ID
Figure 18: Sparse agents exhibit modular and separable encoding of past and fuurr spatial positions. Ridge regression weights for allocentric position decoding. aligned by neuron ID. (A, B) show sparse networks at 20- and 50-timestep prediction horizons; (C, D) show dense networks. While both architectures use overlapping units for past and future decoding, sparse networks show greater separation at longer horizons-consistte with their improved decoding performance.
C.3
Regularization and Train-Test Split for Decoding
Some RNN units remain inactive for entire episodes, making the decoding problem potentially ill-conditioned. Accordingly, we applied ridge regression with an lg regularization term:
Lar()) = 2(2i - all/Il
This regularization improves model stability while preserving interpretability. separate decoder was trained for each prediction offset At using the following linear model:
(he) Ahe + b. A R2x522
E ERR
We selected ridge regression for its favorable tradeoffs among interpretability. numerical stability, and computational efficiency. With time complexity O( O(N2) and direct access to regression coeffi- cients, this approach enables fine-grained analysis of how individual RNN units contribute to spatial encoding.
Train- Split Design
To prevent information leakage due to temporal continuity in ht, we employed a structured train-test split. Each decoder was trained on the first 75% of timesteps within an episode and evaluated on the remaining 25%. This design ensures that models cannot rely on short-range temporal correlations and must generalize across broader regions of feature space. All decoders were trained across multiple episodes to ensure robustness to variation in arena layouts and agent trajectories.
32

---

D Generalized linear model for predicting neural activity from current position
As supplementary analysis, we explored how position was represented at the single neu- ron level in the RNNs. NeMoS (Neural ModelS) was used for fitting generalized linear models (GLMs) that predicted neural activity from the current position of agent, see https://itthb..c cww. http://airooonsstttteeeemana neuron, but multiple episodes (arena configurations) were used for each model fit, typically 5-10 episodes minimum to avoid overtraining 96 x 96 arena was coarse- coarse-grained grained into 14 x 14 position bins, and a binary variable per bin was encoded that reported whether not the agent was in given bin (196 total regressor coefficients). Neural data was also shifted into the positive axis (lowest value zeroed), and normalized to the range of 0-1 per neuron to extract the activity changes relative to given neuron's baseline for easier cross-neuron comparison.
for extended documentation [98]. One GLM fit per
arena configuration. Prior to setting up the model, the
The GLM fit the first 70% of each episode in the training set of episodes Figure 19, The majority of neurons fit had comparable performance between the test and training sets Figure 20
Example train/test split for one episode (multiple episodes are stitched together during GLM fitting)
EpiD
40
20
train
40
60
test
Example test/train split for single episode to be used in GLM fitring. GLM models, one model per neuron, with 5-10 episodes per model, predicted neural activity from position. The first 70% of an episode trained, while the last 30% the test split.
Figure 19:
Approximately 100/512 neurons had good fit performance using just position regressors across runs, with values typically in the range of 60-120 position-sensitive neurons per training run. An example good fit is shown Figure 21
We then plotted the number of neurons with GLM coefficients in the three largest mean-valued k-means clusters out of five clusters, FeeuF for for the selection. We also plotted the average GLM coefficient per position bin Figure 23] The number of strongly position encoding neurons increased with radial distance from the origin of the arena, and the average GLM coefficient magnitude also increased with radial distance from the origin Figure 24 This trend was seen in all main conditions, PPO-RNN with and without path integration, and PQN-RNN. Thus, an accumulation circuit may ramp with distance from origin and pressure the agent to return the origin the farther away it gets, partially explaining the periodic origin revisiting behavior observed across conditions. This pattern was present without the position predicting auxiliary objective Figure 25 and Figure 26 and also in the PQN-RNI runs Figure 27 and Figure 28, suggesting it fundamental to the task solution. Example scripts to this analysis method are in the github repository for this paper.
33

---

Test -ogliikllhood
-0.2
-0.4
-0.6
-0.8
-1.0
-1.2
-0.6
0.0
Training log-likelihood
Figure 20: Log likelihood scores for the same GLM model (single neuron fit), showing the difference in log likelihood between test and train data, one model per data point. The data here is a PPO-RNN baseline run.
Ep1
3
12
0.8
Rezeroed Activity
20000
40000
60000
80000
100000
120000
Time
single neuron (train) single (train)
Example of single neuron, multi-episode model fit. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each 196 position bins (14 x 14) that the arena grids coarse-grain binned into for this analysis. We show example good fit for single neuron's data (black), fit to 12 different episodes for one model (fit red).
Figure 21:
34

---

Number of Pos. Sensitive Neurons Per Bin
0
0
65
12
22
14
31
49
51
44
42
37
39
35
70
14
30
35
25
44
45
44
25
14
13
49
42
52
77
64
31
39
16
18
8
39
18
21
12
29
30
60
16
29
24
26
22
26
39
19
19
24
46
50
39
34
16
16
22
13
14
7
12
23
16
45
14
23
9
29
17
10
22
17
60
13
46
47
44
Bin
40
30
15
9
15
16
17
22
22
28
25
16
34
22
27
18
45
28
24
26
11
13
20
40
19
32
30
17
10
26
14
35
12
10
17
5
15
16
36
52
53
34
46
21
36
24
15
14
33
43
40
78
20
44
24
8
18
54
36
27
14
35
32
38
52
67
28
32
42
21
18
42
28
16
44
10
42
41
25
45
25
27
59
37
60
3
5
8
10
11
12
X Bin
Figure 22: Number of position-sensitivv neurons per position bin in the arena for PPO-RNN. Each of the 512 GLM models fit, per neuron, has GLM coefficient for each of 196 position bins (14 14) that the arena grids were coarse-grand grain binned into for this analysis. The heatmap of the number of well-fit neurons per position bin is shown in the plot. The data here is PPO-RNN baseline run. The O's in the uppermost row and rightmost column of bins are due to the agents never going out to those position bins in these runs.
Original matrix ave. GLM coeff
Smoothed matrix of ave. GLM coeff
6
4-
Arena BiniD
Arena BinID
10
12
Arena X BinlD
Arena X BinlD
Figure 23: Heatmap of average GLM coefficient magnitude per position for PPO-RNN. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each of 196 position bins (14 14) that the arena grids were coarse-grain binned into for this analysis. These plots show the average GLM coefficient value across well-fit neurons per bin (left), and the Gaussian smoothed shape the same heatmap (right). The data here is PPO-RNN baseline run. The 0's in the lowermost row and rightmost column bins are due to the agents never going out to those position bins in these runs.
35

---

Position Selectivity Value vs Distance Origin
Average GLM Position Coeff.
0.26
0.24
0.22
0.20
0.16
0.14
5 6
Radial Distance from Origin (# of Bins)
Figure 24: Average position bin GLM coefficient versus radial distance from the arena origin (center) for PPO-RNN. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each coarse-grain binned into for this analysis. The
of 196 position bins (14 14) that the arena grids average GL.M coefficient for the well-fit neurons per radial bin distance from the origin was plotted, finding the neural response position change increases with distance from the origin. The data here is PPO-RNN baseline run. Data points are the mean across neurons while error bars are 95% CI.
Original matrix of ave. GLM coeff
Smoothed matrix of GLM coeff
6
10
12
6
8
10
12
0
Arena Y BinlD
Arena Y BinID
12
BinlD
Arena
Figure 25: Heatmap of average GLM coefficient magnitude per position bin for PPO-RNN without path integration. Each of the 512 GLM models fit, per neuron, has one GLM coefficient for each of 196 position bins (14 x 14) that the arena grids were coarse-grand grain binned into for this analysis. These plots show the average GLM coefficient value across well-fit neurons per bin (left), and the Gaussian smoothed shape of the same heatmap (right). The data here is a PPO-RNN without ciatiion-prediction (no path integration) run. The 0's in the lowermost and rightmost column of bins are due to the agents never going out to those position bins in these runs.
36

---

Position Selectivity Value Distance From Origin
Average GLM Position Coeff.
0.24
0.22
0.18
0.16
0.14
0.12
6 Radial Distance from Origin (# of Bins)
Figure 26: Average position bin GLM coefficient versus radial distance from the arena origin (center) for PPO-RNN without path integration. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each of 196 position bins (14 x 14) that the arena grids were coarse-grain binned into for this analysis. The average GLM coefficient for the well-fit neurons per radial bin distance from the origin was plotted, finding the neural response to position change increases with distance from the origin. The data here a PPO-RNN without position- prediction (no path integration) run. Data points are the mean across neurons while error bars are 95% CI.
position-prediccion
Original matrix of ave. GLM coeff
Smoothed matrix of ave. GLM coeff
6
6
10
12
6
8
10
12
Arena Y BinlD
Arena Y BinID
10
BinlD
BinID
Figure 27: Heatmap of average GLM coefficient magnitude per position bin for PON-RNN. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each of 196 position bins (14 14) that the arena grids were coarse-grain binned into for this analysis. These plots show the average GLM coefficient value across well-fit neurons per bin (left), and the Gaussian smoothed shape of the same heatmap (right). The data here is PQN-RNN run. The 0's the lowermost row and rightmost column of bins are due to the agents never going out to those position bins in these runs.
37

---

Position Selectivity Value vs Distance From Origin
Average GLM Position Coeff.
0.13
0.12
0.10
0.09
0.08
0.07
6 Radial Distance from Origin (# of Bins)
Figure 28: Average position bin GLM coefficient versus radial distance from the arena origin (center) for PON-RNN. Each of the 512 GLM models fit, one per neuron, has one GLM coefficient for each coarse-graie -rain binned into for this analysis. The
of 196 position bins (14 x 14) that the arena grids average GLM coefficient for the well-fit neurons per radial bin distance from the origin was plotted, finding the neural response position change increases with distance from the origin. The data here is PQN-RNN run. Data points are the mean across neurons while error bars are 95% Cl.
38

---

E Alternative DRL Objective: Parallelized Q-networks (PQN)
Recent work has shown that the off-policy RI. algorithm parallelized Q-networks (PQN) can perform well in vanilla Craftax [5] We explored whether this alternative DRL. algorithm performs differently compared to PPO ForageWorld. We adapted the PQN algorithm to ForageWorld World using the LSTM model architecture and default hy hyperparameters from [75], which differ some ways from our PPO implementation (for example, using an LSTM instead of GRU for the RNN). Except where noted in 3a we used the same environment and training hyperparameters between PPO and PQN. This experiment provides orthogonal test of Forage ForageWorld- World- would different algorithm, model, and training parameters result in different behavior?
We found that the PQN-LSTM models performed comparably to the PPO-GRU baseline for the subset of PQN models that learned the task (Figure 29), that only a subset of PQN runs would converge to this level of performance- across the two successful test conditions that varied the exploration hyperparameters of PQN (in particular controlling the rate of decay of e, the chance taking a random action instead of the optimal action under the current value function of PQN), only 30% of PQN runs performed comparably to the PPO model, with the remainder failing catastrophically in way that was not seen in the baseline PPO runs. We believe this is the result of suboptimal exploration by PQN, ie. the poorly performing PQN runs learned simple and reliable but suboptimal behaviors early in training and then were slow to deviate from them later For the runs that did perform comparably to PPO, we found overall similar learning histories and pathing behavior to the PPO baseline runs (Figure 30 Figure 3Tp, but with the noticeable behavioral differences of much lower rates of predator killing (i.e. reducing predator HP to 0) (Figure 31) in the learning history analysis. Thus, this result suggests that, despite showing comparable overall performance the task, PPO seems to converge to a predator fighting strategy, while PQN converges to predator evasion strategy.
PQN performs similarly on ForageWorld but often fails to learn
PPO-GRU (baseline) - PQN-LSTM (condition 1) PQN-LSTM (condition 2)
PQN-LSTM (condition PON-LSTM (condition failed runs)
20000
Return
15000
10000
5000
run
Iterations
10k
20k
30k
40k
50k
60k
Figure 29: Performance comparison for PQN versus PPO. Individual PQN-LSTM runs performed comparably to the PPO-GRU baseline for two different exploration hyperparameter configurations, but the majority of runs failed to learn more than trivial policy. Condition used the decay parameters shown in Table 3] while condition 2 adjusted e start, finish, and decay parameters so that decayed half fast and ended at final value of 0.1 (twice the chance of random action). No condition resulted in uniformly successful runs. Other exploration hyperparameter values were tested, which are detailed in Table 3 but did not result in any nigh-performing runs.
39

---

A
100
FallPth
Paxh
Eat
DHIKK
Arana Y
Arena Y
Arena
8-
25
100
20
30
1.0
40
60
60
100
Arena X
Arena X
Arena X
B
100
Est
90
90
Arena Y
Arena Y
60
Arena Y
50
40
85
40
100
Arena
X
Arena
C
100
Fath
Reth Path Eet
Dmk
Path
100
DX04
80
80
Arena
Arena
Arena
40
20
60
100
100
Arena X
Arena
Arena
Figure 30: Representative trajectories from three episodes for PQN-RVN. Each row (A-C) illustrates three sequential stages from single episode: early exploratory loops (left), path sampled during the transition to revisitation (middle), and the full episode trajectory with revisited patch regions highlighted in orange (right).
40

---

Spatial Unc./ Distance
Traveled
Spatial Entropy
Angular Variance
0.40
0.30
Mean
0.20
0.15
1.0
0.05
10
0.2
0.00
30
100
125
25
75
1.00
32.5
29
50
100
125
50
100
Pred. Enc. Rate
Pred. Kill Rate
Tool Make
Food & Drink Score
Drink Score
20-
15
0.2
10
0.0
(1000x)
(1000x)
Ep. Number
Ep. (1000x)
Figure 31: Training dynamics and movement patterns learning history for PQN-RNN agents with the auxiliary path integration objective. PQN had overall similar patterns to PPO, but was found to not learn predator killing well (gray arrow).
41

---

Toward Architectures for Cognitive Map Formation (Future Directions)
While our agents do not yet use structured maps, our findings suggest that such architectures may enhance both behavior and interpretability. Building on these findings about memory, planning, and spatial encoding in model-free agents, we highlight broader opportunities for future work. An important next step is to explore architectures that more closely reflect mammalian spatial navigation capabilities. In biological systems, both allocentric and egocentric representations support planning and memory [99] particularly through grid-based spatial computations [100].
architectural direction and outline
These "cognitive maps" 101 104
extend beyond current position encoding -hey support retro- spection, prospective planning, and flexible decision-making [07] Recent models have begun incorporating cognitive map-like representations into reinforcement learning frameworks [108]
In animals, spatial maps are modulated by task variables such as goals, landmarks, and reward-related events [109
110 20 [لЛ These advantages have been recognized machine learning as well, with grid-based representations increasingly integrated into transformers [122] and actor-critic agents improve task performance, efficiency, and interpretability.
Future work should examine how grid-based recurrent architectures solve the ForageWorke losk-probingg what spatial computations emerge and how they differ from standard RNNs. More broadly, our behavioral-neural analysis framework could be applied to other agent architectures and tasks, used to generate hypotheses about navigation and memory in biological systems.
G Craftax licensing permissions
Our Forage World environment is modified from Craftax. The Craftax license is as follows, taken from the URL above:
"Copyright (c) 2024 Michael Matthews
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and permit persons whom the Software furnished do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies substantial portions of the Software."
42

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We tried to carefully ensure the strength of evidence matched the strength of claims in our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- lant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
43

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Environment, model and analysis codes are provided the paper's github repository (https:/ / githuu...h..... com/RileyShi WaalleyeE/raaffeeeeee taattax-Forainng ttte//reee///eee//eeee cree/foraging)
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
44

---

Answer: [Yes].
Justification: Environment, model and analysis codes are provided the paper's 's github repository (https://github com/RileySE. htte///afftaa-Fooraeee maxx-Faagggggggg /tt//treee/ffee foraging). We provide key hyperparameters and other settings to reproduce experiments in Table 3
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
6. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, when needed to justify statistical claims, the authors use 95% CI as error bars unless otherwise specified.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
7. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]..
Justification: Yes, see the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
8. Code of ethics
45

---

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:p::/w.aapppp..ccaaaa///icaa
Answer: [Yes]
Justification: The authors have reviewed the code of ethics and believe our work is not in violation of any rules.
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
9. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, see Section 5
The answer NA means that there is no socictal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
10. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA].
Justification:
We did not scrape any data or release model significant risk of dual-use applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
46

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
11. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: Our work modifies the Craftax environment, whose free use license is given in the appendix: Appendix G Additionally, the proper license terms were followed from the respective host journals for the adapted, previously published figures in Figure We paid for the use of figure (79], and the other figure was published under an unrestricted use license, Creative Commons Attribution License, as long as the author is accredited 99
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, pwppsswipeteatteee thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
12. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes].
Justification: We have released the ForageWorld environment code and ac- companying model and analysis code, which are provided on Github at (https://github. wwwm/oom//eeeeeeeeee Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
13. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA].
Justification: We do not perform human research.
Guidelines:
47

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
14. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: We do not perform human research.
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
15. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA].
Justification: We do not use I.MMs.
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy for what should or should not be described.
cc /
48


Reviews:
[{"review_id": "e7BDsMczjY", "reviewer": "Reviewer_bGpG", "summary": "This paper introduces “ForageWorld”, a procedurally generated open-ended reinforcement learning environment with ecological structure  (built on an existing framework called Craftax). The environment replicates ecological pressures through partial observability, diminishing food sources, shifting predator threats, and biological drives like hunger and fatigue. They conduct joint behavioral and neural analysis using tools adapted from neuroscience, cognitive science and ethology to study reinforcement learning agents trained in ForageWorld environment (using path analysis, generalized linear models, and recurrent state decoding). The presented work demonstrates that analyzing models this way reveals subtle learning strategies and internal representations that standard performance measures often overlook, and shows connections between artificial and biological systems. \n\nThe authors train PPO agents across five configurations (predator blind agent - agent that cannot cause damage to the predators, normal field of view - aerial view of shape 9x11 grid around the agent with the agent in the center, front-facing field of view - an aerial view of the grid rows in-front of the agent instead of 9x11 grid with agent the center as input, auxiliary path integration objective - auxiliary loss function for linear decoding of the position in reference to the start position as the origin from the recurrent state) - \n\n1. Baseline RNN agent (non predator-blind agent with normal field of view and auxiliary path integration objective)\n2. Predator-blind RNN agent\n3. RNN agent without auxiliary path integration objective\n4. RNN agent with front facing field of view\n5. Memoryless agent (feed-forward neural network instead of RNNs)\n\n- The authors highlight the emergence of structured phase like foraging behavior analogous to those observed in animals. Similar to rodents, after initial exploration of the environment, the models show goal-directed revisitation of previously discovered states. Additionally, the authors suggest that the initial exploration qualitatively resembles insect search behavior and show that agents integrate information in a multi-objective manner while making revisitation decisions.\n- In recurrent state decoding analysis the authors find that the past and future allocentric positions (position with respect to the origin/start-state) can be decoded from the recurrent state of the agent better than the egocentric position (position with respect to the agent’s current position), and that the path integration objective is necessary for decodable spatial representations. In addition, decoding accuracy improved over training and removing path integration loss reduced performance on large arenas due to agent’s inability to predict its own position.\n- Pruning for sparse activations improved allocentric decoding in RNN agents.\n- Memoryless agents show reduced survival and poor coverage of the arena in comparison to the RNN agents. They do not exhibit emergent phase like foraging behavior and get trapped in local loops.\n- Both predator-blind agents and non predator-blind agents achieve similar survival performance and converge to similar movement strategies favoring predator avoidance instead of combat.\n- Agents with front-facing field of view performed more dense local exploration in early exploration period, and transitioned faster to longer range exploration compared to normal field of view.", "questions": "- It is unclear how path 1, 2, 3 and 4 were classified in Figure 1, 6, 10 and 14. Do they belong to different episodes or the same episode? If they belong to the same episode, what was the criteria of differentiation?\n- How does survival time evolve across episodes during the training process?\n- What dependencies or correlations exist between survival time and other experimental metrics throughout training?\n- In Figure 4 and similar figures, the 'distance traveled' subplot contains 'Early in Episode' and 'Late in Episode' categories, but it is unclear how these categories were defined. Additionally, the predator related metrics could be better explained. The subplot uses the title “pred. kill” whereas the text mentions “predator defeat”. Is it possible to provide more comprehensive definitions for the metrics used in this figure?\n- The caption for Figure 11 states that \"The absence of structured spatial uncertainty plots (top left) reflects degraded internal spatial representation,\" but this is unclear. How does the absence of the plot indicate degraded spatial representation?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths -\n \nI would like to thank the authors for submitting their interesting work.\n\n- Their work introduces a new benchmark with ecological constraints useful for studying emergent phenomena in deep RL agents which could be of interest across multiple disciplines.\n- The emergent agent behavior observed in this work and its similarities to animal behavior is very interesting.\n- The experimental analysis clearly isolates and demonstrates the emergent phenomena discussed by the authors.\n- The comprehensive evaluation across multiple agent configurations strengthens the validity of the conclusions.\n\n\n\n### Weaknesses -\n\n- While agent behavior in reinforcement learning is significantly influenced by the reward function, the paper does not investigate how variations in reward design affect emergent behaviors.\n- The experiment for agents with front-facing field of view seems incomplete (Figure 15) as it contains only early-to-mid training data.\n- A more comprehensive review of related work on emergent cognitive behaviors and interpretability in reinforcement learning would strengthen the paper's positioning.\n- A more detailed explanation of the metrics in Figure 4 would enhance the paper's clarity.", "was_revised": true, "final_justification": "Issues resolved - methodology clarity, literature coverage, and figure explanations. (I have updated my rating for clarity)\n\nRemaining concern - Reward function sensitivity analysis not addressed - limits understanding of behavior robustness vs. design artifacts. The authors show the emergence of animal behavior, but do not explain in detail how/why these behaviors emerge in relation to the structure of the environment (or) reward.\n\nOverall, it is interesting work with compelling emergent behaviors and rigorous analysis. The unresolved reward sensitivity issue is outweighed by the methodology and clear research value, which makes me maintain my overall rating.", "rebuttal": "Thank you for your insightful comments! To address the specific questions:\n\n### 1 (variations in reward design)\n\nThis is an interesting question, thank you for asking it. We did experiment with several different reward function variants during development of ForageWorld (for example, instantaneous reward for eating when hungry), but did not explore them in detail because the resulting behavior was degenerate/not foraging-like in various ways (the instantaneous reward agent for example would try to be on the edge of hunger as much as possible to maximize reward from eating). As our goal was to develop and study agents performing simulated foraging behavior, we did not discuss these variations in the paper since they do not lead to naturalistic behavior and there are many subtle environment design details that can break naturalistic foraging (for example, the spawn rate of cows relative to the food value per cow must be set carefully to prevent either unavoidable starvation or the agent managing to survive indefinitely in one patch). We will add a brief description of some of these variants to the appendix.\n\n\n### 2 (incomplete figure 15 learning history plot)\n\nThank you for catching that- we had actually standardized on the shorter run durations for most experiments (as the major behavior and dynamics features seem to be established within that duration), but in hindsight that makes the directional vision experiment look out of place in that figure compared to the longer runs. We performed longer runs to show the full time series for that experiment and have amended our manuscript (the trend and conclusions are consistent with the original results).\n\n\n### 3 (related work on emergent cognitive behaviors and interpretability)\n\nThis is a good suggestion- we will enhance our literature review to include work on that topic, including: \n\nAbney, Drew H., et al. \"Advancing a temporal science of behavior.\" Trends in Cognitive Sciences (2025).\n\nVerma, Abhinav, et al. \"Programmatically interpretable reinforcement learning.\" International conference on machine learning. PMLR, 2018.\n\nCheng, Zelei, Jiahao Yu, and Xinyu Xing. \"A survey on explainable deep reinforcement learning.\" arXiv preprint arXiv:2502.06869 (2025).\n\n### 4 (expanded definitions for metrics used in figure 4)\n\nWe apologize if we were unclear about the calculation of metrics that were plotted across the agent’s training period. We will provide more detailed definitions of each metric in the appendix and refer to it. The numbers given are the average per episode value of the metric at a given training time point. For example, predator kills is the mean number of predators killed in one episode (i.e. in one arena configuration). Food score is the mean value of how satiated the agent is in a given episode, with a score of 9 as zero hunger and 0 as max hunger. Predator “kill” and “defeat” are indeed the same phenomenon (reduced the predator HP to zero with repeated attack actions), we will relabel these captions to be consistent. Thank you for your help in clarifying the manuscript!\n\n### 5 (paths used in analysis)\n\nWe thank the reviewer for pointing out this was unclear. Indeed, the paths here belong to the same episode, and are directly consecutive from 1 through 4 in the behavioral time series logs for the episode. Path 1 starts at t=0 of the episode. This is also true for the related appendix figures which show additional representative examples. The path differentiation is done manually for this figure with the purpose of illustrating how a representative agent frequently opens new episodes by doing azimuthally rotating loops as an exploration strategy. We will add this explanation to enhance clarity.\n\n### 6 (survival time across episodes)\n\nSurvival time is closely correlated to total return, and thus we chose to only list returns. This follows from the reward function in section 3.1, which means an agent that survives for longer will very likely be spending more time above the 50% mark on its resource levels and thus receiving positive reward.\n\n### 7 (relationship between survival time and other metrics)\n\nAs can be seen by comparing the Fig. 2 performance plot and the Fig. 4 learning history plot, most behavioral metrics change with improvements in survival time at first, but plateau 30-50% of the way through training. Increase in exploration area correlates with early survival time improvements, but late training improvements seem to be driven by more efficient predator defense plus possibly additional factors that we could not identify, thus leaving an open question for future work. We will expand the current discussion to address this point.\n\n### 8 (unclear Fig. 11 caption)\n\nWe apologize that the caption was not written clearly. The top left plot is missing because there is no position predicting auxiliary objective in this experiment, thus no error in position prediction to report. The quoted sentence about the “absence of structured spatial uncertainty” was supposed to refer to Figures 2 and 17 which show further information about this experiment, where the worse decoder performance and worse survival performance in the absence of the position prediction suggests that internal spatial representations may be worse in the agents which do not predict their own position. We will improve the caption and reference those figures.\n\n### 9 (clarifying the definition of late/early in figure 4 legends)\n\nThank you for the comment and we apologize for the confusion. “Early in Episode” corresponds to the first 1500 time steps of a new episode and “Late in Episode” is after the first 1500 time steps of a new episode. We will clarify this in the paper."}, {"review_id": "lpqTRPQ5L6", "reviewer": "Reviewer_msZZ", "summary": "The authors of the paper \"Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environments\" investigates how model-free reinforcement learning agents internally develop planning-like behaviors. To this end, the authors introduce an environment ForageWorld, which serves as an open-ended task inspired by animal foraging. Based on this the authors investigate properties of the learned GRU policy.", "questions": "* What environment features contribute to the emerging behaviors? I believe this study would benefit from ablations on the environment itself.\n* Can the same findings also be obtained for pre-trained LLMs, since those are generally found to plan several tokens ahead too?", "limitations": "* The study focuses on GRUs trained with PPO only. This is definitely a limitation as it does not support the broad applicability of the analysis pipeline.\n* Unless otherwise convinced, I believe that environments such as Minecraft already capture many aspects of the proposed ForageWorld.\n* It is unclear which environmental aspects cause specific behaviors in the agent", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* ForageWorld exhibits rich dynamics with ecological motifs, which overall provides a nice testing ground.\n* The efforts of this paper are well motivated and follow an interdisciplinary approach.\n* It is notable to find that models such as GRUs that are trained model-free to exhibit planning-like behavior.\n\nWeaknesses:\n\n* The authors study only a single type of agent, which is a GRU trained with PPO, which does not speak for the broad applicability of the methods employed in this work. In particular, one would have hoped that the authors study also LLM-based agents (without or with fine-tuning) on such a task. This would not only strengthen the broader applicability point, but also serve to make the work more visible.\n* There is little discussion or evidence surrounding this environment's advantage as compared to other open-ended environments such as Minecraft (which is mentioned). Could one not perform the same analysis on such environments?", "was_revised": true, "final_justification": "I believe the work is interesting and useful in that a highly optimized environment is presented that allows to easily study neural network agents on Neuroscience-relevant tasks.", "rebuttal": "Thank you for your comments. To respond to specific concerns:\n\n### 1 (why not an LLM agent)\n\nWhile an LLM-based model might be able to learn faster in ForageWorld by using prior knowledge, our goal is not to maximize performance on this task, but to understand what an agent trained de-novo on the task learns, and to prove out analysis techniques for obtaining such understanding. Many of the neuroscience analysis techniques we use would not apply to such a model, and we would not be able to measure learned features such as forward/backward position encoding in the RNN (figure 5). In addition, as ForageWorld is not a language-based task translating between its symbolic observation space and categorical action space and an LLM or VLM would not be straightforward, and would introduce questions of prompting that have no connection to neuroscience or neural analysis but which would greatly impact performance. Further, using a pre-trained model such as an LLM makes connecting interesting behaviors to environment/reward function features (such as the emergence of planning in agent’s activations as a result of our open-ended reward function) impossible, as it is unclear what behavior is induced by the environment and what is derived from prior knowledge in the pre-trained model. Lastly, densely recording and analysing network activations across tens of thousands of timesteps per episode and tens or hundreds of episodes per training run (as we have done here) for an LLM would be intractable in terms of computational and storage requirements for the majority of academic labs.\n\nUsing LLM architectures for computations underlying foraging, navigation, search & rescue, etc. tasks is an interesting future direction, but considerable work needs to be done on developing analysis techniques suitable for those architectures first. We will cite preliminary work in this direction for the future work appendix section, such as:\n\nQiao, Yanyuan, et al. \"Open-nav: Exploring zero-shot vision-and-language navigation in continuous environment with open-source llms.\" arXiv preprint arXiv:2409.18794 (2024).\n\n\n### 2 (why not use Minecraft)\n\nWhile Minecraft represents a more challenging procedurally generated survival environment, and studying the behavior and neural activity of RL agents in it would be worthwhile research, there are a number of factors that would make the contributions of our work difficult or impossible to obtain using Minecraft. \n\n**First**, instrumenting Minecraft to log the range of environment variables that we record would be impossible without assistance from Microsoft to modify the source code/API, which would make it impossible to perform many of our experiments (for example, relating patch revisitation rates to specific environmental factors, as shown in figure 3). This similarly limits the extendability of ForageWorld to modify factors such as arena generation or predator type/behavior.\n\n**Second**, Minecraft’s game logic runs on CPUs, whereas ForageWorld is implemented in JAX and runs completely on GPU, allowing for massive parallelization in rollouts within a single compute node and training for billions of environment timesteps, which was necessary to observe the de-novo emergence of planning, for example. Performing a similar scale of experiment in Minecraft would require vastly more computational resources and software infrastructure to do massively distributed rollouts across a large compute cluster, and would greatly limit the number and duration of experiments that could be run. We also note that such compute resources are out of reach for many research groups, especially computational neuroscientists, which would limit the impact of our environment and analysis tools in follow-on work. This computational accessibility coupled with high task complexity was a major feature of Craftax as a framework to build upon, as it allows future work to easily build upon our work (as we have done with regard to Craftax).\n\n**Third**, while doubtless many interesting phenomena could be observed in Minecraft, none of our main contributions would be significantly enhanced by using Minecraft- a more complex environment is not needed to observe foraging with patch revisitation in a long-time-horizon POMDP and the de-novo emergence of planning, nor for adapting neuroscience analysis tools to deep RL agents.\n\n**Lastly**, we are interested in analyzing agent behavior and neural dynamics when trained from scratch on an open-ended foraging environment, and the complexity of Minecraft environments would make training such an RL agent to do sophisticated temporally-extended foraging involving predator evasion and patch revisitation a formidable challenge technically and scientifically, as to our knowledge prior work demonstrating complex behavior in Minecraft has relied heavily on pre-training on human data (though if there is any recent work training complex tasks from scratch in Minecraft we would be curious to learn about it and study the methods used).\n\n### 3 (environment features that influence behavior)\n\nThis is a very interesting question, which we explore along several dimensions in the paper. We explore how factors such as arena size, agent field of view, and the presence/absence of the position prediction auxiliary loss term affect behavior (figure 2). We found that each of these factors produce changes in agent behavior and/or dynamics which are not obvious from comparison of reward curves. In future work there are of course more factors that could be varied to explore their influence (for example, a multi-agent version of the environment, including learned predators), and we hope to explore such extensions to the core foraging task in future work.\n\n### 4 (additional model(s) for comparison)\n\nWe focused on a single standard DRL architecture because our focus was on developing and demonstrating the environment and analysis techniques, and on demonstrating that planning can emerge without explicit world modelling, not to argue that such emergence is a common feature of many algorithms/models, as making such a claim would demand a much wider range of environments and models to compare. For further comparison, we implemented and tested an additional DRL-RNN model, PQN [*] with an LSTM (changing both RL algorithm and model architecture). We found PQN-RNN models can perform the task comparably well to our PPO models, with similar behavior and learning curves. However, interestingly, relative to PPO agents the PQN agents seem to have higher position prediction performance, prefer curvier paths on short time scales, and are worse at killing predators (prioritizing predator evasion rather than the killing predators with tools as PPO learns to prefer). We will summarize these updates and provide the code in the updated manuscript.\n\n[*] Gallici, Matteo, et al. \"Simplifying deep temporal difference learning.\" arXiv preprint arXiv:2407.04811 (2024)."}, {"review_id": "V7vloZpvfu", "reviewer": "Reviewer_KWpH", "summary": "This paper introduces ForageWorld, a novel open-ended foraging environment inspired by real-world animal behaviors (e.g. depleting resources, predators, partial observability). The authors trained RNN-based, model-free DRL agents (PPO + GRU) to survive in this environment, and after which applied a behavior-first analysis toolkit adapted from neuroscience, cognitive science and ethology and find that trained model-free DRL agents exhibit structured foraging strategies with distinct phases and displays planning-like behavior without explicit planning modules or world models. The authors argue that by conducting joint behavioral and neural analysis for DRL agents, similar to those for animals, yields deeper mechanistic insight into how complex decision-making tasks are solved. Overall, this paper bridges ML and neuroscience by improving interpretability and evaluation of deep RL agents in complex environments with neuroscience insights.", "questions": "1. As mentioned in Weaknesses 1, a recent work [1] causally shows mechanistic evidence that model-free RL agents can exhibit planning-like behaviors. Could you clarify and summarize how this paper advances beyond [1]?\n2. In Sec. 6.1 of [1], the authors inject learned “plan-concept” vectors to the agent’s activations and observe the causal effect of steering the agent from a short to a long route. Do you think there could be an analogous intervention, such as adding a “visit-patch-X” or “evade-predator” vector, that could demonstrate causal control over your agent’s planning-like behavior?\n3. Can you provide several examples of candidate neuroscience tools, aside from those used in this paper’s experiments, that could be applied to evaluate DRL agents in ForageWorld?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. This paper adapted a behavior-first toolkit from neuroscience, cognitive science, and ethology (GLMs, phase segmentation, and linear state decoding) on the trained RNN-based DRL agents. Their analyses expose clear strategy phases (exploration → targeted patch revisitation) and show that allocentric position is linearly decodable up to roughly 50–100 steps into both past and future, demonstrating that planning-like computation can arise from purely model-free RNN agents alone. \n2. Results of this paper draw a connection between DRL agent and animal behavior: For example, in Sec. 4.1, it states that agent's behavior phase shifts are analogous to animal foraging behavior phase shifts, and that their early trajectories resemble the “azimuthally rotating, outward spiraling loops” seen in ants and bees. In Fig. 3, it shows patch-choice GLMs rely on multiple variables such as proximity, predator history and uncertainty, which are commonly used to model rodent foraging decisions. By mapping agent strategies onto these established animal behavioral patterns, the work creates a concrete connection between DRL agent evaluation and behavior neuroscience, which may inspire future neuroAI work and interpretable RL evaluation.\n3. This paper presents ForageWorld, a novel open-ended naturalistic foraging environment built based on Craftax by adding patchy renewable resources, dynamic predators and a survival-oriented reward function. This simulated environment provides a valuable tool for running complex, naturalistic behavioral experiments under realistic ecological constraints and cognitive demands. It can also serve as a platform for testing brain analysis techniques as the current DRL agents are likely simpler than biological agents. \n\n**Weaknesses**\n1. The discussion on existing model-free RNN-based DRL agent work is limited. As the authors claimed, one of the key contributions is showing “model-free RNN-based DRL agents can exhibit structured, planning-like behavior purely through emergent dynamics without requiring explicit memory modules or world models”. Although the authors briefly mentioned several earlier related literatures in Line 269-270, it did not discuss [1], a more recent paper that already claims to provide causal evidence of emergent planning in model-free RL agents. A detailed discussion will help clarify this paper’s novel contribution on this topic.\n\n2. Current neural analysis is limited as it focuses on linear decoders of the GRU hidden state. One possible next step could be incorporating methods such as dynamical-systems analysis or representational-similarity analysis to uncover low-dimensional structure in the hidden activity. Those methods, which are widely used in neuroscience, could offer deeper insights into the trained GRU network. \n\n[1] Bush et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025.", "was_revised": true, "final_justification": "While I disagree with the authors' description of Bush et al. as \"contemporaneous work\", after reading the rebuttal, I believe that this work is sufficiently different from Bush et al. and its results can be viewed as a further and nontrivial generalization of the finding of Bush et al. In addition, I appreciate the paper’s connection to neuroscience analysis tools, which is an aspect that distinguishes it from Bush et al. and bridges DRL and neuroAI. Therefore, I am leaning more towards accept than borderline accept and increase the score to 5.", "rebuttal": "Thank you for your helpful comments! To respond to specific questions:\n\n### 1 (comparison to Bush et al, and causal interventions)\n\nThank you for bringing this work to our attention- we missed this recent paper during preparation, but it is definitely an important point of comparison, and we will discuss it specifically in the manuscript. It is also an interesting paper and was a thought-provoking read, the pointer is much appreciated.\n\nHowever, we must point out the conference rules on contemporaneous work:\n\n*“For the purpose of the reviewing process, papers that appeared online after March 1st, 2025 will generally be considered \"contemporaneous\" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work. Authors are still expected to cite and discuss contemporaneous work and perform empirical comparisons to the degree feasible. Any paper that influenced the submission is considered prior work and must be cited and discussed as such. Submissions that are very similar to contemporaneous work will undergo additional scrutiny to prevent cases of plagiarism and missing credit to prior work.”*\n\n**Bush et al, with an Arxiv date of April 2nd, is considered contemporaneous work and thus a decision regarding our paper should not be based on comparison to this very recent paper.**\n\nThat said, while Bush et al also show evidence of learned planning without a world model, ForageWorld differs from Sokoban in multiple important ways that make the planning evidence we see significant in excess of this prior work. \n\n**First**, ForageWorld is partially observable and not deterministic- our agent can only see a small portion of the arena (9x7 grid within a 96x96 arena, versus the 8x8 grids of Sokoban), and predator and food spawns are random but geographically biased, meaning the agent must plan under uncertainty and unknown information, which add significant barriers to the emergence of planning within the RNN model. \n\n**Second**, ForageWorld requires planning on a much longer timescale than Sokoban, with episodes lasting thousands of timesteps rather than dozens, meaning agents must plan and re-plan continuously within an episode rather than plan once and then execute. \n\n**Third**, our reward function is open-ended (keep food/drink/health/energy levels high), and does not directly reward planning over short time horizons, whereas Sokoban provides short-term feedback if planning is not successful via failed levels. When trying to survive for longer periods of thousands of timesteps our agent seems to plan out its exploration and revisitation of food patches due to the dynamics of patch depletion, and potentially maps out potential routes to evade predators. These behaviors are emergent, and are only indirectly rewarded across long time horizons of hundreds or thousands of timesteps. \n\n**Lastly**, our agent model and learning algorithm are very standard- we use a GRU with input/output fully-connected layers and PPO to train, as they are very commonly used in other research. This means that our finding of planning behavior is likely to generalize easily to other tasks/domains, and specialized models or algorithms are not required for planning to emerge. For example, the ability to perform multiple timesteps of computation per environment timestep is not essential for planning to emerge, per our results, which generalizes the findings of Bush et al.\n\nIn terms of doing a similar causal intervention as Bush et al, it’s conceptually possible, but our environment is larger and more complex than the small fully observable 8x8 grid world used in [1]. Their intervention worked because they were able to handcraft a simple environment to have two possible solutions, and could trigger the agent to pick one over the other. Our open-ended survival task does not have such a way to binarize two “solutions” to a given arena configuration, as there are many patches, paths, and movement strategies an agent could take. Scaling their approach to an environment of our complexity would be very interesting, but is not trivial. Methods to intervene in agents doing long-term planning in complex environments in order to modify or correct their plans would make an interesting topic for future research.\n\n[1] Bush et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025.\n\n\n### 2 (additional analysis techniques)\n\nWe agree that there are a wider range of neuroscience tools that could be applied to ForageWorld in follow-up work. While there are many possible options, to expand the immediate toolkit in the current work, we have added analysis and code (to our anonymized github linked in the paper) for using generalized linear models (GLMs) to predict single GRU neural spiking activity from behavioral variables for each GRU neuron. This addition allows us to better understand the functional circuits that exist within our trained networks. While we are not allowed to provide experimental plots here due to the recent conference guideline emails, this analysis reveals single neuron-level behavioral variable encoding that complements the current decoder results.\n\nThe new neural-focused GLM experiments that we performed add evidence for an accumulation threshold circuit that pressures the agent to return to the origin the farther away it travels radially from the central start position. Specifically, we found the average response differential of position-selective neurons (i.e. the magnitude of the GLM regressor for a given position bin) increases with radial distance from the origin as the agent explores. This circuit could underlie the frequent origin revisitation we see our agents perform, with some sort of attractor driving the agent back to the origin in a ramping way with distance. Our GLM code is flexible, and can support single neural analyses using behavior time series that future researchers may come up with on our task or related ones, provided they use similar recurrent architectures and behavior logging. We have uploaded the code for this analysis to the anonymous github we linked in the manuscript and will add this analysis to the paper appendix.\n\nRepresentational similarity analysis would be difficult because it is designed for a lower number of simpler input states and usually time averages neural activity (the neural GLM predicts the full activity time series in contrast). Our agent has 3 different time-varying physiological levels and highly varying pixel input each time step, thus the conditions one contrasts with RSA and how time series are segmented would have to be carefully considered. We will add a note about this to the paper."}, {"review_id": "3YA5RjsBl7", "reviewer": "Reviewer_se1j", "summary": "In this paper the authors propose 1) a methodology for Reinforcement Learning agent analysis that goes beyond the inspection of reward curves and uses techniques inspired from neuroscience and  ethology to lend insight to agent performance and capabilities and 2) then aim to show that these analyses tools can be applied to a simple RNN based RL agent demonstrating that such an agent possesses some abilities, such as planning, that go beyond what was previously assumed in model free agents.\n\nThe authors make the following contributions:\n\n * Introduce ForageWorld, an environment that simulates ecological systems  for the study of planning and memory agents\n * Demonstrate how to use tools from Neuroscience and ethology (animal behaviour) for analysis and interpretability of RL agents.\n * Show that RNN models, trained in a model free fashion, can learn to plan and use memory.\n * Release of agent and environment code as open source.\n\nThe paper focuses on a set of analysis methodologies that depend on observing jointly the agent's internal representations and behaviour: Goal Inference, Memory Span, Planning Horizon, Spatial Structure, Network Capacity, and Representations.  It is further claimed that RL agents often fail to generalise well in open-ended environments and that these methodologies will provide a clearer indication on which models possess the requisite skills for such environments. \n\nTo study the efficacy of these methods a recurrent network (RNN) based agent trained with PPO with an auxiliary loss on position prediction in the grid along with some methods for introducing sparsity in the network. The agent's hidden state can also be decoded to reveal allocentric positioning throughout acting.\n\nIt is reported that through the use of the analysis methodologies it is determined that the agent explores effectively using revisitation strategies, structures its objectives over the course of an episode utilising its memory, exhibits the emergence of abilities over training, uses its state to support memory and planning, and supports generalisation across environment treatments.", "questions": "Line 28: I assume ML refers to \"Machine Learning\".  Referencing the term in full may add some clarity.\n\nDoes ForageWorld support multi-agent?\n\nIn the paper there is a lot of emphasis on agents trained via Deep RL methods but is this a necessary condition for the methods proposed?  Couldn't this be applied to agents optimised in different ways?  (e.g. behavioural cloning)\n\nIn the abstract, discussion, and possibly a few other times you indicate that there is a view that effective planning and memory use may require world models.  Is there some evidence that this view  is generally held? I was under the impression that model free agents can be fully capable of subsuming such capabilities.  Generative agent based models have certainly seemed to do so without any specific world modelling.\n\nWhy did you choose an RNN based agent?  Why not try feed forward as a baseline? What about a transformer based model?  Pre-trained models?\n\nIn the second paragraph of section 4.2 you refer to two figures (2 and 10), is this intentional?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe work is well motivated toward agent analysis providing a comprehensive breakdown of behavioural competencies (Table 2).  Understanding agent capabilities is critical as agents deploy into more sophisticated and real world settings.\n\nDraws inspiration from biology and neuroscience providing links into biological cognitive behavioural patterns and making use of existing and battle tested methodologies from those domains for understanding how neural activity is jointly represented with behaviour.\n\nThis approach highlights the importance of the interpretability of agents using methods that can gain insight into RL agents (or agents full stop). It is critical to understand agent capabilities especially as large models are increasingly deployed in the world. This paper presents a potential good step in the direction of contrasting and comparing such agents to understand which fundamental capabilities they have and will potentially provide more (or additional) insights than is done via agent evaluation benchmarks.\n\nThe authors demonstrate how a simple agent develops emergent planning and navigation strategies that are well adapted to the environment and generalise across episodes as the agent learns and retains information from its input for later use in an open-ended environment.\n\n**Weaknesses**\n\nI first want to clarify the main purpose of the paper.  It seems to me that the main takeaway is meant to be a novel suite of analysis methods inspired by behavioral and neural inspection methods from neuroscience etc.   There also seems to be a lot of emphasis on the capacity of the model free RNN based agent trained via PPO, is the primary purpose of this choice so as to showcase the efficacy of the analysis methods or is there some claim being made about this  specific agent architecture?  Further, there's some discussion about the sufficiency of neural analysis methods as applied to deep neural networks (as opposed to biological brains), it's unclear if this claim is addressed directly though the results in the paper, although the validity of the approach I think stands independently of this.  Overall, I think adding a bit more clarity about where its main contributions lie will be critical.\n\nI found the overall presentation to be somewhat disorganised throughout the paper. Figures are referenced out of order and many referenced figures are in the appendix. Also, the conclusions drawn in the results section could be better backed up with empirical evidence from the analyses methods used and more clearly linked to their biological counterparts as claimed.  For instance, when discussing revisitation strategies done by the agent and how it develops its objectives throughout an episode it seems clear that the agent has learned these capabilities successfully however, it isn't completely clear how the methods were applied to draw these conclusions. Would it be possible to reference the targets in table 2 and how they were applied as part of an overall methodology?  Further to this point, I believe that I'd have a hard time reproducing your results from what's been put down in the paper. It may be that the appendix adds some clarification here but this type of stuff should be accessible in the main body of the work.  If the methods are the real takeaway of this work then I believe that their use and novel character should be made clearer. \n\nThe ForageWorld environment appears to be a relatively simple toy environment.  Do these results generalise well to other environments?  Are there limits to these analysis methods that depend on this? It would be nice to see whether the thesis about the interpretability of planning and memory capabilities holds up under more complex dynamics.  A few more details about this environment in the main paper and what motivated its choice would be very helpful.  If possible, considering an additional, more natural environment could lend further weight to this work.\n  \nIt would be helpful to see a bit more justification around the model choice along with some additional detail around the architecture, training, RL algorithm and optimisation settings.  Further, some contrast to models that vary in some of these dimensions would be useful as a point of comparison especially from the perspective of the agent abilities that are discoverable through your analysis methods. Finally, like the environment, the choice of agent is somewhat limited compared to what's commonly used vis-a-vis large pre-trained models that can power generative agent based models. Considering such models would help ground the relevance of this work as well as provide a rich set of agent capabilities on which to perform this analysis.", "was_revised": true, "final_justification": "This work presents a novel environment and a methodology for carrying out analysis on RNN based agents that lend concrete insights to biological intelligence.  Overall some of the main contributions include: \n\n * The ForageWorld environment which yields a naturalistic environment that is amenable to carrying out behavioural and neural analysis of agents.\n   * Analysis is done on structured exploration, adoption of multi-objective policies, and the emergence of behavioural competencies of RNN based agents in ForageWorld.\n * Model free deep RL agents can successfully learn complex behaviour in naturalistic environments without explicit world models.\n * Argues that and provides evidence that this methodology can be used for neuro-science style analysis.", "rebuttal": "Thank you for your thoughtful comments! We will respond to specific concerns below:\n\n### 1 (main purpose of the paper)\n\nWe will clarify the language of how we describe our contributions in the manuscript, thank you for pointing out that this could be unclear. Our four main contributions are as you have listed in the summary. The core goal of this work is to demonstrate how studying standard deep RL agents in complex naturalistic environments using analysis methods from neuroscience can reveal complex behaviors and neural patterns that are informative for computational neuroscience and ML research. \n\nFurther, we hope to encourage both computer scientists and computational neuroscientists to dig deeper into the joint behavior and neural data of agents in open-ended environments, as they provide controlled frameworks to test analysis methods intended for more complex real world biological data where controllability and statistical rigor may be lacking. Existing benchmarks do not easily provide the behavior, neural data, and naturalistic structure to be suitable for this, and thus we have provided both a new task and an analysis framework for it. Navigational circuits in large environments are especially poorly understood in both computer and biological sciences, as most well-studied tasks are small and/or fully observable arenas where complex planning with memory is not required. Thus, for novelty and impact we have focused on navigation in our analysis here, but many other aspects of an agent could be studied using similar techniques as well.\n\nWe will add further references to the intro and expand discussion to contextualize this in the paper, such as:\n\nAbney, Drew H., et al. \"Advancing a temporal science of behavior.\" Trends in Cognitive Sciences (2025).\n\n### 2 (paper organization)\n\nGiven the space requirements, we tried to have a coherent and complete narrative in the main text while referencing extensive methodological details and less critical experiments in the appendix.\n\nTo address your specific points, we will fix the figure reference order and revise to better connect our findings in the results section with the analysis methods described in detail in the appendix that were used to obtain them. Thank you for helping us to improve the clarity of the paper!\n\nFor reproducibility, all analysis and task code are available via an anonymized github link present in the main text of the paper, along with basic documentation on how to reproduce our experiments. We are committed to maintaining the project github publicly post-review. If there are specific requests for improvement we would be happy to respond to them.\n\n### 3 (task complexity)\n\nWe designed ForageWorld to find a balance between task complexity, transparency to analysis, and biological plausibility. It is certainly the case that more complex deep RL tasks exist, but many such tasks (for example, Minecraft) are much more difficult to study and in particular more difficult to instrument for behavior analysis. In addition, the increased computational cost of experiments in such environments is prohibitive to some statistical analysis, and many computational neuroscientists in particular do not have access to compute resources sufficient to run experiments in such complex environments. \n\nRelative to common neuroscience tasks used to study foraging, e.g. n-armed bandit tasks which lack spatial or temporal components, ForageWorld pushes the complexity boundary on the neuro-side of neuroAI while still remaining tractable for neuroscience tools and neuroscientists to study. Very complex tasks would be difficult to analyze with the current toolboxes used in neuroscience, which were designed for tasks of low-to-intermediate complexity. Excluding such analyses and many potential users by working with a much more complex environment would decrease the impact of our work. \n\nFurther, ForageWorld as it exists is not trivial to learn- our agents require billions of timesteps of experience to train to competency on the task, and Craftax, the benchmark which we build upon to develop ForageWorld, has not been solved by any agent or architecture to date.\n\n### 4 (model choice and hyperparameters)\n\nCertainly, we are happy to provide clarity on our model design choices. We will revise the manuscript to make explicit why we use an RNN model as well. We use RNN models because such models can be studied readily with neuroscience analysis techniques and have lower complexity than transformer models while still remaining general-purpose learning agents with hundreds of thousands of free parameters, allowing us to strike a balance in terms of compute requirements, amenability to analysis, and model capability. Our baseline PPO-RNN model is largely identical to the best-performing model presented by Craftax, other than the addition of the path integration output head and changes to model hyperparameters. On the original Craftax task, this PPO-RNN model is only moderately worse than the best reported model on the Craftax benchmark, a much larger transformer-based model which would not be compatible with many neuroscience analysis techniques [*].\n\n[*] Dedieu, Antoine, et al. \"Improving transformer world models for data-efficient rl.\" arXiv:2502.01591 (2025). \n\nWhile an LLM-based model might be able to learn faster in ForageWorld by using prior knowledge, our goal is not to maximize performance on this task, but to understand what an agent trained de-novo on the task learns, and to prove out analysis techniques for obtaining such understanding. Many of the neuroscience analysis techniques we use would not apply to such a model, and we would not be able to measure learned features such as forward/backward position encoding in the RNN (figure 5). In addition, as ForageWorld is not a language-based task translating between its symbolic observation space  and an LLM would not be straightforward, and would introduce questions of prompting that have no connection to neuroscience or neural analysis but which would greatly impact performance. Further, using a pre-trained model such as an LLM makes connecting interesting behaviors to environment/reward function features (such as the emergence of planning in agent’s activations as a result of our open-ended reward function) impossible, as it is unclear what behavior is induced by the environment and what is derived from prior knowledge in the pre-trained model. Lastly, densely recording and analysing network activations across tens of thousands of timesteps per episode and tens or hundreds of episodes per training run (as we have done here) for an LLM would be intractable in terms of computational and storage requirements for the majority of academic labs.\n\nTraining and environment details can be found in Appendix A, and model and environment hyperparameters are listed and selected values motivated in Table 3 in the appendix.\n\n### 5 (multi-agent support)\n\nForageWorld currently does not support multi-agent training, but we have ongoing work seeking to develop a multi-agent successor. Craftax also is not multi-agent.\n\n### 6 (focus on deep RL training)\n\nNone of our analysis is specific to deep RL training in particular, and we hope that future researchers interested in our work try many other types of training algorithms. We use deep RL here because we are interested in what emergent capabilities and features an agent will learn from training on a relatively simple reward function (maintain high resource levels and avoid death), and because such RL algorithms combined with memory networks (like RNNs) are of particular interest to neuroscientists as the best known computational approximation of animal learning. However, given a pre-existing behavioral cloning dataset (which, as ForageWorld is a novel task, did not exist prior to developing our deep RL agent) it is entirely possible to apply similar analysis to an agent trained via BC, or to any other agent with an RNN-type model. We will note these potential extensions in our discussion of future work, thank you for the suggestion!\n\n### 7 (necessity of world models for planning)\n\nThis view (that world models are necessary for explicit planning) is often stated in neuroscience and adjacent computational biology fields, although as written in the Bush et al (2025) paper [*] suggested by reviewer KWpH this topic is understudied considerably even in ML/AI (see the KpWH response about that contemporaneous work, which had a simpler task, shorter time horizons, full observability, and a reward structure that directly incentivizes planning). We will clarify this in the paper. On the neuro side, it was only recently appreciated that insects are individually quite intelligent, rather than just being swarm-governed or simple sensory-feedback automatons (see “Lars Chittka. The mind of a bee. Princeton University Press, 2022.” which we cited). The sort of planning a small insect brain can do is currently unknown, but their brains are only 1-2 orders of magnitude larger than our small DRL agent models, and the commonly-held historical view is that a brain of such size does not have the capacity to learn explicit planning without a world model (the existence of which has not been discovered by experimental neuroscience to date).\n\n[*] Bush et al. Interpreting Emergent Planning in Model-Free Reinforcement Learning. ICLR 2025.\n\n### 8 (Feed-forward comparison)\n\nWe did compare our RNN model to a feedforward agent, trained identically via PPO but with the RNN removed and replaced by an MLP of roughly similar parameter count. This ablation could not learn the task, as shown in Figure 2. This demonstrates that memory is required to learn our task to a non-trivial level of competency.\n\n### 9 (Referring to two figures in section 4.2)\n\nYes this is intentional. Figure 2 refers to the performance of feedforward (memoryless) agents quantitatively, while Figure 10 shows an example of the suboptimal pathing of such an agent."}]

Author rebuttal:

Meta-review: This paper presents a valuable study of emergent planning-like behavior in standard model-free deep RL agents trained in a novel open-ended environment, ForageWorld. The authors show that GRU-based agents trained with PPO can exhibit long-horizon behavior resembling planning, despite the absence of explicit memory or world models. They further support their claims using neuroscience-inspired tools to analyze internal representations and behavioral strategies.

The paper’s key strengths include the well-motivated task design, comprehensive behavioral analysis, and the use of interpretable neural probing techniques. The authors provide detailed implementation and analysis resources to strengthen reproducibility.

Reviewers initially raised concerns regarding novelty in analysis methodology and the lack of comparison to Bush et al. (2025). The authors highlighted important distinctions in task complexity and experimental design from Bush et al. They also provided code for using generalized linear models (GLMs) to predict single GRU neural spiking activity from behavioral variables for each GRU neuron. Furthermore, the authors provided detailed discussion on the limitations of applying alternative methods like RSA in their setting.

Overall, the authors have convincingly addressed all reviewer concerns during the rebuttal. The paper makes a solid and timely contribution to understanding the emergence of planning-like behavior in RL agents.
Decision: Accept (poster)