Adaptive patch foraging in deep reinforcement learning agents
Authors: Nathan Wispinski
Andrew Butcher
Kory Wallace Mathewson
Craig S Chapman
Matthew Botvinick
Patrick M. Pilarski
Venue: Accepted by TMLR ()

Abstract: Patch foraging is one of the most heavily studied behavioral optimization challenges in biology. However, despite its importance to biological intelligence, this behavioral optimization problem is understudied in artificial intelligence research. Patch foraging is especially amenable to study given that it has a known optimal solution, which may be difficult to discover given current techniques in deep reinforcement learning. Here, we investigate deep reinforcement learning agents in an ecological patch foraging task. For the first time, we show that machine learning agents can learn to patch forage adaptively in patterns similar to biological foragers, and approach optimal patch foraging behavior when accounting for temporal discounting. Finally, we show emergent internal dynamics in these agents that resemble single-cell recordings from foraging non-human primates, which complements experimental and theoretical work on the neural mechanisms of biological foraging. This work suggests that agents interacting in complex environments with ecologically valid pressures arrive at common solutions, suggesting the emergence of foundational computations behind adaptive, intelligent behavior in both biological and artificial agents.

Published in Transactions on Machine Learning Research (4/2023)
Adaptive patch foraging in deep reinforcement learning agents
Nathan J. Wispinski
nathan36uuallerta.cc
University of Alberta, Edmonton, Canada (work conducted while an intern with DeepMind, Edmonton, Canada)
Andrew Butcher
abutcherrrdeeppmind.ccmm com
DeepMond, Edmonton, Conada
Kory W. Mathewson DeepMind, Montreal, Canada
korymath wwwppaaddddddeddddd....oo
Craig S. Chapman
c.s. L.s.chapman@uuuleett..c
University of Alberta, Edmonton, Canada
Matthew M. Botvinick
botriniickkldeepmind..ok
DeepMind, London, UK
Patrick M. Pilarski
pilarski@hullberta.cc
DeepMönd, Edmonton, Conada University of Alberta, Edmonton, Canada
Alberta Machine Intelligence Institute (Amii), Edmonton, Canada
Reviewed on OpenReview: https: // openreview. net/ forum? id= a0T3.OP9BB
Abstract
Patch foraging is one of the most heavily studied behavioral optimization challenges in biology. However, despite its importance to biological intelligence, this behavioral optimiza- lion problem is understudied in artificial intelligence research. Patch foraging is especially amenable study given that it has known optimal solution, which may be difficult to discover given current techniques in deep reinforcement learning. Here, we investigate deep reinforcement learning agents in ecological patch foraging task. For the first time, we show that machine learning agents can learn to patch forage adaptively in patterns similar to biological foragers, and approach optimal patch foraging behavior when accounting for temporal discounting. Finally, we show emergent internal dynamics in these agents that resemble single-cell recordings from foraging non-human primates, which complements ex- perimental and theoretical work on the neural mechanisms of biological foraging, This work suggests that agents interacting in complex environments with ccologically valid pressures arrive at common solutions, suggesting the emergence of foundational computations behind adaptive, intelligent behavior in both biological and artificial agents.
1 Introduction
Patch foraging is one of the most critical behavioral optimization problems that biological agents encounter in nature. Almost all animals forage, and must do so elfectively to survive. In patch foraging theory, spatial patches are frequently modeled as exponentially decaying in resources, with areas outside of patches as having no resources (Charnov] 1976). Agents are faced with a decision about when cease foraging in a depleting patch in order to begin travelling some distance to richer patch. Research has shown that animals are adaptive patch foragers in this context, intelligently staying in patches for longer when the environment is

---

Published in Transactions on Machine Learning Research (4/2023)
resource-scarce, and staying shorter time in patches when the environment is resource-rich (Cowie 1977
Hayden et al. 2011 Kacelnik 1984 Krebs et al. 1974
Despite its importance to biological intelligence, patch foraging is understudied in artificial intelligence re- search. Computational models of patch foraging are often agent- models with fixed decision rules (Pleasants] 1989 Tang Bennett, 20100) although recent work has involved the use of tabular reinforcement
learning models (Constantino & Daw 2015; Goldshtein al.) 2020 Miller al.) 2017 Morimoto] 2019)
Additionally, neural networks trained using methods such Hebbian learning have displayed foraging be-
havior in separate ecological tasks such patch selection (Coleman et al. 2005 Montague et al. 1995
Niv et al. 2002). Many deep reinforcement learning agents are able to succossfully scarch environments for rewarding collectibles like apples while avoiding obstacles and/or enemies (as in gridworlds or popular Platanios et al.) 2020), and even to stay or switch in the However, these environments significantly differ from core principles of theoretical and experimental foraging rescarch. Namely, many environments lack the repeating, lemporally evolving tradeoff' between immediale, decaying resources and delayed, richer ones (Stephens &
video games; e.g., Lin 1991 Mnih et al.
2015
face of decaying rewards (Shuvaev et al.
2020
Krebs 2019
Patch foraging is especially feasible to study computationally given that it has known optimal solution- the marginal value theorem (MVT: Charnoy 1976). In short, the MVT states that the optimal solution is to cease foraging within a patch and begin traveling toward a new patch when the reward rate of the current patch drops below the average reward rate of the environment. Foraging is so important to the survival of biological agents that theorists argue that the foraging behavior of animals is not only adaptive, but approaches this MVT optimal behavior in natural environments because of strong selective pressures
(Charnov & Orians 2006 Pearson et al., 2014 Stephens & Krebs 2019) Many animals, including humans, have been shown to behave optimally in patch foraging tasks in the wild and in the laboratory. For example, human mushroom foragers (Pacheco-Cobos et al.] 2019)) rodents (Lottem et al.
2018
Vertechi et al., 2020).
and birds, fish, and becs (Cowic, 1977 Krebs et al. 1974 Stephens & Krebs 2019
all bohave consistent with the MVT solution of optimal patch foraging (although many examples of suboptimal patch over-slaying exist; see Nonacs 2001).
b
1/30
Reward
250
500
d
Step in patch 12 m
6 m
8 m
10 m
y (m)
-16
-16
16
x (m)
Figure 1: Task. a) Mock-up of the 3D foraging environment and agent with LIDAR rays. b) Overhead view. An agent starts each episode between two equidistant patches. c) The agent receives exponentially decreasing reward on every step it is within patch. When the agent enters one patch, the opposite patch is immediately refreshed to its starting reward state. d) Overhead spatial trajectories of a representative trained agent in each evaluation environment.
2

---

Published in Transactions on Machine Learning Research (4/2023)
However, the optimal patch foraging solution may be difficult to discover using many frequently used deep reinforcement learning approaches. The MVT dictates comparison between the long-run average reward rate of the environment with the instantaneous reward rate of the current patch. This value comparison requires multiscale temporal resolutions, both local and global, which can be difficult to represent using model-free reinforcement learning, but are seen in animal cognition and neural dynamics (Badman et al, 2020). Further, model-free reinforcement learning potentially requires significant temporal exploration in- volved with the trial-and-error learning of leaving patches at different depletion times. Finally, changes in the resource-richness of the environment or the agent's 's movement time between patches impact the optimal patch time, as prescribed by the MVT.
Given the potential difficulty for deep reinforcement learning agents to learn to patch forage, how do biological agents solve the same problem? Theoretical work suggests that a simple evidence accumulation mechanism, similar to one commonly proposed for perceptual and value- based decision making in mammals (Brunton et al. 2013 Gold & Shadlen 2007 Hanks Summerficld, 2017 Wispinski al.) 2020), can approximate MVT solutions in complex environments (Davidson & El Hady 2019). Neural recordings from non-human primate cingulate cortex suggests that such mechanism may underlie decisions in computerized patch foraging task (Hayden et al. 2011]. Past work in deep reinforcement learning has used task paradigms from animal research to probe the abilities of agents to learn abstract rule structures (Wang et al., 2016).
detour problems (Banino et al. 2018). and perceptual decision making (Song et al.) 2017). In these studies, internal dynamics show patterns similar to those recorded from animals completing the same behavioral task, suggesting the discovery of similar mechanisms solely through reward learning (Silver et al. 2021
Further, studying animal intelligence, the behavioral pressures that shaped biological intelligence (Cisek
2019 Stephens & Krebs, 2019) and their neural implementation is likely to expedite progress in artificial intelligence (Hassabis el al. 2017).
Here, we first ask if deep reinforcement learning agents can learn to forage in 3D patch foraging environment inspired by experiments from behavioral ecology. Second, we ask whether these agents forage intelligently- adapting their behavior lo the environment in which they find themselles. Next, we investigale if agent foraging behavior in these environments approaches the known optimal solution. Finally, we investigate how these agents solve the patch foraging problem by interrogating internal dynamics in comparison to theory and neural dynamics recorded from non-human primates.
This paper offers number of novel contributions. We demonstrate:
The first investigation of deep recurrent reinforcement learning agents in a complex ecological patch loraging task;
2. Deep reinforcement learning agents that learn to adaptively trade off travel time and patch reward in patterns similar to biological foragers;
3. That these agents approach optimal patch foraging behavior when accounting for temporal discounting:
4. That the internal dynamics of these agents show key patterns that are similar to neural recordings from foraging non-human primates and patterns predicted by foraging theory.
This paper is an empirical investigation into the emergence of complex patch foraging behavior, and offers a model to the biology community with which to study patch foraging. Additionally, these results add to the neuroscientific literature on how biological agents approximate the optimal solution using a general decision mochanism, and how adaptive behavior ariscs from simple changes in this mechanism.
2 Experiments
2.1 Environment
A continuous 3D environment was selocted to approximate the rich sensorimotor experience involved ecological foraging experiments. The environment consisted of 32 x 32 m flat world with two patches (i.e., half spheres) equidistant from the center of the world (Figure 1 see Cultural General Intelligence
3

---

Published in Transactions on Machine Learning Research (4/2023)
Team et al. 2022). Patches always had diameter of 4 m. Agents started each episode at the middle of the world, facing perpendicular to the direction of the patches. Each episode terminated after 3600 steps. Agents received reward of zero on each step they were outside of both patches. When an agent was within a patch, it received reward according to the exponentially docaying function, r(n) = Noe An where n is the number of non-consecutive sleps the agent has been inside a patch without being inside the alternative patch. In this way; as soon as an agent entered patch, the alternative patch was refreshed to its initial reward state (i.e., n = 0). As such, agents are faced with a decision about how long to deplete the current patch before traveling toward a newly refreshed patch. For all experiments, the initial patch reward, No, was set to 1/30, and the patch reward decay rate, X, was set to 0.01 (Figure D.). The surface color of each patch changed proportional to the reward state of the patch in RGB space. Patches changed color from white (i.e., [1, 1, IJ) to black (i.e., [0, O, 0) following the function, r(n)/No. In this way, agents had access to the instantaneous reward rate of the patch through patch color, rather than having to estimate patch reward rate by estimating the docay function and keeping track of stops spent within a patch. We chose to make instantaneous patch reward information available in the environment as biological agents often have complete or partial sensory information regarding the current reward state of patch (e.g., visual input of apple density on tree).
2.2 Agents
The agents observe the environment through a LIDAR sensor-based observation sensor. L.IDAR-based scnsing is common for physical robots (Malavazi et al. 2018), and agents in other simulated environments (e.g-, Baker et al. 2019 Cultural General Intelligence Team et al. 2022). The LIDAR sensor casts uniformly across 3 rows of 8 rays. Rays are evenly spaced, with azimuth ranging from -45" to +455, and altitude ranging from -300 to +300. Each ray returns an encoding of the first object it intersects with which includes object type, color, and distance of the object from the agent. Object type is encoded as a one-hot vector for each valid object type (ground plane, patch sphere, no-intersection; e.g., [0, 1, 0)). Color encoded for patch spheres only, as [R] where each is a float [0, 1]. Distance is encoded to the maximum LIDAR distance (128 m). This corresponds to LIDAR space with 24 channels. Agents analyzed in the dynamics results section below instead had 14 rows of 14 rays with azimuth ranging from 0 to 360", and altitude ranging from -90º to +90", consistent with (Cultural General Intelligence Team et al. 2022). LIDAR differences did not substantially impact agent behavior. LIDAR inputs were convolved (24 output channels, 2x2 kernel shape), before they were concatenated with the reward and action taken on the previous step. These values were then passed through a MLP (3 layers of 128, 256, and 256 units) and a LSTM layer (256 units). Finally, LSTM outputs were passed to an actor and critic network head.
a float [0, 1] normalized
The action space is 5-dimensional and continuous (adapted from Cultural General Intelligence Team et al. 2022). Each action dimension takes a value in [-1, 1]. The dimensions correspond to 1) moving forward and backwards, 2) moving left and right, 3) rotating left and right, 4) rotating up and down, and 5) jumping or crouching. Agents can take any combination of act ions simultaneously. Movement dynamics are subject to
a
Out of patch
In patch 1
In patch 2
b
Trained agent
Random policy
60
60
6 m
Score
Score
8 m
10 m
12
0
3600
3600
Steps
Steps
Figure 2: Performance. a) Agent behavior from representative evaluation episode. Shaded regions define
when the agent is outside of any patch (white), inside patch 1 (red), inside patch 2 (blue). b) Episode score for a representative trained agent (solid lines), and a representative agent with random uniform action policy (dashed lines) in each evaluation environment. Shaded regions denote standard errors across evaluation episodes.
4

---

Published in Transactions on Machine Learning Research (4/2023)
the inertia of the environment. Actions were taken by sampling from Gaussians parameterized by the policy network head output for each action dimension.
We use a state-of-the-art continuous control deep reinforcement learning (RL) algorithm for training our agents: maximum a posteriori policy optimization (MPO; Abdolmaleki et al. 2018). MPO is an actor- critic, model-free RT, algorithm which leverages samples to compare different actions in a particular state and then updates the policy ensure that better actions have a high probability of being sampled. MPO alternates between policy improvement which updates the policy using fixed Q-function and policy evaluation which updates the estimate of the Q-function. Following previously described methods, agents were trained in distributed manner with each agent interacting with 16 environments in parallel (Cultural General Intelligence Team et al., 2022). Agent architecture and training hyperparameters were the same as in Cultural General Intelligence Team et al. 20222, unless otherwise stated. Experience was saved in an experience buffer. Agent parameter updates were accomplished by sampling batches and using MPO to fit the mean and covariance of the Gaussian distribution over the joint action space (Abdolmaleki et al.) 2018). Agent architecture, observation space, and the MPO algorithm were selected in part because of the success of these implementational choices in more complex environments (see Cultural General Intelligence Team, 2022).
Three agents were trained in each of four discount rate treatments (N= = 12), selected on the basis of MVT simulations (Figure 3d).. Agents were each initialized with a different random seed, and trained for 12e7 steps using the Adam optimizer (Kingma & Ba, 2014) and a learning rate of 3e 3e-4, On each training episode, patch distance was drawn from a random uniform distribution between 5 m and 12 m, and held constant for each episode. Trained agents were evaluated on 50 episodes of each evaluation patch distance (i.e., 6, 8, 10, and 12 m). By manipulating patch distance, we vary the amount of time it takes agents to travel between patches, which in turn varies the resource-richness of the environment similarr to many animal experiments on adaptive patch foraging behavior (Cowie 1977; Kacelnik, 1984). If an agent was within patch at the end of an ovaluation episode (e.g., Figure 2h), this final patch encounter was excluded from all analysos, as no distinct patch leave behavior could be verified to determine the total steps in this patch.
Videos of a representative agent during training and evaluation are available in the supplementary material.
3 Results
For several results below, we fit data using linear regression of the form, y = bæ + a, where free parameter b is the slope of the fitled line, and a is constant. b references the predicted relationship, negative or positive, of the rate of change in y (e.g., steps in patch; Figure 3a) per unit change in x (e.g., patch distance in meters). In text, we report b, along with its standard error (e.g., b =-550 + 0.50). For full statistical reporting, see Appendix A.1
3.1 Environment adaptation
Trained agents displayed behavior consistent with successful patch foraging loagiing--eentss learned to leave patches before they were fully depleted of reward (mean leaving step = 121.7), and traveled for several steps without reward in order to reach a refreshed patch (mean travel steps between patches = 57.7; e.g., Figure Pb. We computed each agent's mean final score in each patch distance evaluation environment (c.g., single representative agent in Figure 2b) to analyze whether agent score varied systematically with patch distance. Analysis showed agents achieved a higher score on episodes where patches were closer together (p < 0.05, linear regression slope b 5-8.2 + 0.21).
In patch foraging theory and experimental animal research, animals intelligently adapt patch leaving times to the environment in which they find themselves. Thcory predicts agents monotonically increase their stops in patch when the distance between patches increases (Charnov] 1976 Stephens & Krebs 2019). Similarly, in biological data, agents stay a longer time in patches when alternative patches are further away (Cowie, 1977 Hayden et al., 2011 Kacelnik 1984; Krebs et al., 1974). This adaptive behavior present in the current agents- trained agents increased their patch leaving times when patch distance increased, leaving patches later when travel distance was higher (p < 0.05, b = 9.60 + 0.87; Figure 3a). These results support
5

---

Published in Transactions on Machine Learning Research (4/2023)
our contribution that the current deep reinforcement learning agents learn to adaptively trade off travel time and patch reward in patterns similar to biological foragers.
3.2 Optimality
Above we show that trained agents are able to successfully forage, and intelligently adapt their foraging behavior to the environment. in accordance with patch foraging theory (Charnov 1976 Stephens & Krebs 2019), and animal behavior (e.g., Cowie, 1977.. However, do these agents adapt optimally according to the marginal value theorem (MVT), like many animals (Stephens & Krebs, 2019)? As stated above, the MVT provides a simple rule for when to leave patches optimally (Charnov] 1976 Shuvaev et al. 2020 That is, an agent should leave a patch when the reward rate of the patch drops below the average reward rate of the environment (Figure 3t).
The current agents however temporal discounting methods, which exponentially diminish rewards in the future. In other words, agents are tasked with solving discounted patch foraging problem. Given that agents are effectively asked to compare the values between the current patch reward on the next step relative to a refreshed patch reward after several travel steps, temporal discounting encourages longer patch leaving times relative to predictions from the MVT. Below, we evaluate agents relative to the MVT, which
animals are typically compared against Cooiie 1977, Charnov & Parker 1995; Hayden et al.) 2011 Krebs al. 2019; Stephens Krebs, 2019). We also attempt to account for temporal discounting in the MVT compare agents against the task that they are given.
et al. 1974 Pacheco-Cobas
For each agent and evaluation environment (e.g., 6 m), environment by calculating the average reward per step for each evaluation episode. Over all evaluation episodes for each agent and environment, this provides an estimate of the optimal patch leaving step (see Figure D and Figure 3g) Here, we take each agent's mean patch leaving time relative to the MVT across patch distance environments, and test these 12 values (one for each agent) are significantly different from
estimate the average reward rate of the
P
200
200
0.04
500
Steps in patch
Steps in patch
Optimal patch steps
0.0157
0
Reward
0.99757
100
100
0.02
250
MVT
0.89177
0.99337
89637
10 12 Patch distance (m)
6
10
250
500
250
500
Patch distance (m)
Step in patch
Travel steps
200
50
Steps relative to
Steps relative to
Steps patch
Steps relative discounted MVT
MVT
MVT
100
-10
-10
0
-50
6
8
10
12
6
8
10 12
6
8
10
12
6
8
10 12
Patch distance (m)
Patch distance (m)
Patch distance (m)
Patch distance (m)
Figure 3: Patch leaving times. a) Average of all agents. b) Average of agents grouped by discount rate. c) Graphical model of the MVT solution. Where the patch reward rate (solid black line) intersects the observed average reward rate the environment as determined by the agent's behavior (dashed horizontal black line), determines the MVT optimal average patch leaving time (solid vertical red line). d) Patch leaving times prescribed by the MVT (black), and simulation results for the MVT considering discount rates. e) Mean difference between the observed and optimal patch lcaving time for all agents, and f) agents grouped by discount rate. g) Represenlative single trained agent patch leaving times (dots) against the MVT solution (red lines). h) Mean difference between the observed and discounted MVT patch leaving time for agents grouped by discount rate. All vertical lines and/or shaded regions denote standard errors over agent means in each patch distance evaluation environment.

---

Published in Transactions on Machine Learning Research (4/2023)
zero (Figure 3-) using a one-sample t-test. Comparing the difference between average observed and MVT optimal patch leaving times, agents tend to overstay in patches relative to the MVT optimal solution (p < 0.05; mean = 15.8 steps above MVT optimal, standard error = 2.7; Figure 3p). These results are consistent with the patch foraging behavior of humans and other animals in computerized laboratory tasks (Cash-
Padgett Hayden 2020; Constantino & Daw] 2015: Hutchinson el al. 2008 Kane et al. 2017 Nonacs 20010, and are also consistent with temporal discounting predictions.
We now ask if agents trained with higher temporal discounting rates behave closer to MVT optimal (Figure D. Here, we take the mean patch leaving time relative to the MVT across patch distance environments, and group these values by agent discount rate, leaving 3 agents per discount rate. Using linear regression, we test whether the difference between observed and MVT optimal patch leaving times decreases with discount rate. As expected, agents trained with higher temporal discounting rates tend to behave closer to MVT optimal (p < 0.05, b 2784.01 + 992.19; Figure 3 for details see Appendix A.1.2
Are agents then optimal after accounting for temporal discounting rales in the MVT solution? We accounted for the temporal discounting rate by simulating individual stay and leave decisions at many patch leaving steps (for details, see Appendix A.2). Agents could either stay for additional step of reward before leaving a patch, or immediately leave the patch, where the subsequent 5000 steps were simulated as alternating between a fixed number of steps in a patch and a fixed number of steps traveling between patches. For example, on step 42 within a patch, and given a future fixed travel time of 50 steps and a future fixed patch time of 100 steps, is it more beneficial to leave immediately or stay in the patch for an additional step of reward? Over a grid of subsequent fixed patch and travel steps, the difference in the discounted return (sum of discounted rewards) between each stay/leave decision provided an indifference curve, where the 5000-step discounted return was equal for staying relative to leaving. Where this stay Slay/leang leavee indifference step matched the fixed patch steps provided an approximation of an average patch time where the value of leaving is about to exceed the value of staying.
After accounting for each agent's temporal discounting rate in the MVT (Figure 31), we ask if agent patch leaving times approach the optimal solution (Figure 3). Similar to above, we take each agent's mean patch leaving time relative to the discounted MVT solution across patch distance environments, and test if these 12 values are significantly different from zero (Figure 3h) using a one-sample t-test. Comparing the difference between average observed and discounted MVT oplimal patch leaving limes, agents were not significantly different from the optimal solution (p = 0.74; mean = 0.9 steps above discounted MVT optimal, standard error- = 2.8; Figure 3 although agents appear to slightly understay or overstay in patches depending on the environment (see Appendix 2). Overall, these results support our contribution that the current agents approach optimal patch foraging behavior when accounting for temporal discounting.
3.3 Dynamics and patch leaving time variability
Above we show that trained agents approach optimal patch leaving time behavior when accounting for temporal discounting. Here, we investigate how these agents decide to leave patch by interrogating internal dynamics in comparison to theory and neural dynamics recorded from non-human primates.
Theory and modeling work have shown that patch foraging decisions can be made using a simple evidence accumulation mechanism, which can approximate MVT solutions in complex enviromments Davidson & El Hady 2019). Evidence accumulation is a general decision making mechanism, which has been proposed to underlic many perceptual- and value-based decisions in animals (Brunton et al., 2013 Gold & Shadlen 2007 Wispinski al.] 2020). In brief, a decision variable representing the degree of evidence in support of decision (i.e., leaving the current patch) accumulates over multiple time steps until it reaches a threshold (Figure 4). When decision variable reaches threshold dictates when decision is made (i.e., when the
agent commits to lcaving a patch). Evidence accumulation is especially useful when considering evidence that is delivered over multiple time points (e.g., rewards in a patch, Davidson & El Hady] 2019 Hayden et al. 2011 or noisy visual input, Gold & Shadlen 2007: Ratcllff et al.) 2016 Wispinski et al. 2020 Key mochanisms in this model typically include the slope with which evidence accumulates, and the distance that evidence needs to travel from baseline to threshold for a decision to be made (Gold & Shadlen, 2007]
Ratcliff et al. 2016). For example, when evidence for a decision is stronger, the decision variable tends to
7

---

Published in Transactions on Machine Learning Research (4/2023)
accumulate with a higher slope, reaching threshold sooner (Figure 4p). Similarly, when decision requires less evidence, the distance between baseline and threshold can be decreased so decisions are made sooner. Both of these mechanisms are typically subject to variability in the internal state of the decision maker (Gold & Shadlen 2007: Ratcliff et al. 2016).
Neural recordings in primate cingulate cortex similar to an evidence accumulation mechanism have been ob- served during computerized patch foraging task, suggesting that such mechanism may underlie biological solutions to the patch foraging problem (Hayden et al., 2011). Here we investigate LSTM layer activity in single trained agent, and show several emergent similarities with neural data in foraging primates. The results described below are consistent with several other agents we investigated.
In this section we ask if variability in the internal state of the agent can explain earlier or later patch leaving times within the same patch distance environment, as in biological agents (Hayden et al.) 2011]. We took patch encounters where an agent had first enterod a newly refreshed patch until it first left that patch. In these encounters, agents experience the same exponential decrease in reward within a patch but displayed variable patch leaving times. As in [Hyden et al. (2011), divided these data into quartiles based patch leaving times. Patch encounters were split into exclusive groups named: Earliest (25th percentile), Early
b
Threshold
Steps in patch
200
Quartile
Proportion of variance explained
0.25
Decision variable
Latest
0.001
100
Late
Slope
Early
+6
Earliest
-B
Steps from patch entry
6
8
10 12
Patch distance (m)
PC
Unit activity
PC activity
-1
25
50
75
-75
-50
-25
0
25
50
75
-75
-50
-25
Steps from patch entry
Steps from patch exit
Steps from patch entry
Steps from patch exit
d
Threshold baseline normalized
1.2.
0.01
10-1
PC
Mean slope normalized
Unit activity
Slope
Unit
-0.02
Steps from patch entry
10a
6 10 12
6
10
Patch distance (m)
Patch distance (m)
Figure 4: Dynamics from a single trained agent. a) Example of an idcalized evidence accumulation process. A decision variable accumulates to threshold, which determines patch leaving time. Here, variability in the slope of the decision variable explains patch leaving time differences. b) Patch encounters for single agent split into patch leaving time quartiles in cach evaluation environment. c) Mcan activity of an example LSTM unit aligned to patch entry (left) and patch exit (right), for each quartile of patch leaving times. Shaded regions along activity traces denote standard errors across patch encounters. Shaded bars at the top of the plot indicate steps where there a significant slope-quartile relationship (negative blue, positive in red). For patch entry-aligned data, traces in each quartile are plotted until median patch leaving time. d) Zoomed-in region of e) Mean slope by patch leaving time quartile for step 20 in d (log-scale). f) Principal components decomposition of LSTM layer dynamics. Sclected principal component (PC) in red. g) Activation of the selected principal component. h) Mean difference in example unit and example PC activity from patch exit to patch entry in each evaluation environment. i) Mean slope of example unit and example PC in each evaluation environment. All vertical lines and/or shaded regions denote standard errors across patch encounters.
8

---

Published in Transactions on Machine Learning Research (4/2023)
(50th percentile), Late (75th percentile), and Latest (100th percentile). These groups were taken equally from each evaluation environment, (6, 8, 10, 12 m) so results were independent of patch distance (Figure p
Using these groups, we can investigate if earlier patch leaving times in each evaluation environment had a higher slope of rising activity, as predicted by theory and biological data (Figure 4p). We perform a linear regression of the slope of LSTM unit change against patch leaving time quartile (e.g., Figure 4 at every time step, using data from all individual patch encounters. We performed these sliding regressions on both patch entry-aligned data (left panel Figure 4 and patch exit-aligned data (right panel Figure 4b) Primate data show that cingulate neurons have higher slope of activity before the Earliest patch leaving times, and lower slope of activity before the Latest patch leaving times (inset of Figure 4.). Here we find the same pattern in a number of LSTM units for several steps after patch entry, but before patch exit (blue shaded bars; left panel of Figure (-). In the example unit shown, there are 19 consecutive steps where there is a significantly higher slope for encounters that had shorter patch leaving times (e.g., step 20: p < 0.001, b = -0.0174 = 0.002; Figure - In other words, the pattern of results in Figure 4 match the idcalized results consistent with primate neural data and theory in the inset. of Figure 4a. Additionally, only 1/10 steps had significant relationship just before and after the patch encounter.
A principal components dimensionality reduction was performed on the LSTM data, and most results for the example unit can also be seen in a component accounting for roughly 14% of LSTM layer variability (Figure -.. The expected negative relationship between activity slope and patch leaving quartile (e.g., Figure A and Figure 4e) was statistically significant for 15 consecutive steps after patch entry (blue shaded bars; left panel of Figure 4g).
Additionally, activity traces appear to begin at a similar level at patch entry, and appear to converge just before patch exit (Figure and g), suggesting that changes in the distance activity must travel from baseline to threshold do not underlie patch leaving time variability (although not significantly, see Appendix A.1 Overall, this suggests that variability in the slope of rising activity can explain variability in the agent's s patch leaving times, similar to primate neural recordings during patch foraging (Hayden et al. 2011]. These results support the first half of our contribution that internal dynamics of these agents show key patterns that are similar to neural recordings from foraging non- hon-human primates and patterns predicted by foraging theory.
3.4 Dynamics and environment adaptation
Above we show that within a patch distance environment, variability in the slope of LSTM layer activity predicts patch leaving time. We now turn to investigate the same relationships, but between patch distance enviromments. The behavioral analysis above showed that agents adapt patch leaving times to the resource- richness of the environment (Figure Bh). Agents stayed in patches for more steps when travel time to a new patch was longer, and stayed in patches for fewer steps when travel time to a new patch was shorter.
Under an evidence accumulation mechanism, patch leaving times can be adapted by changing the slope of activity, or by changing the distance decision variable needs to accumulate from baseline to threshold in each evaluation environment. Primate neural data suggests that when travel times between patches increase, the average slope of neural activity decreases, and the distance between baseline and threshold increases both acting to prolong the time the animals stay in a patch (Hayden et al. 2011). We investigated which of these changes, if any, may drive the increase of patch leaving time when patches are further apart.
Decrcasing the slope with which a decision variable accumulates toward a threshold prolongs patch leaving times. Here we took the average slope of activity during significant time steps (i.e., blue bars; Figure A and g) for every patch encounter, and separated these data by patch distance environment. Using a lincar regression, we find no significant relationship between the average slope of activity and patch distance environment in the example unit (p- = 0.18, = -0.00026 + 0.00020; Figure nor for the selected PC (p = 0.13, b = -0.00 -0.0034 + 0.002; Figure
Inereasing the distance a docision variable needs to travel from bascline to threshold another mechanism which prolongs patch leaving times (Davidson & El Hady] 2019). Here we took the difference between activity at patch exit and activity at patch entry for every patch encounter, and separated these data by
9

---

Published in Transactions on Machine Learning Research (4/2023)
patch distance. This difference in activity approximates the range decision variable may need to span to complete a patch leaving decision (e.g., Figure [.). Using linear regression, we find a positive relationship between activity range and patch distance environment for both the example unit (p < 0.05, b = 0.053+0.0033 Figure 4h), and for the selected PC (p < 0.05, b = 0.20 + 0.020; Figure 4n).
Through an evidence accumulation framework, these results suggest that adaptive behavior across envi- ronments is accomplished via changes in the distance decision variable must travel between baseline and threshold. In other words, when it takes less time to travel to a new patch, distance between baseline and threshold decrease, shortening the agent's time in the current patch. When it takes more time to travel to new patch, distance between bascline and threshold increase, prolonging the agent's time in the current patch. This mechanism is similar to work which suggests that decision thresholds during foraging may be estimated by an exponential moving average of past rewards (Constantino & Daw 2015 Davidson & El Hady 2019; Shuvaev et al. 2020) to approximate the average reward rate the environment the MVT [Charnov] 1976). In contrast, primate neural recordings suggest that adaptation between environments with short long travel times between patches is also driven by changes in the slope of activity (Hayden et al., 2011) different mechanism within an evidence accumulation framework, albeit with very similar behav- ioral results in this context. These results support the sccond half of our contribution that internal dynamics of these agents show key patterns that are similar to neural recordings from foraging non- human primates in this case results are different but are consistent with
and patterns predicted by foraging
the same underlying mechanism.
4 Discussion
Here we tested deep reinforcement learning agents in foundational decision problem facing biological ppanch foraging. We find that these agents successfully learn to forage in a 3D patch foraging
environment. Further, these agents intelligently adapt their foraging behavior to the resource-richness of
the environment in pattern similar to many biological agents (Cowic 1977
Hayden et al. 2011 Kacelnik
1984 Krebs et al. 1974; Stephens & Krebs
2019).
Many animals Cowie
1977
Krebs et al. 1974
Stephens & Krebs 2019 including humans in the wild
Pacheco-Cobos et al. 2019 have been shown to be optimal patch foragers. The deep reinforcement
learning agents investigated here also approach optimal foraging behavior after accounting for discount rate, These results are similar to those from humans and non-human primates in computerized patch foraging tasks participants tended to overstay relative to the MVT solution, but this discrepancy was significantly reduced after accounting for discount rate or risk sensitivity (Constantino & Daw 2015 CassePadgett & Hayden] 2020). Together, these results raise questions as to how and why humans seemingly discount future rewards in lab foraging /Constantino & Daw 2015 Hutchinson et al.] 2008), but show undiscounted patch leaving times nature (Pacheco-Cobon el al., 2019. Although outside the scope of this paper, these
issues may be reconciled with further work on foraging using different reinforcement learning approaches, such average reward reinforcement learning (Kolling & Akam 2017 Sutton & Barto 2018), especially in deep reinforcement learning models (Shuvaev et al. 2020 Zhang & Ross 2021), to potentially better model ecological decision making in biological agents. While the current paper considers only the MPO reinforcement learning algorithm due to its success in a similar environment (Cultural General Intelligence Team et al. 2022). future work may find improvements in learning to solve the patch foraging problem with alternative methods.
In several agents that learned adaptively patch forage, internal dynamics emerged to resemble several key properties of single-cell neural recordings from foraging non-human primates completing computerized patch foraging task (Hayden et al. 2011). These results complement experimental and theoretical work on how adaptive patch foraging behavior may be accomplished in biological agents by general decision making mechanism eeviencee accumulation (Davidson & El Hady] 2019 Gold & Shadlen] 2007 Wispinski let al. 2020). These results however do not necessarily mean that the agents investigated here, nor biological agents, use an evidence accumulation mechanism to solve the patch foraging problem (Blanchard & Hayden
20114] Kane et al.] 2021). Other strategics, or variations on accumulation models have also been proposed
to underlie foraging behavior (Davidson & El Hady 2019 Kilpatrick et al. 2021 Cazettes et al. 2022 As
10

---

Published in Transactions on Machine Learning Research (4/2023)
in biological research, decision making frameworks provide one way to interpret and test predictions about behavioral and neural data. Emergent patterns in artificial agents completing similar tasks to biological
agents provide concrete predictions for neural data and aid in interpretabllity (Banino et al. 2018 Hassabis
ctt al. 2017 Song et al. 2017 Wang et al.) 2016) espocially in ocological tasks like patch foraging where
neura ct al.
data is often more dilficult to interpret relative to in-lab, stationary, computerized Lasks (Pearson
2014).
In this paper we demonstrate agents capable of patch foraging in complex environment using continuous action space. Future experiments may build on the ecological complexity of the environment and interesting agent behavior may arisc in environments where the assumptions and predictions of the MVT start to brcak down (Davidson & El Hady 2019 Stephens & Krebs 2019). For example, biological realism may be increased by adding multiple patches, modeling a biologically realistic patch refresh rate, or adding competitive or collaborative agents to the environment (Bidari et al.
2022 Other biological rescarch argues that there are optimal movement strategics for finding unknown patch locations, and that many animals abide by these
movement. policies (Calhoun et al. 2014 Cisek 2019 Sims et al. 2008 Tello-Ramos el. al., 2015 Woodgate et al. 2017). Given that the current agents generate continuous and complex movement trajectories (Figure Ii), future work may investigate situations in which these optimal movement policies may emerge in artificial agents. The current movement trajectories may also be improved in future experiments by using alternative RL methods or modifying the sensory inputs available to the agents. Finally, foraging frameworks have been successfully extended to explain other aspects of intelligence, such as visual search (Wolfe et al. 2018), or human memory (Hills al.) 2012), which provide another avenue where reinforcement learning models of foraging may aid in intelligence research.
In conclusion, we have trained deep reinforcement learning agents on complex patch foraging task and for the first time observed the emergence of adaptive, optimal behavior, and neural dynamics that resembled those of biological agents. This paper contributes a model with which biological and artificial intelligence rescarchers may further undorstand patch foraging (Frankenhuis ct al. 2019 fundamental decision prob-
lem that strongly guided the evolution of biological intelligence (Cisek 2019
Slephens & Krebs 2019). Such
paradigms have been, and may continue be, critical in the continued development of artificial intelligence Hassabis et al. 2017 Lindsay 2021.
Broader Impact Statement
First, these results suggest the utility of using deep reinforcement learning agents to model foraging behavior and its underlying neural mechanisms in biological agents, especially more complex environments where the optimal solution is not known. Second, these model-free agents and the patch foraging problem can provide insights into other problems requiring the consideration of value at multiple temporal resolutions, such information search and resource management applications. Finally, we argue that patch foraging provides relevant environment from the biological literature to investigate continual lcarning in artificial agents, as biological agents must adapt their behavior to changing environment statisties (e.g., when changing locations from a scarce to a plentiful environment, or during seasonal changes in resources).
Acknowledgments
We are deeply indebted to our colleagues, including Leslie Acker, Andrew Bolt, Michael Bowling, Dylan Brenneis, Adrian Collister, Elnaz Davoodi, Richard Everett, Arne Olav Hallingstad, Nik Hemmings, Ed- ward Hughos, Michael Johanson, Marlos Machado, Drew Purves, Kimberly Stachenfeld, Richard Sutton, Jane Wang, Alexander Zacherl, and the DeepMind Cultural General Intelligence Team for their support, suggestions, and insight regarding this work. We would also like to thank Ben Hayden and Fric Charnov for insightful comments on an earlier version of the manuscript. This work was funded solely by DeepMind. The authors declare no competing interests.
11

---

Published in Transactions on Machine Learning Research (4/2023)
References
Abbas Abdolmalcki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Hoess, and Martin Ried- miller. Maximum posteriori policy optimisation. arviv preprint or orX:::1866.0000 06920, 2018.
Ryan Paul Badman, Thomas Trenholm Hills, and Rei Akaishi. Multiscale computation and dynamic atten- tion in biological and artificial intelligence. Brain Sciences, 10(6))396, 2020.
Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. orXiv preprint Xiv::19909 07528, 2019.
Andrea Banino, Caswell Barry, Benigno Uria, Charles Blundell, Timothy Lillicrap, Piotr Mirowski, Alexan- der Pritzel, Martin J Chadwick, Thomas Degris, Joseph Modayil, et al. Vector-based navigation using grid-like representations in artificial agents. Nature, 557(7705)::29 433, 2018.
Subekshya Bidari, Ahmed El Hady, Jacob D Davidson, and Zachary Kilpatrick. Stochastic dynamics of social patch foraging decisions. Physical Review Research, 4(3):033128, 2022.
Tommy C Blanchard and Benjamin Y Hayden. Neurons in dorsal anterior cingulate cortex signal postdcci- sional variables in a foraging lask. Journal of Neuroscience, 34(2):646 655, 2014.
Bingni W Brunton, Matthew M Botvinick, and Carlos D Brody. Rats and humans can optimally accumulate evidence for decision-making. Science, (5 2013.
Adam Calhoun, Sreekanth H Chalasani, and Taryana Sharpee. Maximally informative foraging by caenorhabditis elegans. eLife, 3:e04220, 2014.
Tyler Cash-Padgett and Benjamin Hayden. Behavioural variability contributes to over-staying in patchy foraging. Biology Letters, 16(3))20190915, 2020.
Fanny Cazettes, Luca Mazzucalo, Masayoshi Murakami, Joao P. Morais, Alfonso Renart, and Zachary F. Mainen. A repertoire of foraging decision variables in the mouse brain. bioRaiv, 2022. doi: 10.1101/2021. 04.01.438090.
Eric L Charnov. Optimal foraging, the marginal value theorem. Theoretical Population Biology, ((2):129-136, 1976.
Eric L Charnov and Gordon H Orians. Optimal foraging: Some theoretical explorations. 2006.
Eric L Charnov and Geoffrey A Parker. Dimensionless invariants from foraging theory's marginal value theorem. Proceedings of the National Academy of Sciences, 92(6):1446-1450, 1995.
Paul Cisck. Rosynthesizing behavior through phylogenetic refinement. Attention, Perception, & Psy-
chophysics, (1(7):2265-228) 2019.
Scott L Coleman, Vincent R Brown, Daniel S Levine, and Roger L Mellgren. A neural network model of foraging decisions made under predation risk. Cognitive, Affective, & Behavioral Neuroscience, 5(4): 434 451, 2005.
Sara M Constantino and Nathaniel Daw. Learning the opportunity cost of time in a patch-foraging task. Cognilive, Affective, & Behavioral Neuroscience, 15(4):837 853, 2015.
Richard J Cowie. Optimal foraging in great Lits (Parus major). Nature, 268(5616):137 139, 1977.
Cultural General Intelligence Team, Avishkar Bhoopchand, Bethanie Brownfield, Adrian Collister, Agustin Dal Lago, Ashley Edwards, Richard Everett, Alexandre Frechette, Yanko Gitahy Oliveira, Ed- ward Hughes, Kory W. Mathewson, Picrmaria Mendolicchio, Julia Pawar, Miruna Pislar, Alex Platonov, Evan Senter, Sukhdeep Singh, Alexander Zacherl, and Lei M. Zhang, Learning robust real-time cultural transmission without human data. Xr preprint arziv. 2203222222111 2022.
12

---

Published in Transactions on Machine Learning Research (4/2023)
Jacob D Davidson and Ahmed El Hady. Foraging an evidence accumulation process. PLoS Computational Biology, 15(7)=e100700) 2019.
Willem Frankenhuis, Karthik Panchanathan, and Andrew G Barto. Euriching behavioral ecology with reinforcement learning methods. Behavioural Processes, 161:94 100, 2019.
Joshua I Gold and Michael N Shadlen. The neural basis of decision making. Annual Review of Neuroscience, 30:535-574, 2007.
Aya Goldshtein, Michal Handel, Ofri Eitan, Afrine Bonstein, Talia Shaler, Simon Collet, Stefan Greif, Ro- drigo A Medellín, Yuval Emek, Amos Korman, et al. Reinforcement learning enables resource partitioning in foraging bats. Current. Biology, 30(20):409--4102, 2020.
Timothy Hanks and Christopher Summerfield. Perceptual decision making rodents, monkeys, and humans. Neuron, 93(1):15 31, 2017.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience- inspired artificial intelligence. Neuron, (5(2):245-22) 258, 2017.
Benjamin Y Hayden, John M Pearson, and Michael I. Platt. Neuronal basis of sequential foraging decisions in patchy environment. Nature Neuroscience, 14(7):933 939, 2011.
Thomas T Hills, Michael N Jones, and Peter M Todd. Optimal foraging in semantic memory. Psychological Review, 119(2):331, 2012.
John MC Hutchinson, Andreas Wilke, and Peter M Todd. Patch leaving in humans: Can generalist adapt its rules to dispersal of items across patches? Animal Behaviour, 75(4):1331-1344 13499, 2008.
Alejandro Kacelnik. Central place foraging in starlings (Sturmus vulgaris). I. Patch residence time. The Journal of Animal Ecology, pp. 283 209, 1984.
Gary A Kane, Flena M Vazey, Robert C Wilson, Amitai Shenhav, Nathaniel D Daw, Gary Aston-Jones, and Jonathan Cohen. Increased locus coeruleus tonic activity causes disengagement from a patch-foraging lask. Coyreilive, Affective, & Behavioral Neuroscience, 17(6):1073 1083, 2017.
Gary A Kane, Morgan H James, Amitai Shenhav, Nathaniel D Daw, Jonathan D Cohen, and Gary Aston- Jones. Rat anterior cingulate cortex continuuusyy signals decision variables in a patch foraging task. bioRrie, 2021.
Zachary P Kilpatrick, Jacob D Davidson, and Ahmed El Hady. Uncertainty drives deviations in normative foraging decision strategies. Journal of the Royal Society Interface, 18(180):20210337, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXio preprint
arXiv:1412.698 2014.
Nils Kolling and Thomas Akam. (Reinforcement?) Learning to forage optimally. Current Opinion in Newrobiology, 46:162-169, 2017.
John R Krebs, John C Ryan, and Eric L Charnov. Hunting by expectation or optimal foraging? A study of patch use by chickadees. Animal Behaviour, 22:953 22:953-IN3, 1974.
Long Ji Lin. Self-improvement based on reinforcement learning, planning and teaching. In Proceedings of the Eighth International Conference on Machine Learning, ML 99, pp. 323 327, 1991. ISBN 1558602003.
Grace W Lindsay. Convolutional neural networks model of the visual system: Past, present, and future.
Journal of Cognitive Neuroscience, 33(10):2017-2031, 2021.
Eran Lottem, Dhruba. Banerjce, Pictro Vertechi, Dario Sarra, Matthijs oude Lohuis, and Zachary F
Mainen. Activation of serotonin neurons promotes active persistence a probabilistic foraging task. Nature Communicaaions, 9-1):1112, 2018.
13

---

Published in Transactions on Machine Learning Research (4/2023)
Flavio BP Malavazi, Remy Guyonneau, Jean- -ran-Baptiste Fasquel, Sebastien Lagrange, and Franck Mercier. Lidar-only based navigation algorithm for an autonomous agricultural robot. Computers and Electronics in Agriculture, 154:71- 79, 2018.
Matt Miller, Kevin M Ringelman, John M Eadie, and Jeffrey C Schank. Time to fly: comparison of marginal value theorem approximations in agent-based model of foraging waterfowl. Ecological Modelling, 351:77 86, 2017.
Volodymyr Maih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):5299 533, 2015.
P Read Montague, Peter Dayan, Christophe Person, and Terrence J Sejnowski. Bee foraging in uncertain environments using predictive hebbian learning. Nature, 377(6551):725-725) 1995.
Juliano Morimoto. Foraging decisions multi-armed bandit problems: Applying reinforcement learning algorithms to loraging data. Journal of Theorelical Biology, 467:48 56, 2019.
Yael Niv, Daphna Joel, Isaac Meilijson, and Eytan Ruppin. Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors. Adaptive Behavior, 2002.
Peter Nonacs. State dependent behavior and the marginal value theorem. Behavioral Ecology, 12(1):71-88 83, 2001.
Luis Racheco-Cobos. Bruce Winterhalder, Cecilia Liatia, Marcos F Rosetti, Robyn Hudson, and Cody T Ross. Nahua mushroom gatherers arca-restricted search strategics that conform to marginal value theorem predictions. Proceedings of the National Academy of Sciences, 116(21):10339 10347, 2019.
John M Pearson, Karli K Watson, and Michael L Platt. Decision making: The neuroethological turn. Neuron, 82(5):950-99) 10195500-555000 2014.
Emnanouil Antonios Plalanios, Abulbair Saparov, and Tom Mitchell. Jelly bean world: A testbed for never-ending learning. arXiv preprint arXiv:2002. 06366, 2020.
John M Pleasants. Optimal foraging by nectarivores: test of the marginal-valu theorem. The American Naturalist, 134(1):51-71, 1989.
Roger Ratcliff, Philip L Smith, Scott D Brown, and Gail McKoon. Diffusion decision model: Current issues and history. Trends in Cognitive Sciences, 20(4):260 281, 2016.
Sergey Shuvaev, Sarah Starosta, Duda Kvitsiani, Adam Kepecs, and Alexei Koulakov. R-learning in actor- critic model offers biologically relevant mechanism for sequential decision-making. Advances in Neural Information Processing Systems, 33:18872 13:18872-18882, 2020.
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021.
David W Sims, Emily J Southall, Nicolas E Humphries, Graeme C Hays, Corey JA Bradshaw, Jonathan W Pitchford, Alex James, Mohammed Z Ahmed, Andrew Brierley, Mark A Hindell, et al. Scaling laws of marine predator search behaviour. Nature, 461(1182):0088-1102 2008.
H Francis Song, Guangyu R Yang, and Xiao-Jing Wang. howard-baser training of recurrent neural networks for cognitive and value- based tasks. eLife, 6:e21492, 2017.
David W Stephens and John Krebs. Foraging theory. Princeton University Press, 2019.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Wenwu Tang and David A Bennett. Agent-based modeling of animal movement: review. Geogrophy Compass, 4(7):682-700 700, 2010.
14

---

Published in Transactions on Machine Learning Research (4/2023)
Maria Cristina Tello-Ramos, T Andrew Hurly, and Susan D Healy. Traplining in hummingbirds: Flying short-distance sequences among several locations. Behavioral Ecology, 26(3):812-8) 1182-19,, 2015.
Pietro Vertechi, Eran Lottem, Dario Sarra, Beatriz Godinho, Isaac Treves, Tiago Quendera, Matthijs N
oude Lohuis, and Zachary F Mainen. Inference-based decisions in hidden state foraging task: Differential contributions of prefrontal cortical arcas. Neuron, 106(1):166-17,, 2020.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforeement learn. ar preprint arXiv:1611.0576 2016.
Nathan J Wispinski, Jason P Gallivan, and Craig S Chapman. Models, movements, and minds: Bridging the gap between decision making and action. Annals of the New York Academy of Sciences, 1464(1):30-35 51, 2020.
Jeremy M Wolfe, Matthew Cain, and Abla Alaoui-Soce, Hybrid value foraging: How the value of targets shapes human foraging behavior. Allention, Perception, Psychophysics, 80(3):609 621, 2018.
Joseph L Woodgate, James C Makinson, Ka S Lim, Andrew M Reynolds, and Lars Chittka. Continuous radar tracking illustrates the development of multi-destination routes of bunblebees. Scientific Reports, 7(1):1-15, 2017.
Yiming Zhang and Keith W Ross. On-policy deep reinforcement learning for the average- criterion. arXiv preprint ar Xix:22106. 07329, 2021.
15

---

Published in Transactions on Machine Learning Research (4/2023)
A
Appendix
A.1 Statistical reporting
A.1.1
Environment adaptation
We computed each agent's mean final score in each patch distance evaluation environment (e.g., single representative agent in Figure 2b) to analyze whether agent score varied systematically with patch distance. Agents achieved a higher score on episodes where patches were closer together (Lincar mixed effects regression with random agent intercept: b =-5.2 + 0.21, p = 3.96 10-166; Figure 2 2p). Data consisted of 12 mean final scores (one for each agent) x 4 patch distances.
Trained agents adapted their patch leaving times to the environment, leaving patches later when travel distance is higher (Linear mixed effects regression with random agent intercept: b = 9.60 + 0.87, p 4.03 x 10 10-28; Figure 3a). Data consisted of 12 mean patch leaving times (one for each agent) x 4 patch distances.
A.1.2 Optimality
We take each agent's mean patch leaving time relative to the MVT across patch distance environments, and test if these 12 values are significantly different from zero (Figure 3P) using a one-sample t-test. Comparing the difference between average observed and optimal patch leaving times, agents tend to overstay in patches relative to the optimal solution (One-sample t-test: f(11) 5.60, p 1.60 10-1; mean 15.8 steps above MVT optimal, standard error 2.7; Figure 3e). Bonferroni-correcteed one-sample t-tests show that agents significantly overstayed relative to the MVT solution in all evaluation environments (ps < 0.0015), except for 12 m (p- = 0.047).
We take the mean patch leaving time relative to the MVT across patch distance environments, and group these values by agent discount rale, leaving agents per discount rate. Using linear regression, we test whether the difference between observed and MVT optimal patch leaving times decreases with discount rate. As expected, agents trained with higher temporal discounting rates tend to behave closer to MVT optimal (Linear regression: b 2784.01 + 992.19, p 0.019; Pearson correlation: r(10) -0.66, p 0.019; Figure A. Data consisted of 3 agent's 'ss mean difference between observed and MVT patch leaving times (averaged across 4 environments) x 4 discount rates.
We take each agent's 's mean patch leaving time relative to the discounted MVT solution across patch distance environments, and test if these values are significantly different from zero (Figure 3)) using a one-sample I-test. Comparing the difference between average observed and discounted MVT optimal patch leaving times, agents were not significantly different from the optimal solution (One-sample t-test: t(11) 0.34, p 0.74: mean 0.9 steps above discounted MVT optimal, standard error 2.8; Figure Bh). Bonferroni-correcter one-sample t-tests show that agents significantly overstayed in the 6 and 8 m evaluation environments (ps < 0.0067), understayed in the 12 m (p 0.0023), and were not significantly different from optimal in the 10 m environment (p- 0.30).
A.1.3 Dynamics and patch leaving time variability
Trained dynamics agents were evaluated on 30 episodes of each evaluation patch distance (iee., 6, 8, 10, and 12 m). For the individual trained agent presented in Figure evaluation data were processed into 3790 unique patch encounters (mean 31.6 unique patch encounters per evaluation episode). While single LSTM units (of 256) were analyzed, principal components analysis (PCA) was also conducted to visualize patterns that accounted for larger amounts of variability in this layer. Exploratory analysis found that PC2 captured several key effects described in Hayden et al. (2011) while also accounting for 14% of LSTM layer variability (the second most. in the network). Other PCs are included in Figure A.2 for completeness.
For singlo-trial dynamics data aligned to patch entry and patch exit (Figure 4 and g), the slope of LSTM unit or PC activity change across consecutive steps was regressed against patch leaving time quartile for 40 in-patch steps and 10 pre- post-patch steps. Significance threshold for linear regressions at each dynamics
16

---

Published in Transactions on Machine Learning Research (4/2023)
a
b
Sleps in patch
150
Steps in patch
150
Steps relative to
Steps relative discounted MVT
50
MVT
199067
88687
80
0.99757
80
-10
-50
6 8
10 12
Eueraz
Broar
0.007ar
0.77777
Patch distance (m)
Discount rate
Discount rate
Discount rate
Figure A.l: Patch leaving times. a) Agents grouped by discount rate. b) Agents grouped by discount rate and collapsed by patch distance environment. c) Mean difference between the observed and MVT patch leaving time collapsed by patch distance environment (Linear regression: b= 27844.0 + 992.19, p 0.019). d) Mean difference between the observed and discounted MVT patch leaving time collapsed by patch distance environment. All vertical lines denote standard errors.
time step was Bonferroni-correetted for 50 steps (i.e., p 0.001) independently for patch entry- and patch exit-aligned data for both the example unit (Figure 4c) and for the example PC (Figure 4g)
We find in several LSTM units a significant relationship between the slope of rising activity and patch leaving time quartile for several steps after patch entry, but before patch exit (blue shaded bars; Figure 4-) In the example unit shown, there are 19 consccutive stops where there is a significantly higher slope for encounters that had shorter patch leaving times (ps < 0.001; e.g. step 20 linear regression:
-0.0174 + 0.002, = 2.36 x 10-14, step 20 Pearson correlation: r(3774) -0.12, p 2.36 x 10-14, Figure 4e). Data for each lincar regression consisted of ~944 activity slopes 4 patch leaving time quartiles (Earliest, Early, Late, Latest). All results were equivalent when using Pearson correlations instead of linear regressions.
Exit step activity was significantly different between patch leaving time quartiles for the example unit (One- way ANOVA: F(3, 3775) = 7.15, p 8.76 x 10-5). Bonferroni-corrected pairwise follow-up tests show no significant differences other than the mean activity of the Latest patch leaving time quartile from all other conditions (ps- < 0.0012). For the selected PC, analysis similarly showed exit step activity was significantly different between patch leaving time quartiles (One-way ANOVA: F(3, 3775) 320.79, p 1.58 10 10-185). Follow-up lests show significant differences between all conditions (ps < 1.13 x 10-11) ") other than bet ween the Late and Latest patch leaving time quartiles (p 0.99).
A.1.4 Dynamics and environment adaptation
We took the difference between activity at patch exit and activity at patch entry for every patch encounter, and separated these data by patch distance. We find positive relationship between activity range and patch distance environment for the example unit (Linear regression: b 0.053 + 0.003, p 5.88 x 10 10-56, Pearson 5.59 x 10-56; Figure 4h, and for the selected PC (Linear regression: b = 0.20 + 0.020, p = 1.70 x 10-20 22; Pearson correlation: r(3774) = 0.16, = 1.70 x 10 10-22, Figure 4n). Data consisted of ~944 differences in exit and entry activity x 4 patch distance evaluation environments (6, 8, 10, and 12 m).
correlation: r(3774) = 0.25,
Decreasing the slope with which a decision variable accumulates toward a threshold tends to prolong patch leaving times. For both the example unit and PC, we took the average slope of activity across all steps for which there was significant predictive relationship between slope and patch leaving time quartile after patch entry (c.g., steps 5 to 23 after patch entry for the example unit; Figure 4p). We find no significant relationship between the average slope of activity and patch distance environment in the example unit (Linear regression: 5 = 0.00026 + 0.00020, = 0.18; Pearson correlation: r(3785) = -0.022, p = 0.18; Figure Q nor for the selected PC (Lincar regression:
-0.0034 0.002, p = 0.13; Pearson correlation: r(3784) -0.025, p = 0.13; Figure 4). Data consisted of 944 average activity slopes x patch distance evaluation environments.
17

---

Published in Transactions on Machine Learning Research (4/2023)
b
PC1 activity
Proportion of variance explained
0.25
5
PC2 activity
0
PC
d
5
PC3 activity
0
0
25
50 75
-75 -50
-25 0
Steps from patch entry
Steps from patch exit
Figure A.2: Major principal components from the LSTM layer of a representative trained agent. a) Propor- tion of LSTM layer variance explained by each of the first 10 PCs. PC2 (c) is highlighted the main text. b, c, d) Activation of the principal components accounting for the most amount of variance in the agent's LSTM layer dynamics. Average activation aligned to patch entry (left), and patch exit (right). Shaded blue and red bars indicate steps where there is significant slope-patch leaving time quartile relationship (negative in blue, positive in red). Traces in each quartile are plotted until median patch leaving time. Shaded regions along activity traces denote standard errors.
A.2 Accounting for discounting the marginal value theorem
We accounted for temporal discounting rate by simulating individual stay and leave decisions at many patch lcaving stops. Agents could cither stay for an additional step of reward before leaving a patch, or immediately leave the patch, where the subsequent 5000 steps were simulated as allernating between a fixed number of steps in patch and fixed number of steps traveling between patches. For example, on step 150 within a patch, and given a future fixed travel time of 50 steps and future fixed patch time of 100 steps, is it more beneficial to leave immediately (Figure A.3h: "leave" in black) stay in the patch for additional step of reward (Figure A.3 3; 150 in green)? By computing the discounted ed return for immediately leaving, and computing the discounted returns for remaining in the patch for one additional step of reward at every level of patch depletion, we generate a curve where the value of leaving can be compared to the value of staying at individual patch depletion points (Figure A.3b).
In Figure A.3 b, we show the simulated indifference step when there are 100 fixed steps in a patch and 50 fixed steps traveling between patches. An indifference step can be estimated for every choice of subsequent fixed patch and travel steps (e.g., in Figure AA.a),, which gives rise to indifference curve over these parameters (Figure A.3b) Using the observed mean travel steps from trained agents, we can instead sweep over only fixed steps in patch in order to evaluate an agent's choice of mean patch time. This assumes that the agent's mean travel time is constant and independent of mean patch time.
Where the stay/leave indifference curve matched the fixed patch steps (i.c., the unity line Figure A.3p) provided an approximation of single mean patch time that maximizes the discounted return (given a discount rate and observed mean travel steps).
18

---

Published in Transactions on Machine Learning Research (4/2023)
b
1/30
6.40
0.015
Indifference step
500
Discounted rewards
Retur
Patch leave step
Stay Leave
75
Leave
150
Intillaenis
Stay
Curve
Leave
250
6.35
500
500
-0.015
Step
Patch leave step
Mean patch time
Figure A.3: Accounting for temporal discounting in the marginal value theorem. a) Simulated discounted rewards for different artificial policies. After the first step, these simulated agents alternate between a fixed number of steps a patch, and a fixed number steps traveling between patches. b) Computed discounted return for artificial policies in a). c) Difference in discounted returns between stay and leave policies at different fixed patch times (assuming discount rate and fixed travel time). White arrow indicates discounted MVT estimate for optimal mean patch time.
A.3 Training details
We generally follow similar training procedures for the models described in Cultural General Intelligence Team et al. 20022.. Each agent was trained internal cluster for roughly 13 days, and used approximately 40 GiB RAM, 8 CPU, and 8 GPUs.
19


Reviews:
[{"review_id": "oClXfV0H7J", "reviewer": "Reviewer_hyzv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The use of a 3D patch foraging environment inspired by experiments from behavioral ecology, which adds ecological complexity and realism to the task.\n* The demonstration that the agents can learn to adapt their behavior in a way that is similar to biological foragers and approach optimal patch foraging behavior when accounting for temporal discounting.\n\nWeaknesses:\n* The paper does not provide a detailed description of the agents' architectures and hyperparameters used in the experiments.\n* The paper does not provide a comparison with other existing methods in the literature to demonstrate the effectiveness of the proposed method.\n* From the perspective of RL, (I guess) it is possible that the environment used in the experiments may not be complex enough to fully capture the real-world patch foraging phenomenon.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the use of deep reinforcement learning in simulating patch foraging behavior in artificial intelligence. The study found that the agents were able to learn adaptive patterns similar to those seen in biological foragers and approached optimal behavior when accounting for temporal discounting. Additionally, the authors observed internal dynamics in the agents that resembled neural recordings from non-human primates. The study suggests that both biological and artificial agents may arrive at similar solutions when faced with similar ecological pressures, indicating a common foundation for adaptive, intelligent behavior.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n* Can you provide a comparison with other existing methods in the literature to demonstrate the effectiveness of the proposed method?\n* It would be better for the authors to provide more information about the ecological complexity of the environment used in the experiments, especially for an audience that is not familiar with patch foraging or biological intelligence.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FAufc5lnzQ", "reviewer": "Reviewer_1JDk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think this paper provides an interesting connection between cognitive science/biology and reinforcement learning. There is a large literature modeling biological behaviors with reinforcement learning, and patch foraging seems well-studied, so I believe that the respective communities will be interested in this paper. One strength of this paper is that the authors clearly articulate the motivation for their work and provide a thorough explanation of the MVT and how it relates to patch foraging behavior in biological systems. The authors also present a detailed description of their experimental setup and method, making it easy for readers to understand how the RL agents were trained and evaluated.\n\nOne weakness of the paper is that the authors do not adequately discuss the limitations of their method or how it might be improved in future work. For example, the authors mention that the optimal patch foraging solution may be difficult to discover using model-free RL, but they do not explore why this might be the case or what alternative approaches might be more successful. In fact, it would be more surprising to me if deep RL with an LSTM wouldn’t be able to learn patch foraging, given recent successes of deep RL on simulated games. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors show that deep reinforcement learning is able to learn optimal patch foraging solutions in a simulated environment. The authors test their method on a simulated 3D foraging environment and compare the performance of their trained RL agent to a random baseline agent. These learned agents are able to skillfully trade off travel time and patch rewards. Furthermore, the internal representations of these agents are similar to those of biological agents.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The authors should provide a more detailed discussion of the limitations of their method and potential avenues for future work. This could include a discussion of the challenges of using model-free RL to solve the patch foraging problem, as well as potential alternative approaches that might be more successful.\n- The authors could also provide a better baseline RL agent than just random and how it compares to the performance of the trained RL agent. Perhaps they could even compare to how humans accomplish this task? I think some literature already looks at this, so maybe just including that data to compare against would be interesting. \n- The trained agent trajectories in figure 1 don’t look optimal. Why is it taking long paths between the two patches? \n- “Patch leaving times, movement policies, and changes in the resource-richness of the environment additionally impact the average reward rate of the environment for an agent, which can create a non-stationary equilibrium.\n    - I'm a bit confused by this sentence. What is meant by a non-stationary equilibrium? I would think this is just an MDP problem, so I don’t see what is non-stationary, and what dynamics are in equilibrium? \n- It would be helpful if the authors provided more context for the human performance data, including details about the sample size and any other relevant characteristics of the participants.\n", "broader_impact_concerns": "There do not appear to be any significant ethical concerns raised by the work presented in this paper. However, the authors should consider adding a Broader Impact Statement to address any potential implications of their work for the use of RL in real-world applications, particularly in situations where foraging behavior is important (e.g., resource management, environmental monitoring).\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fU0QUMMF5n", "reviewer": "Reviewer_DAPD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThis paper appears to be the first to highlight patch foraging as a potentially fruitful environment in which to study the behavior of deep RL agents. From the literature review, this environment contains ample theoretical and experimental biological research results to guide the behavioral investigation of deep RL agents.\n\nThe paper is very well organized and clearly written. The main claims and findings are stated precisely; the empirical results presented in figures are tightly connected to the claims they are meant to support; the results are rigorously analyzed and explained.\n\nThe findings of this paper is relevant and interesting to the broader AI community. I believe this paper falls under the category of \"experimental and/or theoretical studies yielding new insight into the design and behavior of learning in intelligent systems\" in the TMLR solicitation. While this paper rightly constrains its claims to the specific task of path foraging, one may venture to believe that it lends support to the broader hypothesis that deep RL agents can serve as low-fidelity surrogate models of real biological agents.\n\n\nWeaknesses\n\nIn the environment used in this paper, as soon as an agent enters a patch, the other patch is instantaneously refreshed. This does not seem like a realistic assumption. While this paper merely adopts this assumption from the existing literature, it is still a limitation and weakness of this area of study.\n\nOnly one deep RL algorithm (MPO) is used for the entire behavioral analysis. This leaves the paper open to the criticism or concern that the findings may somehow be a fluke of MPO, that other deep RL algorithms could produce different conclusions.\n\nThe relation between this kind of behavioral research and the more mainstream agenda of designing better learning agents is not clear and should be discussed. One may not wish to replicate the behavior of biological agents if doing so inherits undesirable limitations of biological agents.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper conducts an empirical analysis of the behavior of deep reinforcement learning (RL) agents in an ecological patch foraging task. The main findings are that RL agents display behavioral patterns similar to biological agents, find optimal foraging strategies, and display neural network layer activities that are similar to non-human primate neural recordings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(Critical) By using the phrase \"non-stationary equilibrium\" on page 2, the authors seem to intend to convey some additional difficulty of the environment, but this is not true for the actual work done in this paper. I doubt the authors meant the word \"equilibrium\" in the sense of game theory with multiple agents, since this paper is only concerned with a single agent. \"Non-stationary\" does not accurately describe the actual concrete environment used in the paper, since time-dependence of the reward rate is not an issue because the agent observes the instantaneous reward rate. If the authors meant to say that a real-world patch foraging environment is a multi-agent problem where non-stationarity and equilibrium notions apply, then the authors ought to make it clear that these are additional difficulties that are not treated in the current paper.\n\n(Minor) Given that there exists a known theoretical optimal strategy, it is not surprising that a deep RL agent can learn to find it. The paragraph on page 2 that stresses the difficulty of the environment appears excessive and can be toned down.\n\n(Minor) The spatial trajectories shown in Figure 1d currently do not seem to serve much purpose in this paper. The authors should explain what a reader is supposed to learn from it.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers recognized that this work is interesting and technically valid. They had some concerns about clarity, the lack of comparisons to other models, and some of the choice of words. But, they were largely satisfied with the revisions the authors made to address these concerns, and all recommended acceptance.
Decision: Accept (poster)