Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Authors: Jonathan Cook
Chris Lu
Edward Hughes
Joel Z Leibo
Jakob Nicolaus Foerster
Venue: NeurIPS 2024 poster (2024)
Keywords: social learning
cultural accumulation
in-context learning
reinforcement learning
Primary area: neuroscience_and_cognitive_science
TL;DR: We investigate cultural accumulation among reinforcement learning agents and demonstrate that it can improve upon "single-lifetime" learning.

Abstract: Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds an expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches to reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building on the previously demonstrated ability for reinforcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for a single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of a generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in-weights cultural accumulation can be interpreted as analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.

Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning
Jonathan Cook
Chris Lu FLAIR, University of Oxford
FLAIR, University of Oxford
j jonahaan cook20hertford. ox. ac.
ac uk christopher. lu@eng. .
Edward Hughes
Joel Z. Leibo Google DeepMind
Jakob Foerster
Google DeepMind
FLAIR, University of Oxford j jakobBeng ox ac uk
edwarddughhss@goggle.ccm com
jzlogoogle.o com
Abstract
Cultural accumulation drives the open-ended and diverse progress in capabilities spanning human history. It builds expanding body of knowledge and skills by combining individual exploration with inter-generational information transmission. Despite its widespread success among humans, the capacity for artificial learning agents to accumulate culture remains under-explored. In particular, approaches reinforcement learning typically strive for improvements over only a single lifetime. Generational algorithms that do exist fail to capture the open-ended, emergent nature of cultural accumulation, which allows individuals to trade-off innovation and imitation. Building the previously demonstrated ability for rein- forcement learning agents to perform social learning, we find that training setups which balance this with independent learning give rise to cultural accumulation. These accumulating agents outperform those trained for single lifetime with the same cumulative experience. We explore this accumulation by constructing two models under two distinct notions of generation: episodic generations, in which accumulation occurs via in-context learning and train-time generations, in which accumulation occurs via in-weights learning. In-context and in- in-weights cultural accumulation can be interpreted analogous to knowledge and skill accumulation, respectively. To the best of our knowledge, this work is the first to present general models that achieve emergent cultural accumulation in reinforcement learning, opening up new avenues towards more open-ended learning systems, as well as presenting new opportunities for modelling human culture.
Introduction
The capacity learn skills and accumulate knowledge over timescales that far outstrip single lifetime (i.e., across generations) is commonly referred to as cultural accumulation [Hofstede et al.) 1994 Tennie al. 2009 Cultural accumulation has been considered the key human success H continuously aggregating new skills, knowledge and technology. It also sustains generational improvements in the behaviour of other species, such as in the homing efficiency of birds [Sasaki and Biro [017].. The two core mechanisms underpinning cultural accumulation are social learning Banndur and Walters 1977. Hoppitt and Laland, 2013) and independent discovery Mesoudi] [011] Its effectiveness can be attributed to the flexibility with which participants engage these two mechanisms [Enquist et 2008 Rendell et al.) 2010 Mesoudi and Thornton 2018).
"Equal contribution.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Inttaaise generation Rr,
Agents ebserve throughout training Initial probubiiity Pasi
In- In-Context Accumulation
Generation
Generation
Generation
do second genoraion
This observation
impletity enables social loanning
ouperrfomingg the fox.
Train each agent using RI In-Contedt Learning
jaturn
AT fext
Thipprooss continaes geoeersoos ulinonely righet bone line.
endppencortly
Agents independent leorning to improve performance
Set final parameters
& annealed from remaving the dependancy Revi
genenilior or obsenwes praviceo
0
6
Trial
Figure 1: Left: flow chart describing our RL model of cultural accumulation. Right: An annotated,
illustrative plot demonstrating in- in-context accumulation as observed in our results.
Cumulative culture thus executes evolutionary search in the space of behaviours, with higher rates of independent discovery corresponding to a higher mutation rate. Ultimately, how individuals should balance social vs. independent learning depends on the validity and fidelity of social learning signals, i.e,, how successful other agents in the environment are at the same task.
Given the profound success of cultural accumulation in nature, it is natural to explore its applica- bility to artificial learning systems, which remains an under-cxplored research direction. In deep reinforcement learning (RL), the learning problem is usually framed taking place over a single "lifetime". Generational methods that do exist, such as iterated policy distillation (Schmitt et al. [018 Stooke al. [2222] and expert iteration (Anthony al. [2017], explicitly optimise new generations with hand-crafted and explicit imitation learning techniques, as opposed to learning accumulation implicitly, which should ultimately enable greater flexibility. Prior work has successfully demon- strated emergent social learning in RL [Borsa et al., 2017 Ndousse et al. [021] Bhoopchand et al.. 2023). showing that it helps agents solve hard exploration problems and adapt online new tasks. However, these works have only considered one iteration of information transmission from an expert to a student. In this work, we therefore investigate how social learning and exploration can be balanced to achieve cultural accumulation in RL. This would lay the foundations for an open-ended, population- based self-improvement loop among artificial learning agents. It would also establish new tools for modelling cultural accumulation.
In humans, cultural accumulation takes place at which knowledge, skills and technology are acquired [Perreault] 2012] Cochrane et al. 2023). We therefore present two corresponding formulations of cultural accumulation in RI. in-context accumulation, which operates over fast adaptation to new environments, and in-weights accumulation, which operates over the slower process of updating weights (i.c., training). The in-context setting can be interpreted as analogous to short-term knowledge accumulation and the in-weights setting as analogous long-term, skills-based accumulation.
number of timescales due to the different rates
We demonstrate the effectiveness of both the in-context and in in-weights models by showing sustained generational performance gains several tasks requiring exploration under partial observability" On each task, we find that accumulating agents outperform those that learn for a single lifetime of the same total experience budget. This cultural accumulation emerges purely from individual agents maximising their independent rewards, without any additional losses.
2 Background
2.1 Partially-Observable Stochastic Games
A Pobseall-O Stochastic Game (POSG) Kuhn 1953 (1, S. A,T, R, 7,),,) I = {I,..., NN} defines the set of agents, S defines the set of states, and A defines the direct product of action spaces for each agent, i.c., A A' x ... A where A' is the action space for agent i e T defines the stochastic transition function, T Ax + [0, 1]. de-
is defined by the tuple M
Code be found at htt.p//s/ggtttto..////////////aa

---

fines the reward function, R: Sx. A > R. and O defines the observation function, 0x: + [0 where O the observation space. ? E (0. 1] refers the discount factor of returns.
At each timestep i, each agent samples an action from its policy aj ~ TT. TT (at), where 0f corresponds to the policy parameters for agent i. These actions aggregated to form the joint action d; (a),..., of The environment then calculates the next state St41 ~ T(s,, at). The agents then
receive the next observation 0 ((5+-1) and reward Ir R($+41) from the environment. The objective for each agent i in POSG is to maximise its expected discounted sum of returns
[ E B0 (e ) Ei= 7 R(s),a)))
2.2 Partially-Observabbl Markov Decision Processes
A
Markov Decision Processes (POMDP) [Kaelbling et al. 1998) is special case of the POSG with only one agent. For a given agent in POSG, if the policy parameters of all other agents 0- are fixed, the POSG can be reduced to a POMDP for agent i. Informally, this can be done by assuming the other agents are part of the environment. In other words, sampling from other agent policies would be part of the transition function.
To solve POMDPs, we make use of memory-based policy architectures that represent the prior episode interactions at timestep f using o₁. Memory is needed to solve POMDPs because of the state aliasing introduced by partial-ohe ootiiallobervbbiliiyy. For recurrent neural networks (RNNs) Φe refers the hidden state at timestep f. We now condition the policy on this state of ~ Tg: (o' and also update the hidden state with recurrence function hig, such that Φe+1 has (o (of)..
2.3 Meta-RL
In this paper, we consider meta-RL settings where an agent tasked with optimising its return over distribution of POMDPs M.. At each episode POMDP is sampled M ~ .M and the agent is given K trials to optimise its cumulative return. This setting has also been called in-context RI. [Laskin et al. 2022]. We adopt the setting of Ni et al. [202T], where meta-RI. RL is formulated as another POMDP M, which the selection of M now part of the state s. The agent's 's last action aę-1, reward T and trial termination are appended to the observation oz.
One common approach meta-RL uses gradient-based optimisation train meta-policy with respect to M using recurrent network architectures (Duan et al. [016)] Wang er al., 2016]]. Within each episode, updates the agent's 's internal state ø enable static agent parameters implement an in-context RL procedure, allowing the agent to adapt to a specific M. During adaptation to new POMDP, this procedure aggregates and stores information from sequential observations. Over long enough episodes, agents can therefore face an in-context within-episode exploration of the environment could yield higher returns.
-epprottttilloiiiiinnnnnn tradeoff, where
2.4 Generational Training
Generational training refers to training populations of agents in sequence, thus chaining together multiple learning processes [Stooke et al. 2022). We define a generation 9m = {[9....... 9m population of N Npop agents. Each 9% can therefore be model weights, case of in-context RI. We distinguish Nppp from the N used in Section 2.1 because each member of generation of size Nppp may be within its POSG of N agents. We member of a generation. A new generation is generated by process that conditions on the old generation 9m++1 (gm). By modelling cultural accumulation over both in-context RL and RL training, f could capture generational accumulation in both knowledge and skills.
New } as
internal state in the
to index specific
3 Problem Statement
In this work, we investigate how to achieve cultural accumulation in RL agents. We formalise a measure of success in Equation [ where we compare the returns achieved in a held-out set of environments when learning is spread across G total generations to learning in single lifetime with
Our use of "episode" and "trial" aligns with Bauer al. 2023 which is the reverse of Duan al. 2016
3

---

the same total experience budget. This comparison evaluates whether our models capture the benefits of cultural accumulation over singe lifetime learning.
R
(T.
(I)
Policies are parameterised according to To xo(-))) where are parameters of the trained network and is the agent's internal state. We define in-context accumulation as cultural accumulation during online adaptation to new environments. In this setting. 0 are frozen and T is the length of an episode. We assume that meta-RL. training process produced and that the internal state & is used to distinguish between generations, which we elaborate in Section 4.1.2 We define in-weights accumulation as cultural accumulation over training runs. Here, the number of environment steps used for training each generation and each successive generation is trained from randomly initialised parameters 0, meaning that instead are the substrate for accumulation.
3.1 Environments
To study processes of cultural accumulation among humans, Cornish et al. [2017] use random sequence memorisation task in which sequences recalled by one participant become training data for the next in an iterated learning process. In doing so, they show a cumulative increase in recalled string length, thus simulating cultural accumulation. We thus present Memory Sequence as a simple environment for investigating basic cultural accumulation in RI. agents. In Memory Sequence, there is arbitrary sequence made up of digits that agents must infer over the course of in-context learning or training. Each action corresponds to predicting a digit. Episodes are fixed length and agents receive +1 reward for predicting the correct next digit and reward for predicting an incorrect next digit. The sequence is randomly sampled and fixed during in-context adaptation for in-context accumulation, whilst it is fixed across training for in-whights accumulation.
Memory Sequence:
Total Length: 20
Total Langth: 17.4
Total Length: 13
[Auc, www..yooowggree..
Figure 2: Left: A visualization the Goal Sequence Environment. Right: Routes travelled get
shorter across generations in the TSP environment. Visualisation implementation is based on Jumanji [onnet et al. 2024].
Goal Sequence: Previous work on social learning RL uses gridworld Goal Cycle environment [Ndousse et al.] [202T]. In this environment, there are n identical goals and agents must figure the correct eyclic ordering in which navigate through them. Agents receive an egocentric partial observation of the environment state on each timestep. Other prior work uses 3-dimensional Goal Cycle environment [Bhoopchand et al., 2023). Goal Cycle is limited in its ability evaluate generational improvement across learned agent behaviours, because once an agent has identified the correct cyclic order, it should simply repeat the same action sequence of cycling through the goals for the duration of episode. We therefore introduce Goal Sequence (Figure 2 simple adaptation of Goal Cycle, but with more open open-ended properties. Specifically, we replace the cycle with non-repeating sequence of goals that agents must discover by navigating based egocentric, partial observations. We use 7 x grid with goal types and agents observe 3 3 grid of cells directly in front of them. For in-context accumulation, the goal sequence and positions randomly generated and fixed across in-context adaptation. For in-weights accumulation, the goal sequence is fixed across training, but goal positions still randomised between episodes.
Travelling Salesperson: Finally, we use the Travelling Salesperson Problem (TSP). Given that cultural accumulation involves the transmission of information not otherwise immediately accessible

---

to new generations, we specifically focus on the obbervbbbe variant of TSP [Buck and Keller 2008 Noormohammadi-4 Asi and Taghirad 2019] in which some all of the cities' coordinates are not observed. The agent's objective is to visit every city, minimising the total length of the route travelled. The positions of cities are randomly generated and fixed across in-context adaptation for in-context accumulation. City positions are fixed across all of training for in-weights accumulation.
4 Cultural Accumulation in RL
We model cultural accumulation as taking place within POSGs. Each member of given generation 9m++1 learns in parallel on independent instance of the environment, which also contains static members of the previous generation, 9m- Therefore, from the perspective of given agent n in Jn++1 this is POMDP, since the prior, static generation be considered part of the environment. In this work, we focus on generational improvement and have thus introduced a separation between the development phase of agent, where they learning from and improving upon prior generations, and the transmission phase, where they are simply acting and being observed by the next generation. Under this formulation, propose and investigate distinct mechanisms for cultural accumulation in RI. agents based on our in-contexl and in-weights dichotomy introduced in Section 3
4.1 In-Context Accumulation
We first consider in-context accumulation, where aim to achieve cultural accumulation via fast in-context adaptation following a separate phase meta-RL training. Meta-RL. RL. training can be viewed imbuing agents with multiple memory systems (Squire et al., 1993]. By updating o,
in-context RI. executes processes of knowledge acquisition, or declarative memory (Squire) 2004] updating. In-context accumulation extends this knowledge acquísition operate across multiple generations. The corresponding results are in Section 5.1
4.1.1 Training
Algorithm 1 Training Loop for In-Context Accumulation (changes to RL in red)
1: 6 init. agent state
2: 8 init. agent parameters
3: 8 := oracle parameters
4: oracle noise factor
5: 0m + Ô // Initialise the agent parameters
6: for train step / T 1]do do
7: $ $
8:
6" + ò
9: = trajectory buffer
10:
for each trial A 00, K - 1] do
11:
Pots 11 11 (K 1)
12:
while trial not done do
13: 0" (s))
14:
if IsVisible Bernulli(poos) then
15:
Set oracle visible in o" end if
16:
17:
a ~ (s), a" t ~ To- (o") dt
18:
for (o)
19:
8, - T(s,a)
20:
B (0" r Append transition buffer
21:
end while
end for
23:
Hth < updatec@",, (
end for
To train agents capable of in-context accumulation, we require three attributes: (1) agents must learn to leam from the behaviour of other agents (i.c., social learning) within the context of single episode,
5

---

(2) agents must be able to act independently by their last trial, so that their behaviour can provide useful demonstrations for the next generation, (3) agents must use sufficient independent exploration to improve on the the previous generation, or each new generation will only be as good as the last.
To facilitate (1) during training, but not test time, assume
to oracle 0 with fixed parameters 8 0 is able to condition on the full state s, which we refer as privileged information. Agent n can observe the behaviours of O as they jointly act in an environment. To achieve (2), we include this representation of 0 in o" with probability Pots each timestep and linearly anneal Pebs across the K trials from - O. This ensures that the agent learns to act independently by the final trial. Perez al. 2023 show that increasing opportunities learn socially leads less diversity in homogeneous populations, which could be seen as limiting independent discovery in our context, further motivating this constraint. For (3), we add some random noise the oracle policy according to tunable parameter 4 This encourages the agent to learn that the behaviour of other agents 0 in the environment may be sub-optimal, which case agent n should engage in selective social learning (Poulin-Dubois and Brosseau-Liard 2016). This approach could be seen as imposing an information rate limit [Prystawski et al. 2023) between the oracle and learning agent. We present this process
altogether Algorithm with changes to standard RL2 in red.
4.1.2 Evaluation
Algorithm 2 In In-Context Accumulation During Evaluation
1: Ø == init. agent state
2: 8 = parameters from training
3: 8o $ // Sample initial state
4: o t à Define an initial reset state
5: for each generation m E [1, MM do
6: for each population member
Npppl do
7:
6 + Φ
8:
80 for each trial & E [0, K - do
8"
9:
if k K 1 then
11:
dt f- 6% // Store agent state at beginning of last trial as reset state
end if
13:
Pobs = 1 k/(K- 1)
14:
Φm < o - // Set previous generation state to best reset state from that generation
15:
while trial not done do
16:
0m 0% O(s)
17:
if IsVisible ~ Bernoulli(pos)) then
18:
Set previous generation visible in o",
19:
end if
20:
a To(0" l 1) a To ( t fo
21:
6 1), 6n + fo(o 1
m-
22:
Ts"" a")
23:
end while
24:
end for
25:
n² = [I,,
E, I[k = K 1)r // Select best population member based on final
trial performance
end for end for
27:
Having trained agents via Algorithm [ we evaluate their in-context accumulation. During this evaluation phase, is replaced by the best of the previous generation n². as we do not assume
'For the oracle capture sub-optimal, but goal-directed behaviour, inject comelated noise by corrupting the privileged information o.. 'This is an external selection mechanism. We investigate the emergence of selective social learning in Appendix C

---

In-Context Accumulation Memory Sequence
In-Context Accumulation Memory Sequence
RL3:
Notey Oracle Prob. 0.33
Generation
RL2:
32 Thats
10
Nolsy Oracle
Noay Oracie Prot: 0.67
Noiy Oradle
Figure 3: Left: In-context accumulation during evaluation Memory Sequence. Right: Evaluation results following training with different oracle accuracies.
access to privileged information at test time. This previous best agent, now in transmission phase, has its internal state reset between trials. A member of the new generation, in development phase, continuously updates its internal state across trials. The same meta- meta-parameters B from training are used across generations and only the internal states & are used to update each generation. Generations are made up of N agents that interact with N independent environments.
For each generation Im, we store the internal state of each agent on the first step of its final trial and call this the "reset state" 6 After the final trial, we then keep the previously stored reset state of the best performing agent ō"." in terms of rewards received in its last trial. When introducing the next generation +m+1+ generation 9m transitions into transmission phase and we represent the adapted behaviour of the best performing agent by resetting the internal states 1::N + 6% at the beginning of each trial. Thus, the next generation gm++1 can essence observe the best of 9m repeating its final trial. In practice, we mask 9m+11 from the observations of Jm in order to reflect the fact that agents do not expect to see other agents their final trial, as guaranteed by training attribute (2) above. Since the observations of 9m++1 can include information about the learned behaviours of Im, we implicitly have gm+11 = f(gm). ). establishing purely in-context equivalent to generational training (Section 2.4 This process is presented in Algorithm 2
4.2 In-Weights Accumulation
Finally, we present algorithm for cultural accumulation to take place over the course of training successive generations of policies (i.e., in-weights) rather than through solely updating the internal state (i.e., in-context). This is akin to generational training (Section 2.4). but without modifying the policy-gradies loss, by simply including the previous generation within environments on which the new generation is training, that their actions be observed and learned from. This alternative model of cultural accumulation assumes an agent's 's lifetime to be an entire training run, rather than a single episode. Here, we anneal the probability of observing the previous generation Pobs on a given environment timestep linearly over training. The corresponding approach is detailed in Algorithm 3 presented in Appendix A In-weights RL (i.e., RL training) can interpreted updating procedural memory [Johnson 2003 via skill acquisition. We thus interpret in-weights accumulation as gradual, basel accumulation. Note that there is no separation between "training" and "evaluation" in this case, because the in- weiwesghts algorithm views cultural accumulation as part of the training. This is in contrast to in-context accumulation, which an evaluation-time algorithm executed by appropriately trained melaceol The results that correspond to this setting are in Section 5.2
5 Results
For each of our experiments, we use Simplified Structured State Space Model (S5) (mithh et al.. 2022) modified for RL [Lu et al. 2024 to encode memory, building the PurelaxRI. codebase [Lu et al. 2022a]. This model architecture runs asymptotically faster than Transformers in sequence
length and outperforms RNNs in memory tasks [Morad et al., 2024 PPO [Schulman et al. 2017 is
7

---

In In-Coneext Accumulation: Goal Sequence
In Context Accumulation TSP
3.5
3.3
R1: F Trals
Gancrot
RLT Noly Orade
Cencriotio on Kelly Orade
33
Figure 4: Left: In-context accumulation during evaluation on Goal Sequence. Right: In-context accumulation during evaluation TSP.
used the RL training algorithm. For Goal Sequence experiments, observations processed by CNN, whilst for Memory Sequence and TSP, they are processed by feed forward layers. Architectural details provided in Appendix [ and algorithmie hyperparameters in Appendix [F] For all in-context experiments, a one-hot encoding of the reward received on each timestep and a one-hot encoding of the current trial number are included each agent's observation. We provide further details on the configuration of each environment in Appendix D. All results are averaged across 10 seeds and the shaded regions of plots show standard error.
5.1 In-Context Results
As single-lifetime baselines, we use RL trained on: (a) episodes of equivalent length to one generation and (b) episodes of equivalent length to all cumulative generations. All in-context evaluations are performed on held-out environment instances for the same number trials.
Memory Sequence: In Figure 3 we show that in-context learners trained according to Algorithm) D capable of accumulating beyond single-lifetime RL baselines and even beyond the performance of the noisy oracles with which they were trained when evaluated on new sequence. Interestingly, when evaluating the accumulation performance of agents trained with oracles of different noise levels, we see that it degrades when oracles are too accurate. This is directly inverse to results in imitation learning [Sasaki and Yamashinal 202T]. where the quality of expert trajectories positively impacts the performance achieved when training on these trajectories. We attribute this result to an over-reliance on social learning when oracles are too accurate, which therefore impedes the progress of independent in-context learning during training. Conversely, if oracles are too random, agents do not acquire the necessary social learning skills to effectively make use of prior generations.
inWWights Accumulation: Memory Sequence
Resatting with Accumulation: Memory Sequence
20.0
27.5
15.0
0-
13.5
1e5um
20.0
-10
7.
50
-15
2.5
-20
Accunstaton
100
125
175
200
Figure 5: Left: In-weights accumulation on Memory Sequence. Right: In-weights accumulation compounds with resetting. Error bars represent 95% confidence intervals.

---

Goal Sequence: Figure 4 (left) shows that in in-context accumulation also significantly outperforms single-lifetime RL when evaluated on a new goal sequence. On this more challenging, partially- observable navigation task, we found that higher, but still imperfect oracle accuracies during training produced the most effective accumulating agents. This likely due to the fact that learning to learn socially in this environment is much harder problem that involves finding and actively following another agent. We additionally observe that the performance of each generation typically drops slightly the last trial two, where agents are entirely mostly independent. This indicates that although agents are able to independently recall and navigate most of the sequence they have learned, improving on the previous generation, they are still somewhat over-reliant on demonstrations.
4 (right). (right), Again, we see that cultural accumulation enables sustained improvements considerably beyond RL² over single continuous context (i.e., lifetime). We only show the RI.² baseline trained on shorter episodes, as the baseline trained on longer episodes failed to make meaningful learning progress. In Figure 2 we show routes traversed by different generations for an example set of city locations. Notably, these routes become more optimised across generations, with later generations exploiting decreasing subset of edges.
TSP: We present these results in Figure
5.2 In- In-Weights Results
In Weights Accumulation: Goal Sequence
Figure 5 (left) shows that agents can accumulate over the course of train- ing via Algorithm 3 For in-weights accumu- lation, the sequence is kept fixed over train- ing, meaning that agents are trained to learn as much of a single sequence as possible. After one generation of accumulation, these agents outperform single-lifetime training for dura- tion equivalent to complete generations. This demonstrates that single-lifetime learners suc- cumb to primacy bias (Nikishin et al. [2022] and converge prematurely. In light of this, we inves- tigate whether in weights accumulation can be paired with other methods to overcome primacy bias. In particular, we use the simple approach of resetting some policy network layers [Nikishin al. 2022 opting for resetting the final two feed-forward layers, and present these results in Figure (right). We observe that resetting alone helps mitigate premature convergence, but that when we apply resetting to accumulating agents the improvements are compounded and agents reach higher returns than via either method independently.
Memory Sequence:
Restur
Generation Generation Single Lifetime
0
5000
10000
15000
20000
25000
30000
25000
40000
Step
Figure 6: In-weights accumulation on Goal Se- quence.
Goal Sequence: Figure 6 shows that in-weights accumulation exceeds the performance of training a single agent for a continuous run equivalent to the total duration of 4 accumulation steps.
TSP: We show results for in neighghtts accumulation in TSP in Figure 7 within Appendix B
Related Work
Social Learning: Considerable focus has been placed on using social learning overcome hard exploration problems, imitate human behaviour, and generalise to unseen environments [Ndousse] al. 2021 Bhoopchaad et al., 2023 Ha and Jeong, 2023 Ndouuse et al. [202]] trains agents that achieve reasonable performance without demonstrator by increasing the proportion of episodes with no demonstrator in handpicked increments over training. We employ a similar approach for training agents to eventually act independently by annealing the probability of observing the previous generation across training. To obtain independently capable agents in-context, we anneal the probability of observing the previous generation between trials, which is similar to the expert dropout used in [hhopphhand et al. [2023].
Machine Learning Models of Cumulative Culture: A growing body of work uses advances in
machine learning to model cultural accumulation in-silico [Prystawski al. 2023 Perez et al.) 2023
We initialise the second generation entirely from scratch so that it can learn to learn socially and use resetting with accumulation thereafter.

---

Explicit Imitation Learning
Implicit Third- Social Learning
Observational Learning by RL
Behavioral Cloning Pomerleau
Borsa al. 2017
I 1988
Emergent Social Learning
Non-Generational
Inverse RL
Ndousse et al. 2021
Russell
1998
carning Few- Imitation
Bhhoophhad et al. 2023
Kickstarting Schmitt et al. 2018
Generational Training
Our Work
X-Land
Stooke et al.
2022
Table I: Comparisons to prior work. Generational Training helps overcome local optima to tackle open-ended tasks [Stooke et al., 2022]. Implicit third- persod social learning more general than hand-crafted imitation learning algorithms. Our work combines both to achieve cultural accumulation.
2024]. Perez et al. 2024 use Bayesian RL with constrained inter-generational communication to reproduce social learning processes documented in human populations. This communication is represented in a domain specific language. Rather than requiring explicit communication channel between agents, we use social learning to facilitate accumulation in a more domain agnostic manner, demonstrating performance gains in multiple distinct settings. [Prystawski et al. (2023) demonstrate cultural evolution in populations of Large Language Models, where language- communication is used the mechanism for knowledge transfer between generations.
Generational RL: The generational process of cultural accumulation can be seen as iterations of implicit policy distillation and improvement. Iterative policy distillation (Schmitt et al. 2018 Stooke eet al. 2022 is therefore related to our work. Unlike these methods, we do not assume that the new learner has access to the policies of experts or past generations, but only observations of their behaviour as they interact with shared environment. We also do not explicitly modify the learning objective, leaving the agent free learn how much how little to imitate past generations.
7 Conclusion
We take inspiration from the widespread success of cumulative culture in nature and investigate cultural accumulation in reinforcement learning. We construct in-context model of accumulation, which operates on episodic timescales, and an in-weights model, which operates over entire training generational process that exceeds the perfor-
We define successful cultural accumulation mance of independent learning with the same total experience budget. We then present in-context and in-weights algorithms that give rise successful cultural accumulation on several tasks requiring exploration under partial observability. We find that in-context accumulation can be impeded by training agents with oracles that are either too reliable too unreliable, highlighting the need to balance social learning and independent discovery. We also show that in in-weights accumulation effectively mitigates primacy bias and is further improved by network resets.
Limitations and Future Work: An interesting future direction would be to use learned auto- curricula [Dennis et al., 2020 Tiang et al.) 2021] for deciding when agents should learn socially or independently, instead of linearly annealing the observation probability. Another extension would be investigate cumulative culture settings with different incentive structures, such as heterogeneous or cooperative rewards |Rutherford et al. 2023]. Future work could also study ways we can evolve the process of cultural accumulation itself [u al. 2023a or learn influence the learning process of other agents (Lu et al., 2022b 2023b for cultural transmission. Finally, we note that whilst understanding cumulative culture has benefits for both machine learning and modelling human society, the consequences of self- self-inpro systems should warrant consideration.
Acknowledgments
Jonathan Cook supported by the ESPRC Centre for Doctoral Training Autonomous Intelli- gence Machines and Systems EP/S024050/1. Jakob Foerster is partially funded by the UKI grant EPP/Y224481/I (originally selected for funding by the FRC). Jakob Foerster is also supported by the JPMC Research Award and the Amazon Research Award.
10

---

References
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search, 2017.
Albert Bandura and Richard Walters. Social learning theory, volume 1. Englewood cliffs Prentice Hall, 1977.
Jerome Barkow, Akinsola A Akiwowo, Tushar K Barua, MRA Chance, Eliot D Chapple, Gouranga P Chattopadhyay, Daniel G Freedman, WR Geddes, BB Goswami, PAC Isichei, et al. Prestige and culture: biosocial interpretation [and comments and replies]. Current Anthropology, 16(4):553-572, 1975.
Bauer, K. Baumli, S. Baveja, et al. Human- Human-timescale adaptation an open-ended task space. In International Conference Machine Learning. 2023.
A. Bhoopehand, B. Brownfield, A. Collister, al. Learning few-shot imitation as cultural transmission. In Nature Communications 14, page 7536, 2023.
Clément Bonnet, Daniel Luo, Donal Byrne, Shikha Surana, Sasha Abramowitz, Paul Duckworth, Vincent Coyette, Laurence I. Midgley, Elshadai Tegegn, Tristan Kalloniatis, Omayma Mahjoub, Matthew Macfarlane, Andries Smit, Nathan Grinsztajn, Raphael Boige, Cemlyn N. Waters, Mo- hamed A. Mimouni, Ulrich A. Mbou Sob, Ruan de Kock, Siddarth Singh, Daniel Furelos-Blane Victor Le, Arnu Pretorius, and Alexandre Laterre. Jumanji: a diverse suite of scalable reinforcement learning environments in jax, 2024. https:s////aaxx wwwpeiiivvrrrr.ooooooooo 223/
2306. 09884
Diana Borsa, Bilal Piot, Rémi Munos, and Olivier Pietquin. Observational learning by reinforcement learning. arXiv preprint arXiv: neee::1066666111 2017.
A. R. Buck and M. Keller. A myopic monte carlo strategy for the partially observable travelling salesman problem. 2008.
Aaron Cochrane, Chris R Sims, Vikranth R Bejjanki, C Shawn Green, and Daphne Bavelier. Multiple timescales of learning indicated by changes in wvidence-accumulation processes during perceptual decision-making. npj Science of Learning, 8(1):19, 2023.
H. Cornish, R. Dale, S. Kirby, and M. H. Christiansen. Sequence memory constraints give rise to language-like structure through iterated learning. In PLoS ONE, 2017.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment 2020.
design. Advances in neural information processing systems, 33:1
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and Abbeel. RL Fast reinforcement learning via slow reinforcement learning. arXiv preprint, arXiv:1611.c. 1.10.777,, 2016.
M. Enquist, S. Ghirlanda, and A. Jarrick. Why does human culture increase exponentially? 2008.
Seungwoong Ha and Hawoong Jeong. Social learning spontaneously emerges by searching optimal heuristics with deep reinforcement learning. In International Conference on Machine Learning, pages 12319-12338. PMLR, 2023.
J. Henrich. The secret of our success: How culture is driving human evolution, domesticating our species, and making us smarter. Princeton University Press, 2019.
G. Hofstede, G. J. Hofstede, and M. Minkov. Cultures and organizations: Software of the mind. McGraw-Hill Professional, 1994.
William Hoppitt and Kevin N Laland. Social learning: an introduction to mechanisms, methods, and models. Princeton University Press, 2013.
Victoria Horner, Darby Proctor, Kristin Bonnie, Andrew Whiten, and Frans BM de Waal. Prestige affects cultural learning in chimpanzees. PloS one, 5(5) 10625, 2010.

---

Minqi Jiang, Michael Dennis, Jack Holder,, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Replay-guided adversarial environment design. Advances Neural Information Processing Systems, 34:1884-18977 2021.
Addie Johnson. Procedural memory and skill acquisition. Handbook of psychology, pages 499-523 2003.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. 101:99-134, 1998.
H. W. Kuhn. Extensive games and the problem of information. 28:193- 1953.
M. Laskin, L. Wang, J. Oh, et al. In-context reinforcement learning with algorithm distillation. In Advances in Neural Information Processing Systems, 2022.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation, Advances in Neural Information Processing Systems, 35: 35:16455- 16468, 2022a.
Chris Lu, Sebastian Towers, and Jakob Foerster. Arbitrary order meta-learning with simple population- based evolution. ALIFE 2023: Ghost in the Machine: Proceedings of the 2023 Artificial Life Conference. MIT Press, 2023a.
Chris Lu, Timon Willi, Alistair Letcher, and Jakob Nicolaus Foerster. Adversarial cheap talk. In International Conference on Machine Learning, pages 22917-22941, PMLR, 2023b.
Chris Lu, Yannick Schroecker, Albert Gu, Emilio Parisotto, Jakob Foerster, Satinder Singh, and Feryal Behbahani. Structured state space models for in-context reinforcement learning, Advances in Neural Information Processing Systems, 36, 2024.
Christopher Lu, Timon Willi, Christian A Schroeder De Witt, and Jakob Foerster. Model-free opponent shaping. In International Conference on Machine Learning, pages 144111 PMLR, 2022b.
Alex Mesoudi. Variable cultural acquisition costs constrain cumulative cultural evolution. PloS one, 6(3)ee1823) 2011.
Alex Mesoudi and Alex Thornton. What is cumulative cultural evolution? Proceedings of the Royal Society B, (101188))2018777))) 2018.
Steven Morad, Chris Lu, Ryan Kortvelesy, Stephan Liwicki, Jakob Foerster, and Amanda Prorok. Re- visiting recurrent reinforcement learning with memory monoids. arXiv preprint arXiv:2402.09900 2024.
K. Ndousse, D. Eck, S. Levine, and N. Jaques. Emergent social learning via multi-agent reinforcement learning. In International Conference Machine Learning, 2021.
Ni, Eysenbach, and R. Salakhutdinov. Recurrent model-free rl can be a strong baseline for many pomdps. 2021.
E. Nikishin, M. Schwarzer, P. D'Oro, P-L. Bacon, and Courville. The primacy bias in deep reinforcement learning. In International Conference on Machine Learning, 2022.
A. Noormohammadi-AAt and D. Taghirad. Multi-goal motion planning using traveling salesman problem in belief space. 2019.
J. Perez, M. Sánchez-Fibla, and C. Moulin-Frier. Cultural evolution in populations with heterogenous and variable preferences. 2023. URL https:/iirrii hal. www.ssciinceiieeee cccience/hal-04400000
J. Perez, C. Léger, M. Oveldo-Tellez. C. Foulon. J. Dussauld, P. Oudayer, and C. Moulin-Frier. Cultural evolution in populations of large language models. 2024,
Charles Perreault. The pace of cultural evolution. 2012.
12

---

Dean A Pomerleau. Alvinn: An autonomous land vehicle in neural network. Advances in neural information processing systems, 1, 1988.
D. Poulin-Dubois and Brosseau-Liard. The developmental origins of selective social learning. Current Directions in Psychological Science, 25(1):60-644 2016.
B. Prystawski, D. Arumugam, and N. D. Goodman. Cultural reinforcement learning: framework for modeling cumulative culture on a limited channel. 2023.
Luke Rendell, Robert Boyd, Daniel Cownden, Marquist Enquist, Kimmo Eriksson, Mare W Feldman, Laurel Fogarty, Stefano Ghirlanda, Timothy Lillicrap, and Kevin N Laland. Why copy others? insights from the social learning strategies tournament. Science, 328(5975):208-113 2010.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pages 101-103, 1998.
Alexander Rutherford, Benjamin Ellis, Matteo Gallici, Jonathan Cook, Andrei Lupu, Gardar Ing- varsson, Timon Willi, Akbir Khan, Christian Schroeder de Witt, Alexandra Souly, et al. Jaxmarl: Multi-agent environments in jax. arXiv preprint arXiv:231..1099, 2023.
Sasaki and R. Yamashina. Behavioural cloning from noisy demonstrations. In International Conference on Learning Representations, 2021.
Sasaki and D. Biro. Cumulative culture can emerge from collective intelligence in animal groups. 2017.
Simon Schmitt, Jonathan J Hudson, Augustin Zidek, Simon Osindero, Carl Doersch, Wojeiech M Czarnecki, Joel Z. Leibo, Heinrich Kuttler, Andrew Zisserman, Karen Simonyan, and M. Ali Eslami. Kickstarting deep reinforcement learning. arXiv preprint, arXiv:/803.03835, 2018.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint, arXiv: 1707.0634777 2017.
J. T. H. Smith, A. Warrington, and S. W. Linderman. Simplified state space layers for sequence modelling, arXiv preprint, arXiv:2208.0m 244499333 04933, 2022.
Larry Squire. Memory systems of the brain: brief history and current perspective. Neurobiology of learning and memory, 82(3):171-177, 2004.
Larry R Squire, Barbara Knowlton, and Gail Musen. The structure and organization of memory. Annual review of psychology, 44(1):453-445 1993.
A. Stooke, A. Mahajan, C. Barros, et al. Open-ended learning leads to generally capable agents. arXiv preprint, arXiv:2107. 12808, 2022.
C. Tennie, J. Call, and M. Tomasello. Ratcheting up the ratchet: On the evolution of cumulative culture. In Philosophical Transactions of the Royal Society B: Biological Sciences, 2009.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning reinforcement learn, arXiv preprint arXiv: 1rX1v:1611.05763, 2016.
13

---

end for A In-Weights Accumulation
Algorithm 3 In Weights Accumulation
1: 0 := init. agent parameters
2: 0% 0 // Initialise the "random" first generation parameters
3: for each generation m E [1, G] do
4: for each population member [1, Npopl do
5:
B = trajectory buffer 0t
6:
7:
for train step E 00, 1] do
8:
Pobs = 1 1/(T" -
9:
~~S S
10:
while not done do 0 0t ((s)
11:
12:
if p ~ (prmouull(Pots) then Set previous generation visible in o
13:
14:
end am- Tzn' (0=-1)
15:
16:
a Ton (0t)
17:
8 T(s,a)
18:
B+ (o am mh) Append transition buffer
19:
end while
8 t update B)
21: end for
argmax Ei=r-1 Pm Select best population member based on final episode performance
23:
24: end for
14

---

B In- Weights Accumulation in TSP
In this setting, after two generations, performance exceeds agents trained for a single lifetime equivalent to generations. However, we do not observe as much improvement on subsequent generations.
In-Weight Accumulation:
Generation
Generation Single Lifetime
100
200
300 Updaite
400
500
600
Figure 7: In-weights accumulation on TSP.
15

---

C Selective Social Learning
In the Goal Sequence experiments, we select the best performing agent of the last generation for the current generation of agents to observe during training, automating the selection process. In human and animal cultural accumulation, this selection instead learned through prestige cues (Barkow et al.) [1975]. Horner et al. [2010]. Thus, in the Memory Sequence experiments, we do not automatically select the best of the past generation for agents to observe. Instead, the agents can observe the entire last generation. For In-Context Accumulation,
sort the observed oracles and agents by their performance during pre-training and evaluation. Thus, the agents ideally learn weight the last generation by their relative performance. For In- Weights Accumulation, sorting does not make difference, as random network initialization agnostic to the observation ordering. Thus, the agent learns implicitly through observation, which of the past generation is the best to imitate.
16

---

D Further Environment Details
D.1 Memory Sequence
For in-context accumulation in Memory Sequence, we use sequences comprised of three digits, with maximum length of 24, and give the agents four trials per episode. Each generation has population size of three. We perform initial meta- meta-trainin training for 8e6 timesteps. Agents get reward of 1.0 for correct response and reward for 1.0 for an incorect one. The trial ends when an incorrect response is given or the maximum length is reached.
For in-weight accumulation in Memory Sequence we use sequences comprising of ten digits, with a maximum length of 24. Agents train for le5 timesteps. Each generation has population size of five. Agents get reward of 1.0 for correct response and reward for 1..0 for incorect one. The episode ends when the maximum length is achieved.
D.2 Goal Sequence
We use 7 x 7 grids with 3 goal types. Agents observe 3 grid of cells directly in front of them. Observations are symbolic with 4 channels (one for each goal type and one for agents). On each timestep, agent take of three actions: receive +1 reward for hitting the correct next goal and - reward for hitting the incorrect next goal. For in-context experiments, trials are 30 steps long and there 4 trials in episode. For in-weights experiments, episodes are 50 steps long.
forward, turn left, turn right. Agents
D.3 TSP
For in-context accumulation in TSP, six cities and provide eight trials per episode. The city positions are uniformly sampled in the unit square. Each generation has a population size three. We perform initial meta-training for 8e6 timesteps. Agents get reward of distdis(euur city next eity))
v2 the selected next city valid. Agents get reward 1.0 and the trial ends they select an already-visited city.
For in- weight accumulation in TSP, we use twenty-four cities. Each generation has population size of eight. Agents train for 3e5 timesteps. The reward setup is equivalent to above.
17

---

E Architecture Details
E.1 Memory Sequence and TSP
The policy and value networks share two fully layers and a GRU layer. Values and policies are computed with fully connected layers. All layers use leaky ReLU activations. Here, we take the example of a 4-dimensional input for the 4-dimensional embedding of an action observation and assume the in-weights setting, so no trials. The final output is 10-dimensional in the case of random sequence made up of 10 distinct digits.
Shared input layers:
FC (4, 128)
FC (128, 256)
FC (256, 256)
GRU (256, 256)
Value MIP:
FC (256, 128)
FC (128, 128)
FC (128, 1)
Policy MLP:
- FC (256, 128)
FC (128, 128)
FC (128, 10)
E.2
Goal Sequence
The policy and value networks share three convolutional layers, a fully connected layer, and an S5 layer. Values and policies are computed with two fully connected layers. Convolutions use leaky ReLUU activation functions and all other layers use tanh activation functions. Inputs are 5-dimensional for one-hot encodings of 3 goal types, walls and agents. A 3-dimensional vector for one-hot encodings of reward concatenated to the output the last convolutional layer. one-hot encoding the trial would also be included in the in-context setting.
Shared input layers:
Conv (5, 32), 1 x filters, stride 1, padding
Conv (32, 64), 1 filters, stride 1, padding 0
Conv (64, 64), 1 filters, stride 1, padding 0
FC (576 3, 256)
S5 (256, 256)
Value MLP:
FC (256, 64)
FC (64, 64)
FC (64,
Policy MIP:
FC (256, 64)
FC (64, 64)
FC (64.
18

---

Hyperparameters
F1
Memory Sequence and TSP
population size
5
learning rate batch size rollout length update epochs minibatches
2.5
10-5
4
128
4
4
7 AGAE clip entropy coefficient
0.99
0.95
0.2
0.01
value coefficient
0.5
max gradient norm anneal learning rate
0.5
False
F.2 Goal Sequence
population size
4
learning rate
10
10-5
batch size
128
rollout length
32
update epochs
8
minibatches
8
0.99
AGAE clip entropy coefficient
0.95
0.2
0.01
value coefficient
0.5
max gradient norm
0.5
anneal learning rate
False
19

---

G
Compute Resources
Memory Sequence and TSP experiments were run on single NVIDIA RTX A40 GPU (40GB memory) in under 20 minutes. Training of in-context learners in Goal Sequence was run in under minutes on 4 A40s (oracle training runs in the same time). In- In-weights accumulation Goal Sequence was run in 30 minutes on 4 A40s.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction motivate the study of cultural accumulation in artificial learning agents and claim that, under our models, cultural accumulation can be achieved over RL training and in-context RL the settings we consider, both of which are backed up by our results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In our conclusion we outline the limitations of this work and highlight direc- tions for future research.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Experimental details required to understand and interpret the results included in the main text and all further details required to reproduce the results are included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
22

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide an anonymous open-sourced repository containing all code used to experiments and instructions needed to reproduce results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nipss wttpcc/ccccccccccccccooooo gwwcc//gu/es//eee / wwwaanisssiooooooiooiooiooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Key training and test details needed to interpret results are in the main text any further details are included in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: All plots include shaded regions indicating standard error or error bars repre- senting 95% confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources used for each experiment are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Ethics.
This research was conducted with strict adherence to the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the paper's conclusion, we discuss the potential positive and negative societal impacts of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not include the risk of any data models that have risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: No existing assets were used this paper.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The open-sourcing of our code includes the release of the environments used with supporting documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing, nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing, nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "EOHAFmv0fM", "reviewer": "Reviewer_2ZBi", "summary": "The paper studies cultural accumulation within the context of RL agents. The techniques involve social learning based on in-context learning and in-weights learning. Each agent is modelled as a POMDP within a larger POSG. The techniques are applied to memory sequence, goal sequence, and TSPs, where they are compared to RL^2.", "questions": "The authors mention the two variants of RL^2 show the learning for the length of a single lifetime as well as the full length combining generations (equivalent to the length of their own algorithm). If so, then how come Figure 4 shows the same length for RL^2 in both cases? \n\nWhy does the noisy oracle work well? Is this because in the evaluation phase, only an observation can be provided so there is less discrepancy between the phases?\n\n“We model cultural accumulation as taking place within POSGs. Each member of a given generation gm+1 learns in parallel on an independent instance of the environment, which also contains static members of the previous generation, gm.  Therefore, from the perspective of a given agent i in gm+1 this is a POMDP, since the prior, static generation can be considered part of the environment.”\nCan this be clarified? For example, does this mean the policies of other agents are frozen? And why do we need the framework of POSGs if actually the problem to solve is only POMDP.\n\nWhat is the importance of having the POMDPs being sampled from a distribution to demonstrating the proposed approach? I don’t clearly get this from the text. Also, it appears that actually also the environment is just a POMDP with observations indicating the task.\n\nWhat are the hyperparameters for RL^2\n\nWhat is the benefit of sharing the recurrent state at one step vs sharing the observation at each time step?\n\nHow would you implement this in a physical setup?", "limitations": "The authors mention limitations but not the strongest ones: \n    • the presence of the full-state oracle.\n    • The applicability", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The concept of transmitting information across generations other than the genotype is underexplored, which the authors study using in-context accumulation (i.e. passing on hiddent states).\n\nThe results show a positive performance trend.", "weaknesses": "It does not appear obvious how to implement such a training setup on a physical system.\n\nTraditional evolutionary algorithms pass policy parameters from the previous generation to the next, and these techniques are not discussed. And indeed, there is a big field integrating RL techniques with evolutionary algorithms. \n\nObserving other agents is also not new and this is commonly seen in multi-agent reinforcement learning. For instance, sharing observations:\nhttps://arxiv.org/pdf/1812.00922\nThe benefit of sharing hidden states is not clear.\n\nTSPs are best solved by evolutionary algorithms. There is no comparison to such algorithms.\n\nMy initial reading of the POSG section was that there is an error in the definition of the reward function as it does not index the individual. However, as mentioned in the Limitations section, the authors basically use the same R for all agents. So I believe that indexing R_i and then mentioning that all the R_i’s are the same would be more clear. It does cast some doubt on why the POSG formalism is more suitable.\n\nIt is said that each member of a given generation learns on an independent instance of the environment with frozen agents from the previous generation and that for each agent inidividually, the problem is a POMDP. So there appears to be no benefit in introducing POSGs.\n\nBaselines: only one baseline RL^2. And the performance difference can likely be explained by the assumption of a full state oracle. It does not seem a fair comparison to assess the algorithms with different environment access assumptions.\n“To facilitate (1) during training, but not at test time, we assume access to an oracle −i with fixed −i parameters θ̂ . This oracle is able to condition on the full state s, which we refer to as privileged information. Agent i can observe the behaviours of −i as they jointly act in an environment.”\n\nThe RL^2 seems to be an essential part of the algorithm (Appendix A) yet it is not explained. Moreover, RL^2 too passes on hidden state information so the comparative benefit should be explained as well.\n\nThe need for two separate phases of evolution is not clearly explained. Also it is not clear how they connect.\n\nIt is not clear how RL^2 and the proposed technique are comparable in terms of the total number of evaluations but also in the observations and learning setting.\nAlgorithm 1: \n p_obs, phi, and theta are not being used\n \n there is no need to repeat line 19 for all i. It can be put at the end of the generation since it is the same for all i.\n\n l.11: index n is undefined\n\nThere is limited theory development.\n\nThere is no explanation of why the noisy oracle can work well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough and extensive review. We are pleased that the reviewer finds that “the concept of transmitting information across generations other than the genotype is underexplored” and that “the results show a positive performance trend.”\n\n# On related work\n\n> Traditional evolutionary algorithms...\n\nThe core distinction between *cultural evolution* and traditional evolutionary algorithms (i.e., genetic algorithms) is that cultural evolutionary algorithms [1] accumulate knowledge and skills, as opposed to parameters or DNA, that are relevant to the population and explore directly in this space of this knowledge and skills. We additionally explore passing on policy parameters in Figure 5 (right). \n\n> Observing other agents is also not new...\n\nWe acknowledge that observing other agents is not new. As seen in the paper in Table 1, we very deliberately highlight where our work sits in relation to the literature, as acknowledged by reviewers h1KJ and R6qB. There we draw attention to the fact that our work uniquely combines implicit, third person imitation with generational training to achieve cultural accumulation. \n\n# On the formalism\n\n> My initial reading of the POSG section was that there is an error in the definition of the reward function.\n\nThanks for catching this! We’ve fixed it in our manuscript.\n\n> The problem is a POMDP, so there appears to be no benefit in introducing POSGs.\n\nWe introduce POSGs to introduce the notation we will be using to refer to other agents and their policies. Indeed, our setting is a POMDP from the perspective of any individual agent. However, when describing the implementation and method with formal notation, it is far easier to use the notation from POSGs.\n\n# On baselines\n\n> Only one baseline RL^2. And the performance difference can likely be explained by the assumption of a full state oracle.\n\nAs stated in Section 3 of the paper, the goal was to compare to single-lifetime baselines using an otherwise equivalent algorithm, as the method of accumulation is additive (i.e., builds atop the same underlying RL algorithm). As RL^2 is the meta-RL algorithm underlying our in-context accumulation implementation, that same algorithm without accumulation is the natural baseline for showing the impact of test time accumulation. The full-state oracle is only for training in the in-context regime. Training with an oracle for the RL^2 baselines actually impedes performance because we do not assume access to oracles at test time. \n\n> The RL^2 seems to be an essential part of the algorithm (Appendix A)...\n\nIn the background section on Meta-RL (Section 2.3), we introduce the concepts underpinning RL^2 and cite the paper appropriately, but do not explicitly introduce it as RL^2 there, which we now do in our updated manuscript.  Appendix A provides the algorithm for RL^2 in detailed pseudocode, with our adaptations to enable in-context accumulation in red.\n\n# Points of clarification\n\n> The need for two separate phases of evolution is not clearly explained...\n\nCould the reviewer please clarify what they mean by this? If referring to the difference between in-context and in-weights accumulation, the former involves accumulation over in-context learning agents, whilst the latter involves accumulation over agents that are training and is most similar to prior work on generational training [3].\n\n> It is not clear how RL^2 and the proposed technique are comparable in terms of the total number of evaluations...\n\nWe apologise for the insufficient clarity here. As Algorithm 2 is referenced first, but appears in the Appendix, we used the line “rollout agents as in training” to refer to the lines in Algorithm 2 that show how p_obs, phi and theta are used. We have rectified this by including those lines of pseudocode explicitly in Algorithm 1 as well. The total number of evaluations is the same for RL^2 and in-context accumulation. We have updated our manuscript to make this more explicit.\n\n> There is no explanation of why the noisy oracle can work well.\n\nThe noisy oracle has full state information. We add some noise to this state information so that the oracle is sub-optimal. Its performance is plotted simply to illustrate that in-context accumulation at test time goes beyond the performance of anything seen in training, which provides further evidence that in-context accumulation outperforming RL^2 baselines is not simply “explained by the full-state oracle”.\n\n# Addressing questions:\n\n1. The two RL^2 baselines have different contexts (i.e., number of trials within an episode) during training. For fairness, we evaluate over the same total length in case the shorter-context baseline generalises zero-shot to longer contexts, which we do not observe happen.\n\n2. See Points of clarification.\n\n3. See On clarity.\n\n4. Sampling from a space of POMDPs corresponds to randomly sampling a specific environment instance.\n\n5. They are the same as for the corresponding accumulation algorithm. This is because the underlying RL algorithm is the same between the baselines and accumulation. We have updated our manuscript to explicitly state this.\n\n6. The recurrent state is not shared. It is used to distinguish between generations for in-context accumulation and plays no role in in-weights accumulation. Sharing observations is an instance of explicit communication/ information transmission, whereas we consider implicit third person social learning for its flexibility in allowing agents to learn from the behaviour of other agents, or independently explore or exploit.\n\n7. An example would be teams of agents at deployment time figuring out how to improve on solutions to tasks e.g. in a warehouse order fulfilment setting. \n\n[1] A comprehensive survey on cultural algorithms, Alireza Maheria, et. al, 2011\n\n[2] A Survey of meta-reinforcement learning, Jacob Beck, et. al, 2023\n\n[3] Open-ended learning leads to generally capable agents, Stooke, et. al, 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lgk7Y7i0AH", "reviewer": "Reviewer_9itL", "summary": "The paper \"Artificial Generational Intelligence: Cultural Accumulation in Reinforcement Learning\" introduces the concept of cultural accumulation in reinforcement learning (RL), where agents benefit not only from their own experiences but also from the knowledge passed down from previous generations, akin to human cultural evolution. The authors propose two models for this accumulation: in-context accumulation, which occurs within single episodes, allowing fast adaptation, and in-weights accumulation, which embeds knowledge in neural network weights over longer training periods. Through experiments on tasks such as Memory Sequence, Goal Sequence, and the Traveling Salesperson Problem (TSP), the paper demonstrates that agents trained with cultural accumulation outperform those trained without it. This work represents the first demonstration of general models achieving emergent cultural accumulation in RL, opening new directions for more open-ended learning systems and providing fresh insights into modelling human cultural processes with artificial agents.", "questions": "1. How do you anticipate the scalability of your models to more complex or real-world tasks beyond the experimental environments used in this paper?\n2. What are the computational resources required to train these models, and how do they compare to traditional RL methods?\n3. How do you determine the optimal balance between social learning and independent discovery during training?\n4. How do your models handle diverse or rapidly changing environments where cultural knowledge might quickly become outdated?", "limitations": "1. While the paper demonstrates the models on specific tasks, there is limited discussion on the scalability and complexity of the models in more complex or real-world scenarios.\n2. The authors do not provide a detailed analysis of the computational resources required for their models, which could be a limitation for practical applications.\n3. The potential real-world applications and implications of the proposed models are not thoroughly explored. Discussing these aspects would provide a clearer picture of the practical relevance of the work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Innovative Concept: The introduction of cultural accumulation in RL is novel and aligns well with how learning and knowledge transfer occur in human societies.\n2. Robust Experimental Design: The experiments are well-designed to test the hypotheses, with clear evidence showing the benefits of cultural accumulation.\n3. Comprehensive Analysis: The paper provides a thorough analysis of both in-context and in-weights accumulation, exploring different environments and scenarios.", "weaknesses": "1. Complexity of Models: The proposed models may be complex to implement and require significant computational resources, which might limit their applicability.\n2. Scalability Concerns: While the models work well in the presented tasks, it is unclear how they will scale to more complex or real-world scenarios.\n3. Limited Real-World Applications: The paper primarily focuses on theoretical and controlled environments. More discussion on potential real-world applications and implications would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their positive and informative review. We are pleased that the reviewer finds that the paper exhibits an “innovative concept”, “robust experimental design” and “comprehensive analysis”. Whilst we appreciate the reviewer’s acknowledgement of these strengths, we would like to address the claimed weaknesses.\n\n# On complexity\n\n> The proposed models may be complex to implement and require significant computational resources, which might limit their applicability.\n\nWe in fact make very minimal changes to existing RL algorithms. For evidence of this, see Algorithm 2 (in-context accumulation training), which has 3 simple lines of difference relative to vanilla RL^2. The core approach of in-weights accumulation simply involves training agents in successive generations and keeping around the previous generation in a shared environment, rather than using explicit policy distillation as in [1].\n\nA distinct advantage of cultural evolution is that it more efficiently explores the space of behaviours than genetic algorithms, which rely on cumulative mutations in genotype (e.g., network weights). This means that our implementations of cultural accumulation in RL require only very limited compute resources. We detail the compute resources used in Appendix H. All experiments can be run on a single 40 GB A40 GPU in under 1 hour thanks to our JAX-based algorithm and environment implementations. We cut this down to a maximum train time of 8 minutes by using 4 GPUs for Goal Sequence experiments. \n\n# On scalability\n\n> While the models work well in the presented tasks, it is unclear how they will scale to more complex or real-world scenarios.\n\nWe view this work as an important first step in a new direction, as recognised by the acknowledgements of novelty by the reviewer as well as reviewers h1KJ and R6qB. As such, we begin by seeking to demonstrate and understand cultural accumulation in RL using environments that are amenable to doing so. As a first step in considering additional environments, we present results for in-weights accumulation on Atari Breakout and Freeway in Figure 1 of the attached pdf document, and Brax Half-Cheetah in Figure 2.\n\nWe also believe that the use of language models in addition to RL is a natural direction for scaling up this work, but one that we think warrants its own paper.\n\n# On real-world applications\n\n> More discussion on potential real-world applications and implications would strengthen the paper.\n\nOne application is modelling human cultural accumulation “in-silico”, which we believe is a key contribution of this work. To the best of our knowledge, we are the first to show the effect of demonstrator ability (i.e., oracle noise) on emergent cultural accumulation, an effect also documented amongst humans [2].\n\nWe do however acknowledge that more discussion of the real-world applications that might arise from this new line of work would be of value to the paper. One can imagine teams of agents at deployment time figuring out how to improve on solutions to tasks e.g. in a warehouse order fulfilment setting, in a home assistant setting, in a delivery drone setting etc. Our setup is particularly amenable to putting a human in the loop too, because in-context accumulation only requires observational data of the human.\n\nFinally, in the era of large scale pretraining in AI, it is useful to study the mechanisms by which learning systems can learn from and improve upon data representing other agents.\n\n# Addressing questions\n\n1. See “On scalability” and “On real-world applications” above.\n\n2. See “On complexity” above.\n\n3. The agent discovers this entirely through learning to balance imitation and independent learning in the in-weights setting. In the in-context setting, we find that oracle noise level will bias it in one direction or another and therefore needs to be considered. As with other hyperparameters, the best way to discover the optimal oracle noise level is via empirical search.\n\n4. We would expect that independent exploration would be the most effective strategy in this case, depending on the learning sample efficiency of the task at hand. Here cultural accumulation should excel relative to imitation learning or policy distillation because the social vs asocial tradeoff is learned online.\n\n [1] Open-ended learning leads to generally capable agents, Stooke, et. al, 2021\n\n [2] Cumulative cultural learning: Development and diversity, Cristine H. Legare, 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UFTY7O18bU", "reviewer": "Reviewer_h1KJ", "summary": "This paper introduces the problem of modelling cultural accumulation in populations of deep RL agents, with evolution happening through non-communicative social learning. The paper introduces 2 setups for cultural accumulation: one where the agents learn in-context from other agents within an episode composed of several trials through meta-RL (called the in-context setup), and one where the agents learn from other agents whose behavior is sometimes visible in the agent's observations (called the in-weight setup).\n\nIn the in-context setup, agents are first trained alongside a noisy oracle (with access to the oracle annealed to 0 over the course of the episode) but imitating the oracle is not enforced in the loss; social learning is implicit. The oracle is kept noisy to encourage agents to learn on their own, so that cultural accumulation can occur. At evaluation time oracles are replaced by the previous generation of agents. There is no separate phase for training in the in-weights case.\n\nThe paper tests algorithms for cultural accumulation in three experimental environments. The first is a sequence memorization task, the second is a goal sequence memorization task in a gridworld with egocentric view, and the third a multi-agent, partially-observable traveling salesperson problem. Baselines are RL$^2$ algorithms with no social learning (single lifetime) for the in-context setup, and single lifetime RL training for the in-weight setup. The paper shows consistent improvement on all tasks when cultural accumulation happens with respect to single-lifetime baselines.", "questions": "* What is the deep RL algorithm underlying both the in-context and the in-weights setup? I am not sure this is mentioned in the main body of the paper;\n\n* What is the extent to which these results could serve as a model for human evolution, knowing how much RL agents and humans differ, and how different the current setup is compared to human learning setups? How do such experiments compare, as far as modelling of human evolution is concerned, to simple models from the cultural evolution literature like those of [Enquist et al, 2008] (or of [Fogarty et al 2015] in innovation)? Could more realistic experiments be devised?\n\n* Couldn't there have been simpler ways to model cultural evolution using LLMs instead of RL agents, since they already have meta-learning capabilities? (for instance, LLM agents with skill repertoires like LMA3 [Colas et al 2023] or Voyager [Wang et al 2023]).\n\n* (minor) L181 the claim about populations should say whether this is a simulation or human result;\n\n* Why does the TSP agent not benefit that much from more than 1 transmission event, do you think? Is there an exploration problem?\n\n* Why is cultural accumulation better than single lifetime for these tasks? Is there a plasticity problem with the single-lifetime agent? In evolution a part of the advantage of generational transmission is that environments are changing and populations need to be adaptable. There are also aging processes and the difficulty of maintaining lifeforms for arbitrary amounts of time. This is not the case in simulated agents for fixed environments, so how come cultural accumulation provides an advantage in this setting? Are there differences in the amount of exploration in the generational vs single lifetime experiments?\n\n### Notations\n(minor)\n\n* l68 what is the observation space? shouldn't the observation function be defined on $O: \\mathcal{O} \\times S \\rightarrow [0, 1]$, defining a joint distribution on observations and states (assuming that $\\mathcal{O}$ is the observation space)? \n\n* The $g_m$ notation is not clear when first encountered, are these model weights?\n\n* l70 do the parameters depend on the time t?\n\n* l105 Not sure I get this notation: $g_{m+1} = f (·|g_m)$, is $g_m$ a distribution? Why not just $f(g_m)$?", "limitations": "* The paper is concerned with tasks where a clear reward function is defined beforehand. While this is important and models some aspects of tasks early humans had to face, they are not open ended and not subject to variation. Studies of cultural evolution in simulation must eventually tackle open-ended domains, and even better, domains where the population of agents themselves alter the task landscape as they evolve (see for instance how language or the invention of water containers has influenced human evolution).\n\n* The skills learned by agents are not compositional, and thus do not model the stepping-stone aspect of human cultural evolution that allows populations to build upon existing sets of knowledge or combine them to create larger cultural adaptations (the way the Voyager agent implements skill compositionlity by writing programs could be a source of inspiration here);\n\n* The exploration/innovation aspect of cultural evolution is under-represented in this paper. Could exploratory behavior also be meta-learned as it leads to higher long-term innovation and thus reward? Are the test environments complex and variable enough to test this? \n\n* The in-context setup requires access to an oracle to steer agents towards learning to imitate other agents behavior. I acknowledge that this paper does not claim to model the emergence of social learning during evolution, but it would have been significant to get the agents to imitate one another without access to optimal behavior on the task.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The questions being studied are worthwhile and timely; as our understanding of the role of cultural evolution in humanity's success improves it is important that AI researchers investigate related questions with artificial agents.\n\n* The positioning with respect to the existing literature is good and differences between this setup and existing works studying cultural evolution in agent populations are highlighted.\n\n* The background section does a good job of refreshing readers' minds with respect to POMDPs, meta-RL and generational learning;\n\n* The experiments and their presentation are of excellent quality with adequate error bars and offsets, and support the hypothesis that cultural accumulation allows populations of agents to reach better policies;\n\n* The results are significant, and I can see other researchers being inspired by this approach.", "weaknesses": "* The paper could be clearer: the fact that there are multiple approaches on multiple environments makes the paper complex and difficult to parse. Maybe the in-weights results could go in the appendix and more space could be devoted in the main text to giving examples or longer explanations of the tasks, as well as of the meta-RL setup? I think that some notations might need polishing, see Questions section below.\n\n* The paper distinguishes itself from [Perez et al 2023, 2024] by adopting a model of cultural evolution based only on imitation of non-pedagogic behavior, i.e. agents imitating other agents doing the same task. While this is a completely valid form of social learning and definitely present in human learning, the first introduction paragraph suggests that this is the main mechanism leading to humanity's success as a species. This is not the case, and it is known that language as a means for instruction and rapid communication of knowledge, explicit teaching, and shared norms and institutions are an important (and human-specific, whereas imitation of behavior occurs in great apes) component of cultural evolution (see Henrich 2019 as well as Tomasello 2019). This should be discussed in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their extensive and detailed review. We are pleased that the reviewer finds that “the questions being studied are worthwhile and timely”, “the positioning with respect to the existing literature is good”, “the background section does a good job of refreshing readers' minds”, “the experiments and their presentation are of excellent quality” and they “can see other researchers being inspired by this approach”.\n\nIn their summary, the reviewer demonstrates a deep and thorough understanding of the contents of the paper, and we would like to thank them for this attention to detail.\n\nWe appreciate the reviewer’s constructive feedback and would like to address the weaknesses they state below.\n\n# On clarity\n\n> The paper could be clearer: the fact that there are multiple approaches on multiple environments makes the paper complex and difficult to parse.\n\nWhilst we consider investigating both the in-context/ in-weights settings to be an important contribution, especially as in-context learning becomes an increasingly used paradigm, we acknowledge that there is a lot for the reader to keep in mind. Moving results for some environments, and descriptions of the corresponding environments, to the appendix could be an appropriate solution that would also enable us to include more detail on the algorithms themselves in the main text.\n\n# On the setting\n\n> Language as a means for instruction and rapid communication [...] should be discussed in the paper.\n\nWe thank the reviewer for pointing this out and emphatically agree that language is a core transmission mechanism underpinning cultural accumulation in humans. We agree that acknowledging this in the paper would add clarity to that portion of the introduction.\n\n# Addressing questions\n\n1. PPO. We apologise for this being missing in the main text and will ensure to include it there.\n\n2. The Memory Sequence environment is actually lifted directly from a controlled study of human cultural accumulation [1]. We believe that settings such as this, which allow for investigating the core mechanisms of the focal phenomenon without additional complexity, are useful for an initial work of this kind. Some verification of this work’s utility as a model of human cultural accumulation is shown by the emergent impact of the “skill level” of other agents on social learning and accumulation, which is observed among humans [2]. The relevance of the in-context and in-weights distinction to multiple-memory systems [3] and the role this plays in cultural accumulation [4] is also of value. Scaling to settings that more closely resemble cultural accumulation “in the wild” is important future work. \n\n3. Meta-learning arises analogously in both LLM pretraining and RL^2-style meta-RL; we call it “in-context” to be explicit about that parallel. In this work, we focused on  tabula-rasa, communication-free cultural accumulation, but see work leveraging LLMs as an important direction. As a preliminary step in this direction, we ran an analogous experiment to in-context accumulation on Memory Sequence with GPT-4 and show these results in Figure 3 of the attached pdf document.\n\n4. Acknowledged. This is firstly a human result that is then demonstrated in artificial agents in the cited paper.\n\n5. We believe this is likely an exploration challenge as the reviewer suggests.\n\n6. Yes, we verify that there is a plasticity problem in the in-weights setting by showing that this is partially mitigated by the common approach to dealing with plasticity loss of partial network resets. Importantly, partial resets compound with in-weights accumulation when used together, as Figure 5 (right) shows. In the in-context setting, we expect that accumulation is an effective way of overcoming the challenges of learning across long contexts.\n\n# Addressing notation\n\n1. We thank the reviewer for picking up on this, their definition is correct.\n\n2. These are model weights (in-weight) or hidden state (in-context). We will make this clearer in a camera ready version.\n\n3. No, that is a typo and we thank the reviewer for pointing this out.\n\n4. You are correct that this interpretation is unintended and we have remedied this by updating our manuscript.\n\n# Addressing limitations\n\n1. We agree that this would be an exciting direction and indicate this in Section 7.\n\n2. In a simple way, they are compositional in that agents can build knowledge of the environment, and corresponding navigational routes in Goal Sequence and TSP, across generations. We agree that Voyager-like skill composition, or better yet a persistent environment would be exciting directions to consider for cultural accumulation in learning agents. \n\n3. The exploration-imitation tradeoff is indeed being meta-learned (in the in-context setting) as agents are learning to balance these as they adapt in-context to new environments. The hard exploration component of these tasks (as shown by lower single-lifetime RL performance) is what enables us to study this. \n\n[1] Sequence memory constraints give rise to language-like structure through iterated learning, Hannah Cornish, et. al, 2017\n\n[2] Cumulative cultural learning: Development and diversity, Cristine H. Legare, 2017\n\n[3] Memory systems of the brain: a brief history and current perspective, Larry R. Squire, 2004\n\n[4] Multiple timescales of learning indicated by changes in evidence-accumulation processes during perceptual decision-making, Aaron Cochrane, et. al, 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhzizwdJHg", "reviewer": "Reviewer_R6qB", "summary": "The authors introduce two methods for learning agents to trade-off imitation and exploration across generations, by incorporating the behavior of noisy oracles and/or the best-performing agents in prior generations into the observations of the next generation of agents. Two settings are studied: in-context learning and in-weights learning.  The in-context learning involves:\n- a training phase, in which a meta-RL algorithm learns a recurrent RL algorithm that uses observations from a noisy oracle to learn to solve a POMDP by updating a hidden state\n- a test phase in which the RL algorithm is frozen but the hidden state can still change, and the oracle is switched out with the agent with the highest performing hidden state from the previous generation. \n\nThe in-weights accumulation has no train-test split or hidden state, and just trains the RL algorithm from scratch at each generation, allowing it to observe both the oracle and the best agent from the previous generation. Noise is added to the oracle and the probability of observing it is decreased over time, so the agents must learn to explore and act independently.\nThey demonstrate that their methods outperform non-cumulative/single life baselines in three simple partially observable tasks.", "questions": "- How exactly is the oracle’s behavior incorporated into the observation? \n\n- How is a^{-i} used by the in-weights algorithm? \n\n- How sensitive is the approach to the population size?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The approach is novel and interesting\n- The work is well situated among related prior work in generational methods and social learning.", "weaknesses": "- The writing about the algorithms is quite unclear, especially the training algorithm of the in-context accumulation and the in-weights accumulation algorithm are only partially explained in the main text while the pseudocode is relegated to the appendix (see questions section) the paper could benefit from more thorough explanations of the methods in the main text \n\n- The ability to observe an oracle with access to privileged information (even with the addition of noise) seems like an unrealistic choice, artificially making the problem easier especially in the in-weights setting, since the oracle can be observed throughout rather than only at training time. Even worse, the method is highly sensitive to how much noise is added to the oracle, and the optimal noise amount also varies significantly across settings. Why can’t the agents in this approach learn to learn robustly from prior generations without an oracle?\n\n- There is only one baseline used in each setting, and it is unclear how much of an even playing field they’re tested on. How much were they tuned, e.g. by adjusting the learning rate decay if premature convergence was a major factor? Could they access the oracle demonstrations in any way, or could only the accumulation algorithms observe the oracle? Equation 1 does not account for the population size N, just the number of generations- shouldn’t the baseline’s experience budget be multiplied by another factor of N?\n\n- Cultural accumulation is not just passing down information about the unobserved state, but can also involve passing down procedural knowledge (e.g. mathematics or dance). the authors claim that in-weights accumulation can be interpreted as updating procedural memory, but the environments they test in do not seem to have the right type of complexity to demonstrate this. It would help if the authors tested in a setting that is fully observable but with more complex dynamics requiring the development of skills, such as half-cheetah\n\n- Only observing the best performing agent from the previous generation seems unnaturally limited - humans get to observe a range of behaviors from other humans and may learn from the mistakes of those who perform poorly, or could combine together two suboptimal agents’ strategies to make an optimal one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their clear and focused review. We are pleased that the reviewer finds that “the approach is novel and interesting” and that “the work is well situated among related prior work in generational methods and social learning.” \n\nThe reviewer seems to have grasped the core contributions of our work, however there is an important mistake made in their summary. They claim that during in-weights accumulation we allow new generations “to observe both the oracle and the best agent from the previous generation,” however this is distinctly false as we make no use of an oracle for in-weights accumulation. The oracle is solely used for training social learning abilities ahead of *in-context* accumulation. \n\n# Points of clarification\n\n> The paper could benefit from more thorough explanations of the methods in the main text.\n\nWe thank the reviewer for pointing this out and agree to bring further algorithmic details into the main text.\n\n> The ability to observe an oracle with access to privileged information seems like an unrealistic choice, especially in the in-weights setting.\n\nWe clarify again that we make no use of oracles in the in-weights setting. In conjunction with the point above, we appreciate that further details on the in-weights algorithm in the main text would be of value in preventing misunderstandings such as this.\n\nAs the reviewer indicates, agents can learn to learn from prior generations when training (i.e., in-weights), but this is not possible from scratch during purely in-context accumulation, which is what necessitates the use of an oracle during training on the train subset of environments. We would like to note that this is a less restrictive assumption than access to expert agents on test environments [1, 2].\n\n> The method is highly sensitive to how much noise is added to the oracle.\n\nThis work is meant as an investigation into cultural accumulation in RL and so we perceive insights such as this to be of immense value; to the best of our knowledge, no existing work has revealed how social learning in different settings (i.e., in environments shared with agents of different abilities) impacts downstream cultural accumulation. This result is counterintuitive when compared, at face value, to the need for high quality expert trajectories in imitation learning [3]. \n\n> How much were the baselines tuned, e.g. by adjusting the learning rate decay if premature convergence was a major factor? \n\nThe baselines use the same underlying RL algorithm and have the same hyperparameters. The best hyperparameters found for baselines and accumulation were the same.\n\n> Could the baselines access the oracle demonstrations in any way, or could only the accumulation algorithms observe the oracle? \n\nFor in-context accumulation, we tried training the RL^2 baselines with an oracle, but since oracles are not seen at test time, they in fact then do worse.\n\n> Equation 1 does not account for the population size N, just the number of generations- shouldn’t the baseline’s experience budget be multiplied by another factor of N?\n\nPopulations train in parallel in independent environments for baselines and the main algorithm. Because of this equivalence, N would cancel out on the left and right sides of the equation.\n\n# On environments\n\n> Cultural accumulation is not just passing down information about the unobserved state, but can also involve passing down procedural knowledge [...] It would help if the authors tested in a setting that is fully observable but with more complex dynamics.\n\nWhile it's true that many people who use the term 'procedural' in neuroscience are interested in continuous motor control, it's certainly not the case that neuroscientists intend the term only to apply in the continuous case. The term is also used in the literature on the neuroscience of multiple memory systems [4], which is what inspires our in-context/ in-weights distinction. \n\n\nTo address the reviewer’s request for results on a more complex, fully-observable environment, we ran in-weights accumulation on MinAtar Breakout and Freeway, and Half-Cheetah as requested, and yielded results in Figure 1 and Figure 2 of the attached pdf document.\n\n\n> Only observing the best performing agent from the previous generation seems unnaturally limited.\n\n\nWe in fact have positive results addressing this exact concern. Appendix D details how the results on Memory Sequence assess the selective social learning described by the reviewer. We excluded this discussion from the main text due to space constraints, but will ensure some of it is included in the main body of a camera-ready version. \n\n\n# Addressing questions\n\n1. This is done simply through being in a shared environment. For Goal Sequence, this corresponds to observing the other agent whenever it enters the field of view. For Memory Sequence and TSP, this corresponds to simply observing the last action taken by the other agent.\n\n2. Inferences about a^{-i} (either from direct observation or partially observed state transitions depending on the constraints of the environment) can implicitly be used by the learning agent to inform its own actions. There is no explicit use of a^{-i}, which is the unique quality of social learning and what allows learners to flexibly balance social learning and independent learning to achieve generational accumulation.\n\n3. For all in-context experiments, the population size is only three. For in-weights experiments, the population size is only five. The consistency of results within a population can further be seen by the reasonably small standard error regions in our plots.\n\n[1] Emergent social learning via multi-agent reinforcement learning, Kamal Ndousse, et. al, 2021\n\n[2] Learning few-shot imitation as cultural transmission, Bhoopchand, et. al, 2023\n\n[3] Behavioural cloning from noisy demonstrations, Fumihiro Sasaki, 2021\n\n[4] Memory systems of the brain: a brief history and current perspective, Larry R. Squire, 2004", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to the reviewers for their insightful feedback. We appreciate the consensus that our work is exploring an important, understudied area and that our results indicate positive progress by demonstrating that cultural accumulation can outperform single-lifetime baselines. This is the key takeaway of our work, and we hope it will accelerate future research on cultural accumulation in learning agents. 

In particular, we are glad that reviewers found that “the approach is novel and interesting” (R6qB), “the work is well situated among related prior work” (R6qB), “the experiments are well-designed to test the hypotheses, with clear evidence showing the benefits of cultural accumulation” (9itL), and “the results are significant” (h1KJ), showing “a positive performance trend” (2ZBi). 

Some reviewers raised concerns with clarity (h1KJ, R6qB) and there were a few misinterpretations of parts of the paper, further indicating that the presentation of the algorithms could be clearer. We strongly believe that this is addressable and would like to emphasise that reviewers have not raised any further common issues with our work. 

# On clarity

We greatly appreciate the reviewers’ perspectives on how the paper, which “provides a thorough analysis of both in-context and in-weights accumulation, exploring different environments and scenarios” (9iTL), can be made clearer in its presentation of the different algorithms and results. In particular, we agree that more algorithmic details (many of which currently feature in pseudocode within the appendix) should be included within the main text. We will ensure to do so for a camera-ready version of the paper, in favour of moving some results to the Appendix and/or slightly shortening the Background section if necessary.

We are confident that this will strengthen the manuscript’s overall quality and hope that it adequately addresses this common concern raised by some of the reviewers.

For new results corresponding to specific reviewer's comments, please see the attached pdf document.

Meta-review: This paper investigates the hypothesis that cultural accumulation may be possible in RL. In other words, agents may benefit not only from their own experiences while interacting with a given environment but also from knowledge implicitly passed down from previous generations. The authors proposed, investigated, and empirically analyzed two ways in which accumulation could occur.

All reviewers had an overall positive impression of this work, highlighting the well-conducted experiments designed to test the hypotheses investigated by the authors (as mentioned, e.g., by 9itL, 2ZBi, and h1KJ). Reviewer R6qB and others also argued that the proposed approach is novel and interesting. The reviews were thorough and insightful, and the discussion phase helped clarify various important concerns raised by the reviewers—particularly claims made by the authors that were perceived as overly broad and unclear. Some reviewers, such as 2ZBi, also argued (even post-rebuttal) that it is not immediately apparent how well the proposed approach compared to other methods that made similar assumptions.

Overall, however, the consensus among the reviewers was that this paper does contribute important insights that may benefit the NeurIPS community and that points of contention that were brought up were not critical and are addressable in a revised version of the paper. The reviewers strongly encouraged the authors to improve their document by adding further discussion and improving clarity to ensure that the insights introduced in this work become more apparent to a broader audience.
Decision: Accept (poster)